nils-herrmann/clean_scirepeval huggingface dataset:
Dynamic contrast-based quantization for lossy wavelet image compression

This paper presents a contrast-based quantization strategy for use in lossy wavelet image compression that attempts to preserve visual quality at any bit rate. Based on the results of recent psychophysical experiments using near-threshold and suprathreshold wavelet subband quantization distortions presented against natural-image backgrounds, subbands are quantized such that the distortions in the reconstructed image exhibit root-mean-squared contrasts selected based on image, subband, and display characteristics and on a measure of total visual distortion so as to preserve the visual system's ability to integrate edge structure across scale space. Within a single, unified framework, the proposed contrast-based strategy yields images which are competitive in visual quality with results from current visually lossless approaches at high bit rates and which demonstrate improved visual quality over current visually lossy approaches at low bit rates. This strategy operates in the context of both nonembedded and embedded quantization, the latter of which yields a highly scalable codestream which attempts to maintain visual quality at all bit rates; a specific application of the proposed algorithm to JPEG-2000 is presented.
Main subject detection via adaptive feature selection

In this paper we present an algorithm which uses adaptive selection of low-level features for main subject detection. The algorithm first computes low-level features such as contrast and sharpness, each computed in a block-based fashion. Next, the algorithm quantifies the usefulness of each feature by using both statistical and geometric information measured across blocks. Finally, the saliency of each block is determined via a weighted linear combination of the features, where the weights are chosen based on each feature's estimated usefulness. Our results demonstrate that the adaptive nature of this algorithm allows it to perform competitively with other techniques, while maintaining very low computational complexity.
Image-adaptive contrast and entropy based model of regions of visible distortion

This paper presents an image adaptive algorithm for predicting regions of natural images that visually mask distortions created as a result of modification of wavelet domain coefficients. The visual error perception algorithm adaptively predicts the visual detection threshold of distortions added in the wavelet subband. Contrast masking and entropy masking properties of an image are used to generate an activity map indicating the local masking ability of the image. Ground truth data from psychophysical experiments were used to optimize the adaptive activity parameter in order to evaluate the masking thresholds of image regions. The use of this algorithm can increase the amount of watermarking data applied to an image, it is applicable for general wavelet quantization error detection and HVS based compression.
Predicting the Perceived Interest of Object in Images

This paper presents the results of a psychophysical experiment and an associated algorithm designed to compute the perceived interest of objects in images. We measured likelihood functions via a psychophysical experiment in which subjects rated the perceived visual interest of each of 408 objects in 100 images. These results were then used to determine the likelihood of interest given various factors such as size, location, contrast, color, and edge-strength. The resulting likelihood functions are used as part of a Bayesian formulation in which perceived interest is inferred based on these factors. Results demonstrate that our algorithm can perform well in predicting perceived interest.
On the quality assessment of enhanced images: A database, analysis, and strategies for augmenting existing methods

Most methods of image quality assessment (QA) have been designed for QA of degraded images. This paper presents the results of a study designed to investigate whether existing QA methods can be adapted to succeed on enhanced images. We developed a database containing digitally enhanced images and associated subjective quality ratings. Next, we analyzed the efficacy of select QA methods and their reverse-mode versions in predicting the ratings. Given the fact that an enhanced image makes the original image appear degraded, we tested both normal and reverse-mode versions, where the latter were implemented by specifying the enhanced image as the reference and the original image as the “degraded” image. Our results demonstrate that this reverse-mode approach improves QA of enhanced images. We present a strategy for further improving the QA methods by using measures of contrast, sharpness, and color saturation.
Region-of-importance detection based on fusion of audio and video

In this paper, a new framework is proposed for autonomous universal surveillance based on video and audio data. “Universal” indicates no specification of targets of interest. Instead, regions of importance (ROIs) in a scene should be detected. Specifically, in the video domain, a frame-based main subject detection is proposed based on adaptive selection of low-level features. In the audio domain, a time-delay-based direction of arrival estimation scheme is adopted. The outputs of video and audio processing are fused in a probabilistic framework to generate more refined ROIs. Experimental results demonstrate the effectiveness of the proposed scheme in ROI detection.
A Fast Wavelet-Based Algorithm for Global and Local Image Sharpness Estimation

In this letter, we present a simple, yet effective wavelet-based algorithm for estimating both global and local image sharpness (FISH, Fast Image Sharpness). FISH operates by first decomposing the input image via a three-level separable discrete wavelet transform (DWT). Next, the log-energies of the DWT subbands are computed. Finally, a scalar index corresponding to the image's overall sharpness is computed via a weighted average of these log-energies. Testing on several image databases demonstrates that, despite its simplicity, FISH is competitive with the currently best-performing techniques both for sharpness estimation and for no-reference image quality assessment.
Saliency Detection Based on Multiscale Extrema of Local Perceptual Color Differences

Visual saliency detection is a useful technique for predicting, which regions humans will tend to gaze upon in any given image. Over the last several decades, numerous algorithms for automatic saliency detection have been proposed and shown to work well on both synthetic and natural images. However, two key challenges remain largely unaddressed: 1) How to improve the relatively low predictive performance for images that contain large objects and 2) how to perform saliency detection on a wider variety of images from various categories without training. In this paper, we propose a new saliency detection algorithm that addresses these challenges. Our model first detects potentially salient regions based on multiscale extrema of local perceived color differences measured in the CIELAB color space. These extrema are highly effective for estimating the locations, sizes, and saliency levels of candidate regions. The local saliency candidates are further refined via two global extrema-based features, and then a Gaussian mixture is used to generate the final saliency map. Experimental validation on the extensive CAT2000 data set demonstrates that our proposed method either outperforms or is highly competitive with prior approaches, and can perform well across different categories and object sizes, while remaining training-free.
${\bf S}_{3}$: A Spectral and Spatial Measure of Local Perceived Sharpness in Natural Images

This paper presents an algorithm designed to measure the local perceived sharpness in an image. Our method utilizes both spectral and spatial properties of the image: For each block, we measure the slope of the magnitude spectrum and the total spatial variation. These measures are then adjusted to account for visual perception, and then, the adjusted measures are combined via a weighted geometric mean. The resulting measure, i.e., S3 (spectral and spatial sharpness), yields a perceived sharpness map in which greater values denote perceptually sharper regions. This map can be collapsed into a single index, which quantifies the overall perceived sharpness of the whole image. We demonstrate the utility of the S3 measure for within-image and across-image sharpness prediction, no-reference image quality assessment of blurred images, and monotonic estimation of the standard deviation of the impulse response used in Gaussian blurring. We further evaluate the accuracy of S3 in local sharpness estimation by comparing S3 maps to sharpness maps generated by human subjects. We show that S3 can generate sharpness maps, which are highly correlated with the human-subject maps.
VSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images

This paper presents an efficient metric for quantifying the visual fidelity of natural images based on near-threshold and suprathreshold properties of human vision. The proposed metric, the visual signal-to-noise ratio (VSNR), operates via a two-stage approach. In the first stage, contrast thresholds for detection of distortions in the presence of natural images are computed via wavelet-based models of visual masking and visual summation in order to determine whether the distortions in the distorted image are visible. If the distortions are below the threshold of detection, the distorted image is deemed to be of perfect visual fidelity (VSNR = infin)and no further analysis is required. If the distortions are suprathreshold, a second stage is applied which operates based on the low-level visual property of perceived contrast, and the mid-level visual property of global precedence. These two properties are modeled as Euclidean distances in distortion-contrast space of a multiscale wavelet decomposition, and VSNR is computed based on a simple linear sum of these distances. The proposed VSNR metric is generally competitive with current metrics of visual fidelity; it is efficient both in terms of its low computational complexity and in terms of its low memory requirements; and it operates based on physical luminances and visual angle (rather than on digital pixel values and pixel-based dimensions) to accommodate different viewing conditions.
Can visual fixation patterns improve image fidelity assessment?

This paper presents the results of a computational experiment designed to investigate the extent to which metrics of image fidelity can be improved through knowledge of where humans tend to fixate in images. Five common metrics of image fidelity were augmented using two sets of fixation data, one set obtained under task-free viewing conditions and another set obtained when viewers were asked to judge image quality. The augmented metrics were then compared to subjective ratings of the images. The results show that most metrics can be improved using eye fixation data, but a greater improvement was found using fixations obtained in the task-free condition (task-free viewing).
Contrast based quantization and rate control for wavelet coded images

A visually-optimal quantization and rate-control strategy based on results of recent contrast sensitivity and suprathreshold summation experiments is proposed. At suprathreshold contrasts, masked detection thresholds for wavelet subband quantization distortions were approximately equal for scale-3, 4, and 5 distortions; approximately 52% greater for scale-2 distortions; and approximately 84% greater for scale-1 distortions. Based on a suprathreshold error-pooling model, contrasts for individual subbands are selected to match these contrast ratios, and are adjusted to account for changes in relative sensitivity at suprathreshold contrasts. Quantization step sizes are then computed from the adjusted base contrasts. A target contrast is estimated from the desired rate, and rate control is performed by adjusting this contrast until the rate is met. Images compressed with the proposed algorithm show improved visual quality at low bit rates.
Spatially-selective quantization and coding for wavelet-based image compression

Recent developments in psychovisual modeling have led to improvements in wavelet-based coder performance. A spatially selective quantizer based on texture masking sensitivities is introduced, which hides distortion in high-contrast portions of images. Unlike other spatial quantization schemes, this method requires explicit side information to convey stepsizes. A simple coder is presented which leverages this side information to reduce the rate required to code the quantized data. Side information coding is also discussed. With respect to visual quality, this compression scheme performs competitively with a CSF-optimized JPEG-2000 coder at equivalent rates.
A computational model for predicting local distortion visibility via convolutional neural network trainedon natural scenes

A crucial requirement for modern image coding is the ability to accurately and efficiently predict the local visibility of coding artifacts. Such predictions could help guide the allocation of bits or the determination of quality for each spatial region. This paper presents a convolutional-neural-network-based (CNN-based) model to predict local distortion visibility in natural scenes. Although CNNs have recently emerged as a powerful tool for many computer vision applications due to its deep learning abilities and computational efficiency, CNNs have never been tested for predicting continuous values such as visibility thresholds. We optimized the model's parameters on our recently published large dataset on local masking in natural scenes [Alam et al.,  2014]. Testing results demonstrate that our CNN-based model: (1) can indeed succeed in this task; (2) can more accurately predict thresholds than modern gain-control-based models; (3) is competitive in terms of prediction accuracy with a gain-control model tuned to the same dataset; and (4) is significantly more computationally efficient than modern gain-controls models.
Quality Assessment of Screen Content Images via Convolutional-Neural-Network-Based Synthetic/Natural Segmentation

The recent popularity of remote desktop software and live streaming of composited video has given rise to a growing number of applications that make use of the so-called screen content images that contain a mixture of text, graphics, and photographic imagery. Automatic quality assessment (QA) of screen-content images is necessary to enable tasks, such as quality monitoring, parameter adaptation, and other optimizations. Although QA of natural images has been heavily researched over the last several decades, the QA of screen content images is a relatively new topic. In this paper, we present a QA algorithm called convolutional neural network-based screen content image quality estimator (CNN-SQE), which operates via a fuzzy classification of screen content images into plain-text, computer-graphics/cartoons, and natural-image regions. The first two classes are considered to contain synthetic content (text/graphics), and the latter two classes are considered to contain naturalistic content (graphics/photographs), where the overlap of the classes allows the computer graphics/cartoons segments to be analyzed by both text-based and natural-image-based features. We present a CNN-based approach for the classification, an edge-structure-based quality degradation model, and a region-size-adaptive quality-fusion strategy. As we will demonstrate, the proposed CNN-SQE algorithm can achieve better/competitive performance as compared with the other state-of-the-art QA algorithms.
Learning natural statistics of binocular contrast for no reference quality assessment of stereoscopic images

Algorithms for no-reference (NR) stereoscopic image quality assessment (SIQA) aim to evaluate the perceptual quality of a stereoscopic/3D image without the assistance of its reference. Current NR SIQA models often require training on 3D distorted images and their associated human opinion scores, which ultimately restrict their further application. In this paper, we present a simple yet effective NR SIQA model that does not require training on existing 3D image databases. Instead, we train our model on a large dataset of natural stereoscopic images based on learning the local statistics of the Cyclopean contrast maps, and then use the existing 2D NR IQA model to help guide the NR SIQA task. Experimental results demonstrate the efficacy of our proposed method.
Data handling inefficiencies between CUDA, 3D rendering, and system memory

While GPGPU programming offers faster computation of highly parallelized code, the memory bandwidth between the system and the GPU can create a bottleneck that reduces the potential gains. CUDA is a prominent GPGPU API which can transfer data to and from system code, and which can also access data used by 3D rendering APIs. In an application that relies on both GPU programming APIs to accelerate 3D modeling and an easily parallelized algorithm, the hidden inefficiencies of nVidia's data handling with CUDA become apparent. First, CUDA uses the CPU's store units to copy data between the graphics card and system memory instead of using a more efficient method like DMA. Second, data exchanged between the two GPU-based APIs travels through the main processor instead of staying on the GPU. As a result, a non-GPGPU implementation of a program runs faster than the same program using GPGPU.
Digital watermarking using local contrast-based texture masking

Digital image watermarking algorithms embed identifying marks that can be used for authentication. To make the distortions induced by the embedding process imperceptible, the watermarking algorithm must determine their visual threshold. This paper presents: (1) the results of a psychophysical detection experiment for wavelet-distortion targets presented against textured backgrounds of varying contrasts; and (2) an associated wavelet-based image watermarking algorithm that embeds the watermark at the threshold of visibility by weighting it using a local masking measure prior to embedding. The embedded watermark is shown to be robust to JPEG compression at quality factors as low as 20.
Quantifying the visual quality of wavelet-compressed images based on local contrast, visual masking, and global precedence

The paper presents a two-stage metric which quantifies the visual quality of images that have undergone wavelet-based compression. The first stage operates via a model of visual pattern masking, which takes as input original and distorted images, and which outputs masked contrast detection thresholds. For distortions beyond the threshold of detection, the images and the thresholds are fed into a second stage which estimates visual quality based on the distance between the distribution of assumed ideal and actual contrast signal-to-noise ratios across scale-space. Results indicate that the proposed metric yields a higher correlation with subjective-rating data than other visual quality metrics when applied to a sample of wavelet-coded images (with rates ranging from approximately 0.08-0.85 bits/pixel) for which peak signal-to-noise ratio correlates poorly with subjective quality.
Vehicle tracking using a human-vision-based model of visual similarity

In this paper we propose an automatic vehicle tracking method for monitoring traffic intersections. The method uses a weighted combination of low-level features and low-level human-visual-system (HVS) modeling. Given an input video, moving vehicles are first detected from the scene and low-level features are extracted from the detected vehicles. Next, each detected region in the current video frame is compared with each detected region in the next frame by using an HVS-based similarity model. Finally, tracking is performed by locating the vehicle with the closest matching low-level features and greatest visual similarity. We demonstrate that combining low-level features with an HVS-based model can be an effective strategy for vehicle tracking.
A spatiotemporal most-apparent-distortion model for video quality assessment

This paper presents an algorithm for video quality assessment, spatiotemporal MAD (ST-MAD), which extends our previous image-based algorithm (MAD ) to take into account visual perception of motion artifacts. ST-MAD employs spatiotemporal “images” (STS images ) created by taking time-based slices of the original and distorted videos. Motion artifacts manifest in the STS images as spatial artifacts, which allows one to quantify motion-based distortion by using classical image-quality assessment techniques. ST-MAD estimates motion-based distortion by applying MAD's appearance-based model to compare the distorted video's STS images to the original video's STS images. This comparison is further adjusted by using optical-flow-derived weights designed to give greater precedence to fast-moving regions located toward the center of the video. Testing on the LIVE video database demonstrates that ST-MAD performs well in predicting video quality.
Local masking in natural images measured via a new tree-structured forced-choice technique

It is widely known that natural images can hide or mask visual signals, and that this masking ability can vary across different regions of the image. Previous studies have quantified masking by measuring image-wide detection thresholds or local thresholds for select image regions; however, little effort has focused on measuring local thresholds across entire images so as to achieve ground-truth masking maps. Such maps could prove invaluable for testing and refining masking models; however, obtaining these maps requires a prohibitive number of trials using a traditional forced-choice procedure. Here, we present a tree-structured forced-choice procedure (TS-3AFC) designed to efficiently measure local thresholds across images. TS-3AFC requires fewer trials than normal forced-choice by employing recursive patch subdivision in which the child patches are not tested individually until the target is detectable in the parent patch. We show that TS-3AFC can yield masking maps which demonstrate both intrasubject and inter-subject repeatability, and we analyze the performance of a modern masking model and two quality estimators in predicting the obtained ground-truth maps for a small set of images.
3D-MAD: A Full Reference Stereoscopic Image Quality Estimator Based on Binocular Lightness and Contrast Perception

Algorithms for a stereoscopic image quality assessment (IQA) aim to estimate the qualities of 3D images in a manner that agrees with human judgments. The modern stereoscopic IQA algorithms often apply 2D IQA algorithms on stereoscopic views, disparity maps, and/or cyclopean images, to yield an overall quality estimate based on the properties of the human visual system. This paper presents an extension of our previous 2D most apparent distortion (MAD) algorithm to a 3D version (3D-MAD) to evaluate 3D image quality. The 3D-MAD operates via two main stages, which estimate perceived quality degradation due to 1) distortion of the monocular views and 2) distortion of the cyclopean view. In the first stage, the conventional MAD algorithm is applied on the two monocular views, and then the combined binocular quality is estimated via a weighted sum of the two estimates, where the weights are determined based on a block-based contrast measure. In the second stage, intermediate maps corresponding to the lightness distance and the pixel-based contrast are generated based on a multipathway contrast gain-control model. Then, the cyclopean view quality is estimated by measuring the statistical-difference-based features obtained from the reference stereopair and the distorted stereopair, respectively. Finally, the estimates obtained from the two stages are combined to yield an overall quality score of the stereoscopic image. Tests on various 3D image quality databases demonstrate that our algorithm significantly improves upon many other state-of-the-art 2D/3D IQA algorithms.