nils-herrmann/clean_scirepeval huggingface dataset:
A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms

Measurement of visual quality is of fundamental importance for numerous image and video processing applications, where the goal of quality assessment (QA) algorithms is to automatically assess the quality of images or videos in agreement with human quality judgments. Over the years, many researchers have taken different approaches to the problem and have contributed significant research in this area and claim to have made progress in their respective domains. It is important to evaluate the performance of these algorithms in a comparative setting and analyze the strengths and weaknesses of these methods. In this paper, we present results of an extensive subjective quality assessment study in which a total of 779 distorted images were evaluated by about two dozen human subjects. The "ground truth" image quality data obtained from about 25 000 individual human quality judgments is used to evaluate the performance of several prominent full-reference image quality assessment algorithms. To the best of our knowledge, apart from video quality studies conducted by the Video Quality Experts Group, the study presented in this paper is the largest subjective image quality study in the literature in terms of number of images, distortion types, and number of human judgments per image. Moreover, we have made the data from the study freely available to the research community . This would allow other researchers to easily report comparative results in the future
Visual Importance Pooling for Image Quality Assessment

Recent image quality assessment (IQA) metrics achieve high correlation with human perception of image quality. Naturally, it is of interest to produce even better results. One promising method is to weight image quality measurements by visual importance. To this end, we describe two strategies-visual fixation-based weighting, and quality-based weighting. By contrast with some prior studies we find that these strategies can improve the correlations with subjective judgment significantly. We demonstrate improvements on the SSIM index in both its multiscale and single-scale versions, using the LIVE database as a test-bed.
Boundary detection in speckle images

Optimal statistical procedures are formulated for detecting object boundaries in speckle noise imagery. Although speckle noise is found in many applications, our principal interest is in synthetic aperture radar (SAR). The first procedure described is parametric. The ratio of local neighborhood averages is thresholded, which is a statistically natural approach since the noise is often modeled as multiplicative. A nonparametric procedure based on a linear rank statistic is also described, which can be shown to be locally most powerful (among rank tests). Examples are given, and comparisons are offered. The parametric scheme performed slightly better than the rank-order method in the examples, but the inherent robustness of the latter may recommend it for the practical application.
Rate Bounds on SSIM Index of Quantized Image DCT Coefficients

In this paper, we derive bounds on the structural similarity (SSIM) index as a function of quantization rate for fixed-rate uniform quantization of image discrete cosine transform (DCT) coefficients under the high rate assumption. The space domain SSIM index is first expressed in terms of the DCT coefficients of the space domain vectors. The transform domain SSIM Index is then used to derive bounds on the average SSIM index as a function of quantization rate for Gaussian and Laplacian sources. As an illustrative example, uniform quantization of the DCT coefficients of natural images is considered. We show that the SSIM index between the reference and quantized images fall within the bounds for a large set of natural images. Further, we show using a simple example that the proposed bounds could be very useful for rate allocation problems in practical image and video coding applications.
Quality Assessment of Perceptual Crosstalk on Two-View Auto-Stereoscopic Displays

Crosstalk is one of the most severe factors affecting the perceived quality of stereoscopic 3D images. It arises from a leakage of light intensity between multiple views, as in auto-stereoscopic displays. Well-known determinants of crosstalk include the co-location contrast and disparity of the left and right images, which have been dealt with in prior studies. However, when a natural stereo image that contains complex naturalistic spatial characteristics is viewed on an auto-stereoscopic display, other factors may also play an important role in the perception of crosstalk. Here, we describe a new way of predicting the perceived severity of crosstalk, which we call the Binocular Perceptual Crosstalk Predictor (BPCP). BPCP uses measurements of three complementary 3D image properties (texture, structural duplication, and binocular summation) in combination with two well-known factors (co-location contrast and disparity) to make predictions of crosstalk on two-view auto-stereoscopic displays. The new BPCP model includes two masking algorithms and a binocular pooling method. We explore a new masking phenomenon that we call duplicated structure masking, which arises from structural correlations between the original and distorted objects. We also utilize an advanced binocular summation model to develop a binocular pooling algorithm. Our experimental results indicate that BPCP achieves high correlations against subjective test results, improving upon those delivered by previous crosstalk prediction models.
Generalized order statistic filters

The authors explore generalizations of order statistic (OS) filters, a class of finite-width discrete windowing filters defined by linearly weighting the samples in the window according to their natural ordering as real numbers. The concept can be extended by defining generalized ordering rules generating different permutations of the sample prior to weighting. Although the resulting class of generalized OS filter is very broad (including, e.g. the linear filters), local signal ordering properties relating to local signal monotonicity unify them. It is envisaged that both the framework for filter/signal description and the class of filters generated will find use in many signal shaping applications.<<ETX>>
A bound involving n-dimensional instantaneous frequency

The author states and proves an integral inequality that bounds the absolute difference epsilon (x)= mod m(x) -m(x) mod where m(x) is the response of a modulated n-dimensional real linear filter w to a complex exponential signal with n-dimensional instantaneous frequency Delta u(x) and m(x)= mod W( Delta u(x)-u/sub 0/) mod where W is the Fourier transform of w. The quantity of epsilon (x) provides an estimate of the error incurred by using m(x) as an estimate of m(x), e.g., if Delta u(x) is unknown. Such estimates may be applied to the problem of measuring the n-dimensional instantaneous frequency of certain nonstationary phase-modulated signals.
GAFFE: A Gaze-Attentive Fixation Finding Engine

The ability to automatically detect visually interesting regions in images has many practical applications, especially in the design of active machine vision and automatic visual surveillance systems. Analysis of the statistics of image features at observers' gaze can provide insights into the mechanisms of fixation selection in humans. Using a foveated analysis framework, we studied the statistics of four low-level local image features: luminance, contrast, and bandpass outputs of both luminance and contrast, and discovered that image patches around human fixations had, on average, higher values of each of these features than image patches selected at random. Contrast-bandpass showed the greatest difference between human and random fixations, followed by luminance-bandpass, RMS contrast, and luminance. Using these measurements, we present a new algorithm that selects image regions as likely candidates for fixation. These regions are shown to correlate well with fixations recorded from human observers.
The Effect of Median Filtering on Edge Estimation and Detection

In this paper we consider the effect of median prefiltering on the subsequent estimation and detection of edges in digital images. Where possible, a quantitative statistical comparison is made for a number of filters defined with two-dimensional geometries; in some cases one-dimensional analyses are required to illustrate certain points. Noise images prefiltered by median filters defined with a variety of windowing geometries are used to support the analysis, and it is found that median prefiltering improves the performance of both thresholding and zero-crossing based edge detectors.
Modeling the Time—Varying Subjective Quality of HTTP Video Streams With Rate Adaptations

Newly developed hypertext transfer protocol (HTTP)-based video streaming technologies enable flexible rate-adaptation under varying channel conditions. Accurately predicting the users' quality of experience (QoE) for rate-adaptive HTTP video streams is thus critical to achieve efficiency. An important aspect of understanding and modeling QoE is predicting the up-to-the-moment subjective quality of a video as it is played, which is difficult due to hysteresis effects and nonlinearities in human behavioral responses. This paper presents a Hammerstein-Wiener model for predicting the time-varying subjective quality (TVSQ) of rate-adaptive videos. To collect data for model parameterization and validation, a database of longer duration videos with time-varying distortions was built and the TVSQs of the videos were measured in a large-scale subjective study. The proposed method is able to reliably predict the TVSQ of rate adaptive videos. Since the Hammerstein-Wiener model has a very simple structure, the proposed method is suitable for online TVSQ prediction in HTTP-based streaming.
Stereo disparity from multiscale processing of local image phase

We present a stereo algorithm employing new modulation (multicomponent AM-FM) models for image representation, a disparity channel model for depth computation, and a multichannel (Gabor wavelet like) processing for multi scale (coarse to fine) computation of stereo disparity from local image phase. The algorithm generates a dense, subpixel accuracy disparity map without sophisticated feature extraction and interpolation.
Very low bit rate foveated video coding for H.263

Recently, foveated video has been introduced as an important emerging method for very low bit rate multimedia applications. In this paper, we develop several rate control algorithms, and measure the performance of foveated video. We utilize H.263 video, and compare the performance with regular video based on the SNRC (signal-to-noise-ratio in curvilinear coordinates). In order to maximize compression, we use a maximum quantization parameter (QP=31) for the regular video, and code a foveated video sequence at the equivalent bit rate. In simulation, we improve the PSNRC to 3.64 (1.62)dB under 30 (14) kbits/sec for P pictures in CIF "News" ("Akiyo") standard video sequence.
Perceiving distortions in visual signals

This paper and the associated talk are largely an opinion piece on current research on visual quality assessment: where it is at, and where it is going. The paper and talk follow three themes. First I'll discuss full-reference (FR) visual quality assessment (QA), and will opine that future gains on this largely-solved problem are likely to be found in modeling human behavior, which is a hard problem quite different from perceptual modeling. Then I will discuss the no-reference (NR) or blind QA problem, where satisfying results are now being obtained on still images. However, the blind video QA problem remains far from solution owing to a lack of regular statistical video models, among other things. Lastly, I will discuss the exceedingly difficult stereo (3D) visual QA problem. It is my hope that the talk and this paper will stimulate promising avenues of research.
Multiresolution 3-D range segmentation using focus cues

This paper describes a novel system for computing a three-dimensional (3-D) range segmentation of an arbitrary visible scene using focus information. The process of range segmentation is divided into three steps: an initial range classification, a surface merging process, and a 3-D multiresolution range segmentation. First, range classification is performed to obtain quantized range estimates. The range classification is performed by analyzing focus cues within a Bayesian estimation framework. A combined energy functional measures the degree of focus and the Gibbs distribution of the class field. The range classification provides an initial range segmentation. Second, a statistical merging process is performed to merge the initial surface segments. This gives a range segmentation at a coarse resolution. Third, 3-D multiresolution range segmentation (3-D MRS) is performed to refine the range segmentation into finer resolutions. The proposed range segmentation method does not require initial depth estimates, it allows the analysis of scenes containing multiple objects, and it provides a rich description of the 3-D structure of a scene.
Making image quality assessment robust

We develop a robust framework for natural scene statistic (NSS) model based blind image quality assessment (IQA). The robustified IQA model utilizes a robust statistics approach based on L-moments. Such robust statistics based approaches are effective when natural or distorted images deviate from assumed statistical models, and achieves better prediction performance on distorted images relative to human subjective judgments. We also show how robustifying the model makes IQA approach resilient against deviation in model assumptions, small variations in the distortions and amount of data the model is trained on.
Automated Facial Feature Detection from Portrait and Range Images

We propose a novel technique to detect feature points from portrait and range representations of the face. In this technique, the appearance of each feature point is encoded using a set of Gabor wavelet responses extracted at multiple orientations and spatial frequencies. A vector of Gabor coefficients, called a jet, is computed at each pixel in the search window on a fiducial and compared with a set of jets, called a bunch, collected from a set of training data on the same type of fiducial. The desired feature point is located at the pixel whose jet is the most similar to the training bunch. This is the first time that Gabor wavelet responses were used to detect facial landmarks from range images. This method was tested on 1146 pairs of range and portrait images and high detection accuracies are achieved using a small number of training images. It is shown that co-localization using Gabor jets on range and portrait images resulted in better accuracy than using any single image modality. The obtained accuracies are competitive to that of other techniques in the literature.
A hierarchical Bayesian-map approach to computational imaging

We present a novel approach to inverse problems in imaging based on a Hierarchical Bayesian-MAP (HB-MAP) formulation. In this paper we specifically focus on the difficult and basic inverse problem of multi-sensor (tomographic) imaging wherein the source image of interest is viewed from multiple directions by independent sensors. We employ a Probabilistic Graphical Modeling extension of the Compound Gaussian (CG) distribution as a global image prior into a Hierarchical Bayesian inference procedure. We first demonstrate the performance of the algorithm on Monte-Carlo trials followed by empirical data involving natural (optical) images. We demonstrate how our algorithm outperforms many of the previous approaches in the literature including Filtered Back-projection (FBP) and a variety of state-of-the-art compressive sensing (CS) algorithms.
An unequal error protection scheme for multiple input multiple output systems

Rapid progress has been made in the use of spatial diversity for reliable communication over wireless channels. A large amount of work has been done in the field of unequal error protection and joint source-channel coding for reliable and efficient transmission of data over wireline and wireless channels. However, the use of unequal error protection in the temporal domain, combined with the advantages of multiple input multiple output communication systems has yet to be explored. In this paper an unequal error protection scheme based on the combined use of turbo codes and space-time codes for communication over wireless channels is proposed. This scheme assigns more coding gain and temporal diversity to important bits but assigns more spatial diversity and less coding gain to less important bits. Simulations illustrate a 25% reduction in average transmission time and a 15 dB improvement compared with no spatial diversity.
On eigenstructure-based direct multichannel blind image restoration

Existing eigenstructure-based direct multichannel blind image restoration techniques include nullspace-based and direct deconvolver estimation techniques. The nullspace-based approach can be formulated as an optimization problem. We show that this formulation implies a new subspace-based approach that uses matrix operations. This new approach has the same advantages as the nullspace-based one but requires less computational complexity. Under some mild conditions, its complexity is equal to that of the FFT. Furthermore, the relation among the nullspace-based approach, the direct deconvolver estimation and the new subspace-based approach is studied.
Study of Temporal Effects on Subjective Video Quality of Experience

HTTP adaptive streaming is being increasingly deployed by network content providers, such as Netflix and YouTube. By dividing video content into data chunks encoded at different bitrates, a client is able to request the appropriate bitrate for the segment to be played next based on the estimated network conditions. However, this can introduce a number of impairments, including compression artifacts and rebuffering events, which can severely impact an end-user’s quality of experience (QoE). We have recently created a new video quality database, which simulates a typical video streaming application, using long video sequences and interesting Netflix content. Going beyond previous efforts, the new database contains highly diverse and contemporary content, and it includes the subjective opinions of a sizable number of human subjects regarding the effects on QoE of both rebuffering and compression distortions. We observed that rebuffering is always obvious and unpleasant to subjects, while bitrate changes may be less obvious due to content-related dependencies. Transient bitrate drops were preferable over rebuffering only on low complexity video content, while consistently low bitrates were poorly tolerated. We evaluated different objective video quality assessment algorithms on our database and found that objective video quality models are unreliable for QoE prediction on videos suffering from both rebuffering events and bitrate changes. This implies the need for more general QoE models that take into account objective quality models, rebuffering-aware information, and memory. The publicly available video content as well as metadata for all of the videos in the new database can be found at http://live.ece.utexas.edu/research/LIVE_NFLXStudy/nflx_index.html.
A Subjective and Objective Study of Stalling Events in Mobile Streaming Videos

Over-the-top mobile adaptive video streaming is invariably influenced by volatile network conditions, which can cause playback interruptions (stalling or rebuffering events) and bitrate fluctuations, thereby impairing users’ quality of experience (QoE). Video quality assessment models that can accurately predict users’ QoE under such volatile network conditions are rapidly gaining attention, since these methods could enable more efficient design of quality control protocols for media-driven services such as YouTube, Amazon, Netflix, and many others. However, the development of improved QoE prediction models requires data sets of videos afflicted with diverse stalling events that have been labeled with ground-truth subjective opinion scores. Toward this end, we have created a new mobile video quality database that we call LIVE Mobile Stall Video Database-II. Our database contains a total of 174 videos afflicted with distortions caused by 26 different stalling patterns. We describe the way we simulated the diverse stalling events to create a corpus of distorted videos, and we detail the human study we conducted to obtain continuous-time subjective scores from 54 subjects. We also present the outcomes of our comprehensive analysis of the impact of several factors that influence subjective QoE, and report the performance of existing QoE-prediction models on our data set. We are making the database (videos, subjective data, and video metadata) publicly available in order to help the advance state-of-the-art research on user-centric mobile network planning and management. The database may be accessed at http://live.ece.utexas.edu/research/LIVEStallStudy/liveMobile.html.
Improving visible-surface reconstruction

Techniques are described for improving the speed and accuracy of iterative visible-surface reconstruction algorithms. In particular, the importance of obtaining accurate early information is emphasized, both in the initial surface approximation and in the early localization of surface discontinuities. The first of these goals is attained using a simple technique known as constraint expansion, which yields a good initial approximation while accounting for detected discontinuities. The second goal is attained by using the locations of the image intensity edges as a guide in locating potential discontinuities.<<ETX>>
A Linear Estimator Optimized for the Structural Similarity Index and its Application to Image Denoising

We use a perceptual distortion metric-the structural similarity (SSIM) index, to derive a new linear estimator for estimating zero-mean Gaussian sources distorted by additive white Gaussian noise (AWGN). We use this estimator in an image denoising application and compare its performance with the traditional linear least squared error (LLSE) estimator. Although images denoised using the SSIM-optimized estimator have a lower peak signal-to-noise ratio (PSNR) compared to their LLSE counterparts, the SSIM-optimized estimator clearly outperforms the LLSE estimator in terms of the visual quality of the denoised images.
H.264 visually lossless compressibility index: Psychophysics and algorithm design

Although the term ‘visually lossless’ (VL) has been used liberally in the video compression literature, there does not seem to be a systematic evaluation of what it means for a video to be compressed visually lossless-ly. Here, we undertake a psychovisual study to infer the visually lossless threshold for H.264 compression of videos spanning a wide range of contents. Based on results from this study, we then propose a compressibility index which provides a measure of the appropriate bit-rate for VL H.264 compression of a video given texture (i.e., spatial activity) and motion (i.e., temporal activity) information. This compressibility index has been made available online at  in order to facilitate practical application of the research presented here and to further research in the area of VL compression.