nils-herrmann/clean_scirepeval huggingface dataset:
Crowdsourced study of subjective image quality

We designed and created a new image quality database that models diverse authentic image distortions and artifacts that affect images that are captured using modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we are using to conduct a very large-scale, on-going, multi-month image quality assessment (IQA) subjective study, wherein a wide range of diverse observers record their judgments of image quality. Our database currently consists of over 320,000 opinion scores on 1,163 authentically distorted images evaluated by over 7000 human observers. The new database will soon be made freely available for download and we envision that the fruits of our efforts will provide researchers with a valuable tool to benchmark and improve the performance of objective IQA algorithms.
Making a “Completely Blind” Image Quality Analyzer

An important aim of research on the blind image quality assessment (IQA) problem is to devise perceptual models that can predict the quality of distorted images with as little prior knowledge of the images or their distortions as possible. Current state-of-the-art “general purpose” no reference (NR) IQA algorithms require knowledge about anticipated distortions in the form of training examples and corresponding human opinion scores. However we have recently derived a blind IQA model that only makes use of measurable deviations from statistical regularities observed in natural images, without training on human-rated distorted images, and, indeed without any exposure to distorted images. Thus, it is “completely blind.” The new IQA model, which we call the Natural Image Quality Evaluator (NIQE) is based on the construction of a “quality aware” collection of statistical features based on a simple and successful space domain natural scene statistic (NSS) model. These features are derived from a corpus of natural, undistorted images. Experimental results show that the new index delivers performance comparable to top performing NR IQA models that require training on large databases of human opinions of distorted images. A software release is available at http://live.ece.utexas.edu/research/quality/niqe_release.zip.
MICA: A Multilinear ICA Decomposition for Natural Scene Modeling

We refine the classical independent component analysis (ICA) decomposition using a multilinear expansion of the probability density function of the source statistics. In particular, we introduce a specific nonlinear system that allows us to elegantly capture the statistical dependences between the responses of the multilinear ICA (MICA) filters. The resulting multilinear probability density is analytically tractable and does not require Monte Carlo simulations to estimate the model parameters. We demonstrate the MICA model on natural image textures and envision that the new model will prove useful for analyzing nonstationarity natural images using natural scene statistics models.
Visibility prediction of flicker distortions on naturalistic videos

We conducted a series of human subjective studies where we found that the visibility of flicker distortions on naturalistic videos is strongly reduced when the speed of coherent object motion is large. Based on this, we propose a model of flicker visibility on naturalistic videos. The model predicts target-related activation levels in the excitatory layer of neurons for a video using spatiotemporal backward masking. The target-related activation level is then shifted and scaled according to video quality level changes that cause flicker distortions. Finally, flicker visibility is predicted based on neural flicker adaptation processes. Results show that the predicted flicker visibility using the model correlates well with human perception of flicker distortions on naturalistic videos.
Design of Linear Equalizers Optimized for the Structural Similarity Index

We propose an algorithm for designing linear equalizers that maximize the structural similarity (SSIM) index between the reference and restored signals. The SSIM index has enjoyed considerable application in the evaluation of image processing algorithms. Algorithms, however, have not been designed yet to explicitly optimize for this measure. The design of such an algorithm is nontrivial due to the nonconvex nature of the distortion measure. In this paper, we reformulate the nonconvex problem as a quasi-convex optimization problem, which admits a tractable solution. We compute the optimal solution in near closed form, with complexity of the resulting algorithm comparable to complexity of the linear minimum mean squared error (MMSE) solution, independent of the number of filter taps. To demonstrate the usefulness of the proposed algorithm, it is applied to restore images that have been blurred and corrupted with additive white gaussian noise. As a special case, we consider blur-free image denoising. In each case, its performance is compared to a locally adaptive linear MSE-optimal filter. We show that the images denoised and restored using the SSIM-optimal filter have higher SSIM index, and superior perceptual quality than those restored using the MSE-optimal adaptive linear filter. Through these results, we demonstrate that a) designing image processing algorithms, and, in particular, denoising and restoration-type algorithms, can yield significant gains over existing (in particular, linear MMSE-based) algorithms by optimizing them for perceptual distortion measures, and b) these gains may be obtained without significant increase in the computational complexity of the algorithm.
Disparity Estimation on Stereo Mammograms

We consider the problem of depth estimation on digital stereo mammograms. Being able to elucidate 3D information from stereo mammograms is an important precursor to conducting 3D digital analysis of data from this promising new modality. The problem is generally much harder than the classic stereo matching problem on visible light images of the natural world, since nearly all of the 3D structural information of interest exists as complex network of multilayered, heavily occluded curvilinear structures. Toward addressing this difficult problem, we formulate a new stereo model that minimizes a global energy functional to densely estimate disparity on stereo mammogram images, by introducing a new singularity index as a constraint to obtain better estimates of disparity along critical curvilinear structures. Curvilinear structures, such as vasculature and spicules, are particularly salient structures in the breast, and being able to accurately position them in 3D is a valuable goal. Experiments on synthetic images with known ground truth and on real stereo mammograms highlight the advantages of the proposed stereo model over the canonical stereo model.
Motion estimation and compensation for foveated video

Utilizing the nonuniform resolution property of the human visual system, foveated video can provide high visual quality relative to non-foveated video by allocating more bits to the central foveation area. In this paper, we present a motion estimation and compensation algorithm for foveated video and measure the performance using a new measure of visual fidelity termed foveal mean absolute distortion. The computation redundancy reduction is achieved by subsampling searching area dependent on the local bandwidth in the sense of Nyquist sampling criterion. In addition, we reduce motion compensated errors by increasing temporal correlation when single or multiple foveation points are added or subtracted.
Assessment of video naturalness using time-frequency statistics

Successful video quality analysers make use of a reference video to compare against or by training on a database of human rated distorted videoes as priors, both of which are either not available or difficult to obtain in many practical scenarios. Although efforts have been made towards designing still picture quality analyzers that are `completely blind' and do not require any prior training on, or exposure to, distorted images or human opinions of them , we are attempting to fill an important but challenging gap by designing a `completely blind' video naturalness analyser. The principle of this new approach is based on the regularties observed in time-frequency relationships of natural vidoes across time. Our experimental results on the LIVE VQA (video quality assessment) database  show that, even with no prior knowledge, the new VQA algorithm performs better than the full reference (FR) quality measure PSNR. The approach is very lean in computational expense which makes it a very good candidate for real time signal processing applications.
Non-stationary texture segmentation using an AM-FM model

We present a novel method for segmenting non-stationary textures. Our approach uses a multidimensional AM-FM representation for the texture, and provides the FM features to an SOFM-LVQ neural network system that performs the segmentation. For the segmentation, we use the eigenvalues of the instantaneous frequency gradient tensor; and show how these eigenvalues capture the non-stationary structure of a texture. For a woodgrain image, the segmentation results are shown to capture the essential non-stationary nature of the grain.
Cross-Layer Optimization for Downlink Wavelet Video Transmission

Cross-layer optimization for efficient multimedia communications is an important emerging issue towards providing better quality-of-service (QoS) over capacity-limited wireless channels. This paper presents a cross-layer optimization approach that operates between the application and physical layers to achieve high fidelity downlink video transmission by optimizing with respect to a quality criterion termed “visual entropy” using Lagrangian relaxation. By utilizing the natural layered structure of wavelet coding, an optimal level of power allocation is determined, which permits the throughput of visual entropy to be maximized over a multi-cell environment. A theoretical approach to optimization using the Shannon capacity and the Karush-Kuhn-Tucker (KKT) conditions is explored when coupling the application with the physical layers. Simulations show that the throughput gain for cross-layer optimization by visual entropy is increased by nearly 80% at the cell boundary as compared with peak signal-to-noise ratio (PSNR).
Multiresolution feature extraction based on multifrequency decomposition

Edges and regions are important features for various computer vision and pattern recognition applications. We propose simple schemes for generating edge and region features based on multifrequency decomposition. These methods involve two steps. First, a multiresolution image structure is obtained by multifrequency decomposition. Second, multiresolution features are generated from this multiresolution image structure by some simple operations. Since these features have a hierarchical structure, we can use the coarse-to-fine strategy to improve the reliability and computational efficiency for matching and for other applications.<<ETX>>
No-reference image blur assessment using multiscale gradient

The increasing number of demanding consumer video applications, as exemplified by cell phone and other low-cost digital cameras, has boosted interest in no-reference objective image and video quality assessment (QA). In this paper, we focus on no-reference image and video blur assessment. There already exist a number of no-reference blur metrics, but most are based on evaluating the widths of intensity edges, which may not reflect real image quality in many circumstances. Instead, we consider natural scenes statistics and adopt multi-resolution decomposition methods to extract reliable features for QA. First, a probabilistic support vector machine (SVM) is applied as a rough image quality evaluator; then the detail image is used to refine and form the final blur metric. The algorithm is tested on the LIVE Image Quality Database; the results show the algorithm has high correlation with human judgment in assessing blur distortion of images.
Efficient Video Quality Assessment Along Temporal Trajectories

We propose a new video quality assessment (VQA) algorithm - the motion compensated structural similarity index - that assesses not only spatial quality but also quality along temporal trajectories. Drawing inspiration from the motion-compensated approach followed for video compression, we propose a motion-compensated approach to temporal quality assessment. The proposed algorithm is computationally efficient as compared to other VQA algorithms that utilize motion information from extracted optical flow and correlates well with human perception of quality. In order to exemplify the utility of the algorithm in a practical setting, we evaluate the quality of H.264/AVC compressed videos. Efficiency of computation is enabled by the novel motion-vector re-use concept.
Oriented Correlation Models of Distorted Natural Images With Application to Natural Stereopair Quality Evaluation

In recent years, bandpass statistical models of natural, photographic images of the world have been used with great success to solve highly diverse problems involving image representation, image repair, image quality assessment (IQA), and image compression. One missing element has been a reliable and generic model of spatial image correlation that reflects the distributions of oriented and relatively oriented spatial structures. We have developed such a model for bandpass pristine images and have generalized it here to also capture the spatial correlation structure of bandpass distorted images. The model applies well to both luminance and depth images. As a demonstration of the usefulness of the generalized model, we develop a new no-reference stereoscopic/3D IQA framework, dubbed stereoscopic/3D blind image naturalness quality index, which utilizes both univariate and generalized bivariate natural scene statistics (NSS) models. We first validate the robustness and effectiveness of these novel bivariate and correlation NSS features extracted from distorted stereopairs, then demonstrate that they are predictive of distortion severity. Our experimental results show that the resulting 3D image quality predictor based in part on the new model outperforms state-of-the-art full- and no-reference 3D IQA algorithms on both symmetrically and asymmetrically distorted stereoscopic image pairs.
Statistics of natural fused image distortions

The capability to automatically evaluate the quality of long wave infrared (LWIR) and visible light images has the potential to play an important role in determining and controlling the quality of a resulting fused LWIR-visible image. Extensive work has been conducted on studying the statistics of natural LWIR and visible light images. Nonetheless, there has been little work done on analyzing the statistics of fused images and associated distortions. In this paper, we study the natural scene statistics (NSS) of fused images and how they are affected by several common types of distortions, including blur, white noise, JPEG compression, and non-uniformity (NU). Based on the results of a separate subjective study on the quality of pristine and degraded fused images, we propose an opinion-aware (OA) fused image quality analyzer, whose relative predictions with respect to other state-of-the-art metrics correlate better with human perceptual evaluations.
Maximum-Likelihood Decomposition of Overlapping and Touching M-Fish Chromosomes Using Geometry, Size and Color Information

Since the birth of chromosome analysis by the aid of computers, building a fully automated chromosome analysis system has been the ultimate goal. Along with many other challenges, automating chromosome classification and segmentation has been one of the major challenges especially due to overlapping and touching chromosomes. In this paper we present a novel decomposition method for overlapping and touching chromosomes in M-FISH images. To overcome the limited success of previous decomposition methods that use partial information about a chromosome cluster, we have incorporated more knowledge about the clusters into a maximum-likelihood frame work. The proposed method evaluates multiple hypotheses based on geometric information, pixel classification results, and chromosome sizes, and a hypothesis that has a maximum-likelihood is chosen as the best decomposition of a given cluster. About 90% of accuracy was obtained for two or three chromosome clusters, which consist about 95% of all clusters with two or more chromosomes
Automated Region of Interest Detection of Spiculated Masses on Digital Mammograms

We have developed a novel technique to automatically identify a region of interest (ROI) surrounding a spiculated lesion on a mammogram. Our proposed approach for determining the size of the ROI depends on the response of a set of unique spiculation filters (SF). The design of these filters is based on manually annotated physical characteristics of spicules. The accuracy of our algorithm is measured in terms of the percentage of spicule pixels located inside the identified ROI. Spicules on each image were identified by an experienced radiologist to serve as a reference to determine the percentage of spicules located in the ROI. On average, 94 percent of spicule pixels were located inside the ROI identified by our algorithm.
Generating a statistical shape model of the AIDS virus spike

We introduce a method to automatically extract the spike features of the AIDS virus imaged through an electron microscope. This method detects the location of the spikes and extracts a sub-volume enclosing the spike with a sensitivity of 80%. The extracted spikes are further aligned and combined to build a 4D statistical shape model, where each voxel in the shape model is assigned a probability density function. This is a fully automated process that can extract sub-volumes of the AIDS virus spike making it possible to build a statistical model without the need for any user supervision. The AIDS virus spike is the primary target of drug design as it is directly involved in infecting host cells. We envision that this new tool will significantly enhance the overall process of shape analysis of the AIDS virus spike imaged through the electron microscope.
Temporal hysteresis model of time varying subjective video quality

Video quality assessment (QA) continues to be an important area of research due to the overwhelming number of applications where videos are delivered to humans. In particular, the problem of temporal pooling of quality sores has received relatively little attention. We observe a hysteresis effect in the subjective judgment of time-varying video quality based on measured behavior in a subjective study. Based on our analysis of the subjective data, we propose a hysteresis temporal pooling strategy for QA algorithms. Applying this temporal strategy to pool scores from PSNR, SSIM  and MOVIE  produces markedly improved subjective quality prediction.
Learning a River Network Extractor Using an Adaptive Loss Function

We have created a deep-learning-based river network extraction model, called DeepRiver, that learns the characteristics of rivers from synthetic data and generalizes them to natural data. To train this model, we created a very large database of exemplary synthetic local channel segments, including channel intersections. Our model uses a special loss function that automatically shifts the focus to the hardest-to-learn parts of an input image. This adaptive loss function makes it possible to learn to detect river centerlines, including the centerlines at junctions and bifurcations. DeepRiver learns to separate between rivers and oceans, and therefore, it is able to reliably extract rivers in coastal regions. The model produces maps of river centerlines, which have the potential to be quite useful for analyzing the properties of river networks.
Joint Source-Channel Distortion Modeling for MPEG-4 Video

Multimedia communication has become one of the main applications in commercial wireless systems. Multimedia sources, mainly consisting of digital images and videos, have high bandwidth requirements. Since bandwidth is a valuable resource, it is important that its use should be optimized for image and video communication. Therefore, interest in developing new joint source-channel coding (JSCC) methods for image and video communication is increasing. Design of any JSCC scheme requires an estimate of the distortion at different source coding rates and under different channel conditions. The common approach to obtain this estimate is via simulations or operational rate-distortion curves. These approaches, however, are computationally intensive and, hence, not feasible for real-time coding and transmission applications. A more feasible approach to estimate distortion is to develop models that predict distortion at different source coding rates and under different channel conditions. Based on this idea, we present a distortion model for estimating the distortion due to quantization and channel errors in MPEG-4 compressed video streams at different source coding rates and channel bit error rates. This model takes into account important aspects of video compression such as transform coding, motion compensation, and variable length coding. Results show that our model estimates distortion within 1.5 dB of actual simulation values in terms of peak-signal-to-noise ratio.
Color and Depth Priors in Natural Images

Natural scene statistics have played an increasingly important role in both our understanding of the function and evolution of the human vision system, and in the development of modern image processing applications. Because range (egocentric distance) is arguably the most important thing a visual system must compute (from an evolutionary perspective), the joint statistics between image information (color and luminance) and range information are of particular interest. It seems obvious that where there is a depth discontinuity, there must be a higher probability of a brightness or color discontinuity too. This is true, but the more interesting case is in the other direction - because image information is much more easily computed than range information, the key conditional probabilities are those of finding a range discontinuity given an image discontinuity. Here, the intuition is much weaker; the plethora of shadows and textures in the natural environment imply that many image discontinuities must exist without corresponding changes in range. In this paper, we extend previous work in two ways - we use as our starting point a very high quality data set of co-registered color and range values collected specifically for this purpose, and we evaluate the statistics of perceptually relevant chromatic information in addition to luminance, range, and binocular disparity information. The most fundamental finding is that the probabilities of finding range changes do in fact depend in a useful and systematic way on color and luminance changes; larger range changes are associated with larger image changes. Second, we are able to parametrically model the prior marginal and conditional distributions of luminance, color, range, and (computed) binocular disparity. Finally, we provide a proof of principle that this information is useful by showing that our distribution models improve the performance of a Bayesian stereo algorithm on an independent set of input images. To summarize, we show that there is useful information about range in very low-level luminance and color information. To a system sensitive to this statistical information, it amounts to an additional (and only recently appreciated) depth cue, and one that is trivial to compute from the image data. We are confident that this information is robust, in that we go to great effort and expense to collect very high quality raw data. Finally, we demonstrate the practical utility of these findings by using them to improve the performance of a Bayesian stereo algorithm.