nils-herrmann/clean_scirepeval huggingface dataset:
Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos

There has recently been a great deal of interest in the development of algorithms that objectively measure the integrity of video signals. Since video signals are being delivered to human end users in an increasingly wide array of applications and products, it is important that automatic methods of video quality assessment (VQA) be available that can assist in controlling the quality of video being delivered to this critical audience. Naturally, the quality of motion representation in videos plays an important role in the perception of video quality, yet existing VQA algorithms make little direct use of motion information, thus limiting their effectiveness. We seek to ameliorate this by developing a general, spatio-spectrally localized multiscale framework for evaluating dynamic video fidelity that integrates both spatial and temporal (and spatio-temporal) aspects of distortion assessment. Video quality is evaluated not only in space and time, but also in space-time, by evaluating motion quality along computed motion trajectories. Using this framework, we develop a full reference VQA algorithm for which we coin the term the MOtion-based Video Integrity Evaluation index, or MOVIE index. It is found that the MOVIE index delivers VQA scores that correlate quite closely with human subjective judgment, using the Video Quality Expert Group (VQEG) FRTV Phase 1 database as a test bed. Indeed, the MOVIE index is found to be quite competitive with, and even outperform, algorithms developed and submitted to the VQEG FRTV Phase 1 study, as well as more recent VQA algorithms tested on this database.
Spatio-temporal tracking of material shape change via multi-dimensional splines

A new method for modeling shape changes in multidimensional image sequences of deforming objects is presented. The method consists of defining an orthogonal coordinate system on the segmented object in the first image frame and then calculating the deformations of this coordinate system over time. The deformations are found by minimizing an energy functional that consists of a linear combination of a data fidelity term and a shape-change constraint term. The shape-change constraint is based on the differential geometric properties of the parametrized material coordinate system. The deforming material coordinate system models an object's shape changes both locally and globally. The mathematics is first developed for an n-dimensional sequence of images. Examples are given for both 2D and 3D image sequences of both real and synthetic images.<<ETX>>
Image features that draw fixations

The ability to automatically detect 'visually interesting' regions in an image has many practical applications especially in the design of active machine vision systems. This paper describes a data-driven approach that uses eye tracking in tandem with principal component analysis to extract low-level image features that attract human gaze. Data analysis on an ensemble of image patches extracted at the observer's point of gaze revealed features that resemble derivatives of the 2D Gaussian operator. Dissimilarities between human and random fixations are investigated by comparing the features extracted at the point of gaze to the general image structure obtained by random sampling in Monte-Carlo simulations. Finally, a simple application where these features are used to predict fixations is illustrated.
Robust detection of object boundaries in Weibull radar imagery

Radar image speckle is often modeled as having a negative-exponential, or more generally, gamma distribution. However, studies of noise in coherent radar systems suggest that the first-order statistics may be more accurately modeled using the two-parameter Weibull density, the parameters of which vary with the surface being imaged. Techniques for detecting object boundaries in noisy radar images are proposed and compared. The images are assumed to be coarsely sampled, so that the (multiplicative) radar noise can be modeled as uncorrelated and identically distributed. Edge detection in multiplicative noise is effectively accomplished by thresholding ratios of locally adjacent image estimates. The efficacies of edge detectors defined as ratios of single order statistics, ratios of averages and ratios of best linear unbiased estimators (BLUEs) are compared. The comparisons are based on computed error probabilities as the Weibull parameters are varied. Several example images are provided for empirical comparison as well.<<ETX>>
Subjective and objective quality assessment of Mobile Videos with In-Capture distortions

We designed and created a new video database that models a variety of complex distortions generated during the video capturing process on hand-held mobile capturing devices. We describe the content and characteristics of the new database, which we call the LIVE Mobile In-Capture Video Quality Database. It comprises a total of 208 videos that were captured using eight different smart-phones and were affected by six common in-capture distortions. We also conducted a subjective video quality assessment study using this data, wherein each video was assessed by 36 unique subjects. We evaluated several top-performing No-Reference IQA and VQA algorithms on the new database and find insights on how real-world in-capture distortions challenge both human subjects as well as automatic perceptual quality prediction models.
A Structural Similarity Metric for Video Based on Motion Models

Quality assessment plays a very important role in almost all aspects of multimedia signal processing such as acquisition, coding, display, processing etc. Several objective quality metrics have been proposed for images, but video quality assessment has received relatively little attention and most video quality metrics have been simple extension of metrics for images. In this paper, we propose a novel quality metric for video sequences that utilizes motion information in video sequences, which is the main difference in moving from images to video. This metric is capable of capturing temporal artifacts in video sequences in addition to spatial distortions. Results are presented that demonstrate the efficacy of our quality metric by comparing model performance against subjective scores on the database developed by the video quality experts group.
Spatio-temporal quality pooling accounting for transient severe impairments and egomotion

With the increasing popularity of video applications, the reliable measurement of perceived video quality has increased in importance. We study methods for pooling video quality scores over space and time. The method accounts for localized severe impairments of the signal which exhibit significant influence on the subjective impression of the overall signal quality. It also accounts for the effect of camera motion (egomotion) on perceived quality. The method arrived at is tested on the LIVE Video Quality Database and is shown to perform quite well.
Relating analog and digital order statistic filters

Recent efforts have been directed towards the development of analog versions of the digital median filter. Here, the concept is extended to the more general rank-order and order statistic (OS) filters. By identifying analog and digital signals that are in some sense equivalent, such that the outputs of analog and digital OS filters applied to equivalent analog and digital signals are also equivalent, the properties of discrete-time and continuous-time OS filters can be related, making possible a richer mathematical framework for analysis and design of rank-order and OS filters in both domains. While some theory for digital rank-order and OS filters has been developed, the definition of the analog versions is nontrivial. Suitable definitions are offered for analog versions of the rank-order and OS filters, and a formalism is developed for relating the analog and digital filters.<<ETX>>
Towards naturalistic depth propagation

We propose a two-stage depth propagation algorithm for semi-automatic 2D-to-3D video conversion that forces the solution towards statistical “naturalness”. First, both forward and backward motion vectors are estimated and compared to decide initial depth values, then a compensation process is adopted to further improve the depth initialization. Secondly, the luminance and initial depth are decomposed into a wavelet pyramid. Each sub-band of depth is inferred using a Bayesian formulation under a natural scene statistic prior assumption. This is incorporated into a propagation target function as a prior regularizing term. The final depth map associated with each frame of the input 2D video is optimized by composing all the sub-bands. Experimental results obtained on various sequences show that the presented method outperforms several state-of-the-art depth propagation methods.
Massive Online Crowdsourced Study of Subjective and Objective Picture Quality

Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. However, images captured using typical real-world mobile camera devices are usually afflicted by complex mixtures of multiple distortions, which are not necessarily well-modeled by the synthetic distortions found in existing databases. The originators of existing legacy databases usually conducted human psychometric studies to obtain statistically meaningful sets of human opinion scores on images in a stringently controlled visual environment, resulting in small data collections relative to other kinds of image analysis databases. Toward overcoming these limitations, we designed and created a new database that we call the LIVE In the Wild Image Quality Challenge Database, which contains widely diverse authentic image distortions on a large number of images captured using a representative variety of modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we have used to conduct a very large-scale, multi-month image quality assessment (IQA) subjective study. Our database consists of over 350 000 opinion scores on 1162 images evaluated by over 8100 unique human observers. Despite the lack of control over the experimental environments of the numerous study participants, we demonstrate excellent internal consistency of the subjective data set. We also evaluate several top-performing blind IQA algorithms on it and present insights on how the mixtures of distortions challenge both end users as well as automatic perceptual quality prediction models. The new database is available for public use at <;uri xlink:href="http://live.ece.utexas.edu/research/ChallengeDB/index.html" xlink:type="simple">http://live.ece.utexas.edu/research/ChallengeDB/index.html<;/uri>.
Temporal pooling of video quality estimates using perceptual motion models

Emerging multimedia applications have increased the need for video quality measurement. Motion is critical to this task, but is complicated owing to a variety of object movements and movement of the camera. Here, we categorize the various motion situations and deploy appropriate perceptual models to each category. We use these models to create a new approach to objective video quality assessment. Performance evaluation on the Laboratory for Image and Video Engineering (LIVE) Video Quality Database shows competitive performance compared to the leading contemporary VQA algorithms.
Localized measurement of emergent image frequencies by Gabor wavelets

The authors derive, implement, and demonstrate a computational approach for the measurement of emergent image frequencies. Measuring emergent signal frequencies requires spectral measurements accurate in both frequency and time or space, conflicting requirements that are shown to be balanced by a generalized uncertainty relationship. Such spectral measurements can be obtained from the responses of multiple wavelet-like channel filters that sample the signal spectrum, and that yield a locus of possible solutions for each locally emergent frequency. It is shown analytically that this locus of solutions is maximally localized in both space and frequency if the channel filters used are Gabor wavelets. A constrained solution is obtained by imposing a stabilizing term that develops naturally from the assumptions on the signal. The measurement of frequencies is then cast as an ill-posed extremum problem regularized by the stabilizing term, leading to an iterative constraint propagation algorithm. The technique is demonstrated by application to a variety of 2-D textured images.
AM-FM Texture Segmentation in Electron Microscopy Muscle Imaging

Describes the application of an amplitude modulation-frequency modulation (AM-FM) image representation in segmenting electron micrographs of skeletal muscle for the recognition of: (1) normal sarcomere ultrastructural pattern and (2) abnormal regions that occur in sarcomeres in various myopathies. A total of 26 electron micrographs from different myopathies mere used for this study. It is shown that the AM-FM image representation can identify normal repetitive structures and sarcomeres, with a good degree of accuracy. This system can also detect abnormalities in sarcomeres which alter the normal regular pattern, as seen in muscle pathology, with a recognition accuracy of 75%-84% as compared to a human expert.
Tasking on Natural Statistics of Infrared Images

Natural scene statistics (NSSs) provide powerful, perceptually relevant tools that have been successfully used for image quality analysis of visible light images. Since NSS capture statistical regularities that arise from the physical world, they are relevant to long wave infrared (LWIR) images, which differ from visible light images mainly by the wavelengths captured at the imaging sensors. We show that NSS models of bandpass LWIR images are similar to those of visible light images, but with different parameterizations. Using this difference, we exploit the power of NSS to successfully distinguish between LWIR images and visible light images. In addition, we study distortions unique to LWIR and find directional models useful for detecting the halo effect, simple bandpass models useful for detecting hotspots, and combinations of these models useful for measuring the degree of non-uniformity present in many LWIR images. For local distortion identification and measurement, we also describe a method for generating distortion maps using NSS features. To facilitate our evaluation, we analyze the NSS of LWIR images under pristine and distorted conditions, using four databases, each captured with a different IR camera. Predicting human performance for assessing distortion and quality in LWIR images is critical for task efficacy. We find that NSS features improve human targeting task performance prediction. Furthermore, we conducted a human study on the perceptual quality of noise-and blur-distorted LWIR images and create a new blind image quality predictor for IR images.
Model-driven, probabilistic level set based segmentation of magnetic resonance images of the brain

Accurate segmentation of magnetic resonance (MR) images of the brain to differentiate features such as soft tissue, tumor, edema and necrosis is critical for both diagnosis and treatment purposes. Region-based formulations of geometric active contour models are popular choices for segmentation of MR and other medical images. Most of the traditional region-based formulations model local region intensity by assuming a piecewise constant approximation. However, the piecewise constant approximation rarely holds true for medical images such as MR images due to the presence of noise and bias field, which invariably results in a poor segmentation of the image. To overcome this problem, we have developed a probabilistic region-based active contour model for automatic segmentation of MR images of the brain. In our approach, a mixture of Gaussian distributions is used to accurately model the arbitrarily shaped local region intensity distribution. Prior spatial information derived from probabilistic atlases is also integrated into the level set evolution framework for guiding the segmentation process. Our experiments with a series of publicly available brain MR images show that the proposed active contour model gives stable and accurate segmentation results when compared to the traditional region based formulations.
Continuous Prediction of Streaming Video QoE Using Dynamic Networks

Streaming video data accounts for a large portion of mobile network traffic. Given the throughput and buffer limitations that currently affect mobile streaming, compression artifacts and rebuffering events commonly occur. Being able to predict the effects of these impairments on perceived video quality of experience (QoE) could lead to improved resource allocation strategies enabling the delivery of higher quality video. Toward this goal, we propose a first of a kind continuous QoE prediction engine. Prediction is based on a nonlinear autoregressive model with exogenous outputs. Our QoE prediction model is driven by three QoE-aware inputs: An objective measure of perceptual video quality, rebuffering-aware information, and a QoE memory descriptor that accounts for recency. We evaluate our method on a recent QoE dataset containing continuous time subjective scores.
Antisymmetric biorthogonal coiflets for image coding

Wavelet techniques have achieved a tremendous success in image data compression. In designing wavelet coding algorithms, the choice of wavelet systems is of great importance for compression performance. We design the entire class of antisymmetric biorthogonal coiflet systems, whose filterbanks have even lengths and are linear phase. We show that one of the novel filterbanks achieves noticeably better rate-distortion performance than several state-of-the-art filterbanks in image coding.
Deep Visual Discomfort Predictor for Stereoscopic 3D Images

Most prior approaches to the problem of stereoscopic 3D (S3D) visual discomfort prediction (VDP) have focused on the extraction of perceptually meaningful handcrafted features based on models of visual perception and of natural depth statistics. Toward advancing performance on this problem, we have developed a deep learning-based VDP model named deep visual discomfort predictor (DeepVDP). The DeepVDP uses a convolutional neural network (CNN) to learn features that are highly predictive of experienced visual discomfort. Since a large amount of reference data is needed to train a CNN, we develop a systematic way of dividing the S3D image into local regions defined as patches and model a patch-based CNN using two sequential training steps. Since it is very difficult to obtain human opinions on each patch, instead a proxy ground-truth label that is generated by an existing S3D visual discomfort prediction algorithm called 3D-VDP is assigned to each patch. These proxy ground-truth labels are used to conduct the first stage of training the CNN. In the second stage, the automatically learned local abstractions are aggregated into global features via a feature aggregation layer. The learned features are iteratively updated via supervised learning on subjective 3D discomfort scores, which serve as ground-truth labels on each S3D image. The patch-based CNN model that has been pretrained on proxy ground-truth labels is subsequently retrained on true global subjective scores. The global S3D visual discomfort scores predicted by the trained DeepVDP model achieve the state-of-the-art performance as compared with previous VDP algorithms.
Optimal power allocation for minimizing visual distortion over MIMO communication systems

A recent dynamic increase in demand for wireless multimedia services has greatly accelerated the research on cross layer optimization techniques for transmitting multimedia data over wireless channel. In this paper, we explore a novel theoretical approach for joint optimization between the rate distortion (RD) of H.264/AVC video and the link-capacity of MIMO parallel subchannels. We obtain the optimal power level of subchannels through an optimization problem to minimize total visual distortion. In the simulation results, compared to the water filling (WF) method, the proposed scheme provides better results in aspects of visual quality in the face of sum rate loss.
Extracting essential modulated image structure

We present a new theory of multidimensional AM-FM image modeling and derive algorithms for extracting AM-FM sub-image information from digital images. In contrast to Fourier components, AM-FM image functions admit arbitrarily varying amplitude and phase modulations. Thus, they are inherently capable of efficiently capturing essential nonstationary image structures. Often, such nonstationarities contribute significantly to visual perception and interpretation. We describe a practical approach for computing AM-FM image representations using nonlinear demodulation operators. A Gabor filterbank isolates components locally, and optimal filters based on a statistical state-space component model are used to trade image multi-components across the filterbank channel responses. We present dramatic examples where the essential structure of natural images is successfully recovered from their computed AM-FM representations.
FOVEA: a foveated vergent active stereo system for dynamic three-dimensional scene recovery

We introduce FOVEA: a Foveated Vergent Active stereo vision system. FOVEA actively directs a pair of vergent stereo cameras to fixate on surfaces in a scene, performing multiresolution surface depth recovery at each fixation point, and accumulating and integrating a multiresolution map of surface depth over multiple successive fixations. Several features of the system are novel: a foveated image sampling and processing strategy is shown to greatly simplify the problem of establishing; a probabilistic fixation strategy is developed that is driven by the scene structure; the system uses the fixation strategy to recover local depth maps at a high resolution at multiple fixation points, eventually mapping the entire scene; and finally, the local maps are integrated as they are acquired into a global depth map.
Full-reference quality assessment of stereoscopic images by modeling binocular rivalry

We develop a framework for assessing the quality of stereoscopic images that have been afflicted by possibly asymmetric distortions. An intermediate image is generated which when viewed stereoscopically is designed to have a perceived quality close to that of the cyclopean image. We hypothesize that performing stereoscopic QA on the intermediate image yields higher correlations with human subjective judgments. The experimental results confirm the hypothesis and show that the proposed framework significantly outperforms conventional 2D QA metrics when predicting the quality of stereoscopically viewed images that may have been asymmetrically distorted.
Blind image quality assessment on real distorted images using deep belief nets

We present a novel natural-scene-statistics-based blind image quality assessment model that is created by training a deep belief net to discover good feature representations that are used to learn a regressor for quality prediction. The proposed deep model has an unsupervised pre-training stage followed by a supervised fine-tuning stage, enabling it to generalize over different distortion types, mixtures, and severities. We evaluated our new model on a recently created database of images afflicted by real distortions, and show that it outperforms current state-of-the-art blind image quality prediction models.