nils-herrmann/clean_scirepeval huggingface dataset:
Comparative image quality assessment using free energy minimization

It is a straightforward task for human observers to judge the relative quality of two visual signals of the same content, but subject to different type/level of distortions. However, this comparative image quality assessment (C-IQA) problem remains a difficult challenge for the current research of image quality assessment (IQA). In this paper, we propose a C-IQA approach to predict the relative perceptual quality of a pair of images that are possibly subject to different artifact types/levels. The C-IQA algorithm is designed to emulate the process of comparing the relative quality of two visual stimuli as performed by the human visual system (HVS) within the framework of free energy minimization. The brain's internal generative models initialized on the inputs are used to explain the two images. And their relative quality can then be determined through comparing the free energy level of this model-data fitting process. In the existing work of IQA, the full-reference (FR) and reduced-reference (RR) methods need the prior knowledge of the original images while the no-reference (NR) algorithms usually work with a single input image. The C-IQA approach is inherently different from those existing methods in that it takes as input an image pair and predicts their relative quality without using any knowledge about the original image. A computationally efficient solution to the proposed C-IQA scheme based on a linear autoregressive image model is also introduced. Experimental results show that the proposed method achieves about 98% accuracy in line with the subjective ratings when applied on over 300,000 image pairs sampled from the LIVE database, outperforming the FR metrics such as PSNR, SSIM, and some of the most advanced NR IQA algorithms.
Noise estimation using statistics of natural images

We develop a framework for estimating noises of natural images using two important properties of natural image statistics: high kurtosis and scale invariance of natural images in certain transform domains. We examine the effects of additive independent noise on the third and fourth moments of the transformed image signal (skewness and kurtosis). By exploring the said priors of high kurtosis and scale invariance of natural image statistics in 2D discrete cosine transform domain and random unitary transform domain, we derive constrained nonlinear optimization algorithms for accurate estimation of noise variance. Simulation and comparative study show that the proposed approach is capable of estimating the variance of Gaussian additive noise with a relative error as low as one percent. Moreover, the new estimation approach is shown to be effective on multiplicative-additive compound noises as well. This work can significantly improve the performance of existing denoising techniques that require the noise variance as a critical parameter.
Using JPEG and JPEG2000 compressions for fast image quality metrics based on free energy theory

In this paper, we propose a fast, effective and practical algorithm for image quality assessment (IQA). Recently, a new free energy theory was revealed in the field of brain science, which illustrates that the human visual system (HVS) always strives to comprehend the input visual signal by reducing the undetermined portions. Inspired by this, our previous work recently designed a valid reduced-reference (RR) free energy based distortion metric (FEDM) using linear autoregressive model. Despite of fairly well performance, the FEDM is yet difficult to work in real-time applications owing to its weak portability and considerable computational load. Using an alternative way, this paper approaches the free energy related mechanism in human brains with JPEG and JPEG2000 compressions. The proposed metric can work quickly and practically in that it is realized with highly developed and widely employed image compression methods. Results of experiments on the most popular publicly-available LIVE database demonstrate the effectiveness of the proposed RR algorithm over classical full-reference PSNR and SSIM as well as state-of-the-art RR IQA metrics.
Simultaneous dual-subtitles exhibition via Spatial Psychovisual Modulation

Dual-language annotation for in-theatre movie exhibition is a useful technique to facilitate the understanding of the movie for audiences from different nations and with different cultural backgrounds. Currently, the most popular solution is the direct superimposition of subtitles in a pair of different languages over the movie, e.g. English + Chinese which can often be seen in cinema in China today. An obvious drawback of this straightforward solution is that the subtitle area often occludes the movie contents and even becomes annoying for the audience. In this paper, we propose a new Spatial Psychovisual Modulation (SPVM) based solution to the dual-language subtitling problem. SPVM is a new paradigm of information display exploiting the mismatch between high resolution of the modern optoelectronic displays and limited spatial resolution of the human visual system (HVS). In this work, we design a simultaneous dual-subtitle exhibition system using 2 synchronized projectors with linear polarization filters and polarization glasses. Most audiences can enjoy the movie as usual with a default subtitle language, say English, while others have the option of only seeing subtitle in the other language, e.g. Chinese using the polarization glasses. We have implemented the system and experimental results will be demonstrated to justify the effectiveness and robustness of the proposed dual-subtitle exhibition system.
Uncrowded Window Inspired Ultra High Definition Television Display

Ultra high definition television (UHDTV) has gradually entered our daily life. However, because of the large data of UHDTV, it is hard to render images in real time. We proposed an eye tracking based solution using the concept of uncrowned window from vision research. The theory of uncrowded window suggests that human vision can only effectively recognize objects inside a small window. Object features outside the window cannot be combined properly and therefore they will not be recognizable. We use eye tracker to locate fixation points of the user in real time, and the area inside the uncrowded window displays the results of some advanced image processing algorithms such as deblurring, up scaling, tone mapping of high dynamic range (HDR) imaging and contrast enhancement. Outside of the window we can just process the images simply. Since the user can only see within the uncrowded window, detrimental impact of those poor images outside of the window is almost negligible. The proposed prototype system was written in C++ with SDKs of DirectX, Tobii Gaze SDK, OpenCV, CEGUI and etc. Demonstration of the system will be provided to show that the proposed method is an effective solution to UHDTV display.
No-reference quality assessment for JPEG compressed images

JPEG is a most commonly used standard of compression for digital images. Quality factor (Qfactor) for JPEG compressed image is actually a suitable indicator to the perceptual quality. However, the information of the compressor might be unknown due to various reasons. To evaluate the Qfactor, we recompress the formerly compressed image and measure the consistency between them. Then we define the fixed points (the points on the Qfactor-axis where the content of recompressed images are almost the same with that of directly compressed images) by following the Qfactor based specifications and form the image set. The quality of JPEG compressed images are measured by combining the estimated Qfactor with the features extracted from the image set. The experimental results confirm that the proposed image quality assessment technique, which is no-reference, is able to faithfully predict the visual quality of JPEG compressed images.
Quality Assessment Considering Viewing Distance and Image Resolution

Viewing distance and image resolution have substantial influences on image quality assessment (IQA), but this issue has been highly overlooked in the literature so far. In this paper, we examine the problem of optimal resolution adjustment as a preprocessing step for IQA. In general, the sampling of visual information by human eyes' optics is approximately a low-pass process. For a given visual scene, the amount of the extractable information greatly depends on the viewing distance and image resolution. We first introduce a novel dedicated viewing distance-changed image database (VDID2014) with two groups of typical viewing distances and image resolutions to promote the IQA study for this issue. Then we design a new effective optimal scale selection (OSS) model in dual-transform domains, in which a cascade of adaptive high-frequency clipping in the discrete wavelet transform domain and adaptive resolution scaling in the spatial domain is used. Validation of our technique is conducted on five image databases (LIVE, IVC, Toyama, VDID2014, and TID2008). Experimental results show that the performance of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) can be substantially improved by applying these metrics to OSS model preprocessed images, superior to classical multi-scale-PSNR/SSIM and comparable to the state-of-the-art competitors.
Example-based image contrast enhancement

In this paper, a novel example-based contrast enhancement algorithm is proposed. The proposed approach enhances the contrast by learning some important informative priors from the histogram of the example image. The experimental results indicate that the proposed Example-based Dist-Stretched (ExDS) contrast enhancement algorithm can boost the image contrast effectively. And thanks to the example-based learning process, the output images from the ExDS algorithm have more natural looking than those of traditional histogram equalization based methods. The proposed ExDS algorithm can also be extended to the applications of contrast correction for old film restoration as well as tone mapping for image and video post-productions.
Bayesian Error Concealment With DCT Pyramid for Images

In this paper, the problem of concealing missing image blocks is casted into a framework of Bayesian estimation. The conditional expectation of the missing block vector is taken over a pilot vector of correctly decoded pixels near the missing block. Multiple observations of the missing vector and pilot vectors obtained in a neighborhood are used to approximate the expectation. We design a multiscale estimation approach with discrete cosine transform pyramid to improve estimation efficiency. The DC image of the missing block is recovered first, and then more details related to high-frequency AC coefficients are recovered successively. Moreover, the algorithm operates in an iterative mode through using estimated block to refine the searching process for the next estimation. The algorithm is found to perform very well for a wide range of block loss rates. Substantial improvement over 14 existing error concealment (inclusive of inpainting) algorithms on various images is demonstrated in our extensive experiments, under different test conditions inclusive of high-loss rates and large block sizes.
Global quality of assessment and optimization for the backward-compatible stereoscopic display system

The backward-compatible stereoscopic display is a technology that a stereoscopic view is perceived with 3D glasses while a 2D version of the 3D image is concurrently available for naked-eye viewers on the same physical display medium. This unique functionality is achieved by an information display technology Temporal Psychovisual Modulation (TPVM), an interesting interplay between high refresh rate optoelectronic display, signal processing and psychophysics. However, the current performance of the system is not satisfactory, and it is a trade-off keeping simultaneously the best performance of the 3D view and the 2D view. The global quality of 3D scene and 2D scene plays a great important impact on user's quality-of-experience. In this paper, we are the first to put forward the concept of global quality, including two components: the quality of 3D view and 2D view. Then we have constructed the display system prototype and first proposed quality assessment criteria to evaluate the quality of both 3D scene and 2D scene, towards as guidance to improve their performances. We conduct subjective experiments to figure out when the system runs the best under the criteria. Experimental results demonstrate that we can effectively calculate the optimal quality of this system using the quality assessment criteria.
Information security display system on android device

With the development of the modern technology, there is increasing a lot of important secret information to be protected. In this paper, an information security display system based on spatial psychovisual modulation (SPVM) is introduced on the android mobile platform. SPVM is a new way of information display technology which is conceived as an interplay of signal processing, optoeletronics and psychophysics. Nowadays modern displays support very high pixel density. Meanwhile the human visual system (HVS) cannot distinguish the image signals with spatial frequency more than a certain threshold. By this idea, we can design a new information security display system by utilizing the inconformity between HVS and the modern displays' resolutions. In this system, two types of viewers are defined, the first viewers wearing a pair of polarized glasses are authorized to see the secret information, however the second unauthorized naked-eye viewers without glasses can only see the disguise images. Thus, the different contents on the same display can be seen through different viewing devices. What is more, the increasing stereoscopic display technologies drive amounts of polarization based spatial multiplexing available. So we choose the polarization based stereoscopic screen as the display in this information security display system. The information security display system, as a guidance and concrete implementation of SPVM, attains remarkable performance in our experiments.
A new reduced-reference image quality assessment using structural degradation model

Image quality assessment (IQA) is an important research area in image processing. Reduced-reference (RR) IQA methods contained therein mainly aim to estimate image quality degradations with partial information about the reference image. Following the remarkable achievement of SSIM, structural information has been recognized as one key factor, and has aroused many image quality metrics so far. In this paper, we design a structural degradation model (SDM). Then, the quality score of an image is defined as a nonlinear combination, or SVM based integration, of distance between the structural degradation information of the original and distorted images. Accordingly, a new RR IQA approach using the SDM model is exploited. Experimental results on LIVE database are provided to justify the superior prediction accuracy performance of the proposed method as compared to three significant image quality metrics, PSNR, SSIM and FEDM.
Movie piracy tracking using temporal psychovisual modulation

Nowadays, camcorder piracy has great impact on the motion picture industry. Although some watermarking technologies can track the movie pirate, the video content viewed in the theater may be affected and they cannot obstruct the need of pirated movie because the watermarks in pirated moves are invisible. This paper presents a new method to defeat camcorder piracy and realize content protection in the theater using a new paradigm of information display technology, called Temporal Psychovisual Modulation (TPVM), which utilizes the differences between the human-eye perception and digital camera imageforming to stack an invisible pattern on digital screen and projector. The images formed in human vision are continuous integration of the light field, while discrete sampling is used in digital video acquisition which has “blackout” period in each sampling cycle. Based on this difference, we can decompose a movie into a set of display frames with specific patterns and broadcast them out at high speed so that the audience cannot notice any disturbance, while the video frames captured by camcorder will contain highly objectionable artifacts (i.e., the patterns). The pattern embedded in the movies can also serves as tracking information to reveal the one responsibility for the camcorder piracy.
Perceptually meaningful quadtree decomposition using dual-homogeneous criteria

Saliency detection is a powerful tool for many applications because it can provide valuable information reflecting human visual attention. Quadtree decomposition is a classical and efficient image operator which can divide image into square blocks in different scales. The blocks in smaller size contain valuable information such as edges and texture. In image lossy compression or similar applications, we want to reserve valuable regions (salient objects) as much as we can, meanwhile, we also want to suppress non-salient regions human are not interested in. For these reasons, we propose an ingenious saliency guided quadtree decomposition model. The proposed model outperforms traditional methods in two aspects: It reserves the salient regions well, and the edges can be protected simultaneously.
Scalable visual sensitivity profile estimation

We propose a computational model for estimating scalable visual sensitivity profile (SVSP) of video, which is a hierarchy of saliency maps that simulates the bottom-up and top- down attention of the human visual system (HVS). The bottom- up process considers low level stimulus-driven visual features such as intensity, color, orientation and motion. The top-down process simulates the high level task-driven cognitive features such as finding human faces and captions in the video. The nonlinear addition model has been used for integrating low level visual features. A full center-surrounded receptive field profile is introduced to provide spatial scalability of the model. Due to the hierarchical nature, the proposed SVSP can be directly used to augment the visual quality of codings with spatial scalability. To justify the effectiveness of the proposed SVSP, extended experiments of its application in visual quality assessment are conducted.
Hybrid modeling of natural image in wavelet domain

Natural image is characterized by its highly kurtotic and heavy-tailed distribution in wavelet domain. These typical non-Gaussian statistics are commonly described by generalized Gaussian density (GGD) or α-stable distribution. However, each of the two models has its own deficiency to capture the variety and complexity of real world scenes. Considering the statistical properties of GGD and α-stable distributions respectively, in this paper we propose a hybrid statistical model of natural image's wavelet coefficients which is better in describing the leptokurtosis and heavy tails simultaneously. Based on a linearly weighted fusion of GGD and α-stable functions, we derive the optimal parametric hybrid model, and measure the model accuracy using Kullback-Leibler divergence, which evaluates the similarity between two probability distributions. Experiment results and comparative studies demonstrate that the proposed hybrid model is closer to the true distribution of natural image's wavelet coefficients than single GGD or α-stable modeling.
Hybrid parametric-nonparametric modeling with application to natural image upsampling

Linear autoregressive (AR) model is widely used in signal processing. Usually the AR models are solved by classical least square (LS) method. An important issue with the LS solution of the AR model, which has been seemingly overlooked, is its numerical stability. The issue is related to the rank condition of the design matrix. We observed, in case of natural images, that the probability of numerical rank deficiency is rather high, roughly thirty-five per cent, due to discrete nature and structures of the digital images. Without care numerical rank deficiency can adversely affect the parameter estimation of the AR model. In this paper we use the rank revealing QR (RRQR) factorization to select optimal subset from the design matrix so as to effectively lower the condition number of the system. By removing the ill conditioned part of the right orthogonal matrix of the RRQR decomposition, we obtain a robust truncated solution to the linear system. On the other hand, for natural images, the unselected data tend to highly correlate with the pixel being modeled, and their exclusion from the modeling process waste valuable information. To avoid this loss we recycle the data including those discard by the parametric AR estimator into a nonparametrgic model of nonlocal type. Interestingly, the data that cause ill condition to the parametric AR model are of high quality for the non-local nonparametric modeling. Therefore, an approach of hybrid parametric-nonparametric modeling can make the best use of data and improve the model performance. The hybrid modeling approach is applied to image resolution upconversion, and it greatly improves the performance of the state-of-the-art image interpolator, achieving a gain of 3dB or more in PSNR in some cases.
Perceptual image quality assessment combining free-energy principle and sparse representation

Since the purpose of objective image quality assessment is to be consistent with subjective image quality assessment as highly as possible, the understanding of the mechanisms of human visual system will certainly benefit the study of objective image quality assessment. Recent developments in brain theory and neuroscience, particularly the free-energy principle, account for the perception and understanding of visual scenes. As the free-energy principle conjectures, the brain tries to generate the corresponding prediction for its encountered scene by an internal generative model. On the other hand, sparse representation is evidenced to resemble the neural response properties of simple cells in the primary visual cortex. Conjunctively, in this paper, we suppose the prediction manner of the internal generative model in free-energy principle follows sparse representation and propose an image quality metric accordingly. Experiments on LIVE, TID2008 and CSIQ image databases demonstrate the effectiveness of the proposed image quality metric. Noteworthily, our metric needs little information (only a single scalar) of the reference image and is training-free.
Sound influences visual attention discriminately in videos

In the long term research of visual attention, various computational models have been proposed. However, most of those works do not take audio into consideration. In practice, visual signals often come along with audio. Therefore, it is natural to investigate the influence of audio on visual attention. In this paper, we focus on the problem of when will audio influence visual attention during video watching. We perform eye-tracking experiments on a set of 60 videos in audio-visual (AV) and visual (V) conditions. Based on the eye movement data, we find that the influence of audio on visual attention depends on the consistency between the visual and audio signals. If the salient objects from the visual perspective are not consistent with the salient objects from the audio perspective, audio will influence visual attention. Otherwise, audio has little influence on visual attention.