nils-herrmann/clean_scirepeval huggingface dataset:
Using Free Energy Principle For Blind Image Quality Assessment

In this paper we propose a new no-reference (NR) image quality assessment (IQA) metric using the recently revealed free-energy-based brain theory and classical human visual system (HVS)-inspired features. The features used can be divided into three groups. The first involves the features inspired by the free energy principle and the structural degradation model. Furthermore, the free energy theory also reveals that the HVS always tries to infer the meaningful part from the visual stimuli. In terms of this finding, we first predict an image that the HVS perceives from a distorted image based on the free energy theory, then the second group of features is composed of some HVS-inspired features (such as structural information and gradient magnitude) computed using the distorted and predicted images. The third group of features quantifies the possible losses of “naturalness” in the distorted image by fitting the generalized Gaussian distribution to mean subtracted contrast normalized coefficients. After feature extraction, our algorithm utilizes the support vector machine based regression module to derive the overall quality score. Experiments on LIVE, TID2008, CSIQ, IVC, and Toyama databases confirm the effectiveness of our introduced NR IQA metric compared to the state-of-the-art.
Efficient image sensor noise estimation via iterative re-weighted least squares

Noise estimation is crucial in many image processing algorithms such as image denoising. Conventionally, the noise is assumed as signal-independent additive white Gaussian process. However, for the real raw-data of imaging sensors, the present noise is better modeled as signal-dependent noise. In this work, we propose an efficient image sensor noise estimation method based on iterative re-weighted least squares optimization. Specifically, the image patches are first clustered into different groups, each of which will generate a data sample. To fit those observations robustly, we introduce a weighting matrix to reflect the credibility of each sample. Unfortunately, this setting of weighting matrix in turn depends on the unknown noise parameters. We then develop an iterative re-weighted least squares optimization procedure, in which the weighting matrix and parameter estimates can be updated alternately. Experimental results show that our method outperforms the state-of-the-art works, in terms of both estimation accuracy and computational efficiency.
A general histogram modification framework for efficient contrast enhancement

In this paper we propose a new general histogram modification framework for contrast enhancement. The proposed model works with a hybrid transformation technique to improve image brightness and contrast based on an optional histogram matching in terms of reassigned probability distribution and S-shaped transfer mapping. Experimental results conducted on natural, dimmed, and tone-mapped images show that the proposed technique creates enhanced images efficiently with equivalent or superior visual quality to those produced by classical and state-of-the-art enhancement approaches.
Subjective and objective quality assessment for images with contrast change

It is widely known that, for most natural images, appropriate contrast enhancement can usually lead to improved subjective quality. Despite of its importance to image processing, contrast change has largely been overlooked in the current research of image quality assessment (IQA). To fill this void, in this paper we first report a new and dedicated contrast-changed image database (CID2013). The CID2013 database is composed of four hundred contrast-changed images of fifteen original natural images and the mean opinion scores (MOSs) recorded from twenty-two inexperienced viewers. We then proposed a novel reduced-reference image quality metric for contrast-changed images (RIQMC) using entropies and order statistics of the image histograms. Experimental results on the CID2013, TID2008, and CSIQ databases demonstrate that the proposed RIQMC metric outperforms some mainstream image quality assessment methods.
Spatial error concealment with adaptive linear predictor

In this paper, we propose a novel spatial error concealment algorithm based on adaptive linear predictor. The predictor automatically tunes its order and support shape according to local contexts. The optimal order is determined using Bayesian Index Criterion (BIC). The order-adaptive predictor is able to recover more important features or structures. Simulation results show that compared to the state-of-the-art error concealment algorithms, the proposed method gives better reconstruction performance in terms of objective and subjective evaluations.
Sample-based image completion using structure synthesis

Image completion technique is widely used in image processing applications such as textural recovery, object removal, image edit, etc. When filling in the missing areas of an image, it is often a challenge to keep local consistency of image structures while avoiding ambiguity and visual artifacts. To tackle with this problem, we propose a robust sample-based image completion scheme which is a cascade of two major procedures. First, we extract structural information from both source and sample images and then perform template matching under constraints of boundary band map and contour consistency to reconstruct the damaged structures. Second, a weighted exemplar-based image synthesis algorithm is further devised taking the previous structural information and matching results into account. Extensive experiments and comparative study show the reliability and superiority of our image completion algorithm.
Using image signature for effective and efficient reduced-reference image quality assessment

Reduced-reference (RR) image quality assessment (IQA) with only partial information of the reference image available, has aroused increasing research interests nowadays. Many efforts have been devoted to this area for years, and have introduced various kinds of effective models. However, those arithmetics are usually extremely complicated. Therefore, we in this paper propose a new RR Image-Signature (RRIS) induced IQA metric by estimating the similarity between the transformed images that are induced by the image signature. The main design principle of the proposed method is that the image signature can capture the main information of an image with very few features and computational cost. The proposed algorithm is tested and verified on four large-size image quality databases (TID2008, CSIQ, LIVE and CID2013). Experimental results demonstrate the effectiveness and efficiency of our algorithm over those competing full- and reduced-reference IQA methods.
IVQAD 2017: An immersive video quality assessment database

This paper presents a new database, Immersive Video Quality Assessment Database 2017 (IVQAD 2017), intended for immersive video quality assessment in virtual reality environment. Video quality assessment (VQA) plays an important role in video research fields. Nowadays virtual reality technology have been widely used and playing videos in virtual reality visual system is becoming more and more popular. However, existing research in VQA fields mainly focus on traditional videos. In this paper, we build the IVQAD which contains 10 raw videos and 150 distorted videos. Bit rate, frame rate and resolution were considered as quality degradation factors. All the videos were encoded with MPEG-4. Subjects were asked to assess the video under virtual reality environment and mean opinion score (MOS) was derived by computing. Using IVQAD 2017, researchers can explore the influence of resolution, video compression and video packet loss on immersive videos' quality.
Adapting hierarchical ALS algorithms for temporal psychovisual modulation

Temporal psychovisual modulation (TPVM) is a new information display technology, which aims to generate multiple visual percepts for different viewers on a single display simultaneously. In TPVM system, the viewers with different active liquid crystal (LC) glasses (i.e., different modulation weights) which are synchronized with the display can see different images (called personal views). TPVM can be implemented by nonnegative matrix factorization (NMF) with three additional constrains: the values of images and modulation weights should have upper bound; a special view (called shared view) without using viewing devices should be considered (i.e., the sum of all basis images should be a meaningful image); the sparsity of modulation weights should be considered because of the material property of LC. In this paper, we solve the constrained NMF problem by the modified hierarchical alternating least squares (HALS) algorithms. Through experiments, we analyse the influence of different parameters of TPVM to provide a guideline for parameter selection. This paper will provide an algorithmic guidance for the applications of TPVM.
Defeating Camcorder Piracy by Temporal Psychovisual Modulation

We present a new video projection technique to defeat camcorder piracy in movie theaters using the newly emerged display paradigm of temporal psychovisual modulation. The technique exploits the difference in image formation mechanisms between human eyes and digital cameras: in the human visual system, images are formed via continuous integration of the light field, whereas digital video acquisition is based on discrete sampling of light sensors. A movie frame is decomposed into multiple so-called shale frames such that, when these shale frames are displayed at a frame rate higher than 60 Hz, the human viewers will experience normal movie presentation without any noticeable artifacts, but the video frames captured by camcorders will contain severe, highly objectionable artifacts, depriving the pirated video contents of any commercial values.
Dual-view medical image visualization based on spatial-temporal psychovisual modulation

Medical imaging technologies such as magnetic resonance imaging (MRI) and computerized tomographic (CT) are used to diagnose a wide range of medical diseases. Medical images are generated by detecting density differences between different tissues in the body. Multiple medical image visualization is of critical importance to diagnosis. This paper introduces a dual-view medical image visualization prototype based on spatial-temporal psychovisual modulation (STPVM). Temporal psychovisual modulation (TPVM) enables a single display to generate multiple visual content for different viewers. Spatial psychovisual modulation (SPVM) extends the idea of TPVM to spatial domain. STPVM combines TPVM and SPVM by exploiting both temporal and spatial redundancy of modern displays. Based on STPVM technology, one display can present even more images simultaneously. In this demo, two kinds of medical images e.g. T1 and T2 weighted MRI images, are presented simultaneously. Physicians can switch between either image by just moving the eye fixations. Since T1 and T2 are shown simultaneously and are aligned on the screen, it is more convenient for the physicians to get different information of the same spot from the T1 and T2 images. The developed demo is useful for physicians during surgery navigation and effectively reduces the burden of mental transfer.
Image quality assessment metrics based on multi-scale edge presentation

We propose two image quality assessment metrics named multi-scale modular similarity (M/sup 2/S) and multi-scale modular maxima similarity (M/sup 3/S). It has been well known 1) multi-scale analysis is an effective decomposition technique in image processing, and 2) contours and edges analyses are crucial in the understanding of natural scenes. Motivated by these two facts, we attempt to develop quality assessment metrics using multi-scale edges presentation. We decompose an image with un-decimated dyadic wavelet transform, and then develop M/sup 2/S metric to evaluate the quality of images by comparing the modulus across scales of wavelet transform. Multi-scale edges are defined as local maxima of modulus, which often contain the most important information of the image. As a further step of M/sup 2/S metric, M/sup 3/S only uses the multi-scale edge information. M/sup 3/S is therefore essentially a reduced-reference image quality metric. Extensive experiments indicate that in most cases, the prediction abilities of these two proposed metrics are similarly excellent and both outperform the widely used PSNR and the simple structural similarity metrics.
Unified Blind Quality Assessment of Compressed Natural, Graphic, and Screen Content Images.

Digital images in the real world are created by a variety of means and have diverse properties. A photographical natural scene image (NSI) may exhibit substantially different characteristics from a computer graphic image (CGI) or a screen content image (SCI). This casts major challenges to objective image quality assessment, for which existing approaches lack effective mechanisms to capture such content type variations, and thus are difficult to generalize from one type to another. To tackle this problem, we first construct a cross-content-type (CCT) database, which contains 1,320 distorted NSIs, CGIs, and SCIs, compressed using the high efficiency video coding (HEVC) intra coding method and the screen content compression (SCC) extension of HEVC. We then carry out a subjective experiment on the database in a well-controlled laboratory environment. Moreover, we propose a unified content-type adaptive (UCA) blind image quality assessment model that is applicable across content types. A key step in UCA is to incorporate the variations of human perceptual characteristics in viewing different content types through a multi-scale weighting framework. This leads to superior performance on the constructed CCT database. UCA is training-free, implying strong generalizability. To verify this, we test UCA on other databases containing JPEG, MPEG-2, H.264, and HEVC compressed images/videos, and observe that it consistently achieves competitive performance.
Adaptive high-frequency clipping for improved image quality assessment

It is widely known that the human visual system (HVS) applies multi-resolution analysis to the scenes we see. In fact, many of the best image quality metrics, e.g. MS-SSIM and IW-PSNR/SSIM are based on multi-scale models. However, in existing multi-scale type of image quality assessment (IQA) methods, the resolution levels are fixed. In this paper, we examine the problem of selecting optimal levels in the multi-resolution analysis to preprocess the image for perceptual quality assessment. According to the contrast sensitivity function (CSF) of the HVS, the sampling of visual information by the human eyes approximates a low-pass process. For images, the amount of information we can extract depends on the size of the image (or the object(s) inside) as well as the viewing distance. Therefore, we proposed a wavelet transform based adaptive high-frequency clipping (AHC) model to approximate the effective visual information that enters the HVS. After the high-frequency clipping, rather than processing separately on each level, we transform the filtered images back to their original resolutions for quality assessment. Extensive experimental results show that on various databases (LIVE, IVC, and Toyama-MICT), performance of existing image quality algorithms (PSNR and SSIM) can be substantially improved by applying the metrics to those AHC model processed images.
Stereoscopic image quality assessment with the dual-weight model

Three-dimensional (3D) imaging technology has been growingly prevalent in today's world. But objective quality assessment of 3D images is a challenging task. In this paper, we try to investigate and develop the 2D image quality metrics into stereoscopic image quality assessment using a dual-weight model within the concept of free energy principle and human visual system. On the basis of a psychological measure, the free energy is a principle telling where supervises more and attracts human attention. We believe that the “surprise” can account for the binocular rivalry and introduce the inter-weight. Also, the intra-weight is computed according to the physiology of human visual system. We first do the intra-weighted 2D image quality assessment. With inter-weights, we then calculate the combining score of stereopairs. Experiments are performed on the symmetric LIVE3D-I and asymmetric LIVE3D-II databases. Results confirm that the proposed 3D IQA technique, appropriately computing and combining the inter- and intra-weights, is able to faithfully predict the visual quality of stereopairs.
The impact of phase transition on quality assessment of natural images

Recent research indicates that the well known scale invariance property of natural images may originate from the existence of the critical image layer exhibiting phase transition from disordered to ordered states as the intensity scale increases. In this paper we examine the impact of phase transition on perceptual quality assessment of images and confirm that using only the ordered or structured information in the image as identified by phase transition can lead to improved performance, i.e. “part is better than all”. To explicitly locate the phase transition layer, we collected 200,000 10×10 patches from the hierarchically decomposed 8-bit images from the Kodak database and trained a 2D Ising model. Analysis of the Ising model parameters suggests that phase transition happens approximately at the 4-th layer (4-th bitplane from the least significant bit). We have also tested the performance of image quality metrics using different combinations of the intensity layers. It is interesting to notice that for all the test scenarios, the combined 4-th to 8-th layers, corresponding to the ordered or structured image information give the best performance. This observation confirms that our visual perception does rely on structured information when performing visual quality assessment. And the structured information of images can be extracted by the hierarchical decomposition with a phase transition model to facilitate perceptual image quality assessment.
Bayesian error concealment with DCT pyramid

In this paper, the problem of concealing missing image/video blocks is casted into a framework of Bayesian estimation. The conditional expectation of the missing block vector is taken over a pilot vector of correctly decoded pixels near the missing block. Multiple observations of the missing vector and pilot vectors obtained in a nonlocal manner are used to approximate the expectation. We design a multiscale estimation approach with DCT pyramid to improve estimation efficiency. The DC image of the missing block is recovered first, and then more details related to high frequency AC coefficients are recovered successively. The algorithm is found to be quite competitive among state-of-the-art, and more substantial improvement over existing algorithms on image with heavy loss rate/large block size is observed in our experiments.
Single image detail enhancement via residual homogeneity

In this paper, we put forward a novel method in single image detail enhancement — residual homogeneity (RH). Residual homogeneity is a statistic result when we test different kinds of blocks in database, finding that blocks in residual images are with similar structure, that is, they are homogenous. Based on residual homogeneity, we make a simple assumption that detail layer of a single image is residual component after fast in-place search and block match. Then images are enhanced by applying framework of RH. Unlike many popular algorithms that need to adjust parameters by manual operation to get best performance, our approach is adaptive. Besides, many algorithms output images with intensity change, our RH can keep images from over enhancement with a nature look. RH is also robust to low bit rate H.265 system and runs faster than most popular methods. Moreover, it can be easily FPGA implemented as well. Numbers of experiments testify that our algorithm is robust with good performance both subjectively and objectively.
Factorization Algorithms for Temporal Psychovisual Modulation Display

Temporal psychovisual modulation (TPVM) is a new information display technology which aims to generate multiple visual percepts for different viewers on a single display simultaneously. In a TPVM system, the viewers wearing different active liquid crystal (LC) glasses with varying transparency levels can see different images (called personal views). The viewers without LC glasses can also see a semantically meaningful image (called shared view). The display frames and weights for the LC glasses in the TPVM system can be computed through nonnegative matrix factorization (NMF) with three additional constrains: the values of images and modulation weights should have upper bound (i.e., limited luminance of the display and transparency level of the LC); the shared view without using viewing devices should be considered (i.e., the sum of all basis images should be a meaningful image); and the sparsity of modulation weights should be considered due to the material property of LC. In this paper, we proposed to solve the constrained NMF problem by a modified version of hierarchical alternating least squares (HALS) algorithms. Through experiments, we analyze the choice of parameters in the setup of TPVM system. This work serves as a guideline for practical implementation of TPVM display system.
A psychovisually tuned image codec

A psychovisual quality driven image codec exploiting the psychological and neurological process of visual perception is proposed in this paper. Recent findings in brain theory and neuroscience suggest that visual perception is a process of fitting brain's internal generative model to the outside retina stimuli. And the psychovisual quality is related to how accurately visual sensory data can be explained by the internal generative model. Therefore, the design criterion of our psychovisually tuned image compression system is to find a compact description of the optimal generative model from the input image on the encoding end, which is then used to regenerate the output image on the decoding end. By noting an important finding from empirical natural image statistics that natural images have scale invariant features in the pixels' high order statistics, the generative model can be efficiently compressed through model preserving spatial downsampling on the encoder. And the decoder can reverse the process with a model preserving upsampling module to generate the decoded image. The proposed system is fully standard complaint because the downsampled image can be compressed with any exiting codec (JPEG2000 in this work). The proposed algorithm is shown to systematically outperform JPEG2000 in a wide bit rate range in terms of both subjective and objective qualities.