nils-herrmann/clean_scirepeval huggingface dataset:
Deep learning network for blind image quality assessment

Nowadays, blind image quality assessment (BIQA) has been intensively studied with machine learning, such as support vector machine (SVM) and k-means. Existing BIQA metrics, however, do not perform robust for various kinds of distortion types. We believe this problem is because those frequently used traditional machine learning techniques exploit shallow architectures, which only contain one single layer of nonlinear feature transformation, and thus cannot highly mimic the mechanism of human visual perception to image quality. The recent advance of deep neural network (DNN) can help to solve this problem, since the DNN is found to better capture the essential attributes of images. We in this paper therefore introduce a new Deep learning based Image Quality Index (DIQI) for blind quality assessment. Extensive studies are conducted on the new TID2013 database and confirm the effectiveness of our DIQI relative to classical full-reference and state-of-the-art reduced- and no-reference IQA approaches.
Image quality metric with an integrated bottom-up and top-down HVS approach

A novel image quality assessment technique is proposed that incorporates both a ‘bottom-up’ and a ‘top-down’ simulation of the mechanism of the human visual system (HVS) into a unified quality metric, called top-down/bottom-up quality metric (TBQM). The Importance Map and region-based structural similarity index is employed as the bottom-up and the top-down approach of HVS, respectively. These two approaches are then integrated into the TBQM. Extensive experiments indicate that the proposed metric is highly correlated with subjective quality grading and is significantly better than the widely used image distortion metric, such as mean squared error. This method also outperforms simple top-down structural similarity measures.
An Image Augmentation Method for Quality Assessment Database

Image databases for quality assessment are helpful to evaluate the performance of objective assessment methods. Recommendations in regard to the constitution of databases and experimental methods of the subjective assessment have been proposed to ensure the database a good ground truth for the validation of objective quality assessment methods. However, these restrictions make databases scale-limited by covering small number of scenes distorted by few levels. To enrich IQA databases and increase the generalization capability of IQA models, we devise an effective image augmentation method. The two-stages scheme consists of the image-label pairs generation by minimizing the free energy between the pristine image and its augmentation as well as the distortion level interpolation which is based on the monotonicity of the perceptual quality with the severity of distortion. The experimental results show the ability of the augmented database to improve the prediction accuracy of learning-based no-reference image quality assessment metrics which in turn demonstrates the effectiveness of our method.
An eye-friendly dual-view projection system using temporal psychovisual modulation

In this paper, we propose an enhanced projector using the emergent display technology of Temporal Psycho-Visual Modulation (TPVM) which exploits the difference in image formation mechanisms of human eyes and imaging sensors. The projector have two significant advantages compared to traditional projector: 1) parts of the information (e.g., speech notes) projected can be hidden from the audience but the speaker can see the hidden parts; 2) the speaker's eyes can be protected from exposing to the light emitter directly. In the proposed system, every projected image is split into two subimages, where one contains both the original and hidden information, and another is opposite to the first subimage to achieve the purpose of hiding some information. To avoid the harmful stimulation from projection light to eyes, the enhanced projector detect the position of the speaker's face and adaptively modify the subimages to protect the speaker's eyes from directly exposing to the light source. The proposed prototype system uses the traditional projectors which support 3D display mode, and the proposed eye-protecting and information-hiding scheme is implemented by software. So it is transportable and practicable.
A Psychovisual Quality Metric in Free-Energy Principle

In this paper, we propose a new psychovisual quality metric of images based on recent developments in brain theory and neuroscience, particularly the free-energy principle. The perception and understanding of an image is modeled as an active inference process, in which the brain tries to explain the scene using an internal generative model. The psychovisual quality is thus closely related to how accurately visual sensory data can be explained by the generative model, and the upper bound of the discrepancy between the image signal and its best internal description is given by the free energy of the cognition process. Therefore, the perceptual quality of an image can be quantified using the free energy. Constructively, we develop a reduced-reference free-energy-based distortion metric (FEDM) and a no-reference free-energy-based quality metric (NFEQM). The FEDM and the NFEQM are nearly invariant to many global systematic deviations in geometry and illumination that hardly affect visual quality, for which existing image quality metrics wrongly predict severe quality degradation. Although with very limited or even without information on the reference image, the FEDM and the NFEQM are highly competitive compared with the full-reference SSIM image quality metric on images in the popular LIVE database. Moreover, FEDM and NFEQM can measure correctly the visual quality of some model-based image processing algorithms, for which the competing metrics often contradict with viewers' opinions.
Three Dimensional Scalable Video Adaptation via User-End Perceptual Quality Assessment

For wireless video streaming, the three dimensional scalabilities (spatial, temporal and SNR) provided by the advanced scalable video coding (SVC) technique can be directly utilized to adapt video streams to dynamic wireless network conditions and heterogeneous wireless devices. However, the question is how to optimally trade off among the three dimensional scalabilities so as to maximize the perceived video quality, given the available resource. In this paper, we propose a low-complexity algorithm that executes at resource-limited user end to quantitatively and perceptually assess video quality under different spatial, temporal and SNR combinations. Based on the video quality measures, we further propose an efficient adaptation algorithm, which dynamically adapts scalable video to a suitable three dimension combination. Experimental results demonstrate the effectiveness of our proposed perceptual video adaptation framework.
Constrained NMF for multiple exhibition on a single display

Temporal Psycho Visual Modulation (TPVM) was proposed recently as a new display paradigm that enables multi-exhibition on a single screen: people with different viewing devices can see independent and personalized views. Early studies have shown that TPVM can be implemented by nonnegative matrix factorization (NMF) of image sequences with additional constraints. In this paper, we introduce an algorithm that makes possible the glass-free viewing in a TPVM display system. More specifically, we design new constraints for the NMF problem and derive the solution using the block principal pivoting technique. The proposed algorithm can be readily implemented in a TPVM display system to generate the display frames. Experimental results are provided to show the effectiveness of the proposed algorithm.
ECG data compression based on wave atom transform

In this paper, a new ECG signal compression algorithm based on wave atom transform is presented. According to an assumption that ECG is oscillatory signal, we decompose ECG signal by wave atoms and trimmed insignificant coefficients. The wave atom decomposition has been proved to have a significantly sparser solution than other existing transform methods when it comes to oscillatory signal. In our experiment, the convergence of the energy of wave atoms' coefficients is faster than that of wavelet indeed. The most significant advantage of our algorithm is that unlike many conventional methods, the performance of our algorithm is not dependent on QRS detection, which simplifies the architecture of compression system and is beneficial to telemedicine application. After wave atom transform, the data stream is divided and coded by a hybrid entropy coding strategy combining delta coding, run-length-coding and arithmetic coding. The experimental results on MIT-BIH arrhythmia database proved that our algorithm has high compression ratio (CR > 10) with percentage root mean square difference (PRD) under 1%.
Fixation prediction through multimodal analysis

In this paper, we propose to predict human fixations by incorporating both audio and visual cues. Traditional visual attention models generally make the utmost of stimuli's visual features, while discarding all audio information. But in the real world, we human beings not only direct our gaze according to visual saliency but also may be attracted by some salient audio. Psychological experiments show that audio may have some influence on visual attention, and subjects tend to be attracted the sound sources. Therefore, we propose to fuse both audio and visual information to predict fixations. In our framework, we first localize the moving-sounding objects through multimodal analysis and generate an audio attention map, in which greater value denotes higher possibility of a position being the sound source. Then we calculate the spatial and temporal attention maps using only the visual modality. At last, the audio, spatial and temporal attention maps are fused, generating our final audio-visual saliency map. We gather a set of videos and collect eye-tracking data under audio-visual test conditions. Experiment results show that we can achieve better performance when considering both audio and visual cues.
Nonlinear additive model based saliency map weighting strategy for image quality assessment

Most state-of-the-art image quality metrics are based on the two-step approach: local distortion/fidelity measurement and pooling. During the pooling stage, many weighting strategies have been proposed incorporating properties of the distortion itself, various masking effects and visual attention. Recently, researchers have devoted great enthusiasm and effort to the improvement of image quality assessment using visual saliency models. In this research, it is noticed that visual saliency features of both the original image and the distorted one have impacts on the process of image quality assessment. To reduce the overlapping effects, a nonlinear additive model is proposed to integrate saliency features from the original and distorted images towards improved error weighting results. Our extensive experimental studies on four publicly available image databases (LIVE, TID2008, CSIQ and A57) indicate that the proposed improved nonlinear additive model based saliency map weighting strategy constantly leads to higher prediction accuracy for image quality assessment than traditional methods.
Single Image Super-resolution With Detail Enhancement Based on Local Fractal Analysis of Gradient

In this paper, we propose a single image super-resolution and enhancement algorithm using local fractal analysis. If we treat the pixels of a natural image as a fractal set, the image gradient can then be regarded as a measure of the fractal set. According to the scale invariance (a special case of bi-Lipschitz invariance) feature of fractal dimension, we will be able to estimate the gradient of a high-resolution image from that of a low-resolution one. Moreover, the high-resolution image can be further enhanced by preserving the local fractal length of gradient during the up-sampling process. We show that a regularization term based on the scale invariance of fractal dimension and length can be effective in recovering details of the high-resolution image. Analysis is provided on the relation and difference among the proposed approach and some other state of the art interpolation methods. Experimental results show that the proposed method has superior super-resolution and enhancement results as compared to other competitors.
Image inpainting with adaptive linear predictor

In this paper, a novel examplar-based inpainting algorithm with adaptive linear predictor is proposed. The patches in the damaged region are sequentially estimated with a linear combination of several nearest neighboring patches. The number of candidate patch is automatically tuned to local contexts based on Bayesian Information Criterion (BIC). The flexibility of the order-adaptive predictor makes the proposed algorithm suitable for both structural regions and detailed textures. The multi-scale framework and a novel propagation order are also involved to further improve the inpainting performance. Compared to the state-of-the-art image inpainting algorithms, experimental results show that the proposed method gives comparative or better performance.
On sparse representations of color images

We investigate an intrinsic and useful form of sparsity of color images that was largely overlooked in the literature of image/video processing. This sparsity of multispectral images is revealed and formulated by modeling the image formation process. The underlying new sparse representations of color images are general and can be exploited to improve the performance of existing image restoration algorithms, such as denoising, deblurring, and resolution upconversion.
Video super-resolution for dual-mode digital cameras via scene-matched learning

Many consumer digital cameras support dual shooting mode of both low-resolution (LR) video and high-resolution (HR) image. By periodically switching between the video and image modes, this type of cameras make it possible to super-resolve the LR video with the assistance of neighboring HR still images. We propose a model-based video super-resolution (VSR) technique for the above dual-mode cameras. A HR video frame is modeled as a 2D piecewise autoregressive (PAR) process. The PAR model parameters are learnt from the HR still images inserted between LR video frames. By registering the LR video frames and the HR still images, we base the learning on sample statistics that matches the scene to be constructed. The resulting PAR model is more accurate and robust than if the model parameters are estimated from the LR video frames without referring to the HR images or from a training set. Aided by the powerful scene-matched model the LR video frame is upsampled to the resolution of the HR image via adaptive interpolation. As such, the proposed VSR technique does not require explicit motion estimation of subpixel precision nor the solution of a large-scale inverse problem. The new VSR technique is competitive in visual quality against existing techniques with a fraction of the computational cost.
Saliency-Guided Quality Assessment of Screen Content Images

With the widespread adoption of multidevice communication, such as telecommuting, screen content images (SCIs) have become more closely and frequently related to our daily lives. For SCIs, the tasks of accurate visual quality assessment, high-efficiency compression, and suitable contrast enhancement have thus currently attracted increased attention. In particular, the quality evaluation of SCIs is important due to its good ability for instruction and optimization in various processing systems. Hence, in this paper, we develop a new objective metric for research on perceptual quality assessment of distorted SCIs. Compared to the classical MSE, our method, which mainly relies on simple convolution operators, first highlights the degradations in structures caused by different types of distortions and then detects salient areas where the distortions usually attract more attention. A comparison of our algorithm with the most popular and state-of-the-art quality measures is performed on two new SCI databases (SIQAD and SCD). Extensive results are provided to verify the superiority and efficiency of the proposed IQA technique.
Details preservation inspired blind quality metric of tone mapping methods

High dynamic range (HDR) images are extremely meaningful, especially in the space and medical fields. For visualization of HDR images on standard low dynamic range (LDR) display devices, how to convert HDR to LDR images naturally becomes a valuable issue, which has aroused a variety of tone-mapping operators (TMOs). To compare different LDR images created by distinct TMOs, researchers have recently provided a subject-rated tone-mapped image database, and then developed a full-reference objective tone-mapped image quality index (TMQI) based on the measurement of multi-scale signal fidelity and statistical naturalness. Instead, the basic property of HDR images about details preservation is studied in this paper. With it, a natural inference is that higher-quality tone-mapped images are capable of displaying much more details. We therefore propose a blind quality metric by estimating the amount of details in images generated by darkening/brightening an original tone-mapped images. Experimental results on the above tone-mapped image database confirm that the proposed method, despite of no reference, is robust and statistically superior to the currently optimal full-reference TMQI algorithm, and remarkably outperforms state-of-the-art no-reference IQA metrics.
Information security display system based on temporal psychovisual modulation

This paper introduces an information security display system using temporal psychovisual modulation (TPVM). TPVM was proposed as a new information display technology using the interplay of signal processing, optoelectronics and psychophysics. Since the human visual system cannot detect quick temporal changes above the flicker fusion frequency (about 60 Hz) and yet modern display technologies offer much higher refresh rates, there is a chance for a single display to simultaneously serve different contents to multiple observers. A TPVM display broadcasts a set of images called atom frames at a high speed, and those atom frames are then weighted by liquid crystal (LC) shutter based viewing devices that are synchronized with the display before entering the human visual system and fusing into the desired visual stimuli. And through different viewing devices, people can see different information. In this work, we develop a TPVM based information security display prototype. There are two kinds of viewers, those authorized viewers with the viewing devices who can see the secret information and those unauthorized viewers (bystanders) without the viewing devices who only see mask/disguise images. The prototype is built on a 120 Hz LCD screen with synchronized LC shutter glasses that were originally developed for stereoscopic display. The system is written in C++ language with SDKs of Nvidia 3D Vision, DirectX, CEGUI, MuPDF and etc. We also added human-computer interaction support of the system using Kinect. The information security display system developed in this work serves as a proof-of-concept of the TPVM paradigm, as well as a testbed for future research of TPVM technology.
Visual attention analysis and prediction on human faces with mole

Nowadays visual attention has been applied to many research and application problems. Different algorithms from low level to high level have been developed to detect the saliency map. For images with human face, high-level factors like mole may influence the visual attention. To investigate visual attention on human face with mole, we construct a Visual Attention database for Faces with Mole (VAFM) that contains face images, fixation density maps (FDM), landmark points as well as eye tracking data. Then we build visual attention model for face images with mole combining low-level saliency algorithms and high-level feature. Compared with the traditional low-level saliency algorithms, the proposed model perform better on our dataset.
GES: a new image quality assessment metric based on energy features in Gabor transform domain

We propose Gabor energy similarity (GES), a new full reference image quality assessment metric based on the measuring of Gabor energy features of images. It has been recognized that: 1) 2D Gabor filters can attain the theoretical conjoint resolution limit in spatial and frequency domain defined by Heisenberg's uncertainty principle; 2) it is widely reported that simple cells in visual cortex can be well modeled by 2D Gabor functions; and 3) the feature of Gabor energy is closely related to the model of complex cells in primary visual cortex that are very sensitive to minor changes in nature scene. Motivated by these facts we attempt to design a new image quality assessment by exploring the similarity in Gabor energy functions between the original and the distorted image. The images are firstly decomposed by a filter bank consists of 48 Gabor filters (6 directions, 4 spatial resolutions, and symmetric or anti-symmetric). Then the local energy feature is extracted by computing the modulus of the responses of symmetric and anti-symmetric kernel filters at each point. We finally propose an image quality metric based on averaged cross correlation between two images. Extensive experimental results are used to justify the effectiveness of the proposed GES
Simultaneous triple subtitles exhibition via Temporal Psychovisual Modulation

With the quick acceleration of cultural exchange and globalization, multi-language annotation for in-theatre movie exhibition is gaining more attention. The existing solution overlays a stack of subtitles in different languages over the movie, for example, the familiar combination of English + local tongue as seen in the cinema today. However, a major drawback of this straightforward piling-up type of solution is that as the required number of languages increases, the subtitle area grows and it may eventually span over the movie contents. An effective multi-language subtitling system without compromising the movie watching experience is therefore highly desired. In this paper, we propose a new Temporal Psychovisual Modulation (TPVM) based solution to the multi-language subtitling problem. TPVM is a new paradigm of information display exploiting the mismatch between high refresh rate of the modern optoelectronic displays and limited temporal resolution of the human visual system (HVS). In this work, we design a simultaneous triple subtitle exhibition system using 2 120 Hz stereoscopic DLP projectors with linear polarization filters and LCD shutter glasses. Wearing different pairs of glasses, the audience can enjoy the movie with 3 optional subtitles without interfering with the audience without the glasses and see the movie directly. Extended experimental results will be given in this paper to justify the effectiveness and robustness of the proposed multiple subtitling system.
A reduced-reference quality assessment scheme for blurred images

In this paper, we propose a reduced-reference scheme for evaluating the quality of blurred images under the theory of free-energy principle. Specifically, the free-energy principle indicates that the brain tries to account for the input image with an internal generative model and the discrepancy between the image and its model-explained version, which can be measured by free energy, is related to the image's perceptual quality. Accordingly, we define a visual distance between the blurred image and its original image in free energy to evaluate the quality of the blurred image. Therefore, the proposed quality scheme belongs to reduced-reference methods, which needs some information from the original image for quality assessment. Experimental results on public databases, LIVE, TID2013 and C-SIQ, demonstrate the proposed method works in high consistency with subjective assessment results and outperforms representative image quality assessment approaches.