nils-herrmann/clean_scirepeval huggingface dataset:
Quality evaluation of HEVC and VP9 video compression in real-time applications

Video consumption over Internet has increased significantly over the recent years and occupies the majority of the overall data traffic. To decrease the load on the Internet infrastructure and reduce bandwidth taken by video, higher efficiency video codecs, such as H.265/HEVC and VP9, have been developed. The availability of these two new competing video coding formats raises the question of which is more efficient in terms of rate-distortion and by how much they outperform the current state-of-the-art coding standard, H.264/AVC. This paper provides an answer to this difficult question for low-delay video applications, e.g., real-time video streaming/conferencing or video surveillance. The benchmarking of HEVC and VP9 video compression was conducted by means of subjective evaluations, assuming web browser playback, an uncontrolled environment, and HD video content. Considering a wide range of bit rates from very low to high bit rates, corresponding to low quality up to transparent quality (when compared to the original video), results show a clear advantage of HEVC with average bit rate savings of 59.5% when compared to AVC and 42.4% when compared to VP9.
How to benchmark objective quality metrics from paired comparison data?

The procedures commonly used to evaluate the performance of objective quality metrics rely on ground truth mean opinion scores and associated confidence intervals, which are usually obtained via direct scaling methods. However, indirect scaling methods, such as the paired comparison method, can also be used to collect ground truth preference scores. Indirect scaling methods have a higher discriminatory power and are gaining popularity, for example in crowdsourcing evaluations. In this paper, we present how the classification errors, an existing analysis tool, can also be used with subjective preference scores. Additionally, we propose a new analysis tool based on the receiver operating characteristic analysis. This tool can be used to further assess the performance of objective metrics based on ground truth preference scores. We provide a MATLAB script with an implementation of the proposed tools and we show one example of application of the proposed tools.
Benchmarking of quality metrics on ultra-high definition video sequences

The performance of objective quality metrics for high-definition (HD) video sequences is well studied, but little is known about their performance for ultra-high definition (UHD) video sequences. This paper analyzes the performance of several common objective quality metrics (PSNR, VSNR, SSIM, MS-SSIM, VIF, and VQM) on three different 4K UHD video sequences using subjective scores as ground truth. The findings confirm the content-dependent nature of most metrics (with VIF being the only exception), which has been reported previously for standard and high resolution video sequences. PSNR showed the lowest correlation with ground truth quality scores when the analysis was performed for all contents at once and thus is not recommended as a general metric for video quality, while VIF showed the highest Pearson (0.83) and Spearman (0.87) correlation coefficients and may be used as a general purpose metric. On the other hand, all studied metrics were accurate in distinguishing different quality levels for the same content. The results of several fittings between metric values and subjective ground truth scores demonstrated that logistic fitting provides the highest correlation. The results also indicated a shift in metrics values between synthetic and natural contents.
Cross-lab subjective evaluation of the MVC+D and 3D-AVC 3D video coding standards

Consistent and imitable subjective measurement 3D video quality assessment is investigated for evaluating 3D service parameters and as an essential criterion towards the development of objective models. This paper analyzes the results obtained from three test laboratories to evaluate the performance of the MVC+D and 3D-AVC 3D video coding standards using identical video content and following similar methodologies and instructions. The correlation between laboratories was investigated using similar analysis to benchmarking of objective metrics. Analyses show that test laboratories employing different displays and different subjects could still produce highly correlated results, as they follow similar guidelines to carry out assessments.
Crowd-based quality assessment of multiview video plus depth coding

Crowdsourcing is becoming a popular cost effective alternative to lab-based evaluations for subjective quality assessment. However, crowd-based evaluations are constrained by the limited availability of display devices used by typical online workers, which makes the evaluation of 3D content a challenging task. In this paper, we investigate two possible approaches to crowd-based quality assessment of multiview video plus depth (MVD) content on 2D displays: by using a virtual view and by using a free-viewpoint video, which corresponds to a smooth camera motion during a time freeze. We conducted the crowdsourcing experiments using seven MVD sequences encoded at different bit rates with the upcoming 3D-AVC video coding standard. The results demonstrate high correlation with subjective evaluations performed using a stereoscopic monitor in a controlled laboratory environment. The analysis shows no statistically significant difference between the two approaches.
Subjective and objective evaluation of HDR video coding technologies

This paper reports the details and results of a subjective and objective quality evaluation assessing responses to an MPEG call for evidence (CfE) on high dynamic range (HDR) and wide color gamut video coding. Five HDR video contents, compressed at four bit rates by each proponent responding to the CfE, were used in the subjective assessments. To be able to evaluate the performance of objective quality metrics, the double stimulus impairment scale (DSIS) method was used for subjective assessments instead of previously published paired comparison to an anchor. Subjective results show evidence that coding efficiency can be improved in a statistically noticeable way over the HEVC anchor in terms of perceived quality. However, when compared to paired comparison, less statistically significant differences are observed because of the lower discrimination power of the DSIS method. The collected subjective scores were used as a ground truth to benchmark and analyze the performance of objective metrics. Results show that HDR-VDP-2 and PQ2VIFP have the highest correlation with subjective scores and outperform other investigated metrics.
Motion compensated prediction with geometry padding for 360 video coding

The paper presents a new geometry padding method for motion compensated prediction in 360 video coding. Unlike the conventional padding method for 2D video coding, which extends samples outside of a picture's boundary by simply copying (repeating) those at the boundaries, the proposed geometry padding method considers the spherical nature of the 360 video and the specific geometry projection format when extending samples outside of a picture's or a face's boundary. The proposed geometry padding method is implemented in the HEVC reference software HM-16.12 for the equirectangular and cubemap projection formats. Simulation results show that, when compared with the conventional 2D padding method in HEVC, geometry padding performs better; the proposed geometry padding gives on average luma (Y) BD-rate reduction of 0.3% for equirectangular projection format coding and 1.0% for cubemap projection format coding all in terms of the spherical PSNR (S-PSNR) metric. The proposed geometry padding is especially effective for high motion sequences, where up to 4.3% BD-rate reduction can be achieved.
Free-viewpoint video sequences: A new challenge for objective quality metrics

Free-viewpoint television is expected to create a more natural and interactive viewing experience by providing the ability to interactively change the viewpoint to enjoy a 3D scene. To render new virtual viewpoints, free-viewpoint systems rely on view synthesis. However, it is known that most objective metrics fail at predicting perceived quality of synthesized views. Therefore, it is legitimate to question the reliability of commonly used objective metrics to assess the quality of free-viewpoint video (FVV) sequences. In this paper, we analyze the performance of several commonly used objective quality metrics on FVV sequences, which were synthesized from decompressed depth data, using subjective scores as ground truth. Statistical analyses showed that commonly used metrics were not reliable predictors of perceived image quality when different contents and distortions were considered. However, the correlation improved when considering individual conditions, which indicates that the artifacts produced by some view synthesis algorithms might not be correctly handled by current metrics.
A quality assessment protocol for free-viewpoint video sequences synthesized from decompressed depth data

In this paper, the analysis of a subjective quality experiment consisting in assessing the quality of free-viewpoint video sequences generated from decompressed depth data is presented. In the absence of a dedicated subjective assessment protocol for the evaluation of such 3D systems, a subjective quality assessment methodology is proposed for the context of MVD compression. The proposed methodology includes the assessment of free-viewpoint video sequences generated from decompressed depth data and from view synthesis processes. The proposed methodology is meant to assess the performances of depth map compression and view synthesis algorithms.
Predicting subjective sensation of reality during multimedia consumption based on EEG and peripheral physiological signals

Sensation of reality refers to the ability of users to feel present in a multimedia experience. As 3D technologies target to provide more immersive and higher quality multimedia experiences, it is important to understand Quality of Experience (QoE) and sensation of reality. Recently, there have been efforts to measure brain activity in order to understand implicitly QoE for various multimedia contents. However, brain activity accounting for sensation of reality has not been adequately investigated. The goal of this paper is twofold. First, we investigate how various aspects, such as perceived quality, perceived depth, and content preference affect subjective sensation of reality through explicit subjective ratings. Second, we construct subjective classification systems to predict sensation of reality from multimedia experiences based on electroencephalography (EEG) and peripheral physiological signals such as heart rate and respiration.
HDR image compression: A new challenge for objective quality metrics

High Dynamic Range (HDR) imaging is able to capture a wide range of luminance values, closer to what the human visual system can perceive. It is believed by many that HDR is a technology that will revolutionize TV and cinema industry similar to how color television did. However, the complexity of HDR requires reinvention of the whole chain from capture to display. In this paper, HDR images compressed with the upcoming JPEG XT HDR image coding standard are used to investigate the correlation between thirteen well known full-reference metrics and perceived quality of HDR content. The metrics are benchmarked using ground truth subjective scores collected during quality evaluations performed on a Dolby Pulsar HDR monitor. Results demonstrate that objective quality assessment of HDR image compression is challenging. Most of the tested metrics, with exceptions of HDR-VDP-2 and FSIM computed for luma component, poorly predict human perception of visual quality.
Rate-distortion evaluation for two-layer coding systems

The Bjøntegaard model is widely used to calculate the compression efficiency between different codecs. However, this model is not sufficient to investigate the impact on quality of the interaction of the base and enhancement layer bit rates when comparing two-layer coding systems. Therefore, in this paper, we propose an extension of the Bjøntegaard model from rate-distortion (R-D) curve fitting to rate-rate-distortion (R2-D) surface fitting. The proposed model uses a cubic surface as fitting function and a more complex characterization of the domain formed by the data points to compute a more realistic estimate of compression efficiency. The proposed model can be used to measure the compression efficiency of two-layer coding systems, as well as for other applications, e.g., to optimize the bit rate allocation between texture and depth in 3D video coding. Two examples of assessment of compression efficiency in JPEG XT are presented as illustrations.
Predicting 3D quality based on content analysis

Development of objective quality metrics that can reliably predict perceived quality of 3D video sequences is challenging. Various 3D objective metrics have been proposed, but PSNR is still widely used. Several studies have shown that PSNR is strongly content dependent, but the exact relationship between PSNR values and perceived quality has not been established yet. In this paper, we propose a model to predict the relationship between PSNR values and perceived quality of stereoscopic video sequences based on content analysis. The model was trained and evaluated on a dataset of stereoscopic video sequences with associated ground truth MOS. Results showed that the proposed model achieved high correlation with perceived quality and was quite robust across contents when the training set contained various contents.
Survey of web-based crowdsourcing frameworks for subjective quality assessment

The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.
Ultra-eye: UHD and HD images eye tracking dataset

Due to the recent advances in ultra high definition (UHD) displays, UHD TV may replace HD TV in a near future. However, little is known about the effect of UHD content on human visual perception, specifically, on human visual attention, viewing strategies, and visual saliency. To help studying these properties of the human visual system and their dynamics when HD content is replaced with UHD content, a publicly accessible dataset is proposed, which is composed of 41 4K UHD and HD images with corresponding eye tracking information. The eye tracking information includes the fixation points and fixation density maps measured during extensive subjective experiments. In this paper, we describe the dataset in details, including the strategy for content selection, the eye-tracking experiments, and the computation of the fixation density maps.
EYEC3D: 3D video eye tracking dataset

Despite efforts of the scientific community in recent years, little is known about the mechanisms of the human visual system that control visual attention when watching 3D content. To help understanding these mechanisms and develop more accurate visual attention models, we created a public 3D video eye tracking dataset. The dataset provides the eye tracking information corresponding to eight stereoscopic video sequences. The eye tracking information includes the fixation points and fixation density maps measured during subjective experiments. This paper describes the dataset in details, including the stereoscopic video sequences, the eye tracking experiments, and the computation of the fixation density maps.
High resolution free-view interpolation of planar structure

In many applications, users desire to access a random viewing angle of a scene in high resolution while this specifically queried imagery is not among the acquired sample images. In theory, 3D reconstruction based rendering can generate such an image. However, accurate camera calibration over large scale photo collections is needed and is highly complex in nature. Image stitching based approachescan also be applied. However, such scheme is unable to provide free view interpolation or resolution enhancement. In this paper, we present a novel free view image super-resolution scheme to interpolate free views for planar structures. We construct a Bayesian model and marginalize it over photometric regulation and geometric registration parameters applying the Lie group theory. Experimental results show that the proposed scheme is able to achieve desired performance against the state-of-the-art image super-resolution approaches and successfully obtain registration in full 6 degree-of-freedom (DOF). Compared with existing image based rendering schemes, the proposed scheme achieves free view interpolation for planar structures with higher resolution and less distortion.
Analysis of left ventricle global deformation based on dynamic CT data

Addresses the problem of estimating left ventricle rigid and nonrigid motions from 3D dynamic CT data via a model-based approach. The authors have developed a two-stage scheme to estimate the motion and deformation of the left ventricle. In the first stage, the rigid motion of the left ventricle is computed from the position and orientation change of an object-centered coordinate system. In the second stage, the global deformations of the left ventricle are estimated by fitting the digital surface to superquadric modeling primitives which are used to model the time-varying and deformable left ventricle shape. Deformation operations, such as tapering and bending, are applied to the basic superquadric modeling primitives in order to capture the complex nature of the left ventricle shape and nonrigid motions.<<ETX>>
Stable Maximum Throughput Broadcast in Wireless Fading Channels

This research considers network coded broadcast system with multi-rate transmission and dual queue stability constraints. Existing network coded broadcast systems consider single rate transmission without receiver queue constraints. First, we shall illustrate that broadcast without network coding cannot support maximum throughput in wireless fading channels. However, the network coded broadcast poses new constraints for the receivers to manage stable queues while the fading channel characteristics suggest the broadcast to operate at multi-rate to achieve higher throughput. In this research, we propose a joint scheduling and network coding (JSNC) strategy for such network coded broadcast system to achieve maximum throughput under queue stability constraint. In a single cell broadcast networks with exogenous arrivals of packets at the base station, we prove that JSNC can stabilize the system as long as the rate of the exogenous arrival flow is within the capacity region. Sufficient control parameters are provided in JSNC for trading off between sender's buffer and the receivers' buffers. JSNC can be viewed as a generalization of the classical backpressure scheduling rule to coded information flow. Alternatively, JSNC can also be viewed as an extension of network coding theory to queuing system.
Cloud-based video streaming with systematic mobile display energy saving: Rate-distortion-display energy profiling

Mobile display has been considered as the major contributor to the energy consumption of the ever-increasing mobile video services. Current practices in display energy reduction (DER) utilize local computing resources to analyze the video content before DER strategies can be applied in a per-device fashion. For a given video, same analytical computations are repeated in millions of individual devices. In this paper, we demonstrate that a paradigm shifting framework can be designed to systematically move the common DER local processing to the streaming server with the emergence of cloud-based video services. This framework has the potential to replace the massive per-device DER computation by a one-time global video processing in the cloud. To accomplish this ultimate goal in DER, a new family of video rate-distortion (R-D) profile with embedded DER strategies shall be properly generated. This family of rate-distortion-display energy (R-D-DE) profiles contains a set of common DER parameters to be directly extracted and employed by individual mobile devices to achieve desired display energy saving without repeated local computation. Performance evaluations of the proposed design are carried out to show that this family of R-D-DE profiles is indeed able to command the systematic DER design based on the intrinsic relationships among bitrate, video quality and display energy saving.
An enhanced trellis coded quantization scheme for robust image transmission

In this paper, we present an enhanced scheme over a trellis coded quantization (TCQ) approach recently proposed for the transmission of compressed images over noisy channels. This original scheme employs block DCT, TCQ, and fixed length bit allocation so that the compressed data is relatively insensitive to channel noise. Although the fixed length coding is able to avoid the catastrophic error propagation, one problem of the approach is the severe distortion of the whole blocks in the case of high channel error rate. To reduce the unpleasant error block effects, we propose to re-arrange the subsources of the image before the process of TCQ. With such a preprocessing, the degradation caused by the channel error can be distributed to the whole image, instead of within a particular block. Preliminary results show that this enhanced TCQ scheme is able to improve both the PSNR and the visual quality of the reconstructed image.
Barycentric coordinates based soft assignment for object classification

For object classification, soft assignment (SA) is capable of improving the bag-of-visual-words (BoVW) model and has the advantages in conceptual simplicity. However, the performance of soft assignment is inferior to those recently developed encoding schemes. In this paper, we propose a novel scheme called barycentric coordinates based soft assignment (BCSA) for the classification of object images. While maintaining conceptual simplicity, this scheme will be shown to outperform most of the existing encoding schemes, including sparse and local coding schemes. Furthermore, with only single-scale features, it is able to achieve comparable or even better performance to current state-of-the-art Fisher kernel (FK) encoding scheme. In particular, the proposed BCSA scheme enjoys the following properties: 1) preservation of linear order precision for encoding which makes BCSA robust to linear transform distortions; 2) inheriting naturally the visual word uncertainty which leads to a more expressive model; 3) generating linear classifiable codes that can be learned with significant less computational cost and storage. Extensive experiments based on widely used Caltech-101 and Caltech-256 datasets have been carried out to show its effectiveness of the proposed BCSA scheme in both performance and simplicity.