nils-herrmann/clean_scirepeval huggingface dataset:
Towards qoe estimation of 3D contents through non-invasive methods

Nowadays the Quality of Experience of multimedia content is largely investigated in order to understand the overall acceptability of the content as perceived subjectively by the end user. In this paper, a non-invasive technique is adopted for estimating the heart rate of a subject while he is watching 3D contents, as a first step towards the ultimate goal of relating heart rate variations to the quality perceived by the observer in order to provide an effective tool for automatically evaluating the Quality of Experience. In order to validate the proposed approach, subjective tests have been performed and the achieved results are promising thus pushing to further investigate the correlation between emotions and the performed heart rate estimation.
Quality evaluation of depth map error concealment using a perceptually-aware objective metric

This paper presents a quality evaluation study on the performance of error concealment methods for depth maps used in multiview video-plus-depth (MVD). The research deals with the problem of decoding corrupted depth maps received from error-prone networks, where the quality of the reconstructed depth data is not always directly related to the quality of the virtual views synthesised by those maps. Even after error concealment such distortions are not particularly perceived as other known types, such as coding distortion. Thus, traditional quality metrics are not adequate to capture all the relevant features. In this work, the performance of two error concealment methods for depth maps is evaluated using a perceptually-aware objective metric. This metric is validated through subjective assessment of virtual views synthesised with concealed depth maps. Each subjective test is performed by comparing the relative quality between between two synthesised images using different error concealment methods. The perceptual impact of reconstruction in corrupted depth of MVD is evaluated under various loss rates, using several colour images and depth maps encoded at multiple quantisation steps. The achieved results reveal that the proposed objective quality metric is mostly inline with user preferences, in respect to the relative performance of each error concealment method.
Preference of Experience in Image Tone-Mapping: Dataset and Framework for Objective Measures Comparison

The popularity of high dynamic range (HDR) imaging has grown in both academic and private research sectors. Since the native visualization of HDR content still has its limitations, the importance of dynamic range compression (i.e., tone-mapping) is very high. This paper evaluates observers’ preference of experience in context of image tone-mapping. Given the different nature of natural and computer-generated content, the way observers perceive the quality of tone-mapped images can be fundamentally different. In this paper, we describe a subjective experiment attempting to determine users’ preference with respect to these two types of content in two different viewing scenarios—with and without the HDR reference. The results show that the absence of the reference can significantly influence the subjects’ preferences for the natural images, while no significant impact has been found in the case of the synthetic images. Moreover, we introduce a benchmarking framework and compare the performance of selected objective metrics. The resulting dataset and framework are made publicly available to provide a common test bed and methodology for evaluating metrics in the considered scenario.
Annealed learning based block transforms for HEVC video coding

Most of the recent video compression standards employ the Discrete Cosine Transform (DCT) for transforming the residual signal in order to remove spatial correlation and to achieve higher compression efficiency. However, by careful adaptation of transforms to the video content, a better set of integer transforms can be obtained. This paper proposes a new on-the-fly block-based transform optimization technique which involves first the classification of the residual blocks based on the cost of encoding the block, and then the generation of new optimized transforms for each class. An annealing based learning technique is further proposed in this paper in order to improve the performance of the optimization algorithm. The algorithm is tested using the latest HEVC test software where an optimized set of transforms is learned on the first frame of the HEVC test sequences and then applied to the subsequent frames in a Random Access (RA) and All Intra (AI) configuration. The results shows that this method can gain over 2% in terms of Bjontegaard Delta (BD)-rate compared to standard HEVC encoder in AI configuration and nearly 1.5% in RA.
Estimation of perceptual redundancies of HEVC encoded dynamic textures

Statistical redundancies have been the dominant target in the image/video compression standards. Perceptually, there exists further redundancies that can be removed to further enhance the compression efficiency. In this paper, we considered short term homogeneous patches that fall into the foveal vision as dynamic textures, for which a psychophysical test was used to estimate their amount of perceptual redundancies. We demonstrated the possible rate saving by utilizing these redundancies. We further designed a learning model that can precisely predict the amount of redundancies and accordingly proposed a generalized perceptual optimization framework.
Influence of HDR reference on observers preference in tone-mapped images evaluation

The most reliable way to assess the quality of images after a dynamic range compression (i.e. tone-mapping) is a subjective study. The goal of this paper is to determine, whether the preference of the participating subjects is significantly affected by the presence of the original high dynamic range (HDR) image displayed on the HDR screen or if we can obtain equivalent results from the test without the reference. For that, an extensive experiment using two different setups has been performed. The statistical analysis showed significant difference between evaluations and a Monte Carlo simulation proved that the absence of the reference is the factor of influence.
Perceived 3D TV Transmission Quality Assessment: Multi-Laboratory Results Using Absolute Category Rating on Quality of Experience Scale

Inspired by the rapidly increasing popularity of 3D movies, there is an industrial push for 3DTV services to the home. One important factor for the success and acceptance by the viewers is a positive quality of experience (QoE) of the new service when delivered. The questions of how to efficiently deliver 3DTV service to the home, and how to evaluate the visual quality perceived by end users are a recent research focus. We have investigated users' experience of stereoscopic 3D video quality by preparing two subjective assessment datasets. The first dataset aimed at the evaluation of efficient transmission in the transmission error free case, while the second focused on error concealment. A total of three subjective assessments, two for first dataset and one for the second, were performed using the Absolute Category Rating with Hidden unimpaired Reference video (ACR-HR) method. The experimental setup allows to show that the ACR-HR subjective method provides repeatable results across labs and across conditions for video quality. It was also verified that MVC is more efficient than H.264 simulcast coding. Furthermore it was discovered that based on the same level of quality of experience spatial down-sampling may lead to better bitrate efficiency while temporal down-sampling is not acceptable. When network impairments occur, traditional error 2D concealment methods need to be reinvestigated as they were outperformed by displaying the same view for both eyes (switching to 2D presentation).
Inpainting-based error concealment for low-delay video communication

Error concealment (EC) is one of the target applications of inpainting techniques. Some methods combine the estimated lost motion vectors (MVs) with the exemplar-based inpainting technique to recover the lost regions. Due to the erroneous motion vectors that might indicate a moving object as background object and vice versa, these methods are still showing visual artifacts in the recovered regions. In this paper, a concept of motion map that can be easily generated in the decoder side is introduced and it is combined with the exemplar-based inpainting technique. The proposed method introduces an adaptive search window size that trades-off the quality and complexity. Moreover, an optional blending technique is proposed to limit the spatio-temporal artifacts. Experiments show that the proposed method improves the visual quality with 5dB on average relative to the state-of-the-art inpainting-based EC method.
3D image quality assessment using Takagi-Sugeno-Kang fuzzy modele

This paper analyzes and presents impact of two types of the objective image quality assessment measures, no-reference and full-reference measures, on perception quality of 3D image using fuzzy logic estimator namely the Takagi-Sugeno-Kang fuzzy model. Based on the choice of two types of model's inputs, a comparative analysis is performed in this paper. All parameters of the no-reference and full-reference Takagi-Sugeno-Kang models are optimized in accordance to the selected sets mapping criteria of the input objective quality measures to the corresponding Differential Mean Opinion Scores (DMOS).
Which saliency weighting for omni directional image quality assessment?

With the explosion of Virtual Reality technologies, the production and usage of omni directional images (a.k.a 360 images) is presenting new challenges in the domains of compression, transmission and rendering. The evaluation of the quality of images generated by these technologies is therefore paramount. As the exploration of 360 images within a Head-Mounted Display (HMD) is non-uniform, current state of the art proposes a saliency weighting of distortions (between a reference and an impaired version) thus allowing us to highlight impairments in frequently attended regions. So far, saliency maps have been generated by tracking head motion alone, and consider that the view-port orientation alone is sufficient to determine saliency. The added value of eye gaze tracking within the viewport has not been studied in this domain yet. In this work, an eye-tracking experiment is performed using a HMD and is followed by subsequent gaze analysis to appreciate the visual attention behavior within a view-port. Results suggest that most eye-gaze fixations are rather far away from the center of the viewport. Across contents and observers, gaze fixations are quasi-isotropically distributed in orientation. The average location of gaze fixation (across contents and observers) from the center of the view-port varies between 14 and 20 visual degrees and these values correspond to a shift in retina beyond the para and peri fovea, into the extra-perifoveal region. A saliency weighting model based on foveation, centered at the middle of the view-port seems to be a correct assumption in only 2.5% of the overall scenarios and is consequently questionable. Therefore there is a need to refine saliency modeling and weighting for quality assessment in case of panoramic viewing.
Priority image and video encoding transmission based on a discrete radon transform

This paper presents an instance of priority encoding transmission system based on an exact and discrete Radon transform called the Mojette transform. The system includes an optimization process constrained jointly by the hierarchy of the source and by the channel property. In the framework of JPEG2000 image wireless transmission, the hierarchy is characterized by a perceptual objective quality assessment which is well correlated with the human judgement and the channel estimation is performed by an exponential loss profile. The solution is compared with the JPEG2000 Wireless (JPWL) system based on the Peak Signal-to-Noise Ratio (PSNR) criterion and the Reed- Solomon codes. We also propose an extension of this work to H.264/AVC videos.
Low-Cost Eye Gaze Prediction System for Interactive Networked Video Streaming

Eye gaze is now used as a content adaptation trigger in interactive media applications, such as customized advertisement in video, and bit allocation in streaming video based on region-of-interest (ROI). The reaction time of a gaze-based networked system, however, is lower-bounded by the network round trip time (RTT). Furthermore, only low-sampling-rate gaze data is available when commonly available webcam is employed for gaze tracking. To realize responsive adaptation of media content even under non-negligible RTT and using common low-cost webcams, we propose a Hidden Markov Model (HMM) based gaze-prediction system that utilizes the visual saliency of the content being viewed. Specifically, our HMM has two states corresponding to two of human's intrinsic gaze behavioral movements, and its model parameters are derived offline via analysis of each video's visual saliency maps. Due to the strong prior of likely gaze locations offered by saliency information, accurate runtime gaze prediction is possible even under large RTT and using common webcam. We demonstrate the applicability of our low-cost gaze prediction system by focusing on ROI-based bit allocation for networked video streaming. To reduce transmission rate of a video stream without degrading viewer's perceived visual quality, we allocate more bits to encode the viewer's current spatial ROI, while devoting fewer bits in other spatial regions. The challenge lies in overcoming the delay between the time a viewer's ROI is detected by gaze tracking, to the time the effected video is encoded, delivered and displayed at the viewer's terminal. To this end, we use our proposed low-cost gaze prediction system to predict future eye gaze locations, so that optimized bit allocation can be performed for future frames. Through extensive subjective testing, we show that bit-rate can be reduced by up to 29% without noticeable visual quality degradation when RTT is as high as 200 ms.
Transform Coding for On-the-Fly Learning Based Block Transforms

Block based video codecs developed in last decades have employed Discrete Cosine Transform (DCT) as a candidate transform for compacting the energy of the predicted residual data into fewer coefficients. However, DCT is not optimal for blocks with directional structure and high texture. Content-adaptive transforms can be used to achieve better compression. However, these adaptive transforms must be sent to the decoder, increasing the bit-rate in return. It is therefore important to take care of the overhead of sending the transforms in order to benefit from the better compaction of the adaptive transforms. This paper addresses this issue and provides methods to efficiently encode these adaptive transforms that are learned from the content. A statistical model is derived to determine the precision required to encode a transform with a negligible loss in the performance of adaptive transforms. Results shows that the overhead can be reduced by an average 70% for low QP and 86% for high QP values compared to encoding the complete transform with a little performance loss.
Towards a New Quality Metric for 3-D Synthesized View Assessment

3DTV technology has brought out new challenges such as the question of synthesized views evaluation. Synthesized views are generated through a depth image-based rendering (DIBR) process. This process induces new types of artifacts whose impact on visual quality has to be identified considering various contexts of use. While visual quality assessment has been the subject of many studies in the last 20 years, there are still some unanswered questions regarding new technological improvement. DIBR is bringing new challenges mainly because it deals with geometric distortions. This paper considers the DIBR-based synthesized view evaluation problem. Different experiments have been carried out. They question the protocols of subjective assessment and the reliability of the objective quality metrics in the context of 3DTV, in these specific conditions (DIBR-based synthesized views), and they consist in assessing seven different view synthesis algorithms through subjective and objective measurements. Results show that usual metrics are not sufficient for assessing 3-D synthesized views, since they do not correctly render human judgment. Synthesized views contain specific artifacts located around the disoccluded areas, but usual metrics seem to be unable to express the degree of annoyance perceived in the whole image. This study provides hints for a new objective measure. Two approaches are proposed: the first one is based on the analysis of the shifts of the contours of the synthesized view; the second one is based on the computation of a mean SSIM score of the disoccluded areas.
Open perceptual binocular and monocular descriptors for stereoscopic 3D images and video characterization

This paper presents a toolbox for monocular and binocular depth estimation and an analysis of performance for some of these. At first, the definition of an appropriate depth indicator (DI) metric for 3D contents is discussed. To this aim, different algorithms from the literature for the characterization of 3D videos are compared. Results show that the simple 7.5 percentile of the disparity map values can already be an indicator, even though it may fail to address several perceptual aspects. In the latter cases, more advanced algorithms presented in this paper may be a better approach. In a second step, monocular depth indicators are described and analyzed in this paper. All code and tools enabling binocular and monocular depth estimation such as depth-map estimation, depth map characterization and monocular depth cue indicator computation is provided open-source. This will enable researchers to further characterize their 3D and 2D contents, for example before running a subjective experiment, or to automatically pre-screen 3D content that is to be presented to a larger number of viewers.
An adaptive Lagrange multiplier determination method for dynamic texture in HEVC

This research proposed a method for adaptive Lagrange multiplier determination for rate-distortion optimization with dynamic texture in High Efficiency Video Coding (HEVC). Inspired by the experimental results of the Lagrange multiplier selection test, the presented approach adaptively predicts the optimum Lagrange multiplier for different dynamic texture sequences, based on the features of the dynamic texture sequences such as normal flow and spatial-temporal information. The Lagrange multiplier among the given values will be chosen based on the Bjontegaard delta measurements. After that, the data of training dynamic texture will be used for Support Vector Machine (SVM) in machine learning for getting the predicting results. The proposed algorithm has been fully integrated into HEVC reference codec. The result shows that the proposed method can improve 0.5 in Structural Similarity Metric (SSIM) and 2 in Peak Signal-to-Noise Ratio (PSNR).
Does H.265 based peri and para-foveal quality flicker disrupt natural viewing patterns?

Region of interest based coders and also those coders that extensively use the Intra coding mode tend to produce flickering artefacts due to the inherent quality fluctuations they produce in the non-salient or homogeneously textured intra coded regions of a scene respectively. These non-salient areas are in turn incident on the para, peri and extra-peri foveal retinal regions that are especially sensitive to temporal artefacts. To study the perceptual effects of such a quality based flicker in the retinal periphery, in this experiment gaze data was collected from 48 observers in a free viewing scenario using a regulated flicker stimulus. The video stimulus in the mentioned peripheral regions were flickered with a predetermined frequency and amplitude by adaptively combining the streams produced using a H.265 video coder, and presenting them in a Gaze Contingent Display(GCD) setup. Utilizing statistical gaze analysis techniques, we observe that, temporal quality flicker in the visual periphery plays a major role in disturbing the natural viewing patterns of an observer. Further, this effect was found to be maximum in the 7.5Hz temporal band thus showing that, encoders that affect temporal quality smoothness must aim to avoid such distraction effects.
Stereoscopic image retargeting based on 3D saliency detection

In this paper, we propose a novel stereoscopic image retargeting algorithm based on 3D visual saliency detection. A new 3D visual attention model is designed based on 2D visual feature detection, depth feature detection and the modeling of various viewing bias in stereo vision. A geometrically consistent seam carving technique is adopted for retargeting stereo image pair. Experimental results demonstrated that both the proposed visual attention model and the proposed retargeting method outperform the state-of-the-art studies.
A Novel Parallel Architecture with Fault-Tolerance for Joining Bi-directional Data Streams in Cloud

Nowadays, there are more and more data sources, such as online stores, wireless sensors, and stock tickers, etc., which produce a mass of data as unbounded streams. Investigating the potential information from these rich statistical data in real time generates enormous values. In most cases, it is essential to join two or more data streams in order to find sufficient information. However, as the volume of these data becomes larger, joining them in a single machine is not an effective way. One of the most practical solutions is to use a shared-nothing cluster built on a cloud to deal with these data streams. In this paper we provide a novel parallel architecture named Dual-Assembly-Pipeline(DAP) with fault-tolerance, in which we join bi-directional data streams by considering the processing nodes' failures. Especially, virtual machines in a cloud may fail so that fault-tolerance becomes more and more important when users issue a continuous query. Our experiments show that our DAP model can effectively join bi-directional data streams in parallel. What's more, when non-adjacent virtual machines fail at the same time, all data can be recovered, while adjacent ones fail, only parts of data can be reinstated.
Contention-aware data caching in wireless multihop ad hoc networks

Data caching is one of the attractive techniques that can increase the efficiency of data access in wireless multi-hop ad hoc networks. However, it is still a challenging task to achieve an optimal trade-off between the total traffic cost and average access delay by properly selecting cache nodes, i.e. cache placement. In this paper, we address the problem of cache placement in wireless multi-hop ad hoc networks. We consider dynamic networks, in which there is a data source that stores one data item and other client nodes require accessing the data item. We define the cache placement problem on a dynamic network topology as Dynamic Cache Placement (DCP). Most of the existing cache placement algorithms use hop counts to measure the total cost of caching systems. Due to the impact of contentions in wireless networks, one hop delay is different from each other and it varies with the traffic load. Thus the previous algorithms cannot achieve their expected performance. We propose two heuristic cache placement algorithms, named Centralized Contention-Aware Caching Algorithm (CCCA) and Distributed Contention-aware Caching Algorithm (DCCA), which detect the variation of contentions to evaluate the benefit of selecting a node as cache node. Simulation results show that the proposed algorithms achieve better performance than other alternative ones in terms of average query delay, caching overheads, and query success ratio.