nils-herrmann/clean_scirepeval huggingface dataset:
Recognizing actions using salient features

Towards a compact video feature representation, we propose a novel feature selection methodology for action recognition based on the saliency maps of videos. Since saliency maps measure the perceptual importance of the pixels and regions in videos, selecting features using saliency maps enables us to find a feature representation that covers the informative parts of a video. Because saliency detection is a bottom-up procedure, some appearance changes or motions that are irrelevant to actions may also be detected as salient regions. To further improve the purity of the feature representation, we prune these irrelevant salient regions using the saliency values distribution and the spatial-temporal distribution of the salient regions. Extensive experiments are conducted to demonstrate that the proposed feature selection method largely improves the performance of bag-of-video-words model on action recognition based on three different attention models including a static attention model, a motion attention model and their combination.
Natural images scale invariance and high-fidelity image restoration

One of the most striking properties of natural image statistics is their scale invariance. Intuitively, a natural image always contains the same contents of different scales and dually the same contents of same scale exist throughout scales of the image. Different from the previous scale invariance related work decomposing an image to its local band-pass filter components, this paper seeks a general model of the natural image paths distribution to describe the scale invariance in the visual world and then a novel strategy for high-fidelity image restoration is presented by characterizing nonlocal self-similarity of natural images throughout scales in a unified statistical manner, which offers a powerful mechanism of combining natural images scale invariance and nonlocal self-similarity simultaneously to ensure a more reliable and robust estimation. Extensive experiments on image restoration from partial random samples manifest that the proposed algorithm achieves significant performance improvements over the current state-of-the-art schemes.
Witsenhausen-Wyner video coding

Inspired by Witsenhausen and Wyner's 1980 (now expired) patent on “interframe coder for video signals”, this paper presents a Witsenhausen-Wyner video codec, where the previously decoded video frame is used at the decoder as side information for joint decoding. Specifically, we replace the predictive Inter coding in H.264/AVC by the syndrome-based coding scheme of Wistenhausen and Wyner, while keeping the Intra and Skip modes unchanged. Within Wistenhausen-Wyner coding mode, we optimize the decisions between syndrome coding and entropy coding among different DCT bands and among different bit-planes within each DCT coefficient. We employ forward motion estimation at the encoder and send the motion vectors to help generate the side information at the decoder, since our focus is not on low-complexity encoding. We also examine the tradeoff between the motion vector resolution and coding performance. Extensive simulations of video transmission over wireless networks show that Witsenhausen-Wyner video coding is much more robust against channel errors than H.264/AVC. The price paid for enhanced error-resilience with Witsenhausen-Wyner coding is a small loss in compression efficiency.
Condition-based early stage diagnosis robust ant colony clustering algorithm and feature mapping

Performance evaluation is crucial to increasing equipment availability and reliability as well as facilitating further proactive maintenance. This paper presents a novel condition-based early stage diagnosis method using robust ant colony clustering algorithm and feature mapping. Vibration signals acquired from equipments are decomposed by wavelet packet transform after wavelet denoising, and then feature measures based on frequency bands are extracted by ant colony clustering algorithm to form an input vector for training performance evaluation model. Operating conditions of equipment are predicted using feature mapping. Furthermore, a robust ant colony clustering algorithm is proposed to adjust comparison probability dynamically. Finally, effectiveness and feasibility of the proposed method are verified by vibration signals acquired from a drilling test bed.
Error Resilient Dual Frame Motion Compensation with Uneven Quality Protection

In this paper, an error resilient JU-DFMC is proposed for video transmission over error-prone channels. In the proposed error resilient JU-DFMC, a new error resilient prediction structure of DFMC is firstly presented. Then an end-to-end distortion model is applied for macroblock (MB) level mode decision. Finally a frame level rate distortion cost scheme is proposed to determine how many times the header information will be transmitted in a high quality frame (HQF). The experimental results show that the proposed method can achieve better performance than the previous DFMC schemes.
Soft mobile video broadcast based on side information refining

Video broadcasting is a popular application of wireless network, whose main challenge is to accommodate different users with different channel conditions. Recently, a novel `D-Cast' approach based on distributed source coding (DSC) is proposed. It can avoid error propagation and still achieve high compression efficiency in inter frame coding by utilizing coset coding and soft broadcast. However, D-CAST is not very efficient because of rough side information. In this work, we present a novel soft mobile video broadcast approach based on side information refinement algorithm (SIR-CAST) to improve the quality of the side information. Moreover, SIR-Cast optimizes the estimate of the quantifying step (Qstep) which is corresponding to the refined side information. Thus, SIR-CAST outperforms D-CAST about 1dB-2dB in video PSNR while maintaining the similar graceful degradation feature as D-CAST.
Universal Steganalysis Based on Statistical Models Using Reorganization of Block-based DCT Coefficients

The goal of stganography is to hide information into media without disclosing the fact of existing communication. Currently, stganography such as least significant bit (LSB), quantization index modulation (QIM) and spread spectrum (SS), has become increasingly widespread. Steganalysis as a counterpart of stganography is to detect the presence of it. In this paper, we present a new universal steganalysis method based on statistical models of the image’s discrete cosine transform (DCT) coefficients. In fact, the block-based DCT by proper reorganization of its coefficients can have similar characteristics to wavelet transforms. The presented universal steganalysis method utilizes these characteristics to build statistical models of the image and its prediction-error image. Features extracted from the re-organization DCT blocks of host images and theirs prediction-error images and features extracted from steg images and theirs prediction-error images are used to train the SVM classifier. In the testing, features from those potential images are inputted the trained-well classifier to determine where the potential images are stego images or not. The experiments have shown that the proposed method outperforms in general prior-arts of steganalysis methods based on wavelet transform domain.
Deep networks for compressed image sensing

The compressed sensing (CS) theory has been successfully applied to image compression in the past few years as most image signals are sparse in a certain domain. Several CS reconstruction models have been recently proposed and obtained superior performance. However, there still exist two important challenges within the CS theory. The first one is how to design a sampling mechanism to achieve an optimal sampling efficiency, and the second one is how to perform the reconstruction to get the highest quality to achieve an optimal signal recovery. In this paper, we try to deal with these two problems with a deep network. First of all, we train a sampling matrix via the network training instead of using a traditional manually designed one, which is much appropriate for our deep network based reconstruct process. Then, we propose a deep network to recover the image, which imitates traditional compressed sensing reconstruction processes. Experimental results demonstrate that our deep networks based CS reconstruction method offers a very significant quality improvement compared against state-of-the-art ones.
Discriminating features learning in hand gesture classification

The advent and popularity of Kinect provides a new choice and opportunity for hand gesture recognition (HGR) research. In this study, the authors propose a discriminating features extraction for HGR, in which features from red, green and blue (RGB) images and depth images are both explored. More specifically, histogram of oriented gradient feature, local binary pattern feature, structure feature and three-dimensional voxel feature are first extracted from RGB images and depth images, then these features are further reduced with a novel deflation orthogonal discriminant analysis, which enhances the discriminative ability of the features with supervised subspace projection. The extensive experimental results show that the proposed method improves the HGR performance significantly.
A ROI quality adjustable rate control scheme for low bitrate video coding

This paper proposed a novel Region of Interest (ROI) based rate control algorithm for video communication systems, in which the subjective quality of ROI can be adjusted according to users' requirements. In the proposed scheme, in order to analyze the relationship between subjective quality and encoding parameters, a Structural Similarity Index Map — quantization parameter (SSIM-QP) model is established. Through this relation, the possible visual quality range of ROI is defined according to the range of ROI QP, which is predicted by rate control algorithm. Then, with interest levels being identified within the visual quality range, resources allocated to ROI is determined. Finally, considering both the quality of ROI and the entire frame, resource allocation is slightly adjusted. Experimental results demonstrate that the subjective visual quality of ROI in our proposed scheme can be adjustable improved compared to the existing rate control algorithms.
Visual attention based image quality assessment

Inspired by the success of structural similarity index (SSIM), some image quality assessment (IQA) methods have been developed recently. To achieve better performance, this paper proposes a new visual attention (VA) model that combines saliency based VA and visual importance based VA, under the assumptions that humans often pay more attention to the regions with important content in the beginning of evaluating a given image and then the regions with poor quality. Then the proposed VA model is incorporated into SSIM. The experiments on LIVE database and TID2008 database demonstrate its improvements over the latest state-of-the-art IQA methods and the information content weighted SSIM measure (IW-SSIM).
Motion vector refinement for frame rate up conversion on 3D video

With the rapid development of digital video technology, frame rate up conversion is widely used. In this paper, a novel motion vector refinement method for frame rate up conversion on depth based 3D video is proposed. Our method involves two major stages in frame rate up conversion which are motion estimation and motion vector filtering. In the motion estimation process, the depth constraint to block matching algorithm is introduced in the bi-directional motion estimation method to obtain the motion vectors. In the motion vector filtering process, a depth-guided filter is designed to enhance the consistence of motions in the same depth plane. The refined motion vectors are used for frame interpolation. Experimental results show that the proposed method achieves 0.45 dB gain in terms of PSNR on average and improves the visual quality of the frame rate up-converted video.
Single image super-resolution with dilated convolution based multi-scale information learning inception module

Traditional works have shown that patches in a natural image tend to redundantly recur many times inside the image, both within the same scale, as well as across different scales. Make full use of these multi-scale information can improve the image restoration performance. However, the current proposed deep learning based restoration methods do not take the multi-scale information into account. In this paper, we propose a dilated convolution based inception module to learn multi-scale information and design a deep network for single image super-resolution. Different dilated convolution learns different scale feature, then the inception module concatenates all these features to fuse multi-scale information. In order to increase the reception field of our network to catch more contextual information, we cascade multiple inception modules to constitute a deep network to conduct single image super-resolution. With the novel dilated convolution based inception module, the proposed end-to-end single image super-resolution network can take advantage of multi-scale information to improve image super-resolution performance. Experimental results show that our proposed method outperforms many state-of-the-art single image super-resolution methods.
Early Determination of Zero-Quantized 8 $\,\times\,$8 DCT Coefficients

This paper proposes a novel approach to early determination of zero-quantized 8 × 8 discrete cosine transform (DCT) coefficients for fast video encoding. First, with the dynamic range analysis of DCT coefficients at different frequency positions, several sufficient conditions are derived to early determine whether a prediction error block (8 × 8) is an all-zero or a partial-zero block, i.e., the DCT coefficients within the block are all or partially zero-quantized. Being different from traditional methods that utilize the sum of absolute difference (SAD) of the entire prediction error block, the sufficient conditions are derived based on the SAD of each row of the prediction error block. For partial-zero blocks, fast DCT/IDCT algorithms are further developed by pruning conventional 8-point butterfly-based DCT/IDCT algorithms. Experimental results exhibit that the proposed early determination algorithm greatly reduces computational complexity in terms of DCT/IDCT, quantization, and inverse quantization, as compared with existing algorithms.
A Spatio-Temporal Autoregressive Frame Rate Up Conversion Scheme

A spatio-temporal autoregressive model is proposed in this paper to address the problem of frame rate up conversion. Every pixel in a skipped frame is generated as a linear combination of pixel values from forward and backward reference frames. At the beginning of the presented scheme, the coarse model parameters are computed according to the given initial pixel values for skipped frames. Then the coarse parameters are refined by an iteration process, during which we also interpolate the original low rate frames by the two closest generated skipped frames to derive more accurate parameters. Experimental results verify that the proposed algorithm significantly improves both the subjective and objective quality of the interpolated frames.
Window-level rate control for smooth picture quality and smooth buffer occupancy

Traditionally, rate control consists of bit allocation and QP decision on R-QP model. In bit allocation, target bits is further confined if buffer overflows. Meanwhile, rate control should also give as smooth as possible picture quality. However, there is no explicit relationship between picture quality and encoding parameters, so the coding result on picture quality is usually unpredictable and uncontrollable. On the condition of smooth picture quality, the smooth buffer occupancy is preferable certainly. In our work, we first proposed a “window model” formulating the size of window and variations of picture quality and buffer occupancy. Thus, given the constraint on picture quality and buffer occupancy, the compliant coding result about them can be expected employing window model. Second, a window-level rate distortion (RD) model inspired by the traditional ρ-domain model is introduced. Lastly, the evaluation of our proposal is presented with elaborate experiments.
Video indexing by motion activity maps

Motion based video indexing is an important and active research area in content-based video retrieval. It explores the dynamic characteristics of video content and provides techniques for video representation and retrieval. In this paper, a new motion-based approach is proposed, in which the image generated from motion activity is used to index video. The proposed approach firstly computes the accumulation measurement of motion activity on the grids of video frames along the time axis. Then, the computed measurement is quantized into several gray levels and a gray image is generated from all measurements on the grids. The generated image called motion activity map (MAM) reserves the motion information on different spatial locations of video. The intensity of MAM is corresponding to the magnitude of motion activity. All MAMs are organized into a hierarchical tree according to the video structure. Therefore, user can browse video by the MAMs tree. The proposed approach provides a hierarchical, coarse to fine view of video and thus makes interactive video retrieval more simply and intuitively.
Wavelet based distributed video coding with spatial scalability

In this paper, an improved spatial scalable wavelet domain distributed video coding (DVC) scheme is proposed. In this scheme, we adaptively apply differential pulse code modulation (DPCM) Wyner-Ziv (WZ) coding and discrete cosine transform (DCT) based intra block coding to reduced-resolution layer. Due to the low energy nature of high-pass subbands, WZ coding can be directly used. At decoder, in full-resolution layer WZ frame decoding, an efficient side information (SI) generation technique for low-pass subband is proposed, in which motion compensated interpolation (MCI) is performed on full-resolution signal. On the other hand, a refined technique is also employed to generate the SI for high-pass subbands by taking advantage of inter-band correlation. Experimental results show that the proposed scheme provides an efficient wavelet domain DVC coding paradigm.
GPU-based optimization for sample adaptive offset in HEVC

The latest high efficiency video coding (HEVC) standard achieves about 50% bit-rate reduction at equivalent visual quality compared to H.264/AVC. Sample adaptive offset (SAO) is one of the newly adopted tools right after deblocking filter, which can improve both coding efficiency and visual quality. However, for real-time encoding scenarios, the complexity of SAO is usually too high to be enabled. In this paper, a GPU-based optimization algorithm is proposed to reduce the complexity of SAO. Experiments are conducted based on the state-of-the-art open source HEVC encoder, i.e. X265. Results show that the proposed algorithm can reduce about 70% processing time of SAO on average without sacrifice of coding efficiency.
Window-Level Rate Control for Smooth Picture Quality and Smooth Buffer Occupancy

In rate control, smooth picture quality and smooth buffer occupancy are both important but contrary to each other at a given bit rate. How to get a good tradeoff between them was not devoted much attention previously. To deal with this problem, a theoretical window model is proposed in this paper, in which several adjacent frames grouped as a window are considered together. The smoothness of both picture quality and buffer occupancy can be gracefully achieved by regulating the size of the window. To illustrate the usage of window model, a window-level rate control algorithm cooperated with the traditional ρ-domain rate-distortion model is further introduced. In experiments, we first show how the proposed window model achieves the tradeoff between picture quality smoothness and buffer smoothness, and then demonstrate the significant PSNR improvement, accuracy of bit control and consistency of visual quality of the proposed window-level rate control algorithm.
A unified framework for locating and recognizing human actions

In this paper, we present a pose based approach for locating and recognizing human actions in videos. In our method, human poses are detected and represented based on deformable part model. To our knowledge, this is the first work on exploring the effectiveness of deformable part models in combining human detection and pose estimation into action recognition. Comparing with previous methods, ours have three main advantages. First, our method does not rely on any assumption on video preprocessing quality, such as satisfactory foreground segmentation or reliable tracking; Second, we propose a novel compact representation for human pose which works together with human detection and can well represent the spatial and temporal structures inside an action; Third, with human detection taken into consideration in our framework, our method has the ability to locate and recognize multiple actions in the same scene. Experiments on benchmark datasets and recorded cluttered videos verified the efficacy of our method.
A visual comfort assessment metric for stereoscopic images

Recent studies have shown that one of the main reasons inducing visual discomfort is accommodation-vergence conflict. To evaluate visual discomfort induced by this conflict, this paper proposes a stereo visual comfort assessment (SVCA) metric based on the measurement of accommodation and vergence for stereoscopic image. Here, accommodation corresponds to the monocular focusing process which is modeled by two-view images' joint entropy; vergence corresponds to binocular fusion which is modeled by two-view images' mutual information. The joint entropy and mutual information are calculated by the visual primitives extracted from two-view images. In this paper, accommodation-vergence conflict is expressed as the ratio of the mutual information over joint entropy. To evaluate the proposed metric, a subjective experiment is conducted to construct a ground truth database. The experimental results show that the proposed SVCA metric achieves a highly competitive performance with some state-of-the-art SVCA models.
Joint just noticeable difference model based on depth perception for stereoscopic images

Just noticeable difference (JND) model can reflect the least perceptible distortion from images, including 2D images and stereoscopic images. As we know, for the perception of human visual system (HVS), stereoscopic images have quite different characteristics from 2D images, since stereoscopic images contain not only planar information, but also depth information. This paper proposes a joint JND (JJND) model based on depth perception for stereoscopic images. Firstly, disparity estimation is performed in order to decompose the image into the occlusion region and the non-overlapped region. Then, different JND thresholds are applied on different regions, according to the depth information of the region, which can be derived from the disparity of the region. Experimental results verified our model's validity for stereoscopic images.