nils-herrmann/clean_scirepeval huggingface dataset:
Efficient image/video deblocking via sparse representation

Blocking artifact, characterized by visually noticeable changes in pixel values along block boundaries, is a common problem in block-based image/video compression, especially at low bitrate coding. Various post-processing techniques have been proposed to reduce blocking artifacts, but they usually introduce excessive blurring or ringing effects. This paper proposes a self-learning-based image/ video deblocking framework via properly formulating deblocking as an MCA (morphological component analysis)-based image decomposition problem via sparse representation. The proposed method first decomposes an image/video frame into the low-frequency and high-frequency parts by applying BM3D (block-matching and 3D filtering) algorithm. The high-frequency part is then decomposed into a “blocking component” and a “non-blocking component” by performing dictionary learning and sparse coding based on MCA. As a result, the blocking component can be removed from the image/video frame successfully while preserving most original image/video details. Experimental results demonstrate the efficacy of the proposed algorithm.
Power saving on mobile devices through contrast-aware backlight control

Based on the high penetration rate of mobile devices, reducing power consumption for mobile devices have become increasingly important. In current technology, statistics show that the display of the mobile device dominates the major power consumption of the device (e.g., up to 40% of the total power). Therefore, to efficiently reduce the display power consumption while maintaining the image quality, this paper proposes a novel contrast-aware backlight control algorithm for mobile devices. By transforming the color space for a displayed image from RGB (red, green, blue) to HSI (hue, saturation, intensity), we first calculate the RMS (root mean square) contrast for the image. Then we estimate the "brightness factor" for adjusting the image based on the RMS contrast and the employed HVS (human vision system)-based visual quality assessment technique. At the same time, the estimated brightness factor is used to determine the screen backlight brightness value of the device. As a result, the best tradeoff between the image display quality and the power consumption can be achieved. Our experimental results have demonstrated that 10~40% reduction of backlight brightness can be achieved, resulting in power consumption of 8~25%, while good image display quality can be maintained.
Self-Learning Based Image Decomposition With Applications to Single Image Denoising

Decomposition of an image into multiple semantic components has been an effective research topic for various image processing applications such as image denoising, enhancement, and inpainting. In this paper, we present a novel self-learning based image decomposition framework. Based on the recent success of sparse representation, the proposed framework first learns an over-complete dictionary from the high spatial frequency parts of the input image for reconstruction purposes. We perform unsupervised clustering on the observed dictionary atoms (and their corresponding reconstructed image versions) via affinity propagation, which allows us to identify image-dependent components with similar context information. While applying the proposed method for the applications of image denoising, we are able to automatically determine the undesirable patterns (e.g., rain streaks or Gaussian noise) from the derived image components directly from the input image, so that the task of single-image denoising can be addressed. Different from prior image processing works with sparse representation, our method does not need to collect training image data in advance, nor do we assume image priors such as the relationship between input and output image dictionaries. We conduct experiments on two denoising problems: single-image denoising with Gaussian noise and rain removal. Our empirical results confirm the effectiveness and robustness of our approach, which is shown to outperform state-of-the-art image denoising algorithms.
Cross-camera vehicle tracking via affine invariant object matching for video forensics applications

The recent deployment of very large-scale camera networks consisting of fixed/moving surveillance cameras and vehicle video recorders, has led to a novel field in object tracking problem. The major goal is to detect and track each vehicle within a large area, which can be applied to video forensics. For example, a suspected vehicle can be automatically identified for mining digital criminal evidences from a large amount of video data. In this paper, we propose an efficient cross-camera vehicle tracking technique via affine invariant object matching. More specifically, we formulate the problem as invariant image feature matching among different viewpoints of cameras. To achieve vehicle matching, we first extract invariant image feature based on ASIFT (affine and scale-invariant feature transform) for each detected vehicle in a camera network. Then, to improve the accuracy of ASIFT feature matching between images from different viewpoints, we propose to efficiently match feature points based on our observed spatially invariant property of ASIFT, as well as the min-hash technique. As a result, cross-camera vehicle tracking can be efficiently and accurately achieved. Experimental results demonstrate the efficacy of the proposed algorithm and the feasibility to video forensics applications.
Compressive sensing-based image hashing

In this paper, a new image hashing scheme satisfying robustness and security is proposed. We exploit the property of dimensionality reduction inherent in compressive sensing/sampling (CS) for image hash design. The gained benefits include (1) the hash size can be kept small and (2) the CS-based hash is computationally secure. We study the use of visual information fidelity (VIF) for hash comparison under Stirmark attacks. We further derive the relationships between the hash of an image and both of its MSE distortion and visual quality measured by VIF, respectively. Hence, based on hash comparisons, both the distortion and visual quality of a query image can be approximately estimated without accessing its original version. We also derive the minimum distortion for manipulating an image to be unauthentic to measure the security of our scheme.
A hybrid error concealment scheme for MPEG-2 video transmission based on best neighborhood matching algorithm

For entropy-coded MPEG-2 video frames, a transmission error will not only affect the underlying codeword but also may affect subsequent codewords, resulting in a great degradation of the received video frames. In this study, a hybrid error concealment (EC) scheme for MPEG-2 video transmission is proposed. Based on the fitness function employed for evaluating the candidate concealed blocks of a corrupted block, a corrupted block in an intra-coded I frame is concealed by either the spatial EC algorithm in H.264 or the proposed fast best neighborhood matching (BNM) algorithm. A corrupted block in an inter-coded P or B frame is concealed by the proposed fast motion-compensated BNM algorithm. Based on the simulation results obtained in this study, the proposed scheme can recover high-quality MPEG-2 video frames from the corresponding corrupted video frames up to a video packet loss rate (VPLR) of 20%
Distributed compressive video sensing

Low-complexity video encoding has been applicable to several emerging applications. Recently, distributed video coding (DVC) has been proposed to reduce encoding complexity to the order of that for still image encoding. In addition, compressive sensing (CS) has been applicable to directly capture compressed image data efficiently. In this paper, by integrating the respective characteristics of DVC and CS, a distributed compressive video sensing (DCVS) framework is proposed to simultaneously capture and compress video data, where almost all computation burdens can be shifted to the decoder, resulting in a very low-complexity encoder. At the decoder, compressed video can be efficiently reconstructed using the modified GPSR (gradient projection for sparse reconstruction) algorithm. With the assistance of the proposed initialization and stopping criteria for GRSR, derived from statistical dependencies among successive video frames, our modified GPSR algorithm can terminate faster and reconstruct better video quality. The performance of our DCVS method is demonstrated via simulations to outperform three known CS reconstruction algorithms.
Self-learning-based single image super-resolution of a highly compressed image

Low-quality images are usually not only with low-resolution, but also suffer from compression artifacts (blocking artifact is treated as an example in this paper). Directly performing image super-resolution (SR) to a highly compressed (low-quality) image would also simultaneously magnify the blocking artifacts, resulting in unpleasing visual quality. In this paper, we propose a self-learning-based SR framework to simultaneously achieve single-image SR and compression artifact removal for a highly-compressed image. We argue that individually performing deblocking first, followed by SR to an image, would usually inevitably lose some image details induced by deblocking, which may be useful for SR, resulting in worse SR result. In our method, we propose to self-learn image sparse representation for modeling the relationship between low and high-resolution image patches in terms of the learned dictionaries, respectively, for image patches with and without blocking artifacts. As a result, image SR and deblocking can be simultaneously achieved via sparse representation and MCA (morphological component analysis)-based image decomposition. Experimental results demonstrate the efficacy of the proposed algorithm.
Learning-based movie summarization via role-community analysis and feature fusion

Movie summarization aims at condensing a full-length movie to a significantly shortened version that still preserves the movie's major semantic content. In this paper, we propose a learning-based movie summarization framework via role-community social network analysis and feature fusion. In our framework, scene-based movie summarization is formulated as a 0-1 knapsack problem, where the scene attention value for each significant scene is calculated as its “value” and the length of this scene is used as its “cost.” To identify the significance of each scene, we propose a learning-based approach to fuse the information derived from visual saliency (based on low-level features and high-level cognitive process for an input movie), high-level semantic analysis (based on the global and local social networks constructed from the movie), and user preferences. Our evaluation results show that in most test cases, the proposed method subjectively outperforms attention-based and role-based summarization methods and our previous role-community-based method in terms of semantic content preservation.
A new hybrid error concealment scheme for MPEG-2 video transmission

For entropy-coded MPEG-2 video frames, a transmission error will not only affect the underlying codeword but also may affect subsequent codewords, resulting in a great degradation of the received video frames. In this study, transmission errors in MPEG-2 video frames are first detected and located by the error detection scheme proposed by Shyu and Leou [1999], and then the corrupted blocks are concealed by the proposed hybrid error concealment scheme. Based on the condition of a corrupted block, a corrupted block in an intracoded I frame is concealed by either the spatial error concealment algorithm in H.26L test model long-term number 9 (TML-9) or the best neighborhood matching (BNM) algorithm followed by the proposed modified spatial anisotropic diffusion (SD) algorithm. A corrupted block in an inter-coded P or B frame is concealed by the proposed motion-compensated BNM algorithm. Based on the simulation results obtained in this study, the proposed scheme can recover high-quality MPEG-2 video frames from the corresponding corrupted video frames up to a bit error rate of 0.5%.
A survey of error resilient coding schemes for image and video transmission based on data embedding

For entropy-coded images/video, a transmission error in a codeword results in a great degradation of the received images/video. Recently, several error resilient coding schemes based on data embedding are proposed, in which some important data for error resiliency can be embedded into the compressed bitstream at the encoder. At the decoder, the important data can be extracted for error resiliency so that high-quality images/video can be recovered. In this study, a survey of error resilient coding schemes based on data embedding is presented.
Deterministic detection of node replication attacks in sensor networks

In Wireless Sensor Networks (WSNs), because sensor nodes do not equip with tamper resistance hardwares, they are vulnerable to the capture and compromise performed by the adversary. By launching the node replication attack, the adversary can place the replicas of captured sensor nodes back into the sensor networks so as to eavesdrop the transmitted messages or compromise the functionality of the network. Although several protocols are proposed to defend against node replication attacks, all the proposed methods can only detect the node replication attacks probabilistically. In this paper, we propose Quorum-Based Multicast (QBM) and Star-shape Line-Selected Multicast (SLSM) to detect the node replication attacks, both of which can deterministically detect the replicas.
Privacy-preserving multimedia cloud computing via compressive sensing and sparse representation

Cloud computing is an emerging technology developed for providing various computing and storage services over the Internet. In this paper, we proposed a privacy-preserving cloud-aware scenario for compressive multimedia applications, including multimedia compression, adaptation, editing/manipulation, enhancement, retrieval, and recognition. In the proposed framework, we investigate the applicability of our/existing compressive sensing (CS)-based multimedia compression and securely compressive multimedia “trans-sensing” techniques based on sparse coding for securely delivering compressively sensed multimedia data over a cloud-aware scenario. Moreover, we also investigate the applicability of our/existing sparse coding-based frameworks for several multimedia applications by leveraging the strong capability of a media cloud. More specifically, to consider several fundamental challenges for multimedia cloud computing, such as security and network/device heterogeneities, we investigate the applications of CS and sparse coding techniques in multimedia delivery and applications. As a result, we can build a unified cloud-aware framework for privacy-preserving multimedia applications via sparse coding.
Efficient CTU-based intra frame coding for HEVC based on deep learning

To further improve the compression efficiency of HEVC intra frame coding, in this paper, a deep learning-based framework is proposed. Inspired by recently developed deep learning models for image super-resolution (SR), we propose to train a CNN (convolutional neural network) model to precisely predict the residual information of each CTU (coding tree unit) at the HEVC encoder. As a result, better CTU reconstruction and better prediction for the compression of subsequent CTUs can be achieved. To reduce computational complexity, different from current CNN-based SR works, we propose to skip the non-linear mapping layer, and incorporate the residual learning to obtain better predicted residual for CTU encoding. Experimental results have shown that the proposed method achieves 3.2% bitrate reduction in average BDBR (Bjentegaard delta bit rate) with only 37% encoding complexity increased.
Real-Time Robust Background Modeling Based on Joint Color and Texture Descriptions

Background construction is the base of object detection and tracking for the machine vision system. Traditional background modeling methods often require complicated computations and are sensitive to illumination changes and shadow interference. In this paper, we propose a block-based background modeling method, which fully utilizes the color and texture characteristics of each incoming frame. The proposed method is quite efficient and is capable of resisting illumination changes and shadow disturbance. Experimental results show that our method is suitable for real-word scenes and real-time applications.
Scene-Based Movie Summarization Via Role-Community Networks

Video summarization techniques aim at condensing a full-length video to a significantly shortened version that still preserves the major semantic content of the original video. Movie summarization, being a special class of video summarization, is particularly challenging since a large variety of movie scenarios and film styles complicate the problem. In this paper, we propose a two-stage scene-based movie summarization method based on mining the relationship between role-communities since the role-communities in earlier scenes are usually used to develop the role relationship in later scenes. In the analysis stage, we construct a social network to characterize the interactions between role-communities. As a result, the social power of each role-community is evaluated by the community's centrality value and the role communities are clustered into relevant groups based on the centrality values. In the summarization stage, a set of feasible summary combinations of scenes is identified and an information-rich summary is selected from these candidates based on social power preservation. Our evaluation results show that in at most test cases the proposed method achieves better subjective performance than attention-based and role-based summarization methods in terms of semantic content preservation for a movie summary.
Seeing through the expression: Bridging the gap between expression and emotion recognition

In this paper, we propose a novel approach for visualizing and recognizing different emotion categories using facial expression images. Extended by the unsupervised nonlinear dimension reduction technique of locally linear embedding (LLE), we propose a supervised LLE (sLLE) algorithm utilizing emotion labels of face expression images. While existing works typically aim at training on such labeled data for emotion recognition, our approach allows one to derive subspaces for visualizing facial expression images within and across different emotion categories, and thus emotion recognition can be properly performed. In our work, we relate the resulting two-dimensional subspace to the valence-arousal emotion space, in which our method is observed to automatically identify and discriminate emotions in different degrees. Experimental results on two facial emotion datasets verify the effectiveness of our algorithm. With reduced numbers of feature dimensions (2D or beyond), our approach is shown to achieve promising emotion recognition performance.
Salient object detection on a hierarchy of image partitions

This paper presents a region-based model for salient object detection. The model is defined as a combination of saliency maps constructed on a hierarchy of image partitions. The saliency map at each level in the hierarchy takes into account the color similarity between each region and its neighbors and highlights objects at the scale defined by the partition, with accurate boundaries. Experimental results on a publicly available dataset show that the proposed model outperforms many state-of-the-art saliency methods.
Automatic keyframe selection based on mutual reinforcement algorithm

This paper addresses the problem of video summarization through an automatic selection of a single representative keyframe. The proposed solution is based on the mutual reinforcement paradigm, where a keyframe is selected thanks to its highest and most frequent similarity to the rest of considered frames. Two variations of the algorithm are explored: a first one where only frames within the same video are used (intra-clip mode) and a second one where the decision also depends on the previously selected keyframes of related videos (inter-clip mode). These two algorithms were evaluated by a set of professional documentalists from a broadcaster's archive, and results concluded that the proposed techniques outperform the semi-manual solution adopted so far in the company.
Deep Relative Attributes

Relative attribute (RA) learning aims to learn the ranking function describing the relative strength of the attribute. Most of current learning approaches learn a linear ranking function for each attribute by use of the hand-crafted visual features. Different from the existing study, in this paper, we propose a novel deep relative attributes (DRA) algorithm to learn visual features and the effective nonlinear ranking function to describe the RA of image pairs in a unified framework. Here, visual features and the ranking function are learned jointly, and they can benefit each other. The proposed DRA model is comprised of five convolutional neural layers, five fully connected layers, and a relative loss function which contains the contrastive constraint and the similar constraint corresponding to the ordered image pairs and the unordered image pairs, respectively. To train the DRA model effectively, we make use of the transferred knowledge from the large scale visual recognition on ImageNet  to the RA learning task. We evaluate the proposed DRA model on three widely used datasets. Extensive experimental results demonstrate that the proposed DRA model consistently and significantly outperforms the state-of-the-art RA learning methods. On the public OSR, PubFig, and Shoes datasets, compared with the previous RA learning results , the average ranking accuracies have been significantly improved by about 8%, 9%, and 14%, respectively.
Text2Video: An End-to-end Learning Framework for Expressing Text With Videos

Video creation is a challenging and highly profession-al task that generally involves substantial manual efforts. To ease this burden, a better approach is to automatically produce new videos based on clips from the massive amount of existing videos according to arbitrary text. In this paper, we formulate video creation as a problem of retrieving a sequence of videos for a sentence stream. To achieve this goal, we propose a novel multimodal recurrent architecture for automatic video production. Compared with existing methods, the proposed model has three major advantages. First, it is the first completely integrated end-to-end deep learning system for real-world production to the best of our knowledge. We are among the first to address the problem of retrieving a sequence of videos for a sentence stream. Second, it can effectively exploit the correspondence between sentences and video clips through semantic consistency modeling. Third, it can model the visual coherence well by requiring that the produced videos should be organized coherently in terms of visual appearance. We have conducted extensive experiments on two applications, including video retrieval and video composition. The qualitative and quantitative results obtained on two public datasets used in the Large Scale Movie Description Challenge 2016 both demonstrate the effectiveness of the proposed model compared with other state-of-the-art algorithms.