nils-herrmann/clean_scirepeval huggingface dataset:
G-CAST: Gradient Based Image SoftCast for Perception-Friendly Wireless Visual Communication

Conventional image and video communication systems are usually designed with the objective being to maximize the fidelity of reconstructed images measured by mean square errors (MSE). It is well known that the fidelity metric MSE may not reflect the visual quality perceived by human eyes. Recent advancements in image quality assessment tell us that the structural similarity (SSIM), especially the gradient similarity, reveals the perceptual fidelity of images more reliably. Inspired by this observation, this paper proposes a new image communication approach, which conveys the visual information in an image by transmitting the image gradients and recovers the image from the received gradient data at decoder side using statistical image prior knowledge. In particular, we designed a gradient-based image SoftCast scheme for wireless scenarios. Experimental results show that the proposed scheme can produce reconstruction images with much better perceptual quality. The advantage in perceptual quality is verified by the quality improvement measured by the metrics SSIM and gradient signal-to-noise ratio (GSNR).
Tanner Graph Based Image Interpolation

This paper interprets image interpolation as a channel decoding problem and proposes a tanner graph based interpolation framework, which regards each pixel in an image as a variable node and the local image structure around each pixel as a check node. The pixels available from low-resolution image are "received" whereas other missing pixels of highresolution image are "erased", through an imaginary channel. Local image structures exhibited by the low-resolution image provide information on the joint distribution of pixels in a small neighborhood, and thus play the same role as parity symbols in the classic channel coding scenarios. We develop an efficient solution for the sum-product algorithm of belief propagation in this framework, based on a gaussian auto-regressive image model. Initial experiments show up to 3dB gain over other methods with the same image model. The proposed framework is flexible in message processing at each node and provides much room for incorporating more sophisticated image modelling techniques.
Layered Soft Video Broadcast for Heterogeneous Receivers

Wireless video broadcast poses a challenge to the conventional visual communication in providing simultaneously each receiver the best video quality under its channel condition. Soft video broadcast, as a newly emerged wireless video broadcast scheme, is able to accommodate multiple receivers of different channel SNRs. However, the current soft video broadcast frameworks such as SoftCast require the bandwidth of the wireless channel to match the number of video coefficients per second. When the channel bandwidth is larger, the existing frameworks become not very efficient in bandwidth expansion. More importantly, it is possible that the users in broadcast applications have different bandwidths. However, none of the existing soft video broadcast frameworks considers bandwidth heterogeneity. In this paper, we propose a soft video broadcast framework, called LayerCast, which can simultaneously accommodate heterogeneous users with diverse SNRs and diverse bandwidths. The bandwidth expansion problem is solved by applying layered coset coding. More importantly, we derive a globally optimal power allocation between layers and, within each layer, between each DCT chunk. In simulations, the proposed framework outperforms SoftCast of up to 4 dB in video PSNR, and outperforms H.264-based framework up to 8 dB in broadcast.
A new image deblurring algorithm with less ringing artifacts via error variance estimation and soft decision

Image deblurring is an ill-posed linear inverse problem. Most traditional algorithms suffer from severe ringing artifacts. Recent approaches handle this issue by regularization techniques based on assumed image prior models. This paper presents a new method to reduce the ringing artifacts, without introducing any image prior models. For this purpose, we revisit the deblurring problem, using a probabilistic graph to model the image formation process. We establish the link between iterative back-projection and belief propagation and show that the ringing artifacts are caused by error propagation. Based on these analysis, we introduce a method to measure the variance of an estimation image and further propose an error-variance aware deblurring algorithm. Experimental results demonstrate that the proposed algorithm is very effective in suppressing the ringing artifacts.
Transform domain energy modeling of natural images for wireless SoftCast optimization

The SoftCast scheme recently proposed for wireless visual communication avoids the threshold effect that traditional communication systems usually suffer from. It provides graceful quality transition by sending images in a sequence of whitened transform coefficients using dense-constellation modulation and analog-like transmission. A key point in SoftCast is that it allocates transmission power among coefficients unequally, according to the expected energy of each coefficient. Importantly, the energy diversity utilized by power allocation should be shared with the receiver for correct decoding. Signaling the energy for each coefficient individually is prohibitive since it requires a large set of meta data. Grouping coefficients into a few chunks and signaling the energy at chunk level, on the other hand, may compromise the efficiency of SoftCast remarkably. In this paper, we investigate the energy distribution of natural images in transform domain and propose a model to approximate this distribution. We apply the model to guide the power allocation in SoftCast. Experimental results show that the proposed method outperforms the equal-chunk approach in the original SoftCast by 2~5 dB, and reduces the number of required meta data significantly at the same time.
Study on subjective quality assessment of Screen Content Images

With the coming age of big data, the cloud technology, referred to as the computations or applications through the Internet, is dramatically developed. The screen content has become one of the most common data form due to the extraordinary advance of network communication technology, and the JCT-VC has started to develop new standard focusing on improving the efficiency of screen content based on High Efficiency Video Coding (HEVC). Nevertheless, the research on the quality assessment of screen content is still quite limited at the current stage. In this paper, we present a study on subjective quality assessment of the Screen Content Images (SCIs) and investigate whether the existing objective Image Quality Assessment (IQA) methods can effectively evaluate the quality of distorted SCIs. We construct a new Screen Content Database (SCD) including 24 source SCIs and 492 compressed ones with two codecs including HEVC as well as the HEVC extension. The Single Comparison (SC) method is employed for the subjective viewing to guarantee the reliability of the results. In our experiment, the correlations of eight popular IQA methods with the obtained Mean Opinion Score (MOS) values are evaluated. The result indicates that visual information fidelity method can achieve highest consistency with human visual perception.
Diversity-Based Reference Picture Management for Low Delay Screen Content Coding

Screen content coding plays an important role in many applications. Conventional reference picture management (RPM) strategies developed for natural content may not work well for screen content. This is because many regions in screen content remain static for a long time, causing a lot of repetitive contents to stay in the decoded picture buffer. The repetitive contents are not conducive to inter prediction, but still occupy valuable memory. This paper proposes a diversity-based RPM scheme for screen content coding. The concept of diversity is introduced for the reference picture set (RPS) to help formulate the RPM problem. By maximizing the diversity of RPS, more potentially better predictions are provided. Better compression performance can then be achieved. Meanwhile, the proposed scheme is nonnormative and compatible with existing video coding standards, such as High Efficiency Video Coding. The experimental results show that, for low delay screen content coding, the bit saving of the proposed scheme is 4.9% on average and up to 13.7%, without increasing encoding time.
Wireless Image SoftCast Using Compressive Gradient

Based on observations that the visual quality has strong correlation with image gradients, gradient based image SoftCast (G-Cast) advocates to convey visual information by delivering image gradients. In G-Cast, both horizontal gradients and vertical gradients needs to be transmitted, even if the channel bandwidth is insufficient. This paper propose to send out the random projection measurements of the gradients instead of delivering gradients directly, so that the number of transmitted data can be adjusted according to bandwidth conditions. Experimental results validate the effectiveness of the the proposed transmission scheme under various channel signal-to-noise ratio (CSNR) and bandwidth conditions.
Compression Artifact Reduction by Overlapped-Block Transform Coefficient Estimation With Block Similarity

Block transform coded images usually suffer from annoying artifacts at low bit rates, caused by the coarse quantization of transform coefficients. In this paper, we propose a new method to reduce compression artifacts by the overlapped-block transform coefficient estimation from non-local blocks. In the proposed method, the discrete cosine transform coefficients of each block are estimated by adaptively fusing two prediction values based on their reliabilities. One prediction is the quantized values of coefficients decoded from the compressed bitstream, whose reliability is determined by quantization steps. The other prediction is the weighted average of the coefficients in nonlocal blocks, whose reliability depends on the variance of the coefficients in these blocks. The weights are used to distinguish the effectiveness of the coefficients in nonlocal blocks to predict original coefficients and are determined by block similarity in transform domain. To solve the optimization problem, the overlapped blocks are divided into several subsets. Each subset contains nonoverlapped blocks covering the whole image and is optimized independently. Therefore, the overall optimization is reduced to a set of sub-optimization problems, which can be easily solved. Finally, we provide a strategy for parameter selection based on the compression levels. Experimental results show that the proposed method can remarkably reduce compression artifacts and significantly improve both the subjective and objective qualities of block transform coded images.
Fast Image Super-Resolution via Local Adaptive Gradient Field Sharpening Transform

This paper proposes a single-image super-resolution scheme by introducing a gradient field sharpening transform that converts the blurry gradient field of upsampled low-resolution (LR) image to a much sharper gradient field of original high-resolution (HR) image. Different from the existing methods that need to figure out the whole gradient profile structure and locate the edge points, we derive a new approach that sharpens the gradient field adaptively only based on the pixels in a small neighborhood. To maintain image contrast, image gradient is adaptively scaled to keep the integral of gradient field stable. Finally, the HR image is reconstructed by fusing the LR image with the sharpened HR gradient field. Experimental results demonstrate that the proposed algorithm can generate more accurate gradient field and produce super-resolved images with better objective and visual qualities. Another advantage is that the proposed gradient sharpening transform is very fast and suitable for low-complexity applications.
Multi-scale Spatial Error Concealment via Hybrid Bayesian Regression

In this paper, we propose a novel multi-scale spatial error concealment algorithm to combine the modeling strengthes of the parametric and nonparametric Bayesian regression. We progressively recover missing blocks in the scale space from coarse to fine so that the sharp edges and texture in the finest scale can be eventually recovered. On one hand, in each scale, the nonparametric part of the methodology is used to exploit the intra-scale correlation, which relies on the data itself to dictate the structure of the model. In this procedure, the non-local self-similarity property is utilized as a fruitful resource for abstracting a priori knowledge of images. On the other hand, the parametric part is used to explicitly model the inter-scale correlation, in which the local structure regularity is thoroughly explored to recover the sharp edges and major texture features of images. It is not respected if only the nonparametric modeling is considering. We achieve the best of both worlds within a multi-scale framework. Experimental results on benchmark test images demonstrate that the proposed method achieves very competitive performance with the state-of-the-art error concealment algorithms.
Residual Highway Convolutional Neural Networks for in-loop Filtering in HEVC

High efficiency video coding (HEVC) standard achieves half bit-rate reduction while keeping the same quality compared with AVC. However, it still cannot satisfy the demand of higher quality in real applications, especially at low bit rates. To further improve the quality of reconstructed frame while reducing the bitrates, a residual highway convolutional neural network (RHCNN) is proposed in this paper for in-loop filtering in HEVC. The RHCNN is composed of several residual highway units and convolutional layers. In the highway units, there are some paths that could allow unimpeded information across several layers. Moreover, there also exists one identity skip connection (shortcut) from the beginning to the end, which is followed by one small convolutional layer. Without conflicting with deblocking filter (DF) and sample adaptive offset (SAO) filter in HEVC, RHCNN is employed as a high-dimension filter following DF and SAO to enhance the quality of reconstructed frames. To facilitate the real application, we apply the proposed method to I frame, P frame, and B frame, respectively. For obtaining better performance, the entire quantization parameter (QP) range is divided into several QP bands, where a dedicated RHCNN is trained for each QP band. Furthermore, we adopt a progressive training scheme for the RHCNN where the QP band with lower value is used for early training and their weights are used as initial weights for QP band of higher values in a progressive manner. Experimental results demonstrate that the proposed method is able to not only raise the PSNR of reconstructed frame but also prominently reduce the bit-rate compared with HEVC reference software.
Image restoration via multi-scale non-local total variation regularization

Total-variation (TV) regularization is widely adopted in image restoration problems to exploit the local smoothness of image. However, traditional TV regularization only models the sparsity of image gradient at the original scale. This paper introduces a multi-scale TV regularization method which models the image gradient sparsity at different scales, and constrains the gradient magnitude of different scales jointly. As different scales extract different frequency of image, our proposed multi-scale regularization method provides constraints for different frequency components. And for each scale, we adaptively estimate the gradient distribution at a particular pixel from a group of nonlocally searched similar patches. Finally, experimental results demonstrate that the proposed method outperforms the conventional TV regularization methods for image restoration.
Artifact reduction of compressed video via three-dimensional adaptive estimation of transform coefficients

Block transform compressed videos usually suffer from annoying artifacts at low bit rates, caused by the coarse quantization of transform coefficients. The inter prediction utilized in video coding also induces block boundary artifacts when the neighboring blocks using different motion vectors from previous decoded frames. In this paper, a three-dimensional adaptive estimation method for video transform coefficients is proposed to reduce the artifact in compressed video. In the proposed method, transform coefficients of each block are estimated by adaptively fusing three prediction sources based on their reliabilities. One prediction source is the transform coefficients directly acquired from the decoded video, whose reliability is determined by the distribution of the quantization noise. The other prediction source is derived by a temporal autoregressive model of transform-blocks along motion trajectory among neighboring reference frames. Its reliability is estimated from prediction variance of local blocks. The last prediction source is derived from the nonlocal transform-blocks, whose reliability is estimated based on the distribution of the nonlocal coefficients and their similarity with the estimated block. Experimental results for compressed video sequences by HEVC show that, the proposed method can reduce the compression artifacts and improve both the objective and subjective quality.
Registration of under-sampled images via higher resolution spectrum restoration

High accuracy image registration is critical for the success of multi-frame super-resolution. Conventionally, the shift between images are estimated directly based on the under-sampled low-resolution (LR) image data. However, the high-frequency of LR data is unreliable due to the aliasing effect of sub-sampling, which will deteriorate the accuracy of registration. This paper proposes to resolve the aliasing by converting the LR images to high-resolution (HR) domain and then perform registration on the restored HR spectrum. To recover the HR spectrum for each corresponding LR image, we fuse the LR images into one HR image and project the estimation difference back to the reconstructed HR spectrum iteratively. To address the unequal reliabilities of different restored frequencies, weighted least square is employed to improve the precision of registration. Experimental results show that the proposed method can outperform other existing methods and improve the quality of super-resolution image.
Bayesian frame interpolation by fusing multiple motion-compensated prediction frames

For video frame rate up-conversion, new frames need to be interpolated along motion trajectories and inserted between the original adjacent frames. This paper proposes a Bayesian frame interpolation strategy, which considers both the spatial (i.e. intra-frame) correlation model and the temporal (i.e. inter-frame) correlation model after motion compensation. Different from the conventional schemes that perform interpolation using only one estimated motion vector field (MVF), the proposed strategy estimates multiple MVFs. To cope with different scales of frame contents, we generate the multiple MVFs through variable block size motion estimation. A criterion is adopted to estimate reliability of the motion compensated prediction frames from different motion fields. Experimental results demonstrate that the proposed strategy improves the interpolation performance remarkably.
Intra Prediction Using Multiple Reference Lines for Video Coding

Traditional intra prediction schemes usually only use the nearest adjacent reference lineto generate the prediction. Although the nearest reference line generally has the strongeststatistical correlation with current block, the farther non-adjacent reference lines can stillprovide potentialb etter prediction in some cases. Thus, in this paper, not only the nearestreference line but also the farther reference lines are utilized to help intra prediction. Whenusing the farther reference lines, an additional residue compensation procedure is introducedto further refine the prediction. In particular, this paper designs three solutions to meetdifferent complexity requirements. They are multiple line-based intra prediction (MLIP),fast search for multiple line-based intra prediction (FS-MLIP), and dual line-based intraprediction (DLIP). Experimentalresul ts verify the effectiveness of the proposed methods. When compared with HM-16.9, the proposed MLIP achieves 2.4% bit saving on averagewith the encoding time increasing about 362%. The FS-MLIP achieves 2.0% bit saving onaverage with the encoding time increasing about 114%. The DLIP achieves 0.9% bit savingon average with the encoding time increasing about only 15%.
Adaptive boundary dependent transform optimization for HEVC

High Efficiency Video Coding (HEVC) adopts hybrid transform coding scheme to improve the transform performance. However, it does not consider the influence of the prediction unit boundary. This paper proposes an adaptive boundary dependent transform optimization scheme for HEVC. Based on the transform unit boundary type identification by checking whether it lies at a prediction unit boundary or not, some additional transform kernels, including DCT-IV, are incorporated for transform. Moreover, the additional adopted transform kernels are generated by re-using the existing kernels in HEVC. Therefore, the butterfly structure for transform implementation can be preserved to facilitate parallel computing, and computational complexity increasing can be avoided. Experimental results show that the coding performance improvements can be up to 0.96% and 0.64% for low delay P and random access testing configurations respectively.
Hybridcast: A wireless image/video SoftCast scheme using layered representation and hybrid digital-analog modulation

The recently proposed SoftCast scheme employs analog-like transmission for wireless visual communication, providing graceful reconstruction quality degradation for drastically changing channel conditions. However, the transmission in SoftCast is not always efficient in terms of power usage. In this paper, we propose a wireless image/video SoftCast scheme which employs layered representation with hybrid digital-analog modulation. In this scheme, a coarse approximation of the image is coded in a base layer in digital ways, while the residual image details are delivered in an enhancement layer in analog-like way. The outputs from the two layers are superimposed for transmission, using a hybrid digital-analog modulation scheme. Since a major part of the signal is handled by the base layer, the power efficiency of the SoftCast layer is significantly improved. Experimental results show that the proposed scheme outperforms the original SoftCast remarkably, while still preserving the smooth quality degradation characteristic of the SoftCast scheme.
Image interpolation via regularized local linear regression

In this paper, we present an efficient image interpolation scheme by using regularized local linear regression (RLLR). On one hand, we introduce a robust estimator of local image structure based on moving least squares, which can efficiently handle the statistical outliers compared with ordinary least squares based methods. On the other hand, motivated by recent progress on manifold based semi-supervise learning, the intrinsic manifold structure is explicitly considered by making use of both measured and unmeasured data points. In particular, the geometric structure of the marginal probability distribution induced by unmeasured samples is incorporated as an additional locality preserving constraint. The optimal model parameters can be obtained with a closed-form solution by solving a convex optimization problem. Experimental results demonstrate that our method outperform the existing methods in both objective and subjective visual quality over a wide range of test images.