nils-herrmann/clean_scirepeval huggingface dataset:
A Unified Approach of Bit-Rate Control for Binary and Gray-Level Shape Sequences Coding

This paper describes a unified framework of bit-rate control for binary and gray-level shape coding. An exponential rate-distortion model is proposed for binary shape and used in the bit-rate control. An approximation method is applied for finer distortion scale adjustment by adjusting the conversion ratio at the binary alpha block level. Since the gray-level shape is encoded as its support information and the alpha plane information, the rate-quantizer relationship of the alpha plane is modeled. The bit-rate control is then able to adjust the quantizer of the alpha plane for the bit-rate control purpose. The extension for video object (VO) coding is also studied. In this bit-rate control approach, a buffer feedback controller is employed to determine the target bit rates for different sources, i.e., binary shape, gray-level shape, and texture. Accordingly, the distortion scales of different sources are adjusted based on their rate-distortion relationships to maintain a stable encoder buffer. The advantages of the proposed method are demonstrated through various tests for different scenarios. Overall, the algorithm can efficiently encode the shape information and shows a good performance for VO coding.
Learning Deep Spatial-Spectral Features for Material Segmentation in Hyperspectral Images

In this study, we propose a two-stage method for material segmentation in hyperspectral images. The first stage employs a Convolutional Neural Network (CNN) to predict the material label at individual pixels. The second stage further refines the segmentation by a fully-connected Conditional Random Field (CRF) framework. For the first stage, we experimented with two different network architectures. One is a network with five convolutional layers and three fully connected layers trained on small patches to predict the label of the central pixel of each patch. The other is an encoder-decoder architecture trained on larger image regions to predict the label of every pixel in a region. In the fully connected CRF, the unary term is aimed to respect the predicted label by the CNN while the pairwise term models the label compatibility between two pixels based on their PCA features. Experimental results demonstrate that the two proposed variants are able to outperform several existing methods quantitatively.
Very low bit rate video coding using 3D subband approach

A video coder for very low bit rate applications is designed and simulated. It employs subband analysis technique to first split the video signal into four temporal subbands. Motion detection is performed to classify the co-sited macroblocks of the subbands into temporal activity (TA) macroblocks or no temporal activity (NTA) macroblocks. For the TA macroblocks, motion classification is carried out by further dividing the four temporal subbands into 16 spatiotemporal subbands. The base temporal subband (LL/sub t/) is coded using a H.261-like (European COST 211 ter SIM3) coder with the motion estimation replaced by a reduced-motion search algorithm. The higher temporal subbands (LH/sub t/, HL/sub t/, HH/sub t/) are vector-quantized. The NTA-macroblocks (except in the baseband) are not coded at all. Simulation results are presented at 9.6 kbits per second (kbps) at five frames per second (fps) for a typical videoconferencing sequence. Comparisons are made with results obtained by the SIM3 coder.
A memory learning framework for effective image retrieval

Most current content-based image retrieval systems are still incapable of providing users with their desired results. The major difficulty lies in the gap between low-level image features and high-level image semantics. To address the problem, this study reports a framework for effective image retrieval by employing a novel idea of memory learning. It forms a knowledge memory model to store the semantic information by simply accumulating user-provided interactions. A learning strategy is then applied to predict the semantic relationships among images according to the memorized knowledge. Image queries are finally performed based on a seamless combination of low-level features and learned semantics. One important advantage of our framework is its ability to efficiently annotate images and also propagate the keyword annotation from the labeled images to unlabeled images. The presented algorithm has been integrated into a practical image retrieval system. Experiments on a collection of 10 000 general-purpose images demonstrate the effectiveness of the proposed framework.
Video quality assessment by decoupling additive impairments and detail losses

In this paper, a review on existing methods of extending image quality metric to video quality metric is given. It is found that three processing steps are usually involved which include the temporal channel decomposition, temporal masking and error pooling. They are utilized to extend our previously proposed image quality metric, which separately evaluates additive impairments and detail losses, to video quality metric. The resultant algorithm is tested on subjective video database LIVE and shows a good performance in matching subjective ratings.
Soft transition analysis technique for video sequences

Video indexing is a technique used where objects within the sequence are segmented, indexed and kept in the database so that they can be retrieved for future reference. One of the problems that arise in the indexing is the occurrence of special effects such as fading and wiping. During the indexing, the frames within such effects are discarded, as they are too complex to be segmented. It is desirable to deduce a technique so that these frames can be utilised in indexing. In this paper, we introduce a method that analyses and decodes the frames within fade transition. This utilises the frame pixel values within the transition to reconstruct the frames in their original sequences, which they are made from. The experiment shows that this technique can successfully reconstruct the original frames even with the existence of limited motion within the objects.
An error sensitivity-based redundant macroblock strategy for robust wireless video transmission

Packet video transmission over wireless networks suffers packet loss due to either temporary link outages or fading-induced bit errors. Such packet losses often occur in bursts, which may cause substantial degradation to the transmitted video quality. In this study, a redundant macroblock strategy is proposed. MB (macroblock) PSNR is employed to evaluate the error sensitivity of MBs. The most sensitive MBs are transferred in separate additional slices while coarsely quantized copies of the MBs are placed in the original slice. When working with chessboard style FMO (flexible macroblock reordering) and fixed length slice mode (FMO-slicing), the scheme performs well against packet loss errors with acceptable overhead and it is highly compatible with original H.264 bitstream. The results indicate that the proposed scheme can improve the decoded video quality by 0.8-2.5 dB comparing to H.264 chessboard FMO-slicing scheme over the simulated wireless channel.
Keynote speech 3: Visual quality evaluation for images and videos

Visual quality assessment of images and videos has become important nowadays with the demand of high-definition and high-quality viewing experience. The standards bodies, i.e., ITU-T VCEG and MPEG, now require subjective tests to be performed on all proposals. This talk will give an introduction on subjective and objective visual quality assessments, with focus on the latter. For subjective visual quality assessment, a short review will be given on the standard subjective evaluation procedure and the subjective score analysis methods. For objective visual quality assessment, its applications, its classifications, and the different techniques used in image and video quality metrics, both from existing literature and from our works, will be discussed. The general framework used by the classical HVS-model based image quality metrics (IQMs) will be briefly introduced first, followed by some new methodologies which do not rely on the HVS models, e.g., ones used in SSIM (Structural SIMilarity) and VIF (Visual Information Fidelity) quality measures. For video quality assessment, how temporal/motion information is used to measure the subjective video quality will also be presented.
Optimal bit allocation in HEVC for real-time video communications

Recently, the Lagrange multiplier λ based rate control has been developed in the latest HEVC (High Efficiency Video Coding) codec. It is revealed that while such a rate control approach successfully guarantees a designated bit-rate, the optimal bit allocation has not been investigated or established in the λ-domain. To enhance the overall performance, we propose a novel linear relationship between distortion and λ. With this distortion model, we deduce a closed-form solution to minimize the distortion while satisfying a given rate regulation at the coding tree unit level. Extensive simulation results show that our method can achieve an average 0.20 dB improvement in the low-delay communications.
Classified perceptual coding with adaptive quantization

A new technique of adaptively classifying the scene content of an image block has been developed in the proposed perceptual coder. It measures the texture masking energy of an image block and classifies it into one of four perceptual classes: flat, edge, texture, and fine-texture. Each class has an associated factor to adapt the quantizer with the aim of achieving constant quality across an image. A second feature of the perceptual coder is the visual thresholding, a process that reduces bit rate by discarding subthreshold discrete cosine transform (DCT) coefficients without degrading the image perceived quality. Finally, further quality gain is achieved by an improved reference model 8 (RM8) intramode decision, which removes sticking noise artifacts from newly uncovered background found in H.261 coded sequences. Subjective viewing tests, guided by Rec. 500-5, were conducted with 30 subjects. Subjective results confirm the efficacy of the proposed classified coder over the RMS based H.261 coder in two ways: (i) it consistently produces better quality sequences (with a mean opinion score, MOS, of approximately 2.0) when comparing at any fix bit rate; and (ii) it achieves a bit rate saving of 35% when measuring at the same picture quality (i.e., same MOS).
Co-Salient Object Detection From Multiple Images

In this paper, we propose a novel method to discover co-salient objects from a group of images, which is modeled as a linear fusion of an intra-image saliency (IaIS) map and an inter-image saliency (IrIS) map. The first term is to measure the salient objects from each image using multiscale segmentation voting. The second term is designed to detect the co-salient objects from a group of images. To compute the IrIS map, we perform the pairwise similarity ranking based on an image pyramid representation. A minimum spanning tree is then constructed to determine the image matching order. For each region in an image, we design three types of visual descriptors, which are extracted from the local appearance, e.g., color, color co-occurrence and shape properties. The final region matching problem between the images is formulated as an assignment problem that can be optimized by linear programming. Experimental evaluation on a number of images demonstrates the good performance of the proposed method on co-salient object detection.
An efficient content adaptive transform for video coding

In this paper, we propose a content adaptive transform (CAT) method for video coding. Firstly, CAT obtains the transform kernels in real-time instead of predefined fixed ones, and therefore it is content based transform. Secondly, the transform kernels is obtained by singular value decomposition (SVD) from a typical decoded block without transmitting them to the decoder. Thirdly, the optimality property of CAT for input signal is derived. Finally, a spiral scanning pattern is proposed to rearrange the diagonal distributed transform coefficients. Extensive experimental results show that, compared with the H.264/AVC High Profile, the proposed CAT scheme provides average PSNR gain 0.55 dB for intra frame coding on a large number of multiple-resolutions sequences.
Multiple feature clustering algorithm for automatic video object segmentation

In this paper, we present an automatic video segmentation algorithm for object-based coding, based on a k-medians clustering algorithm and 2D binary model. Firstly, the k-medians algorithm is employed to partition an image into a set of homogeneous regions. Then, a 2D binary model of the moving object is set up, which, combined with temporal and spatial information, guides the extraction process of the video object planes from the video sequence. The performance of the segmentation algorithm is illustrated by simulations carried out on standard video sequences.
Online Temporally Consistent Indoor Depth Video Enhancement via Static Structure

In this paper, we propose a new method to online enhance the quality of a depth video based on the intermediary of a so-called static structure of the captured scene. The static and dynamic regions of the input depth frame are robustly separated by a layer assignment procedure, in which the dynamic part stays in the front while the static part fits and helps to update this structure by a novel online variational generative model with added spatial refinement. The dynamic content is enhanced spatially while the static region is otherwise substituted by the updated static structure so as to favor the long-range spatio-temporal enhancement. The proposed method both performs long-range temporal consistency on the static region and keeps necessary depth variations in the dynamic content. Thus, it can produce flicker-free and spatially optimized depth videos with reduced motion blur and depth distortion. Our experimental results reveal that the proposed method is effective in both static and dynamic indoor scenes and is compatible with depth videos captured by Kinect and time-of-flight camera. We also demonstrate that excellent performance can be achieved by the proposed method in comparison with the existing spatio-temporal approaches. In addition, our enhanced depth videos and static structures can act as effective cues to improve various applications, including depth-aided background subtraction and novel view synthesis, showing satisfactory results with few visual artifacts.
Shot classification for hard transition

Long transition analysis is a post-shot boundary detection technique used to process special effects once they are detected. Shot classification is one of the stages in such a process, used to classify pixels within the hard transitions, such as wipe, mosaic and push, once detected. In this paper, we present the fundamental algorithm and improvement for the technique. Such classification can achieve an accuracy of over 95% for low disturbance sequences, and on average, over 90% for most cases.
Toward a Blind Quality Metric for Temporally Distorted Streaming Video

With the rapid progress of mobile Internet, the streaming video service has boom over wireless networks in recent years. A smooth playback experience is crucial for the popularization of these services. However, limited by fluctuating bandwidth and various network impairments, the streaming video inevitably suffers kinds of stalling events, which significantly distorts its temporal structures and results in annoying jerky playback. In this paper, we propose an efficient quality metric to blindly evaluate the user experience for stalled streaming video without using its original sequence. Instead of requiring buffer or manifest information like existing methods, we only access to the decoded video and extract two complementary image features, i.e., global intensity and local texture, to estimate the stall number and duration. Then, by means of a straightforward and easy-to-use linear combination model, we can map the normalized stall number and duration information to a quantitative quality score. Experimental results on the publicly available LIVE-Avvasi mobile video database show that our predicted video quality is highly consistent with the user experience and outperforms many existing quality-of-experience models.
Depth enhancement using RGB-D guided filtering

Depth maps from low-cost RGB-D system are generally noisy and not accurate enough. Holes often exist in the depth maps. Bilateral filter is commonly utilized to perform depth enhancement. However, it requires high computational time. Its texture transferring property also makes those boundaries between textured and homogeneous regions in the filtered depth map far from satisfactory. In this paper, we present a method to filter raw depth maps using a RGB-D guided filtering in a two-stage framework. Our method not only has a faster computational time than bilateral filter but also avoids the problem of over-texture transfer. We also use RGB-D frames to fill holes in the depth maps. This can effectively prevents depth bleeding artifacts.
Arbitrarily shaped object coding based on H.264/AVC

Recent video coding research gives birth to the latest video coding standard H.264/AVC whose compression performance significantly exceeds previous standards with more than 50%. But as compared with the MPEG-4, the capability of coding arbitrarily shaped objects is absent from the latest standard. In this paper, we propose a new arbitrarily shaped object coding codec including encoder and decoder based on the latest H.264/AVC, and adopts improved binary alpha coding with a novel motion estimation to facilitate the binary alpha blocks prediction and in the texture coding, a new arbitrarily shaped integer transform derivative from 4×4 ICT in H.264 to code texture and associated coding techniques. Experimental results prove the coding efficiency and flexibility of our proposal and the potential applications are demonstrated.
Face segmentation in head-and-shoulder video sequences based on facial saliency map

In this paper, a novel face segmentation algorithm is proposed based on facial saliency map (FSM) for head-and-shoulder type video application. This method consists of three stages. The first stage is to generate the saliency map of input video image by our proposed facial attention model. In the second stage, a geometric model and a eye-map built from chrominance components are employed to localize face region according to the saliency map. The third stage involves the adaptive boundary correction and the final face contour extraction. Experimental evaluation on test sequences shows that the proposed method is capable of segmenting and tracking the face area quite effectively
Foreground/background bit allocation for region-of-interest coding

Two bit allocation strategies, namely, maximum bit transfer (MET) and joint bit assignment (JBA) are proposed for region-of-interest coding. The MBT strategy uses a pair of quantizers to facilitate maximum bit transfer from the background to foreground image region. It assigns the highest quantization parameter to the background quantizer, and then determines the finest value of foreground quantizer that can be used without increasing the bit rate. In this approach, the background is always encoded with the coarsest quantization level, but this is not always desirable. Therefore, the JBA strategy can be used instead. It has two modes: user defined and automatic. In the user defined mode, the user can adjust bit consumption using a scale that ranges from "not coding foreground" to "coding foreground only". In the automatic mode, bit allocation is based on the characteristics of the image regions, which include size, motion and priority. The coding results showed that improved image quality was achieved at the same bit rate.
How does the shape descriptor measure the perceptual quality of the retargeting image?

Perceptual quality evaluation of the retargeting image plays an important role in benchmarking different retargeting methods, as well as guiding or optimizing the retargeting process. The distortions introduced during the retargeting process are mainly categorized into shape distortion and content information loss . The shape distortion measurement is critical to the evaluation of retargeting image perceptual quality. In this paper, the performances of different shape descriptors, such as PHOW [2], GIST , MPEG-7 descriptors , EMD [5], for evaluating the perceptual quality of the retargeting image are examined based on the public image retargeting subjective quality database [6]. Experimental results demonstrated that most of the shape descriptors can hardly capture the characteristics representing the quality of the retargeting image, but the global shape descriptor GIST  presents significant performance gains. Moreover, by incorporating with the measurements from the perspective of content information loss, a better performance is further obtained.
Improved single-video-object rate control for MPEG-4

Rate control plays a central role in constant bit-rate video coding applications using MPEG-4. This paper considers single-video-object rate control for MPEG-4 and presents a new rate-control algorithm based on the quadratic rate-distortion (R-D) model. The major innovations are a novel constraint for the least-mean-square estimation of the model parameters of the R-D function, a new measure for the encoding complexity, novel quantizer control and an efficient frame skipping strategy. Simulation results show that the MPEG-4 coder, using the proposed algorithm, can achieve a higher PSNR than a coder using the conventional rate-control algorithm.
Temporal depth video enhancement based on intrinsic static structure

Depth video enhancement is an essential preprocessing step for various 3D applications. Despite extensive studies of spatial enhancement, effective temporal enhancement that both strengthens temporal consistency and keeps correct depth variation needs further research. In this paper, we propose a novel method to enhance the depth video by blending raw depth frame with the estimated intrinsic static structure, which defines static structure of captured scene and is estimated iteratively by a probabilistic generative model with sequentially incoming depth frames. Our experimental results show that the proposed method is effective both in static and dynamic scene and is compatible with various kinds of depth videos. We will demonstrate that superior performance can be achieved in comparison with existing temporal enhancement approaches.
Unequal error protection of images over Rayleigh fading channels

This paper introduces a method of transmitting error resilient SPIHT coded images over highly error prone Rayleigh fading channels. Firstly, the source significance of the SPIHT coded output is obtained. Based upon the significance of the bits, the channel coding is varied accordingly. Channel coding consists of a mixture of rate compatible punctured convolutional (RCPC) codes and interleaving to combat the burst errors produced by the fading channel. An additional error concealment technique is also introduced into the SPIHT decoder to improve its results in cases where errors cause the corruption of the average luminance level during decoding. Comparison with using RS block codes at a total transmission rate of 1.0 bpp (bits/pixel) is carried out over fading channels with very high error rates to show the superiority of this method over methods using fixed rate burst error codes.