nils-herrmann/clean_scirepeval huggingface dataset:
Image Quality Assessment by Separately Evaluating Detail Losses and Additive Impairments

In the research field of image processing, mean squared error (MSE) and peak signal-to-noise ratio (PSNR) are extensively adopted as the objective visual quality metrics, mainly because of their simplicity for calculation and optimization. However, it has been well recognized that these pixel-based difference measures correlate poorly with the human perception. Inspired by existing works , in this paper we propose a novel algorithm which separately evaluates detail losses and additive impairments for image quality assessment. The detail loss refers to the loss of useful visual information which affects the content visibility, and the additive impairment represents the redundant visual information whose appearance in the test image will distract viewer's attention from the useful contents causing unpleasant viewing experience. To separate detail losses and additive impairments, a wavelet-domain decoupling algorithm is developed which can be used for a host of distortion types. Two HVS characteristics, i.e., the contrast sensitivity function and the contrast masking effect, are taken into account to approximate the HVS sensitivities. We propose two simple quality measures to correlate detail losses and additive impairments with visual quality, respectively. Based on the findings in that observers judge low-quality images in terms of the ability to interpret the content, the outputs of the two quality measures are adaptively combined to yield the overall quality index. By conducting experiments based on five subjectively-rated image databases, we demonstrate that the proposed metric has a better or similar performance in matching subjective ratings when compared with the state-of-the-art image quality metrics.
Fast HEVC Inter CU Decision Based on Latent SAD Estimation

The emerging high efficiency video coding (HEVC) standard has improved compression performance significantly in comparison with H.264/AVC. However, more intensive computational complexity has been introduced by adopting a number of new coding tools. In this paper, a fast inter CU decision is proposed based on the latent sum of absolute differences (SAD) estimation. Firstly, a two-layer motion estimation (ME) method is designed to take advantage of the latent SAD cost. The new ME method can obtain the SAD costs for both the upper CU and its sub-CUs. Secondly, a concept of motion compensation rate- distortion (R-D) cost is defined, and an exponential model is proposed to express the relationship between the motion compensation R-D cost and the SAD cost. Then, a fast CU decision approach is designed based on the exponential model. The fast CU decision is implemented by comparing a derived threshold with the SAD cost difference between the upper and sub SAD costs. Experimental results show that the proposed algorithm achieves an average of 52% and 58.4% reductions of the coding time at the cost of 1.61% and 2% bit-rate increases under the low delay and random access conditions, respectively.
Optical flow based DC surface compensation for artifacts reduction

The block-based prediction, transform and quantization reduce the redundancy in video data efficiently. However, the blocking artifacts are introduced because of quantization error and prediction scheme. Even if the de-blocking filter is applied like that designed in H.264 and AVS, the blocking artifacts are still visible especially in videos with smooth area coded at low bit-rate. In this paper, an optical flow based technique is proposed to further reduce the blocking artifacts by compensating the DC surfaces of the decoded images using joint optimization of optical flow and edge difference. The proposed algorithm automatically classifies the macroblocks in the decoded images according to their texture feature and endures the smooth area with smooth brightness variation by minimizing the joint function of optical flow magnitude and edge smoothness. It produces a more pleasing image without over-filtering, which can be further applied as an in loop filter for video coding.
A Universal Approach to Developing Fast Algorithm for Simplified Order-16 ICT

Simplified order-16 integer cosine transform (ICT) has been proved to be an efficient coding tool especially for high-definition (HD) video coding and is much simpler than ICT and discrete cosine transform (DCT). To further reduce the computational complexity, a universal approach to developing fast algorithm mainly for, but not restricted to, simplified order-16 ICT is proposed in this paper. The fast algorithm developed by the proposed approach involves additions and shiftings only and can save about 90% of the computational time compared with matrix multiplication.
Learning Efficient Binary Codes From High-Level Feature Representations for Multilabel Image Retrieval

Due to the efficiency and effectiveness of hashing technologies, they have become increasingly popular in large-scale image semantic retrieval. However, existing hash methods suppose that the data distributions satisfy the manifold assumption that semantic similar samples tend to lie on a low-dimensional manifold, which will be weakened due to the large intraclass variation. Moreover, these methods learn hash functions by relaxing the discrete constraints on binary codes to real value, which will introduce large quantization loss. To tackle the above problems, this paper proposes a novel unsupervised hashing algorithm to learn efficient binary codes from high-level feature representations. More specifically, we explore nonnegative matrix factorization for learning high-level visual features. Ultimately, binary codes are generated by performing binary quantization in the high-level feature representations space, which will map images with similar (visually or semantically) high-level feature representations to similar binary codes. To solve the corresponding optimization problem involving nonnegative and discrete variables, we develop an efficient optimization algorithm to reduce quantization loss with guaranteed convergence in theory. Extensive experiments show that our proposed method outperforms the state-of-the-art hashing methods on several multilabel real-world image datasets.
Automatic face location detection and tracking for model-based video coding

One major problem in the model-based coding of facial images is the automatic fitting of the initial wireframe model (WFM) to the facial image. In this paper, the solutions for automatic fitting and adaptation of the initial WFM to the facial image are given. The adaptation is performed on a global scale through repositioning and resizing of the WFM. For this purpose, main facial feature contours are extracted using deformable templates and active contour models. The WFM is then altered by affine transforming some of its major control nodes to fit the facial feature points on the facial image, and the rest of the nodes are interpolated relative to the movement of these nodes. The automatically adjusted WFM is found to be accurate when compared to the manually adjusted one.
Spatio-Temporal Just Noticeable Distortion Profile for Grey Scale Image/Video in DCT Domain

In image and video processing field, an effective compression algorithm should remove not only the statistical redundancy information but also the perceptually insignificant component from the pictures. Just-noticeable distortion (JND) profile is an efficient model to represent those perceptual redundancies. Human eyes are usually not sensitive to the distortion below the JND threshold. In this paper, a DCT based JND model for monochrome pictures is proposed. This model incorporates the spatial contrast sensitivity function (CSF), the luminance adaptation effect, and the contrast masking effect based on block classification. Gamma correction is also considered to compensate the original luminance adaptation effect which gives more accurate results. In order to extend the proposed JND profile to video images, the temporal modulation factor is included by incorporating the temporal CSF and the eye movement compensation. Moreover, a psychophysical experiment was designed to parameterize the proposed model. Experimental results show that the proposed model is consistent with the human visual system (HVS). Compared with the other JND profiles, the proposed model can tolerate more distortion and has much better perceptual quality. This model can be easily applied in many related areas, such as compression, watermarking, error protection, perceptual distortion metric, and so on.
An Efficient Intra Mode Selection Algorithm For H.264 Based On Fast Edge Classification

The H.264/AVC is the newest video coding standard recommended by ITU-T and MPEG. Compared with all existing video coding standards, H.264 can achieve superior performance by using many advanced techniques. Intra mode selection is an important feature in H.264 standard and can reduce the spatial redundancy in intra frame significantly. An efficient rate distortion optimization (RDO) technique is employed in H.264 to choose the best mode for each MB, but the computational cost increases drastically. In this paper, a fast intra mode selection algorithm is introduced. By using a fast edge detection method which is based on non-normalized Haar transform (NHT), edge for each sub-block can be extracted. Based on the local edge information, only few intra modes are chosen as mode candidates. A fast RDO algorithm is also proposed in this paper. By combing these two methods, computational load is reduced remarkably. Experimental results show that this fast intra mode selection scheme can lessen about 80% encoding time with little loss of bit-rate and visual quality.
A disocclusion filling method using multiple sprites with depth for virtual view synthesis

Depth image based rendering (DIBR) is an important technique to generate virtual view images with limited 3-D data. However, disocclusion is a critical problem that the regions occluded by foreground objects become visible in virtual views, which is difficult to be visual-plausibly inferred. In this paper, we propose a novel temporally consistent filling method using multiple sprites with depth (MSD) to fill the disocclusions faithfully. MSD stores the background as well as intermediate foreground objects of past frames in multiple sprites, so that amounts of pixels in disocclusions can be recovered from MSD. Moreover, we also introduce a method to recognize the source space for exemplar based inpainting to fill the remaining disocclusions. The experimental results show the proposed method achieves objective and subjective improvement compared with the state-of-the-art methods. The synthesized sequences by the proposed method have higher spatio-temporal consistency.
Automatic segmentation of moving objects for video object plane generation

The new video coding standard MPEG-4 is enabling content-based functionalities. It takes advantage of a prior decomposition of sequences into video object planes (VOPs) so that each VOP represents one moving object. A comprehensive review summarizes some of the most important motion segmentation and VOP generation techniques that have been proposed. Then, a new automatic video sequence segmentation algorithm that extracts moving objects is presented. The core of this algorithm is an object tracker that matches a two-dimensional (2-D) binary model of the object against subsequent frames using the Hausdorff distance. The best match found indicates the translation the object has undergone, and the model is updated every frame to accommodate for rotation and changes in shape. The initial model is derived automatically, and a new model update method based on the concept of moving connected components allows for comparatively large changes in shape. The proposed algorithm is improved by a filtering technique that removes stationary background. Finally, the binary model sequence guides the extraction objects of the VOPs from the sequence. Experimental results demonstrate the performance of our algorithm.
MPEG-4 based stereoscopic video sequences encoder

In this paper, we propose an object-based MPEG-4 compatible stereoscopic video sequence encoder. We aim at efficient stereoscopic video compression for videoconferencing and 3D telepresence systems. The stereoscopic video sequence includes one main view and one auxiliary view. The main view is encoded using an MPEG-4 encoder and the auxiliary view is encoded by joint motion and disparity compensation. After the input sequences are balanced to compensate for lighting conditions and camera differences, the joint disparity and motion regularization is performed on a VOP basis. The output of the encoder contains two bitstreams, a main bitstream, which can be decoded by a standard MPEG-4 decoder, and an auxiliary bitstream. Simulation results show that the joint estimation and regularization of disparity and motion fields provide more accurate vector fields and efficient compression for the auxiliary stream. The proposed system achieves high image quality at lower bitrate than existing stereoscopic video encoders.
Temporal inconsistency measure for video quality assessment

Visual quality assessment plays a crucial role in many vision-related signal processing applications. In the literature, more efforts have been spent on spatial visual quality measure. Although a large number of video quality metrics have been proposed, the methods to use temporal information for quality assessment are less diversified. In this paper, we propose a novel method to measure the temporal impairments. The proposed method can be incorporated into any image quality metric to extend it into a video quality metric. Moreover, it is easy to apply the proposed method in video coding system to incorporate with MSE for rate-distortion optimization.
Fuzzy quaternion approach to object recognition incorporating Zernike moment invariants

A novel approach to 3-D object recognition based on fuzzy subset theory is described. This method uses Zernike moment invariants of the silhouette of the unknown object to form a set of fuzzy-weighted quantities called fuzzy quaternions. These are matched against those of known objects at predetermined viewpoints. The determination of the Zernike moment invariants can be faster if the equivalent contour integrals are calculated instead. By employing a novel rho -correction scheme, errors due to the digitization are reduced. To speed up the recognition process, a modified Nelder-Mead simplex method is used. Preliminary results demonstrate the potential of the fuzzy quaternion as a viable basis for discrimination. It is concluded that the primary merits of this approach are the ease of model formation, the simplicity of the recognition scheme, and the speed of object recognition. Its disadvantages include the inability to recognize occluded objects and a poor object recognition rate for high perspective distortion of objects.<<ETX>>
Unsupervised extraction of visual attention objects in color images

This paper proposes a generic model for unsupervised extraction of viewer's attention objects from color images. Without the full semantic understanding of image content, the model formulates the attention objects as a Markov random field (MRF) by integrating computational visual attention mechanisms with attention object growing techniques. Furthermore, we describe the MRF by a Gibbs random field with an energy function. The minimization of the energy function provides a practical way to obtain attention objects. Experimental results on 880 real images and user subjective evaluations by 16 subjects demonstrate the effectiveness of the proposed approach.
Camshift based real-time multiple faces match tracking

A novel CAMShift based multiple faces tracking algorithm with weighted greedy tracking and match pursuing is proposed in this paper. The greedy tracking overcomes the disadvantage of CAMShift, which tends to lose target or reiterate tracking for overlapped targets. The match pursuing identifies the tracked targets by using additional information. In addition, a whole frame PDI is introduced to monitor the change of object number in the scene and re-initialize the tracked objects if the change happened. Experimental results indicate the proposed method can track multiple faces robustly in real time.
Blind proposal quality assessment via deep objectness representation and local linear regression

The quality of object proposal plays an important role in boosting the performance of many computer vision tasks, such as, object detection and recognition. Due to the absence of manually annotated bounding-box in practice, the quality metric towards blind assessment of object proposal is highly desirable for singling out the optimal proposals. In this paper, we propose a blind proposal quality assessment algorithm based on the Deep Objectness Representation and Local Linear Regression (DORLLR). Inspired by the hierarchy model of the human vision system, a deep convolutional neural network is developed to extract the objectness-aware image feature. Then, the local linear regression method is utilized to map the image feature to a quality score, which tries to evaluate each individual test window based on its k-nearest-neighbors. Experimental results on a large-scale IoU labeled dataset verify that the proposed method significantly outperforms the state-of-the-art blind proposal evaluation metrics.
A VLC/FLC data partitioning scheme for MPEG-4

Compressed video bit stream is very sensitive to transmission errors. There exist many error-resilient techniques that can significantly improve the reconstructed video quality by reducing the spatiotemporal error propagation. In this paper, an error-resilient coding scheme that can decrease the spatial error propagation with little overhead is proposed. It disables intra AC prediction and groups appropriate fixed length coded (FLC) syntaxes in a video packet (or slice) together to form a new partition. With intra AC prediction disables, errors occurred in these FLC syntaxes do not cause spatial error propagation. We evaluate the efficiency of the error-resilient coding scheme by comparing the reconstructed PSNR of the proposed coding scheme with that of the MPEG-4 coding scheme over the AWGN channel and the Rayleigh channel.
Pre- and Post-Shift Filtering for Blocking Removing in Downsizing Transcoding

When high-quality video is delivered via a bandwidth-limited network, one solution is to reduce the spatial resolution of the bitstream before transmission. Downsizing transcoding is adopted as an efficient tool to reduce the spatial resolution of the precoded bitstream. Since the downsizing process and the coding process both apply a low-pass filtering operation on the picture, the reconstructed video may result in serious blocking artifact. In this paper, pre- and post-shift filters are considered in the transcoding process. The block boundary is shifted between the downsizing process and the coding process. This reduces the blocking effect in downsizing transcoding. Experimental results show that the proposed scheme can efficiently remove the blocking effect without introducing high computation.
Spatio-Temporal Disocclusion Filling Using Novel Sprite Cells

Depth image-based rendering is an important technique for virtual view synthesis with limited 3-D data. However, occluded areas result in disocclusions in synthesized images. Filling of the disocclusions in a plausible manner is a critical task in virtual view synthesis. In addition to spatial consistency, temporal consistency of the filled regions also affects the visual quality. In this paper, we propose a novel codebook method called sprite cell for filling the disocclusions with high spatial and temporal consistency. Each codeword consists of a color vector, depth value, frame log, and the confidence score of the corresponding pixel. In contrast with the existing methods that reuse the filling results of previous frames without considering their accuracy, the proposed method estimates the confidence scores of the filling results to prevent temporal continuation of filling errors. Moreover, we introduce a method to correct the luminance of filled disocclusions that compensates for the change of scenes. The experimental results show that the proposed method achieves both objective and subjective improvements over the state-of-the-art methods. The sequences synthesized with the proposed method have higher spatio-temporal consistency.
Unsupervized Video Segmentation With Low Depth of Field

In this paper, a novel segmentation algorithm based on matting model is proposed to extract the focused objects in low depth-of-field (DoF) video images. The proposed algorithm is fully automatic and can be used to partition the video image into focused objects and defocused background. This method consists of three stages. The first stage is to generate a saliency map of the input image by the reblurring model. In the second stage, bilateral and morphological filtering are employed to smooth and accentuate the salient regions. Then a trimap with three regions is calculated by an adaptive thresholding method. The third stage involves the proposed adaptive error control matting scheme to extract the boundaries of the focused objects accurately. Experimental evaluation on test sequences shows that the proposed method is capable of segmenting the focused region effectively and accurately.
From Logo to Object Segmentation

This paper proposes a method to segment object from the web images using logo detection. The method consists of three steps. In the first step, the logos are located from the original images by SIFT matching. Based on the logo location and the object shape model, the second step extracts the object boundary from the image. In the third step, we use the object boundary to model the object appearance, which is then used in the MRF based segmentation method to finally achieve the object segmentation. The key of our method is the object boundary extraction, which is achieved by searching a variation of the shape model that best fits the local edge of the image. Affine transform is used to consider the variations among the objects. Meanwhile, the Nelder-Mead simplex method with a simple initial rough search is used to run the boundary search. To verify the proposed method, we collect a LogoSeg dataset from the web such as Flickr and Google. The MOMI dataset is also used for the verification. The experimental results demonstrate that the proposed logo detection based segmentation method can improve the performance of the object segmentation.
Video Content Dependent Directional Transform for High Performance Video Coding

In Mode-Dependent Directional Transform (MDDT), Karhunen-CLoeve Transform (KLT) was employed to better compress the directional edges of intra prediction residues. The transform bases of MDDT were derived from the Singular Value Decomposition (SVD) of the intra prediction residues with the diversity of video characteristics. MDDT was mode dependent, but not video content dependent. It was expected to be efficient to most video sequences. However, it did not consider the difference of video content for designing transform basis. In this paper, a video content feature is firstly defined as a concatenation of coefficient magnitude, dominant gradient and spatial activity histograms of residues. Secondly, each KLT basis which is obtained from off-line training is associated with a given feature. Thirdly, a histogram-based feature matching algorithm is proposed to select the best transform basis from the provided multiple candidates for encoding a frame. The experiments show that the average Rate-Distortion (R-D) improvement of 0.65dB PSNR can be achieved by the proposed video Content Dependent Directional Transform (CDDT) compared to the state-of-the-art MDDT for inter frame coding. Compared to Rate-Distortion Optimized Transform (RDOT), CDDT also has about 3% bits saving and comparable PSNR improvement.