nils-herrmann/clean_scirepeval huggingface dataset:
The effect of digital image stabilization on coding performance [video coding]

A digital image stabilization system compensates the image movement caused by hand jiggles and thereby improves the perceptual quality of the image sequence captured by a hand-held digital video camera. In this paper, we apply two recently developed video coding standards, MPEG-4 and H.264, to investigate the effect of image stabilization on video coding. The coding performance of these algorithms on the stabilized image sequence is compared to that on the original image sequence. The results show that image stabilization greatly improves the viewing experience and results in a decrease of bit counts for motion vectors, but the difference in the overall coding performance is minor. Details of the video coding experiments and the image stabilization system are described.
Dual-camera HDR synthesis guided by long-exposure image

Dual-camera electronic devices have the potential to deliver additional functionalities, such as depth acquisition and high dynamic range (HDR) imaging, more easily than those with a single camera. In this paper, we focus on the generation of a high dynamic range image for electronic devices equipped with a pair of parallel cameras that have different exposure time. Specifically, we propose a method that uses a long-exposure image to guide the enhancement of a short-exposure image. Unlike previous methods that are sensitive to stereo mismatch, which often occurs in the occluded regions, our method prevents the effect of inevitable disparity estimation errors on image synthesis and thereby enhances the robustness of HDR imaging. Experimental results show that our method generates higher quality HDR images without artifacts in the occluded regions.
Learning-Based Prediction of Visual Attention for Video Signals

Visual attention, which is an important characteristic of human visual system, is a useful clue for image processing and compression applications in the real world. This paper proposes a computational scheme that adopts both low-level and high-level features to predict visual attention from video signal by machine learning. The adoption of low-level features (color, orientation, and motion) is based on the study of visual cells, and the adoption of the human face as a high-level feature is based on the study of media communications. We show that such a scheme is more robust than those using purely single low- or high-level features. Unlike conventional techniques, our scheme is able to learn the relationship between features and visual attention to avoid perceptual mismatch between the estimated salience and the actual human fixation. We also show that selecting the representative training samples according to the fixation distribution improves the efficacy of regressive training. Experimental results are shown to demonstrate the advantages of the proposed scheme.
A Three-Stage Approach to Shadow Field Estimation From Partial Boundary Information

In this paper, we present a system for estimating the shadow field from a single natural image. Unlike previous works that require extensive user assistance, our system only needs the user to roughly specify the shadow boundary with a broad brush. As the user finishes drawing a stroke, the system starts to estimate the shadow field around the stroke and generates pretty accurate result even if the underlying surface is highly textured. We also propose an optimization scheme to propagate the estimated shadow field to the entire image, achieving a further reduction of user effort required for the system. The shadow field estimated by our system can be used to seamlessly remove the shadow from the image. It is also useful for many shadow editing tasks such as pasting an object's shadow from one image to another. Experimental results on a variety of photos are provided to show the effectiveness of the proposed system.
Acceleration of rate-distortion optimized quantization for H.264/AVC

Rate-distortion optimized quantization improves the coding performance of video compression. However, the search process involved in most existing methods is computationally expensive. In this paper, we present a novel method for accelerating the rate-distortion optimized quantization process for an H.264/AVC video encoder. The acceleration is achieved by using a rate model of entropy coding to directly solve the rate-distortion optimization problem. Compared with the H.264/AVC reference encoder, our method achieves an average 5.6% bitrate reduction for IBBP GOP coding structure, with less than 0.1% increase of total encoding time.
Radiometric compensation for ubiquitous projection

Integrating a pico-projector with a multimedia device allows a nearby wall, desk, or human body to be conveniently used as the projection surface. Ubiquitous projection is no longer a fiction, and the miniature of projector brings about a new form of social interaction and augmented reality applications. This paper gives an overview of the key advances in radiometric compensation for ubiquitous projection and discusses the technical issues associated with the physical limitations of projectors and cameras in spatial resolution and dynamic range. Existing techniques that can be applied to address these issues are reviewed, and open issues for future research are discussed.
A visibility model for quality assessment of dimmed images

Dimming the backlight of a liquid crystal display (LCD) reduces the dynamic range and intensity of a displayed image and hence has a profound impact on the perceived image quality. To reflect the actual degradation of image quality perceived by human eye, we take visibility in addition to contrast into account and develop a visibility model for image quality assessment. The proposed visibility model also considers factors related to the display and the ambient light to accommodate the viewing condition. Experimental results are shown to demonstrate the performance of the proposed model.
Automatic transcription of piano music by sparse representation of magnitude spectra

Assuming that the waveforms of piano notes are pre-stored and that the magnitude spectrum of a piano signal segment can be represented as a linear combination of the magnitude spectra of the pre-stored piano waveforms, we formulate the automatic transcription of polyphonic piano music as a sparse representation problem. First, the note candidates of the piano signal segment are found by using heuristic rules. Then, the sparse representation problem is solved by l1-regularized minimization, followed by temporal smoothing the frame-level results based on hidden Markov models. Evaluation against three state-of-the-art systems using ten classical music recordings of a real piano is performed to show the performance improvement of the proposed system.
A practical solution to transform-domain rounding

In many multimedia applications, manipulation of video signals is preferably performed in the transform domain. Most of such operations are derived from their counterparts in the pixel domain. However, the rounding operation can not be obtained this way because it is a nonlinear operation. Without a proper treatment, it may become a major source of errors for multimedia systems, causing a mismatch between the encoder and the decoder. This problem seems to have been largely ignored. In this paper, we propose a practical solution to this problem by a thorough examination of the mathematical property of the rounding error in the context of transform-domain video transcoding and demonstrate its performance.
Directionally weighted color interpolation for digital cameras

Demosaicing is a color interpolation process that converts a raw image generated by a color filter array to a full color image by estimating the missing color components of each pixel from its neighbors. Most demosaicing approaches often introduce false colors or blurred edges to areas of the image where there are dense edges. We present a directionally weighted color interpolation method to address this problem. We consider finer edge orientations to improve the accuracy of edge indicators and exploit the inter-channel correlation of the R, G, and B color planes in the interpolation process to reduce undesirable artifacts and hence improve the perceptual image quality. Experimental results are shown to demonstrate the strengths of the proposed method.
Smooth Playout Control for Video Streaming over Error-Prone Channels

The quality of media streaming over best-effort networks suffers from network delays and packet losses. The latter is more profound for wireless video. To enhance the QoS of streaming services, adaptive media playout (AMP) has been developed to adjust the playout interval. With AMP, the risk of delay and buffer underflow is reduced. However, the smoothness of playback is not guaranteed. In this paper, we propose a novel AMP control that enables smooth playout and meanwhile maintains reliable visual quality. Our AMP control adjusts the playout interval based on an estimation of channel quality, so it is more adaptive than conventional AMP controls that are based on buffer fullness. Experimental results are provided to justify our approach. Even at 20% packet loss rate, the proposed AMP control is still able to provide smooth and reliable playback
A screw motion approach to uniqueness analysis of head-eye geometry

The screw motion theory is used to solve a class of pose determination problems that can be characterized by a homogeneous transform equation of the form AX=XB, where A and B are known motions and X is an unknown coordinate transformation. Unlike existing methods, this method gives rise to a sound geometric interpretation that takes both rotation and translation into consideration. The author derives a screw congruence theorem and shows that the problem is to find a rigid transformation which will bring one group of lines to overlap another. He also provides a complete analysis of the conditions under which the solution can be uniquely determined.<<ETX>>
Error-resilient coding in JPEG-2000 and MPEG-4

The rapid growth of mobile communications and the widespread access to information via the Internet have resulted in a strong demand for robust transmission of compressed image and video data for various multimedia applications and services. The challenge of robust transmission is to protect the compressed image/video data against hostile channel conditions while bringing little impact on bandwidth efficiency. This paper addresses this critical problem and provides an overview of the error-resilient approaches that have been evaluated and inserted into the emerging JPEG-2000 wavelet-based image coding standard. We also review the state-of-the-art techniques adopted in the MPEG-4 standard for robust transmission of video and still texture data. These techniques include resynchronization strategies, data partitioning, reversible VLCs, and header extension codes. The performance of these approaches under various channel conditions is evaluated.
Multipitch Estimation of Piano Music by Exemplar-Based Sparse Representation

Pitch, together with other midlevel music features such as rhythm and timbre, holds the promise of bridging the semantic gap between low-level features and high-level semantics for music understanding. This paper investigates the pitch estimation of a piano music signal by exemplar-based sparse representation. A note exemplar is a segment of a piano note, stored in the dictionary. We first describe how to represent a segment of the piano music signal as a linear combination of a small number of note exemplars from a large note exemplar dictionary and then show how the sparse representation problem can be solved by -regularized minimization. The proposed approach incorporates tuning factor estimation, note candidate selection, and hidden-Markov-model-based smoothing into the estimation process to improve accuracy. Unlike previous approaches, the proposed approach does not require retraining for a new piano. Instead, only a dozen notes of the new piano are needed. This feature is computationally attractive and avoids intense manual labeling. The system performance is evaluated using 70 classical music recordings of two real pianos under different recording conditions. The results show that the proposed system outperforms four state-of-the-art systems.
Stochastic Color Interpolation for Digital Cameras

This paper presents a stochastic estimation approach to adaptive interpolation of color filter array. It models an image as a 2D locally stationary Gaussian process and achieves robustness against aliasing by employing an edge-sensitive weighting policy based on the stochastic characteristics of uniformly oriented edge indicators. Experimental results show that the algorithm can effectively eliminate the occurrence of perceptible artifact. Performance comparison in terms of peak signal-to-noise ratio and mean square error is provided to demonstrate the superiority of the proposed algorithm.
Robust measure of image focus in the wavelet domain

In this paper we propose a robust technique for image focus measure based on discrete wavelet transform (DWT). We suggest the absolute sum of the second-level detailed image of DWT as focus measure. In the experiment the absolute-sum based measure shows equally well performance as the energy-based measure while enhancing computation efficiency. Comparison to other benchmarking measures is also discussed. Our measure exhibits more robustness to Gaussian noise. Moreover, we show that setting a threshold on the wavelet coefficients can dramatically increase the discriminative power of the focus measure in noisy condition.
Frame-Layer Constant-Quality Rate Control of Regions of Interest for Multiple Encoders With Single Video Source

In this work, we develop a constant-quality rate control algorithm for a surveillance system which consists of one ldquobase encoderrdquo that encodes a down-sampled full-view version of the input video sequence, and one ldquoregion of interest (ROI) encoderrdquo that encodes the region of interest of the input video at the original resolution. Exploiting the inter-relationship between these two independent encoders, the algorithm allocates the bits for the ROI encoder according to the distortion obtained from the corresponding region in the base encoder. Simulation results show that the proposed algorithm can achieve significant reduction in the image quality variation. Compared to the rate control algorithm in JM 8.4, the overall quality is improved and the bit rate is saved.
Enhancement of Backlight-Scaled Images

Switching the liquid crystal display (LCD) backlight of a portable multimedia device to a low power level saves energy but results in poor image quality especially for the low-luminance image areas. In this paper, we propose an image enhancement algorithm that overcomes such effects of dim LCD backlight by taking the human visual property into consideration. It boosts the luminance of image areas below the perceptual threshold while preserving the contrast of the other image areas. We apply the just noticeable difference theory and decompose an image into an HVS response layer and a background luminance layer. The boosting and compression processes, which enhance the visibility of the low-luminance image areas, are carried out in the background luminance layer to avoid luminance gradient reversal and over-compensation. The contrast of the processed image is further enhanced by exploiting the Craik-O'Brein-Cornsweet visual illusion. Experimental results are provided to show the performance of the proposed algorithm.
Online Reranking via Ordinal Informative Concepts for Context Fusion in Concept Detection and Video Search

To exploit the co-occurrence patterns of semantic concepts while keeping the simplicity of context fusion, a novel reranking approach is proposed in this paper. The approach, called ordinal reranking, adjusts the ranking of an initial search (or detection) list based on the co-occurrence patterns obtained by using ranking functions such as ListNet. Ranking functions are by nature more effective than classification-based reranking methods in mining ordinal relationships. In addition, the ordinal reranking is free of the ad hoc thresholding for noisy binary labels and requires no extra offline learning or training data. To select informative concepts for reranking, we also propose a new concept selection measurement, wc-tf-idf, which considers the underlying ordinal information of ranking lists and is thus more effective than the feature selection algorithms for classification. Being largely unsupervised, the reranking approach to context fusion can be applied equally well to concept detection and video search. While being extremely efficient, ordinal reranking outperforms existing methods by up to 40% in mean average precision (MAP) for the baseline text-based search and 12% for the baseline concept detection over TRECVID 2005 video search and concept detection benchmark.
Empirical reliability analysis of disparity-based autofocus

Conventional autofocus methods based on contrast detection are often unable to reliably decide the direction of initial lens movement. In this paper, we show that even using the disparity data obtained from blurry stereo images can effectively solve the problem. This approach is developed for stereo cameras with adjustable focal distance. Such stereo cameras provide sharp images over a wide range of object distance. The disparity-based autofocus approach allows the lens to instantly come to a position near the in-focus position or inside the rising zone of the focus profile. We investigate the impact of disparity variation on the first lens movement driven by disparity-based autofocus and suggest how it can be integrated with contrast detection autofocus.
Enhancement of LCD Images Illuminated with Dim Backlight

Dimming the LCD backlight saves the battery power of a portable multimedia device but causes image quality degradation. In this paper, we propose an algorithm to overcome the effect of dim backlight on image quality. This algorithm enhances the backlight-scaled image by exploiting the human visual property. It performs a two-layer image decomposition based on the just noticeable difference theory and enhances the backlight-scaled image by luminance boosting and compression. Experimental results are provided to demonstrate the performance of the proposed algorithm.
Music Emotion Classification: A Regression Approach

Typical music emotion classification (MEC) approaches categorize emotions and apply pattern recognition methods to train a classifier. However, categorized emotions are too ambiguous for efficient music retrieval. In this paper, we model emotions as continuous variables composed of arousal and valence values (AV values), and formulate MEC as a regression problem. The multiple linear regression, support vector regression, and AdaBoost.RT are adopted to evaluate the prediction accuracy. Since the regression approach is inherently continuous, it is free of the ambiguity problem existing in its categorical counterparts.
Fast multi-frame motion estimation for H.264 and its applications to complexity-aware streaming

JVT/H.264 achieves higher compression efficiency than previous video coding standards such as MPEG4 and H.263. However, this comes at the cost of increased complexity due to the use of variable block-size, multiple reference frame motion compensation, and other advanced coding algorithms. In this paper, we present a fast motion estimation algorithm. This algorithm is based on an adaptive search strategy and a flexible multi-frame search scheme. Compared to the H.264 reference software JM 8.5, this algorithm achieves, on average, a 522% reduction of encoding time, with negligible PSNR drop and bit-rate increase. Performance comparison results are described, and the architecture of a complexity-aware streaming server that applies the algorithm to provide differentiated services is discussed.
Research on light field camera and music emotion recognition

In this talk, I will give a presentation of some recent progress we have made at the Multimedia Processing and Communications Lab of National Taiwan University. Specifically, I will describe the multimedia signal processing techniques we have developed for applications to digital video camera and emotion-based multimedia presentation. The demos to be shown in this talk include 1) programmable aperture photography: multiplexed light field acquisition, 2) music emotion recognition for multimedia retrieval and presentation, 3) integration of video stabilizer with digital video codec, 4) auto focusing, 5) auto white balance, 6) color transfer, and 7) Wii-like 3D pointer.
Efficient Quantization Based on Rate–Distortion Optimization for Video Coding

Most rate-distortion (R-D) optimized quantization methods of video coding involve an exhaustive search process to determine the optimal quantized transform coefficients of a coding block and are computationally more expensive than the conventional quantization. In this paper, we present a novel analytical method that directly solves the rate-distortion optimization problem in a closed form by employing a rate model for entropy coding. It has the appealing property of low complexity and is easy to implement. The results show that the proposed method is $4\times $ to $40\times $ faster than the previous methods and 3%-5% more efficient in bitrate than the H.264/AVC reference encoder, which uses the conventional quantization, for video coded with the IBBP structure of group of pictures (GOP) in the normal peak signal-to-noise ratio (PSNR) range (30-40 dB).
Image Enhancement for Backlight-Scaled TFT-LCD Displays

One common way to extend the battery life of a portable device is to reduce the LCD backlight intensity. In contrast to previous approaches that minimize the power consumption by adjusting the backlight intensity frame by frame to reach a specified image quality, the proposed method optimizes the image quality for a given backlight intensity. Image is enhanced by performing brightness compensation and local contrast enhancement. For brightness compensation, global image statistics and backlight level are considered to maintain the overall brightness of the image. For contrast enhancement, the local contrast property of human visual system (HVS) is exploited to enhance the local image details. In addition, a brightness prediction scheme is proposed to speed up the algorithm for display of video sequences. Experimental results are presented to show the performance of the algorithm.
Radiometric compensation for procam system based on anchoring theory

For a procam system consisting of a projector and a camera, the chroma distortion of an image projected on a non-white surface can be compensated by increasing the intensity of the complementary color of the projection surface. However, this inevitably trades image brightness for chroma correctness because the required intensity often outreaches the projector's dynamic range. In this paper, we propose a solution for the compensation problem using the anchoring theory of lightness perception. A notable feature of our technique is that it takes the chromatic adaptation and viewing condition into account, which leads to a better characterization of the image quality perceived by human eyes. As a result, the compensated image appears much closer to the original image. A computational framework for the tradeoff between brightness and chroma distortion is derived, and the performance of the proposed algorithm is verified by subjective experiments.