nils-herrmann/clean_scirepeval huggingface dataset:
AUtomatic accompaniment generation to evoke specific emotion

A music piece consists of melody and accompaniment in most genres. In this paper, we present a system to automatically generate accompaniment that evokes specific emotions for a given melody. In particular, we propose harmonic progression and onset rate as two key features for emotion-based accompaniment generation. The former refers to the progression of chords, and the latter refers to the number of music events (such as notes and drums) in a unit time. The harmonic progression and the onset rate are altered according to the specified emotion represented by the valence and arousal parameters, respectively. The performance of the system is evaluated subjectively, and the result shows a perfect positive Spearman correlation between the specified emotion and the perceived emotion.
SSIM-Based Perceptual Rate Control for Video Coding

The quality of video is ultimately judged by human eye; however, mean squared error and the like that have been used as quality metrics are poorly correlated with human perception. Although the characteristics of human visual system have been incorporated into perceptual-based rate control, most existing schemes do not take rate-distortion optimization into consideration. In this paper, we use the structural similarity index as the quality metric for rate-distortion modeling and develop an optimum bit allocation and rate control scheme for video coding. This scheme achieves up to 25% bit-rate reduction over the JM reference software of H.264. Under the rate-distortion optimization framework, the proposed scheme can be easily integrated with the perceptual-based mode decision scheme. The overall bit-rate reduction may reach as high as 32% over the JM reference software.
A robust DRM system on the DVB multimedia home platform

In a digital home, the copy rights of the high-quality multimedia content broadcasted from a DVB system need to be protected. However, there is no specification on how to enforce the usage rights of digital content in the DVB standards. As a result, even if the digital content is protected under the conditional access sub-system, end users can still copy and redistribute the digital content once it is descrambled. In this paper, we proposed a DRM system for set-top box which supports Multimedia Home Platform middleware. The rights of the protected digital content are described using the MPEG-21 Rights Expression Language and broadcasted with the digital content. In the proposed system, the rights are stored in the smart card. The proposed system is highly renewable and extensible. The service providers can integrate this system with their existing broadcast services without additional hardware cost.
Prediction of the Distribution of Perceived Music Emotions Using Discrete Samples

Typically, a machine learning model of automatic music emotion recognition is trained to learn the relationship between music features and perceived emotion values. However, simply assigning an emotion value to a clip in the training phase does not work well because the perceived emotion of a clip varies from person to person. To resolve this problem, we propose a novel approach that represents the perceived emotion of a clip as a probability distribution in the emotion plane. In addition, we develop a methodology that predicts the emotion distribution of a clip by estimating the emotion mass at discrete samples of the emotion plane. We also develop model fusion algorithms to integrate different perceptual dimensions of music listening and to enhance the modeling of emotion perception. The effectiveness of the proposed approach is validated through an extensive performance study. An average R2 statistics of 0.5439 for emotion prediction is achieved. We also show how this approach can be applied to enhance our understanding of music emotion.
Fast Multi-Frame Motion Estimation and Mode Decision for H.264 Encoders

H.264/AVC achieves higher compression efficiency than previous video coding standards. However, this performance gain comes at the cost of an increased computational complexity due, in part, to the use of variable block-size motion estimation, multiple reference frames, and rate-distortion mode decision. In this paper, we develop a combined algorithm for fast motion estimation and mode decision by exploiting the motion and cost information available from the blocks that have been processed prior to the current block. At the cost of a slight bit rate increase and PSNR drop, this method, on the average, is about 10.5 times faster than the H.264 reference software JM 8.5 in the total encoding time
Generation of Affective Accompaniment in Accordance With Emotion Flow

The emotion expressed by a music piece varies as the music unrolls in time. To create such dynamic expression, we develop an algorithm that automatically generates the accompaniment for a melody according to the emotion flow specified by a user. The emotion flow is given in the form of arousal and valence curves, each as a function of time. The affective accompaniment is composed of chord progression and accompaniment pattern. The chord progression, which controls the valence of the composed music, is generated by dynamic programming using the input melody and valence data as constraints. A mathematical model is developed to describe the temporal relationship between valence and chord progression. The accompaniment pattern, which controls the arousal of the composed music, is determined according to the quantized arousal values. The performance of the system is evaluated by subjective tests. The cross-correlation coefficient between the input arousal (valence) and the perceived arousal (valence) of the composed music is 0.85 (0.52). It rises to 0.92 for arousal and 0.88 for valence if only musician subjects in the test are considered. Overall, the proposed system is capable of generating subjectively appropriate accompaniments conforming to the user specification.
Emotion-flow guided music accompaniment generation

The emotion of a music piece varies as it unrolls in time. We develop a system that takes a melody and an expected emotion flow as input and automatically generates an accompaniment. The accompaniment is composed of chord progression and accompaniment pattern. The former is generated from melody and valence data through dynamic programming, and the latter from arousal data. A mathematical model is developed to describe the relation between valence and chord progression. The performance of the system is evaluated subjectively. The cross-correlation coefficient between the expected arousals and the perceived ones is 0.84, and the cross-correlation coefficient between the expected valences and the perceived ones is 0.52. Both coefficients exceed 0.90 for musician subjects.
Automatic highlights extraction for drama video using music emotion and human face features

The rich emotion part of a drama video is often the center of attraction to the viewer. Emotion-based highlights extraction is useful for applications such as drama video retrieval and automatic trailer generation. In this paper, we propose a system that uses music emotion and human face as features for automatic extraction of the emotion highlights of a drama video. These high-level audiovisual features are used because music invokes emotion response from the viewer and characters express emotion on their faces. To avoid the interference of speech signal and environmental noise, a novel two-stage music emotion recognition scheme is developed. We first detect the presence of incidental music in a drama video using an audio fingerprint technique, and then perform emotion recognition on the noise-free music available from the album of the incidental music. This simple but effective approach greatly improves the accuracy of music emotion recognition. Besides the conventional subjective evaluation, we propose a new metric for quantitative performance evaluation of highlights extraction. Evaluation results are provided to illustrate the performance of the system.
Robust image compression with packetization: the MPEG-4 still texture case

In this paper, we propose a bit-stream packetization approach to make the MPEG-4 still texture bit-stream robust to channel degradation. Our approach does not affect the spatial/quality scalability features of the still texture algorithm, and it has limited effect on coding efficiency. Also, this packetization approach requires minimal changes to the syntax and a small overhead. Finally, the proposed technique is shown to provide good error robustness over a wide range of error conditions.
A simulation model for continuous autofocus design

The design of continuous autofocus algorithms often involves repetitive, time-consuming field tests on real scenes. This paper presents a simulation method to expedite the design by replacing laborious field tests with simulation tests. The modeling of scene dynamics is achieved by decomposing a dynamic scene into a sequence of static scenes and representing the sharpness responses of the video camera to each static scene by a reciprocal focus profile. Three different scenes and the associated dynamic models are incorporated into the simulation framework to obtain effective clues for the refinement of continuous autofocus algorithms. Experimental results are provided to show the effectiveness of this design-by-simulation method.
Robust Video Streaming over Power Lines

The upcoming HomePlug AV standard enables power line communication (PLC) networks to transmit high quality video streams applicable for various video on demand and video relay services in digital homes. However, video streaming over power lines is sensitive to the interference caused by electrical appliances and to the degradation of quality of service (QoS) caused by the increase of transmission distance. In this paper, we evaluate the performance of power line communications for video streaming under various conditions. We also propose a new multiple-description based video coding (MDC) technique for robust video streaming over power lines. This technique divides the video into an even-frame stream and an odd-frame stream in the server. Each stream is composed of two substreams, one at full resolution and the other at quarter resolution. The four substreams are encoded independently and transmitted in an interleaving manner. If packet losses occur in one substream, the decoder can compensate for the lost video by making use of the other substreams so as to maintain the output video quality
Gaussian noise approximation for disparity-based autofocus

An analytical characterization of the accuracy of disparity-based autofocus requires an effective error model of disparity estimation. In this paper, we investigate the approximation of photon shot noise of images by a Gaussian model for disparity error analysis that takes defocus blur and image noise into account. We show that, counterintuitively, defocus blur alone does not affect the disparity estimation. However, its presence makes disparity estimation more vulnerable to image noise. In addition, we show that the Gaussian model is a fair approximation of the photon shot noise for disparity-based autofocus. The Gaussian model allows defocus blur to be mathematically separable from noise and facilitates further analysis of disparity error for disparity-based autofocus.
Compensation of spectral mismatch to enhance WRGB demosaicking

The presence of spectral mismatch between the components of a WRGB color filter array severely affects the performance of demosaicking. This paper presents a novel method that compensates for the spectral mismatch and greatly enhances the accuracy of R, G, and B interpolation. The method is tested on a two-megapixel WRGB CMOS image sensor. The results show that the proposed method brings the performance of WRGB demosaicking to an unprecedented level competitive with that of the state-of-the-art Bayer demosaicking.
Subjective Evaluation of Vector Representation of Emotion Flow for Music Retrieval

Because it simply consists of an initial point and a terminal point in a two dimensional emotion plane, vector representation of music emotion provides an intuitive and instant visualization of the dynamics of music emotion. In this paper, we investigate the performance of this representation for music information retrieval by conducting a series of subjective tests. A music retrieval system is created, and the user experience data are evaluated by seven metrics: learnability, ease of use, affordance, usefulness, joyfulness, novelty, and overall satisfaction. Compared with the point representation, the vector representation performs relatively better in affordance, novelty, and joyfulness but slightly worse in learnability and ease of use. The overall satisfaction score is 5.19 for the point representation and 5.43 for the vector representation. The results suggest that each representation has its own strengths, and the choice between the two representations depends on which metrics carry more weight in an application at hand.
Smooth control of continuous autofocus

For a video camera to continuously focus on a moving object in the autofocus process, accurate estimate of the in-focus lens position is required in order to avoid the so-called bouncing phenomenon that is resulted from back-and-forth lens movements around the in-focus lens position. In this paper, we present a novel continuous focus method that effectively improves the accuracy of in-focus lens position estimate for dynamic scenes by a Kalman filter and provides good user experience by a smooth control of the lens movements in the autofocus process. Experimental results are shown to demonstrate the performance of our method.
Efficient MB and prediction mode decisions for intra prediction of H.264 High Profile

The rate-distortion optimization framework employed in H.264 for the selection of best coding mode is computationally expensive, although it achieves remarkable coding efficiency. In this paper, an integrated intra prediction algorithm for intra-frame coding of the H.264 High Profile is proposed. It consists of two components: a variance-based method for macroblock mode decision and an improved filter-based method for prediction mode decision using contextual information. The two components can work independently or jointly to achieve higher efficiency. The integrated algorithm achieves an average saving of 51.3% (maximum 66.0%) in encoding time for intra coded frames, with negligible effect on PSNR and bit-rate.
Design and Evaluation of a P2P IPTV System for Heterogeneous Networks

NTUStreaming is an overlay P2P-based IPTV system that integrates innovations in both overlay networking and video coding for optimal user experience. The system consists of three key components: partnership formation, robust video coding, and video segment request scheduling. For partnership formation, a graph construction mechanism TYPHOON based on epidemic algorithms is developed to reduce disconnect time and isolated peers. For robust video coding, a multiple description coding (MDC) scheme with spatial-temporal hybrid interpolation (STHI) is proposed to adjust streaming traffic according to the bandwidth and device capability of each peer. For request scheduling, an optimization algorithm is developed by taking the available bandwidth and the video segment type into account. Experimental results show that NTUStreaming is able to deliver optimal video quality in lossy and dynamic networking environments.
Anti-aliasing for light field rendering

Ghosting artifact resulted from angular undersampling is a major issue of light field rendering. In this paper, we propose an anti-aliasing method that compensates the effect of undersampling by making use of depth information. It can be incorporated into most existing refocusing algorithms to render high-quality images from light field. The method yields proper anti-aliasing across all depth range. Mathematical derivation of the method and the physical intuition behind the idea are described. Results are shown to demonstrate the performance of the method compared to previous ones.
Perception-based high dynamic range compression in gradient domain

It is often required to map the radiances of a real scene to a smaller dynamic range so that the image can be properly displayed. However, most such algorithms suffer from the halo artifact or require manual parameter tweaking that is often a tedious process for the user. We propose an automatic algorithm for high dynamic range compression based on the properties of human visual system. The algorithm is performed in the gradient domain to avoid the halo artifact. It automates the parameter adjustment process while preserving the image details. Performance comparison is provided to illustrate the advantages of the proposed algorithm.
Music emotion ranking

Content-based retrieval has emerged as a promising approach to information access. In this paper, we propose an approach to music emotion ranking. Specifically, we rank music in terms of arousal and valence and represent each song as a point in the 2D emotion space. Novel ranking-based methods for annotation, learning, and evaluation of music emotion recognition are developed and tested on a moderately large-scale database composed of 1240 pop songs. Results are provided to show the feasibility of the proposed approach.
Extraction and alignment evaluation of motion beats for street dance

The coordination between the dancer's movement and the accompaniment of music is an important element of dance performance. In this paper, we propose a system for coordination evaluation of street dance. Given a dance video clip as input, the system first extracts motion beats from the video and then measures how well the motion beats correlate with the music beats. The motion beats are obtained by analyzing the speed and the change of direction of the dancer's movement. Unlike most previous work, which mainly focuses on 2D motion trajectory analysis, our system provides a more efficient and accurate dance movement analysis by using the 3D joint data of the dancer acquired by Kinect. Another distinction of our system is that, for beat correlation, it considers not only the underlying steady beat of music but also the groove pattern that gives the propulsive rhythmic feel of the music. We believe the differentiation of music beats into these two categories leads to a finer dance coordination evaluation. The test video clips for performance evaluation are generated by professional dancers. The average F-score of our system is about 80%.
Extracting Blood Vessels From Full-Field OCT Data of Human Skin by Short-Time RPCA

Recent advances in optical coherence tomography (OCT) lead to the development of OCT angiography to provide additional helpful information for diagnosis of diseases like basal cell carcinoma. In this paper, we investigate how to extract blood vessels of human skin from full-field OCT (FF-OCT) data using the robust principal component analysis (RPCA) technique. Specifically, we propose a short-time RPCA method that divides the FF-OCT data into segments and decomposes each segment into a low-rank structure representing the relatively static tissues of human skin and a sparse matrix representing the blood vessels. The method mitigates the problem associated with the slow-varying background and is free of the detection error that RPCA may have when dealing with FF-OCT data. Both short-time RPCA and RPCA methods can extract blood vessels from FF-OCT data with heavy speckle noise, but the former takes only half the computation time of the latter. We evaluate the performance of the proposed method by comparing the extracted blood vessels with the ground truth vessels labeled by a dermatologist and show that the proposed method works equally well for FF-OCT volumes of different quality. The average F-measure improvements over the correlation-mapping OCT method, the modified amplitude-decorrelation OCT angiography method, and the RPCA method, respectively, are 0.1835, 0.1032, and 0.0458.
Singing voice timbre classification of Chinese popular music

Singing voice plays an important role in the listening experience of music. In this paper, we propose to classify popular music by the timbre quality of the singing voice. Specifically, we adopt six singing voice timbre classes as the taxonomy and build a new data set, KKTIC, that contains the expert annotations of 387 Chinese popular songs. To build an automatic classifier, we resort to signal processing and machine learning techniques and extract a number of singing voice-related features such as vibrato and harmonic-to-noise ratio. We also propose the use of vocal segment detection and singing voice separation as preprocessing steps. Our evaluation identifies the relevant acoustic features and validates the importance of these preprocessing steps. The accuracy in timbre classification reaches 79.84% in a five-fold stratified cross validation.
Exploiting Perceptual Anchoring for Color Image Enhancement

The preservation of image quality under various display conditions becomes more and more important in the multimedia era. A considerable amount of effort has been devoted to compensating the quality degradation caused by dim LCD backlight for mobile devices and desktop monitors. However, most previous enhancement methods for backlight-scaled images only consider the luminance component and overlook the impact of color appearance on image quality. In this paper, we propose a fast and elegant method that exploits the anchoring property of human visual system to preserve the color appearance of backlight-scaled images as much as possible. Our approach is distinguished from previous ones in many aspects. First, it has a sound theoretical basis. Second, it takes the luminance and chrominance components into account in an integral manner. Third, it has low complexity and can process 720p high-definition videos at 35 frames per second without flicker. The superior performance of the proposed method is verified through psychophysical tests.
Color enhancement based on the anchoring theory

Providing consistent viewing experience across different reproduction conditions is an important issue for multimedia systems in the real world. In this paper, we propose a method to improve the perceptual quality of color reproduction for backlight-scaled images displayed on a liquid crystal display. Supported by the anchoring theory of lightness perception developed in psychology, this method is able to enhance the color appearance of images even when the backlight is only 5% of the original intensity. The goal is to make the appearance of the resulting images as close as possible to the original images illuminated with full backlight. The concept behind the proposed color enhancement method is general enough for many other applications. The effectiveness of the method is verified by subjective experiments.
Video Object Extraction via MRF-Based Contour Tracking

Video object segmentation is a critical task in multimedia analysis and editing. Normally, the user provides some hints of foreground and background, then the target object is extracted from the video sequence. Most previous methods are either computation-expensive or labor-intensive, and approaches that assume static background have limited applications. In this letter, we propose a novel video segmentation system that integrates Markov random field-based contour tracking with graph-cut image segmentation. The contour tracking propagates the shape of the target object, whereas the graph-cut refines the shape and improves the accuracy of video segmentation. Experimental results show that our segmentation system is efficient and requires less key-frames and user interactions.