Automatic music video generation based on temporal pattern analysis
Music video (MV) is a short film meant to present a visual representation of a popular music song. In this paper, we present a system that automatically generates MV-like videos from personal home videos based on observations that generally there are obvious repetitive visual and aural patterns in MVs. Based on a set of video and music analysis algorithms, the automatic music video (AMV) generation system automatically extracts temporal structures of the video and music, as well as repetitive patterns in the music. And then, according to the structure and patterns, a set of highlight segments from the raw home video footage are selected, aiming at matching the visual content with the aural structure and pattern. And last, the output music video is rendered by connecting the selected highlight video segments with appropriate transition effects, accompanied with the music. Experiments show that the results are compelling and promising.

title: Automated extraction of music snippets
abstract: Similar to image and video thumbnail, music snippet is defined as the most representative or highlight excerpt of a music clip, and can be used efficiently for fast browsing large number of music files. Music snippet is usually a part of the repeated melody, main theme or chorus. In this paper, we present an approach to extracting music snippet automatically. In our approach, the most salient segment of the music is firstly detected based on its occurrence frequency and energy information. Meanwhile, the boundaries of musical phrases are also detected based on the estimated phrase length and phrase boundary confidence of each frame. These boundaries are used to ensure that an extracted snippet does not break musical phrases. Finally, the musical phrases including the most salient segment are extracted as music snippet. User study indicates that the proposed algorithm works very well on our music database.

title: Photo2Video&#8212;A System for Automatically Converting Photographic Series Into Video
abstract: A novel method for browsing and sharing a single and series of photographs is presented, which can be regarded as a system exploring a new medium type between photograph and video. The scheme exploits the rich content embedded in a single photograph, as well as in a photographic series. Based on studying the typical process of a viewer's attention to variations on objects or regions in an image, a photograph can be converted into a motion clip by simulating camera motions on it. For a selected photographic series, an appropriate set of key-frames are determined for each photograph based on content analyses. And then camera motion pattern is selected for each photograph to generate a corresponding motion photograph clip. Finally, the final output video is rendered by connecting a series of motion photograph clips with specific transitions based on the content of the images on either side of the transition. Also, each motion photograph clip is aligned with the selected incidental music based on music content analysis. As the system, named Photo2Video, generates motion photographs in a fully automatic or semi-automatic manner, it can be used for increasing efficiency in many applications, such as automatic walkthroughs of photograph galleries, motion photographs on website, electronic greeting cards, and personalized Karaoke

title: Optimization-based automated home video editing system
abstract: In this paper, we present an optimization-based system that automates home video editing. This system automatically selects suitable or desirable highlight segments from a set of raw home videos and aligns them with a given piece of incidental music to create an edited video segment to a desired length based on the content of the video and incidental music. We developed an approach for extracting temporal structure and determining the importance of a video segment in order to facilitate the selection of highlight segments. Additionally we extract a temporal structure, beats, and tempos from the incidental music. In order to create more professional-looking results, the selected highlight segments satisfy a set of editing rules and are matched to the content of the incidental music. This task is formulated as a nonlinear 0-1 programming problem and the rules, which are adjustable and increasable, are embedded as constraints. The output video is rendered by connecting the selected highlight video segments with transition effects and the incidental music. Under this framework, we can choose the best-matched music for a given video and support different output styles.

title: Speaker change detection and tracking in real-time news broadcasting analysis
abstract: This paper addresses the problem of real time speaker change detection and speaker tracking in broadcasted news video analysis. In such a case, both speaker identities and number of speakers are assumed unknown. A two-step speaker change detection algorithm, including potential change detection and refinement, is proposed. Speaker tracking is performed based on the results of speaker change detection. A Bayesian Fusion method is used to fuse multiple audio features to get a more reliable result. The algorithm has low complexity and runs in real-time with a very limited delay in analysis. Our experiments show that the algorithms produce very satisfactory results.

title: A flexible framework for key audio effects detection and auditory context inference
abstract: Key audio effects are those special effects that play critical roles in human's perception of an auditory context in audiovisual materials. Based on key audio effects, high-level semantic inference can be carried out to facilitate various content-based analysis applications, such as highlight extraction and video summarization. In this paper, a flexible framework is proposed for key audio effect detection in a continuous audio stream, as well as for the semantic inference of an auditory context. In the proposed framework, key audio effects and the background sounds are comprehensively modeled with hidden Markov models, and a Grammar Network is proposed to connect various models to fully explore the transitions among them. Moreover, a set of new spectral features are employed to improve the representation of each audio effect and the discrimination among various effects. The framework is convenient to add or remove target audio effects in various applications. Based on the obtained key effect sequence, a Bayesian network-based approach is proposed to further discover the high-level semantics of an auditory context by integrating prior knowledge and statistical learning. Evaluations on 12 h of audio data indicate that the proposed framework can achieve satisfying results, both on key audio effect detection and auditory context inference.

title: Constructive Negation Using Typed Existence Properties
abstract: A method for extracting positive information from negative goals is proposed. The method makes use of knowledge about the existence of solutions of predicates and about the types of variables. It strictly generalises earlier work in this area. It can be used as a construction rule or as a simplification rule. As a construction rule it does not involve an SLD-derivation of the negated sub-goal and so is applicable during compilation. As a simplification rule it prunes unsatisfiable goals without doing an explicit satisfiability test.

title: A Three Tiered Approach for Articulated Object Action Modeling and Recognition
abstract: Visual action recognition is an important problem in computer vision. In this paper, we propose a new method to probabilistically model and recognize actions of articulated objects, such as hand or body gestures, in image sequences. Our method consists of three levels of representation. At the low level, we first extract a feature vector invariant to scale and in-plane rotation by using the Fourier transform of a circular spatial histogram. Then, spectral partitioning [20] is utilized to obtain an initial clustering; this clustering is then refined using a temporal smoothness constraint. Gaussian mixture model (GMM) based clustering and density estimation in the subspace of linear discriminant analysis (LDA) are then applied to thousands of image feature vectors to obtain an intermediate level representation. Finally, at the high level we build a temporal multi-resolution histogram model for each action by aggregating the clustering weights of sampled images belonging to that action. We discuss how this high level representation can be extended to achieve temporal scaling in-variance and to include Bi-gram or Multi-gram transition information. Both image clustering and action recognition/segmentation results are given to show the validity of our three tiered representation.

title: Constrained planar motion analysis by decomposition
abstract: General SFM methods give poor results for images captured by constrained motions such as planar motion. In this paper, we propose new SFM algorithms for images captured under a common but constrained planar motion: the image plane is perpendicular to the motion plane. We show that a 2D image captured under such constrained planar motion can be decoupled into two 1D images: one 1D projective and one 1D affine. We then introduce the 1D affine camera model for completing 1D camera models. Next, we describe new subspace reconstruction methods, and apply these methods to the images captured by concentric mosaics, which undergo a special case of constrained planar motion. Finally, we demonstrate both in theory and experiments the advantage of the decomposition method over the general SFM methods by incorporating the constrained motion into the earliest stage of motion analysis.