nils-herrmann/clean_scirepeval huggingface dataset:
Automatic Content Placement in Sports Highlights

To be viable advertising platforms, methods for in-program content placement in sports video must balance against clutter. We propose viewer relevance (VR) measures of video frames in the temporal and spatial domain. Video sub-segments with low temporal VR are first selected, within which actual content is emplaced on regions with low spatial VR. We compute VR measures using color, motion, texture and domain features, upon which spatio-temporal techniques are used to segment spatial regions for actual content placement. Results from preliminary subjective viewing trials on soccer and tennis video indicate that our approach is promising
Automatic character identification in feature-length films

This paper presents a novel approach to automatically identify characters in films using audio visual cues and text analysis. The approach consists of three stages: (i) frontal face track detection and clustering, (ii) face track classification, (iii) name assignment. A finite state machine (FSM) method is utilized to filter faces detected on each frame and build face tracks. The face tracks are clustered using constrained K-centers. The tracks located in the center area of each cluster are set as exemplars. The marginal points of each cluster and the newly detected non-frontal face tracks are classified to these exemplars using complementary cues of audio and visual. The names of characters are ranked based on their occurrences in the film script and the face track clusters are ranked based on track counts. The names are assigned to the clusters according to the ranking order. Experiments were conducted on two feature-length films and gave promising results.
Learning Feature Hierarchies: A Layer-Wise Tag-Embedded Approach

Feature representation learning is an important and fundamental task in multimedia and pattern recognition research. In this paper, we propose a novel framework to explore the hierarchical structure inside the images from the perspective of feature representation learning, which is applied to hierarchical image annotation. Different from the current trend in multimedia analysis of using pre-defined features or focusing on the end-task “flat” representation, we propose a novel layer-wise tag- embedded deep learning (LTDL) model to learn hierarchical features which correspond to hierarchical semantic structures in the tag hierarchy . Unlike most existing deep learning models, LTDL utilizes both the visual content of the image and the hierarchical information of associated social tags. In the training stage, the two kinds of information are fused in a bottom-up way. Supervised training and multi-modal fusion alternate in a layer-wise way to learn feature hierarchies. To validate the effectiveness of LTDL, we conduct extensive experiments for hierarchical image annotation on a large-scale public dataset. Experimental results show that the proposed LTDL can learn representative features with improved performances.
Scene and viewpoint based visual summarization for landmarks

Visual summarization of landmarks is an important task for applications, such as landmark organization, search and browsing. In this work, we make the first attempt towards landmark summarization by simultaneously considering both the scenes (e.g., sunny view and night view) and viewpoints (e.g., front-side and close-distant viewpoint). In the proposed framework of landmark summarization, we first group images into different clusters by viewpoints, then the distinctive scenes for each viewpoint cluster are discovered by the proposed scene-viewpoint based theme modeling. Compared with the existing topic models, our model is capable of mining scene-viewpoint themes directly from all viewpoint clusters and meanwhile differentiating among these themes by viewpoints. The landmark summary is generated by the discovered scene-viewpoint themes, where each theme is represented by the selected images with one certain scene and viewpoint. The experimental results validate the proposed method and demonstrate its advantage in improving user experience.
Automatic semantic annotation for video blogs

In recent years, Weblogs (or blogs) have received great popularity worldwide, among which video blogs (or vlogs) are playing an increasingly important role. As vlogs gain in population, how to make them more easily accessible has become a hot research topic. In this paper, we propose a novel automatic annotation model for vlogs. We extract informative keywords from both the target vlog itself and external resources which are semantically and visually relevant to it. We also present a new evaluation criterion, which assigns a score to an annotation according to its accuracy and completeness in representing the vlogpsilas semantics. Experimental results demonstrate the effectiveness of both the annotation model and evaluation criterion.
Discriminative Exemplar Coding for Sign Language Recognition With Kinect

Sign language recognition is a growing research area in the field of computer vision. A challenge within it is to model various signs, varying with time resolution, visual manual appearance, and so on. In this paper, we propose a discriminative exemplar coding (DEC) approach, as well as utilizing Kinect sensor, to model various signs. The proposed DEC method can be summarized as three steps. First, a quantity of class-specific candidate exemplars are learned from sign language videos in each sign category by considering their discrimination. Then, every video of all signs is described as a set of similarities between frames within it and the candidate exemplars. Instead of simply using a heuristic distance measure, the similarities are decided by a set of exemplar-based classifiers through the multiple instance learning, in which a positive (or negative) video is treated as a positive (or negative) bag and those frames similar to the given exemplar in Euclidean space as instances. Finally, we formulate the selection of the most discriminative exemplars into a framework and simultaneously produce a sign video classifier to recognize sign. To evaluate our method, we collect an American sign language dataset, which includes approximately 2000 phrases, while each phrase is captured by Kinect sensor with color, depth, and skeleton information. Experimental results on our dataset demonstrate the feasibility and effectiveness of the proposed approach for sign language recognition.
Learn to Personalized Image Search From the Photo Sharing Websites

Increasingly developed social sharing websites like Flickr and Youtube allow users to create, share, annotate, and comment medias. The large-scale user-generated metadata not only facilitate users in sharing and organizing multimedia content, but provide useful information to improve media retrieval and management. Personalized search serves as one of such examples where the web search experience is improved by generating the returned list according to the modified user search intents. In this paper, we exploit the social annotations and propose a novel framework simultaneously considering the user and query relevance to learn to personalized image search. The basic premise is to embed the user preference and query-related search intent into user-specific topic spaces. Since the users' original annotation is too sparse for topic modeling, we need to enrich users' annotation pool before user-specific topic spaces construction. The proposed framework contains two components: (1) a ranking-based multicorrelation tensor factorization model is proposed to perform annotation prediction, which is considered as users' potential annotations for the images; (2) we introduce user-specific topic modeling to map the query relevance and user preference into the same user-specific topic space. For performance evaluation, two resources involved with users' social activities are employed. Experiments on a large-scale Flickr dataset demonstrate the effectiveness of the proposed method.
Multi-Modal Event Topic Model for Social Event Analysis

With the massive growth of social events in Internet , it has become more and more difficult to exactly find and organize the interesting events from massive social media data, which is useful to browse, search, and monitor social events by users or governments . To deal with this problem, we propose a novel multi-modal social event tracking and evolution framework to not only effectively capture multi-modal topics of social events, but also obtain the evolutionary trends of social events and generate effective event summary details over time. To achieve this goal, we propose a novel multi-modal event topic model (mmETM), which can effectively model social media documents, including long text with related images, and learn the correlations between textual and visual modalities to separate the visual-representative topics and non-visual-representative topics. To apply the mmETM model to social event tracking, we adopt an incremental learning strategy denoted as incremental mmETM, which can obtain informative textual and visual topics of social events over time to help understand these events and their evolutionary trends. To evaluate the effectiveness of our proposed algorithm, we collect a real-world dataset to conduct various experiments. Both qualitative and quantitative evaluations demonstrate that the proposed mmETM algorithm performs favorably against several state-of-the-art methods.
Video Clock Time Reconition Based on Temporal Periodic Pattern Change of the Digit Characters

A novel temporal neighboring pattern similarity (TNPS) measure is proposed to recognize the time of a digital clock overlay. TNPS detects the presence of a clock overlay by monitoring the periodic changes of the clock digit, and infers the clock time by its natural transition cycle. Compared to traditional methods such as OCR, this method is faster and more reliable because it converts a pattern recognition problem to a pattern change detection problem. Experiments show the recognition result is promising and accurate. One of the applications for this method is to detect the start time of soccer game for our real time live soccer video highlights and event alerts
YouTube Video Promotion by Cross-Network Association: @Britney to Advertise Gangnam Style

The emergence and rapid proliferation of various social media networks have reshaped the way how video contents are generated, distributed, and consumed in traditional video sharing portals. Nowadays, online videos can be accessed from far beyond the internal mechanisms of the video sharing portals, such as internal search and front page highlight. Recent studies have found that external referrers, such as external search engines and other social media websites, arise to be the new and important portals to lead users to online videos. In this paper, we introduce a novel cross-network collaborative application to help drive the online traffic for given videos in the traditional video portal YouTube by leveraging the high propagation efficiency of the popular Twitter followees. Since YouTube videos and Twitter followees distribute on heterogeneous spaces, we present a cross-network association-based solution framework. In this framework, we first represent YouTube videos and Twitter followees in the corresponding topic spaces separately by employing generative topic models. Then, the cross-network topic spaces are associated from both semantic-based and network-based perspectives through the collective intelligence of the observed overlapped users. Based on the derived cross-network association, we finally match the query YouTube videos and candidate Twitter followees in the same topic space with a unified ranking method. The experiments on a real-world large-scale dataset of more than 2.2 million YouTube videos and 31.8 million tweets from 38,540 YouTube users and 39,400 Twitter users demonstrate the effectiveness and superiority of our solution in which network-based and semantic-based association are integrated.
Multimodal Spatio-Temporal Theme Modeling for Landmark Analysis

Here, we discuss mining and summarizing landmarks' general themes as well as the local and temporal themes. General themes occur extensively in various landmarks, and include accommodations and other standard features. The local theme implies a specific theme that exists only at a certain landmark, such as a unique physical characteristic. The temporal theme corresponds to the location-time-representative pattern, which relates only to a certain landmark during a certain period-such as fleet week at the Golden Gate Bridge or red maple leaves in Kiyomizu-dera. Local themes are useful in landmark analysis for their discriminative and representative attributes. However, the ability to discover landmark diversity at different moments makes temporal themes equally important in landmark studies. Time dependent diversity shows complete viewing angles over time and complements local themes in landmark understanding. Furthermore, it provides more comprehensive and structured information for landmark history browsing and tourist decision making. We propose a probabilistic topic model called Multimodal Spatio-Temporal Theme Modeling (mmSTTM). The model considers both textual and visual contexts to learn general, local, and temporal themes, which span a low-dimensional theme space. The model also assigns all textual and visual keywords to each theme, along with a probability for each; a keyword with high weight assignment is meaningful for the theme, while low-weighted keywords are considered noise.
Peg-free human hand shape analysis and recognition

The paper addresses the problem of deformable hand shape recognition in biometric systems without positioning aids. We separate and recognize multiple rigid fingers. An elliptical model is introduced to represent fingers and accelerate the matching of them. Technically, our method bridges the traditional handcrafted-feature methods and the shape-distance methods. We have tested it using our 108-person, 540-sample database with significantly increased positive recognition accuracy.
Structural Sparse Tracking

Sparse representation has been applied to visual tracking by finding the best target candidate with minimal reconstruction error by use of target templates. However, most sparse representation based trackers only consider holistic or local representations and do not make full use of the intrinsic structure among and inside target candidates, thereby making the representation less effective when similar objects appear or under occlusion. In this paper, we propose a novel Structural Sparse Tracking (SST) algorithm, which not only exploits the intrinsic relationship among target candidates and their local patches to learn their sparse representations jointly, but also preserves the spatial layout structure among the local patches inside each target candidate. We show that our SST algorithm accommodates most existing sparse trackers with the respective merits. Both qualitative and quantitative evaluations on challenging benchmark image sequences demonstrate that the proposed SST algorithm performs favorably against several state-of-the-art methods.
Collaborate ball and player trajectory extraction in broadcast soccer video

Enormous accessible broadcast soccer videos demand an efficient ball and player trajectory extraction framework to represent the tactic semantics for the automatic analysis. Camera motions, noise and blurs in broadcast videos make it difficult to extract the trajectories with a single existing object tracking algorithm. In this paper, we propose a novel framework for ball and player trajectory extraction in broadcast soccer videos. The framework generates candidate ball trajectories and player trajectory segments, then it searches the optimal ball trajectory with the likelihood ranking and refines player trajectories with MCMC data association. Instead of extracting ball and player trajectories respectively, our framework employs the motion relationship of the ball and players to build a collaborate scheme to improve the tracking and trajectory refinement results. The experimental results show the proposed framework is more effective than previous works.
Online Multimodal Multiexpert Learning for Social Event Tracking

In this paper, we aim to automatically identify and track the interesting social event from vast amounts of social media data. However, there are two existing challenges: 1) how to model multimodal social event data over time and visualize the topic evolution and 2) how to alleviate the tracking drift problem to boost social event tracking accuracy. We propose a novel online multimodal multiexpert learning algorithm for social event tracking. Compared with existing methods, the proposed model has several advantages: First, it has a nonparametric online multimodal tracking module, which is able to not only automatically learn the number of topics from data over time, but also exploit the multimodal property of the social event. Second, it adopts a novel multiexpert minimization restoration scheme and allows the tracked model to evolve backwards to undo undesirable model updates, which helps alleviate the model drift problem of social event tracking. Third, it is able to not only effectively track the multimodal social event, but also automatically exploit the topic evolution of the social event for a deep understanding with multimodal topics. To evaluate the proposed model, we collect a real-world dataset for research on social event tracking with multimodality information. We have conducted extensive experiments, and both qualitative and quantitative evaluation results have demonstrated the effectiveness of the proposed model.
A Matlab-Based Simulation of System Stability In Frequency-Field Analysis

The frequency-field analysis to a control system says the system's steady-state response when it has a sine signal input. Use this analytic method, control system's specification can be found directly and the method is very simple. Under this we can solve many questions such as avoiding resonance, restraining molest, improving system stability and so on. Matlab (Matrix Laboratory) is software designed for researching of matrix theory, linear algebra and numerical value accounting by Ph.D Clever Moler, the doyen scientist of MathWorks. Now Matlab software has developed into an edition of 7.1 and has been adopted widely by more and more researchers and engineers for its wide account ability and OOP GUI. The application of system stability simulation in frequency-field analysis to a linear system is described. We share our experiences and intuitions gained and encourage other control process to develop simulation-based system analysis
Script-to-Movie: A Computational Framework for Story Movie Composition

Traditional movie production has always been a highly professional work that needs team collaboration, advanced devices and techniques, and vast time and money investment. These high threshold requirements not only prevent mass amateur enthusiasts entering this field, but also hinder professionals quickly previewing their conceived story plots. In this paper, we raise a novel application, named script-to-movie (S2M) composition, to automatically produce new movies from existing videos in accordance with user created script. Our motivation is to liberate producers from complex filming and editing operations, thereby people's story idea can be instantly converted into the vivid movie video. To support the novel “What You Dream Is What You See” (WYDIWYS) production mode, we first propose a hierarchical alignment method to automatically construct a video material database with detailed semantic description. Considering diverse story plots in user designed script, the database contains abundant video materials about different characters appearing in various time and places conditions. On this basis, the S2M composition is formulated as a constrained optimization problem, where semantic story plot and syntactic visual content are synthetically considered to identify a group of optimal video segments to narrate the user designed script story. Both quantitative and qualitative experimental results are reported to illustrate the effectiveness of the proposed S2M application.
Word-of-Mouth Understanding: Entity-Centric Multimodal Aspect-Opinion Mining in Social Media

Most existing approaches on aspect-opinion mining focus on the text domain and cannot be applied to social media where the aspects are essentially multimodal and the opinions depend on the specific aspects. To address the problem of multimodal aspect-opinion mining for entities by leveraging multiple cross-collection sources in social media, in this paper we propose a multimodal aspect-opinion model (mmAOM) considering both user-generated photos and textual documents to simultaneously capture correlations between textual and visual modalities, as well as associations between aspects and opinions . By identifying the aspects and the corresponding opinions related to entities, we apply the mmAOM to entity association visualization and multimodal aspect-opinion retrieval. We have conducted extensive experiments on real-world datasets of entities including Flickr photos, Tripadvisor reviews, and news articles. Qualitative and quantitative evaluation results have validated the effectiveness of the multimodal aspect-opinion mining model, and demonstrated the utility of the derived aspects and opinions from mmAOM in applications of entity association visualization and aspect-opinion retrieval.
Identify Sports Video Shots with "Happy" or "Sad" Emotions

Semantic video content extraction and selection are critical steps in sports video analysis and editing. The identification of video segments can be from various semantic perspectives, e.g. certain event, player or emotional state. In this paper, we examined the possibility of automatically identifying shots with "happy" or "sad" emotion from broadcast sports video. Our proposed model first performs the sports highlight extraction to obtain candidate shots that possibly contain emotion information and then classifies these shots into either "happy" or "sad" emotion groups using hidden Markov model based method. The final experimental results are satisfactory
Folksonomy-Based Visual Ontology Construction and Its Applications

An ontology hierarchically encodes concepts and concept relationships, and has a variety of applications such as semantic understanding and information retrieval. Previous work for building ontologies has primarily relied on labor-intensive human contributions or focused on text-based extraction. In this paper, we consider the problem of automatically constructing a folksonomy-based visual ontology (FBVO) from the user-generated annotated images. A systematic framework is proposed consisting of three stages as concept discovery, concept relationship extraction, and concept hierarchy construction. The noisy issues of the user-generated tags are carefully addressed to guarantee the quality of derived FBVO. The constructed FBVO finally consists of 139 825 concept nodes and millions of concept relationships by mining more than 2.4 million Flickr images. Experimental evaluations show that the derived FBVO is of high quality and consistent with human perception. We further demonstrate the utility of the derived FBVO in applications of complex visual recognition and exploratory image search.