nils-herrmann/clean_scirepeval huggingface dataset:
Deep Relative Tracking

Most existing tracking methods are direct trackers, which directly exploit foreground or/and background information for object appearance modeling and decide whether an image patch is target object or not. As a result, these trackers cannot perform well when target appearance changes heavily and becomes different from its model. To deal with this issue, we propose a novel relative tracker, which can effectively exploit the relative relationship among image patches from both foreground and background for object appearance modeling. Different from direct trackers, the proposed relative tracker is robust to localize target object by use of the best image patch with the highest relative score to the target appearance model. To model relative relationship among large-scale image patch pairs, we propose a novel and effective deep relative learning algorithm through the convolutional neural network. We test the proposed approach on challenging sequences involving heavy occlusion, drastic illumination changes, and large pose variations. Experimental results show that our method consistently outperforms the state-of-the-art trackers due to the powerful capacity of the proposed deep relative model.
Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set

In this paper, we address a practical problem of cross-scenario clothing retrieval — given a daily human photo captured in general environment, e.g., on street, finding similar clothing in online shops, where the photos are captured more professionally and with clean background. There are large discrepancies between daily photo scenario and online shopping scenario. We first propose to alleviate the human pose discrepancy by locating 30 human parts detected by a well trained human detector. Then, founded on part features, we propose a two-step calculation to obtain more reliable one-to-many similarities between the query daily photo and online shopping photos: 1) the within-scenario one-to-many similarities between a query daily photo and the auxiliary set are derived by direct sparse reconstruction; and 2) by a cross-scenario many-to-many similarity transfer matrix inferred offline from an extra auxiliary set and the online shopping set, the reliable cross-scenario one-to-many similarities between the query daily photo and all online shopping photos are obtained. We collect a large online shopping dataset and a daily photo dataset, both of which are thoroughly labeled with 15 clothing attributes via Mechanic Turk. The extensive experimental evaluations on the collected datasets well demonstrate the effectiveness of the proposed framework for cross-scenario clothing retrieval.
General Subspace Learning With Corrupted Training Data Via Graph Embedding

We address the following subspace learning problem: supposing we are given a set of labeled, corrupted training data points, how to learn the underlying subspace, which contains three components: an intrinsic subspace that captures certain desired properties of a data set, a penalty subspace that fits the undesired properties of the data, and an error container that models the gross corruptions possibly existing in the data. Given a set of data points, these three components can be learned by solving a nuclear norm regularized optimization problem, which is convex and can be efficiently solved in polynomial time. Using the method as a tool, we propose a new discriminant analysis (i.e., supervised subspace learning) algorithm called Corruptions Tolerant Discriminant Analysis (CTDA), in which the intrinsic subspace is used to capture the features with high within-class similarity, the penalty subspace takes the role of modeling the undesired features with high between-class similarity, and the error container takes charge of fitting the possible corruptions in the data. We show that CTDA can well handle the gross corruptions possibly existing in the training data, whereas previous linear discriminant analysis algorithms arguably fail in such a setting. Extensive experiments conducted on two benchmark human face data sets and one object recognition data set show that CTDA outperforms the related algorithms.
Cross-Platform Multi-Modal Topic Modeling for Personalized Inter-Platform Recommendation

In this paper, we investigate a novel cross- platform multimedia problem: given two platforms, Flickr and Foursquare, we conduct the recommendation between these two platforms, namely the photo recommendation from Flickr to Foursquare users and the venue recommendation from Foursquare to Flickr users. Such inter-platform recommendations enable users from one single platform to enjoy different recommendation services effectively . To solve the problem, we propose a cross- platform multi-modal topic model ( CM3TM), which is capable of: 1) differentiating between two kinds of topics, i.e., platform- specific topics only relevant to a certain platform and shared topics characterizing the knowledge shared by different platforms and 2) aligning multiple modalities from different platforms. Specifically, CM3TM can not only split the topic space into the shared topic space and platform-specific topic space and learn them simultaneously, but also enable the alignment among different modalities through the learned topic space. Given the location information, we applied the proposed CM3TM into two inter-platform recommendation applications: 1) personalized venue recommendation from Foursquare to Flickr users and 2) personalized image recommendation from Flickr to Foursquare users. We have conducted experiments on the collected large-scale real-world dataset from Flickr and Foursquare. Qualitative and quantitative evaluation results validate the effectiveness of our method and demonstrate the advantage of connecting different platforms with different modalities for the inter-platform recommendation.
Naming faces in films using hypergraph matching

In this paper, we aim to address the problem of naming faces in feature-length films using video and film script. Different from the state-of-the-art methods on naming faces in the videos, most of which used a local matching between a visible face and one of the names extracted from the local video transcript, we use a global matching between names and faces as it is not easy to obtain enough local name cues in the films. In the video, we cluster the faces into groups corresponding to characters and build a face network according to face co-occurrence relationship. Similarly in the film script, a name network is also built according to name cooccurrence relationship. The vertices of the two networks are finally matched by a hypergraph matching method. Experiments are conducted on five feature-length films and give encouraging results.
Lower attentive region detection for virtual content insertion in broadcast video

Virtual Content Insertion (VCI) is an emerging application of video analysis. For VCI the spatial position is very important as improper placement will make the insertion intrusive. To choose the spatial position, we propose the notation of Lower Attentive Region (LAR) and provide a generic framework of LAR detection for broadcast video. An LAR is defined, from the cognition point of view, as a region of the video frame which attracts less audiencepsilas attention. It can be changed with little interruption to the main content of the original video. The proposed LAR detection framework includes both bottom-up and top-down modules and can be adapted to all types of videos. Finally we apply the proposed LAR detection approach to broadcast sports video by integrating domain knowledge. The Experiments on LAR detection and VCI in broadcast video demonstrate the effectiveness of the proposed method.
Automatic mobile sports highlights

We report on our development of a real-time system to deliver sports video highlights of a live game to mobile videophones over existing GPRS networks. To facilitate real-time analysis, a circular buffer receives live video data from which simple audio/visual features are computed to detect for highlight-worthiness according to a priori decision scheme. A separate module runs algorithms to insert content into the highlight for mobile advertising. The system is now under trial over new 3G networks.
Saliency Aware Locality-preserving Coding for Image Classification

The Bag-of-Features (BOF) model is widely used for image classification. Most BOF models incorporate a step of maximum pooling to generate the raw image representation, where salient atoms with maximum response are reserved for final representation. However, recent locality-preserving coding schemes do not account for the saliency characteristic during the process of generating the raw image representations. In this paper, we propose a saliency aware locality-preserving coding scheme by explicitly considering saliency into the dictionary creation and feature coding stages. The novel coding scheme guarantees strong response in the pooling operation and thus contributes to a discriminative image representation. Experiments on three benchmark datasets validate the effectiveness of the proposed method.
Singing voice detection using twice-iterated composite Fourier transform

In this paper, we propose a twice-iterated composite Fourier transform (TICFT) technique to detect the singing voice boundaries from acoustical polyphonic music signals. We show that the cumulative TICFT energy in the lower coefficients is capable of differentiating the harmonic structures of vocal and instrumental music in higher octaves. The musical signal is first segmented into frames based on quarter-notes. Then TICFT is used to measure the harmonic structure of each frame. Finally, the vocal and instrumental frames are classified by applying music domain knowledge. Experimental results show over 80% frame level accuracy can be achieved
Object Tracking by Occlusion Detection via Structured Sparse Learning

Sparse representation based methods have recently drawn much attention in visual tracking due to good performance against illumination variation and occlusion. They assume the errors caused by image variations can be modeled as pixel-wise sparse. However, in many practical scenarios these errors are not truly pixel-wise sparse but rather sparsely distributed in a structured way. In fact, pixels in error constitute contiguous regions within the object's track. This is the case when significant occlusion occurs. To accommodate for non-sparse occlusion in a given frame, we assume that occlusion detected in previous frames can be propagated to the current one. This propagated information determines which pixels will contribute to the sparse representation of the current track. In other words, pixels that were detected as part of an occlusion in the previous frame will be removed from the target representation process. As such, this paper proposes a novel tracking algorithm that models and detects occlusion through structured sparse learning. We test our tracker on challenging benchmark sequences, such as sports videos, which involve heavy occlusion, drastic illumination changes, and large pose variations. Experimental results show that our tracker consistently outperforms the state-of-the-art.
Advances in Next-Generation Networking Technologies for Smart Healthcare

The seven articles in this special section examine how 5G mobile communications will impact the smart healthcare industry. With the advancement of 5G communications, smart or "connected" healthcare is getting tremendous attention from academia, governments, industry, and the healthcare community. This will include, for example, 5G wireless networks, mobile edge computing (MEC), software-defined networking (SDN), and cloud radio access networks (C-RANs) can play a significant role in smart healthcare by offering better insight of heterogeneous healthcare media content to support aff ordable and high-quality patient care. While researchers have been making advances in the study of next-generation networking and healthcare services individually, very little attention has been given to make cost-effective and aff ordable smart healthcare solutions. Connected or smart healthcare has the potential to revolutionize many aspects of our society.
A Generic Framework for Video Annotation via Semi-Supervised Learning

Learning-based video annotation is essential for video analysis and understanding, and many various approaches have been proposed to avoid the intensive labor costs of purely manual annotation. However, there lacks a generic framework due to several difficulties, such as dependence of domain knowledge, insufficiency of training data, no precise localization and in efficacy for large-scale video dataset. In this paper, we propose a novel approach based on semi-supervised learning by means of information from the Internet for interesting event annotation in videos. Concretely, a Fast Graph-based Semi-Supervised Multiple Instance Learning (FGSSMIL) algorithm, which aims to simultaneously tackle these difficulties in a generic framework for various video domains (e.g., sports, news, and movies), is proposed to jointly explore small-scale expert labeled videos and large-scale unlabeled videos to train the models. The expert labeled videos are obtained from the analysis and alignment of well-structured video related text (e.g., movie scripts, web-casting text, close caption). The unlabeled data are obtained by querying related events from the video search engine (e.g., YouTube, Google) in order to give more distributive information for event modeling. Two critical issues of FGSSMIL are: (1) how to calculate the weight assignment for a graph construction, where the weight of an edge specifies the similarity between two data points. To tackle this problem, we propose a novel Multiple Instance Learning Induced Similarity (MILIS) measure by learning instance sensitive classifiers; (2) how to solve the algorithm efficiently for large-scale dataset through an optimization approach. To address this issue, Concave-Convex Procedure (CCCP) and nonnegative multiplicative updating rule are adopted. We perform the extensive experiments in three popular video domains: movies, sports, and news. The results compared with the state-of-the-arts are promising and demonstrate the effectiveness and efficiency of our proposed approach.
Topic-Sensitive Influencer Mining in Interest-Based Social Media Networks via Hypergraph Learning

Social media is emerging as a new mainstream means of interacting around online media. Social influence mining in social networks is therefore of critical importance in real-world applications such as friend suggestion and photo recommendation. Social media is inherently multimodal, including rich types of user contributed content and social link information. Most of the existing research suffers from two limitations: 1) only utilizing the textual information, and/or 2) only analyzing the generic influence but ignoring the more important topic-level influence. To address these limitations, in this paper we develop a novel Topic-Sensitive Influencer Mining (TSIM) framework in interest-based social media networks. Specifically, we take Flickr as the study platform. People in Flickr interact with each other through images. TSIM aims to find topical influential users and images. The influence estimation is determined with a hypergraph learning approach. In the hypergraph, the vertices represent users and images, and the hyperedges are utilized to capture multi-type relations including visual-textual content relations among images, and social links between users and images. Algorithmwise, TSIM first learns the topic distribution by leveraging user-contributed images, and then infers the influence strength under different topics for each node in the hypergraph. Extensive experiments on a real-world dataset of more than 50 K images and 70 K comment/favorite links from Flickr have demonstrated the effectiveness of our proposed framework. In addition, we also report promising results of friend suggestion and photo recommendation via TSIM on the same dataset.
Latent support vector machine for sign language recognition with Kinect

In this paper, we propose a novel algorithm to model and recognize sign language with Kinect sensor. We assume that in a sign language video, some frames are expected to be both discriminative and representative. Under this assumption, each frame in training videos is assigned a binary latent variable indicating its discriminative capability. A Latent Support Vector Machine model is then developed to classify the signs, as well as localize the discriminative and representative frames in videos. In addition, we utilize the depth map together with color image captured by Kinect sensor to obtain more effective and accurate feature to enhance the recognition accuracy. To evaluate our approach, we collected an American Sign Language (ASL) dataset which included approximately 2000 phrases, while each phrase was captured by Kinect sensor and hence included color, depth and skeleton information. Experiments on our dataset demonstrate the effectiveness of the proposed method for sign language recognition.
Adaptive filtering approach of color noise and its application in cephalometeric image

In the view of the drawbacks of the conventional Kalman filtering, a kind of adaptive Kalman filtering approach for colored noise based on the correlative method of the system output is proposed to solve the cephalometric images of stomatology. This approach builds the Kalman filtering model of colored noise by adopting the equivalent measurement equation in order to avoid complicated computation and the expansion of the filter dimension. It is also unnecessary to know the variance of measurement noise beforehand so that it is closer to the actual situation. The results of several experiments are presented to demonstrate the feasibility and good performance of this approach.
3D face image acquisition and reconstruction system

This paper deals with a high speed rangefinder to acquire accurate range data of human face and reconstruct 3D face image with high quality. The system is based on laser active triangulation principle. A laser projector emits a laser stripe on human face and two cameras detect it. A transformation matrix is related to the transformation between the laser image in the camera and the related absolute position in space. We use two cameras to solve the missing parts problem and register the gray scale image with range data to reconstruct 3D image with high quality. Theoretical analysis and practical system are described. Experimental results show that this system can reconstruct 3D face image with high measuring speed and accuracy. This system can be applied in medical fields for doctor to acquire and measure the 3D data of patient's face and design the face orthopedic surgery.
Max-Confidence Boosting With Uncertainty for Visual Tracking

The challenges in visual tracking call for a method which can reliably recognize the subject of interests in an environment, where the appearance of both the background and the foreground change with time. Many existing studies model this problem as tracking by classification with online updating of the classification models, however, most of them overlook the ambiguity in visual modeling and do not consider the prior information in the tracking process. In this paper, we present a novel visual tracking method called max-confidence boosting (MCB), which explores a new way of online updating ambiguous visual phenomenon. The MCB framework models uncertainty in prior knowledge utilizing the indeterministic labels, which are used in updating models from previous frames and the new frame. Our proposed MCB tracker allows ambiguity in the tracking process and can effectively alleviate the drift problem. Many experimental results in challenging video sequences verify the success of our method, and our MCB tracker outperforms a number of the state-of-the-art tracking by classification methods.
Understanding Dynamic Cross-OSN Associations for Cold-Start Recommendation

Online social networks (OSNs) have become an essential part of people's daily life, and an increasing number of users are now using multiple OSNs for different social media services simultaneously. As a result, user's interests and preferences usually distribute in different OSNs. While most of the existing work mainly aggregates the distributed user behaviors or features directly, recently very few efforts have been focused on understanding the cross-OSN association from collective user behaviors. In this paper, we go one step further to consider the dynamic characteristic of user behaviors and propose a dynamic cross-OSN association mining framework. In this framework, dynamic user modeling is first conducted to capture the drift of user interest in each OSN. A session-based factorization method is then proposed to establish the cross-OSN association in a dynamic manner, by incrementally updating the derived association each time a new session of data arrives. Based on the derived dynamic association, we finally design a cold-start YouTube video recommendation application, by only utilizing users’ behaviors in Twitter. Experiments are conducted using real-world user data from Twitter and YouTube. The results demonstrate the effectiveness of this proposed framework in capturing the underlying association between different OSNs and achieving superior cold-start recommendation performance.
TVParser: An automatic TV video parsing method

In this paper, we propose an automatic approach to simultaneously name faces and discover scenes in TV shows. We follow the multi-modal idea of utilizing script to assist video content understanding, but without using timestamp (provided by script-subtitles alignment) as the connection. Instead, the temporal relation between faces in the video and names in the script is investigated in our approach, and an global optimal video-script alignment is inferred according to the character correspondence. The contribution of this paper is two-fold: (1) we propose a generative model, named TVParser, to depict the temporal character correspondence between video and script, from which face-name relationship can be automatically learned as a model parameter, and meanwhile, video scene structure can be effectively inferred as a hidden state sequence; (2) we find fast algorithms to accelerate both model parameter learning and state inference, resulting in an efficient and global optimal alignment. We conduct extensive comparative experiments on popular TV series and report comparable and even superior performance over existing methods.
Label localization with weakly spatial constrained graph propagation

Properly utilizing the spatial correlation of regions benefits for improving the performance of label localization task. However, we could not obtain this information directly since we do not have the region level ground truth. In this paper, we propose a weakly spatial constrained graph propagation by mining the spatial correlation from unlabeled regions and integrating it into the graph propagation framework. Our main framework contains two steps: the spatial constrained graph (SCG) construction and label propagation. Firstly, images are over-segmented and each patch is considered as a node. We deem the relatively stable patch combination as a spatial context to construct the SCG, and encourage label propagations where those patches are visually similar as well as spatially consistent. In the second step, we add the dissimilarity constraints and image level label constraints to the label propagation. The propagation procedure is formulated as a constrained optimization problem and it can be efficiently solved by an iteration method. Experiments on three benchmark datasets demonstrate that the spatial correlation mined by our method is effective to the label localization task.