nils-herrmann/clean_scirepeval huggingface dataset:
A cross-modal adaptation approach for brain decoding

Brain decoding has become a hot topic in many recent brain studies. In a typical neuroimaging experiment, participants are presented with different categories of stimuli while their concurrent brain activity is recorded. Then a classifier is trained on the features extracted from the recorded brain data to discriminate different target stimuli classes. It is a common practice to hypothesize that the stimulus-related information exists in the brain data if the decoder can accurately predict the target stimulus category. However, most of the neuroimaging studies suffer from few and noisy samples. These constraints affects the performance of such decoding systems. In order to cope with this limitation, a dictionary learning approach is used in this paper to transfer knowledge from the multimedia domain to the brain domain. We show that such cross-modal domain adaptation yields better performance of the learning algorithm in the brain domain. This is the first study in the direction of cross-modal adaptation by joint dictionary learning on multimedia and brain modality.
Sparse Color Interest Points for Image Retrieval and Object Categorization

Interest point detection is an important research area in the field of image processing and computer vision. In particular, image retrieval and object categorization heavily rely on interest point detection from which local image descriptors are computed for image matching. In general, interest points are based on luminance, and color has been largely ignored. However, the use of color increases the distinctiveness of interest points. The use of color may therefore provide selective search reducing the total number of interest points used for image matching. This paper proposes color interest points for sparse image representation. To reduce the sensitivity to varying imaging conditions, light-invariant interest points are introduced. Color statistics based on occurrence probability lead to color boosted points, which are obtained through saliency-based feature selection. Furthermore, a principal component analysis-based scale selection method is proposed, which gives a robust scale estimation per interest point. From large-scale experiments, it is shown that the proposed color interest point detector has higher repeatability than a luminance-based one. Furthermore, in the context of image retrieval, a reduced and predictable number of color features show an increase in performance compared to state-of-the-art interest points. Finally, in the context of object recognition, for the Pascal VOC 2007 challenge, our method gives comparable performance to state-of-the-art methods using only a small fraction of the features, reducing the computing time considerably.
Comparing Salient Point Detectors

The use of salient points in content-based retrieval allows an image index to represent local properties of the image. Classic corner detectors can also be used for this purpose but they have drawbacks when are applied to various natural images mainly because visual features do not need to be corners and corners may gather in small regions. In this paper, we present a salient point detector using wavelet transform and we compare it with two corner detectors using two criteria: repeatability rate and information content. We determine which detector gives the best results and show that it satisfies the criteria well.
Learning to Group Objects

This paper presents a novel method to generate a hypothesis set of class-independent object regions. It has been shown that such object regions can be used to focus computer vision techniques on the parts of an image that matter most leading to significant improvements in both object localisation and semantic segmentation in recent years. Of course, the higher quality of class-independent object regions, the better subsequent computer vision algorithms can perform. In this paper we focus on generating higher quality object hypotheses. We start from an oversegmentation for which we propose to extract a wide variety of region-features. We group regions together in a hierarchical fashion, for which we train a Random Forest which predicts at each stage of the hierarchy the best possible merge. Hence unlike other approaches, we use relatively powerful features and classifiers at an early stage of the generation of likely object regions. Finally, we identify and combine stable regions in order to capture objects which consist of dissimilar parts. We show on the PASCAL 2007 and 2012 datasets that our method yields higher quality regions than competing approaches while it is at the same time more computationally efficient.
Real-life violent social interaction detection

This paper proposes a method to detect and localize dyadic human interactions in real videos. The idea stems from the significant difference between an action performed by a single subject and an interaction between two persons. In the first case all the visual information is concentrated on the subject, while in the latter case the action of a person is related to the interacting person's attitude, following an action/reaction principle. This kind of behavior is significant especially in natural and real scenarios, in which people are moving freely without the awareness of being recorded. To highlight these features and provide researchers with a common ground for comparisons, we have collected and annotated a new dataset, retrieving from YouTube 30 different videos of a specific type of interaction, namely urban fight situations. The proposed dataset is one of the most challenging annotated video collection concerning dyadic interactions, due to the intrinsic intra-class variability characterizing real fights. In addition, we provide an extensive experimental analysis on this dataset and we demonstrate that the visual information extracted in the area associated to the interpersonal space plays a fundamental role in detecting fights.
Robust Real-Time Extreme Head Pose Estimation

This paper proposes a new framework for head pose estimation under extreme pose variations. By augmenting the precision of a template matching based tracking module with the ability to recover offered by a frame-by-frame head pose estimator, we are able to address pose ranges for which face features are no longer visible, while maintaining state-of-the-art performance. Experimental results obtained on a newly acquired 3D extreme head pose dataset support the proposed method and open new perspectives in approaching real-life unconstrained scenarios.
Generation of the HDL-A-model of a micromembrane from its finite-element-description

A CAD tool for the automated generation of behavioral models in HDL-A is presented. This CAD tool has been implemented in the frame of a project for the automatic modeling of microsystem components for the co-simulation with VHDLor Spice-models. Starting from the finite-element-description of a microcomponent a nonlinear behavioral HDL-A-model is generated by successively adding or deleting effects to the HDL-A-model according to the observed differences between the two models. Using the example of a micromembrane the practicability of this approach is demonstrated. This CAD tool provides a method for decoupling the generation of behavioral models from the finite-element-simulation process.
SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals' personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.
Maximum likelihood stereo matching

In the research literature, maximum likelihood principles were applied to stereo matching by altering the stereo pair so that the difference would have a Gaussian distribution. In this paper we present a novel method of applying maximum likelihood to stereo matching. In our approach, we measure the real noise distribution from a training set, and then construct a new metric which we denote the maximum likelihood metric for comparing the stereo pair. The maximum likelihood metric is optimal in the sense that it maximizes the probability of similarity. In our experiments and discussion, we compared the maximum likelihood metric to other promising algorithms from the research literature using international stereo data sets. Furthermore, we showed that the algorithms from the research literature could be improved by using the maximum likelihood metric instead of the sum of squared differences.
The SocioMetric Badges Corpus: A Multilevel Behavioral Dataset for Social Behavior in Complex Organizations

This paper presents the SocioMetric Badges Corpus, a new corpus for social interaction studies collected during a 6 weeks contiguous period in a research institution, monitoring the activity of 53 people. The design of the corpus was inspired by the need to provide researchers and practitioners with: a) raw digital trace data that could be used to directly address the task of investigating, reconstructing and predicting people's actual social behavior in complex organizations, b) information about participants' individual characteristics (e.g., personality traits), along with c) data concerning the general social context (e.g., participants' social networks) and the specific situations they find themselves in.
The Role of Featural and Configural Information in Face Classification A Simulation of the Expertise Hypothesis

Face recognition in adults is the product of a unique mechanism in the brain and it is based on years of experience. The goal of this paper is to analyze the role of configural and featural information for face classification and to compare the performance of Bayesian network classifiers with the human performance in three experiments: similarity matching, gender, and race classification. Our results show that despite the fact that the machine classification results are worse than the one of the humans, they are consistent with human classification results
Optimized Graph Learning Using Partial Tags and Multiple Features for Image and Video Annotation

In multimedia annotation, due to the time constraints and the tediousness of manual tagging, it is quite common to utilize both tagged and untagged data to improve the performance of supervised learning when only limited tagged training data are available. This is often done by adding a geometry-based regularization term in the objective function of a supervised learning model. In this case, a similarity graph is indispensable to exploit the geometrical relationships among the training data points, and the graph construction scheme essentially determines the performance of these graph-based learning algorithms. However, most of the existing works construct the graph empirically and are usually based on a single feature without using the label information. In this paper, we propose a semi-supervised annotation approach by learning an optimized graph (OGL) from multi-cues (i.e., partial tags and multiple features), which can more accurately embed the relationships among the data points. Since OGL is a transductive method and cannot deal with novel data points, we further extend our model to address the out-of-sample issue. Extensive experiments on image and video annotation show the consistent superiority of OGL over the state-of-the-art methods.
Evaluating Multi-task Learning for Multi-view Head-Pose Classification in Interactive Environments

Social attention behavior offers vital cues towards inferring one's personality traits from interactive settings such as round-table meetings and cocktail parties. Head orientation is typically employed as a proxy for determining the social attention direction when faces are captured at low-resolution. Recently, multi-task learning has been proposed to robustly compute head pose under perspective and scale-based facial appearance variations when multiple, distant and large field-of-view cameras are employed for visual analysis in smart-room applications. In this paper, we evaluate the effectiveness of an SVM-based MTL (SVM+MTL) framework with various facial descriptors (KL, HOG, LBP, etc.). The KL+HOG feature combination is found to produce the best classification performance, with SVM+MTL outperforming classical SVM irrespective of the feature used.
Localize Me Anywhere, Anytime: A Multi-task Point-Retrieval Approach

Image-based localization is an essential complement to GPS localization. Current image-based localization methods are based on either 2D-to-3D or 3D-to-2D to find the correspondences, which ignore the real scene geometric attributes. The main contribution of our paper is that we use a 3D model reconstructed by a short video as the query to realize 3D-to-3D localization under a multi-task point retrieval framework. Firstly, the use of a 3D model as the query enables us to efficiently select location candidates. Furthermore, the reconstruction of 3D model exploits the correlations among different images, based on the fact that images captured from different views for SfM share information through matching features. By exploring shared information (matching features) across multiple related tasks (images of the same scene captured from different views), the visual feature's view-invariance property can be improved in order to get to a higher point retrieval accuracy. More specifically, we use multi-task point retrieval framework to explore the relationship between descriptors and the 3D points, which extracts the discriminant points for more accurate 3D-to-3D correspondences retrieval. We further apply multi-task learning (MTL) retrieval approach on thermal images to prove that our MTL retrieval framework also provides superior performance for the thermal domain. This application is exceptionally helpful to cope with the localization problem in an environment with limited light sources.
Optimal graph learning with partial tags and multiple features for image and video annotation

In multimedia annotation, due to the time constraints and the tediousness of manual tagging, it is quite common to utilize both tagged and untagged data to improve the performance of supervised learning when only limited tagged training data are available. This is often done by adding a geometrically based regularization term in the objective function of a supervised learning model. In this case, a similarity graph is indispensable to exploit the geometrical relationships among the training data points, and the graph construction scheme essentially determines the performance of these graph-based learning algorithms. However, most of the existing works construct the graph empirically and are usually based on a single feature without using the label information. In this paper, we propose a semi-supervised annotation approach by learning an optimal graph (OGL) from multi-cues (i.e., partial tags and multiple features) which can more accurately embed the relationships among the data points. We further extend our model to address out-of-sample and noisy label issues. Extensive experiments on four public datasets show the consistent superiority of OGL over state-of-the-art methods by up to 12% in terms of mean average precision.
Computational Modeling of Affective Qualities of Abstract Paintings

Modeling the emotional response of human observers to specific imagery is an important aspect of affective computing; it both furthers our understanding of the affective qualities of visual stimuli and paves the way for numerous applications. The emotional effect of colors and their combinations have been extensively studied by artists, as color is one of the essential means of conveying the artist's intention and affecting viewers emotionally. This is especially true in abstract art. In this study, the authors show how concepts from art theory can be used to design a computational model that lets a computer predict whether an abstract painting will elicit positive or negative emotions in the observer. Furthermore, their model can be used to generate abstract paintings that elicit intended emotional responses in observers. This article is part of a special issue on quality modeling.
Category Specific Dictionary Learning for Attribute Specific Feature Selection

Attributes, as mid-level features, have demonstrated great potential in visual recognition tasks due to their excellent propagation capability through different categories. However, existing attribute learning methods are prone to learning the correlated attributes. To discover the genuine attribute specific features, many feature selection methods have been proposed. However, these feature selection methods are implemented at the level of raw features that might be very noisy, and these methods usually fail to consider the structural information in the feature space. To address this issue, in this paper, we propose a label constrained dictionary learning approach combined with a multilayer filter. The feature selection is implemented at dictionary level, which can better preserve the structural information. The label constrained dictionary learning suppresses the intra-class noise by encouraging the sparse representations of intra-class samples to lie close to their center. A multilayer filter is developed to discover the representative and robust attribute specific bases. The attribute specific bases are only shared among the positive samples or the negative samples. The experiments on the challenging Animals with Attributes data set and the SUN attribute data set demonstrate the effectiveness of our proposed method.
Event Oriented Dictionary Learning for Complex Event Detection

Complex event detection is a retrieval task with the goal of finding videos of a particular event in a large-scale unconstrained Internet video archive, given example videos and text descriptions. Nowadays, different multimodal fusion schemes of low-level and high-level features are extensively investigated and evaluated for the complex event detection task. However, how to effectively select the high-level semantic meaningful concepts from a large pool to assist complex event detection is rarely studied in the literature. In this paper, we propose a novel strategy to automatically select semantic meaningful concepts for the event detection task based on both the events-kit text descriptions and the concepts high-level feature descriptions. Moreover, we introduce a novel event oriented dictionary representation based on the selected semantic concepts. Toward this goal, we leverage training images (frames) of selected concepts from the semantic indexing dataset with a pool of 346 concepts, into a novel supervised multitask ℓp-norm dictionary learning framework. Extensive experimental results on TRECVID multimedia event detection dataset demonstrate the efficacy of our proposed method.
The Many Shades of Negativity

Complex event detection has been progressively researched in recent years for the broad interest of video indexing and retrieval. To fulfill the purpose of event detection, one needs to train a classifier using both positive and negative examples. Current classifier training treats the negative videos as equally negative. However, we notice that many negative videos resemble the positive videos in different degrees. Intuitively, we may capture more informative cues from the negative videos if we assign them fine-grained labels, thus benefiting the classifier learning. Aiming for this, we use a statistical method on both the positive and negative examples to get the decisive attributes of a specific event. Based on these decisive attributes, we assign the fine-grained labels to negative examples to treat them differently for more effective exploitation. The resulting fine-grained labels may be not optimal to capture the discriminative cues from the negative videos. Hence, we propose to jointly optimize the fine-grained labels with the classifier learning, which brings mutual reciprocality. Meanwhile, the labels of positive examples are supposed to remain unchanged. We thus additionally introduce a constraint for this purpose. On the other hand, the state-of-the-art deep convolutional neural network features are leveraged in our approach for event detection to further boost the performance. Extensive experiments on the challenging TRECVID MED 2014 dataset have validated the efficacy of our proposed approach.
Visual websearching using iconic queries

Imagine a professional Web site designer who constantly has to come up with innovative looks for the client's homepages? How does he find the new content? The major search engines such as Hotbot (http://www.hotbot.com) allow us to find text on the Web, but typically have few or no capabilities for finding visual media. In this article we discuss methods for finding visual media on the WWW. The emphasis is on iconic queries which are essentially drag-and-drop visual concepts or simple semantics. We describe our method for finding static feature sets and then describe a novel method called active feature sets, which chooses a feature set based on the context in the image.
Viraliency: Pooling Local Virality

In our overly-connected world, the automatic recognition of virality &#x2013; the quality of an image or video to be rapidly and widely spread in social networks &#x2013; is of crucial importance, and has recently awaken the interest of the computer vision community. Concurrently, recent progress in deep learning architectures showed that global pooling strategies allow the extraction of activation maps, which highlight the parts of the image most likely to contain instances of a certain class. We extend this concept by introducing a pooling layer that learns the size of the support area to be averaged: the learned top-N average (LENA) pooling. We hypothesize that the latent concepts (feature maps) describing virality may require such a rich pooling strategy. We assess the effectiveness of the LENA layer by appending it on top of a convolutional siamese architecture and evaluate its performance on the task of predicting and localizing virality. We report experiments on two publicly available datasets annotated for virality and show that our method outperforms state-of-the-art approaches.