nils-herrmann/clean_scirepeval huggingface dataset:
Do Colour Interest Points Improve Image Retrieval?

In image retrieval scenarios, many methods use interest point detection at an early stage to find regions in which descriptors are calculated. Finding salient locations in image data is crucial for these tasks. Observing that most current methods use only the luminance information of the images, we investigate the use of colour information in interest point detection. A way to use multi-channel information in the Harris corner detector is explored and different colour spaces are evaluated. To determine the characteristic scale of an interest point, a new colour scale selection method is presented. We show that using colour information and boosting salient colours results in improved performance in retrieval tasks.
Semi-Supervised Face Detection

This paper presents a discussion on semi-supervised learning of probabilistic mixture model classifiers for face detection. We present a theoretical analysis of semi-supervised learning and show that there is an overlooked fundamental difference between the purely supervised and the semisupervised learning paradigms. While in the supervised case, increasing the amount of labeled training data is always seen as a way to improve the classifier’s performance, the converse might also be true as the number of unlabeled data is increased in the semi-supervised case. We also study the impact of this theoretical finding on Bayesian network classifiers, with the goal of avoiding the performance degradation with unlabeled data. We apply the semisupervised approach to face detection and we show that learning the structure of Bayesian network classifiers enables learning good classifiers for face detection with a small labeled set and a large unlabeled set.
FaceCept3D: Real Time 3D Face Tracking and Analysis

We present an open source cross platform technology for 3D face tracking and analysis. It contains a full stack of components for complete face understanding: detection, head pose tracking, facial expression and action units recognition. Given a depth sensor, one can combine FaceCept3D modules to fulfill a specific application scenario. Key advantages of the technology include real time processing speed and ability to handle extreme head pose variations. Possible application areas of the technology range from human computer interaction to active aging, where precise and real-time analysis is required. The technology is available to community.
Plug-and-Play CNN for Crowd Motion Analysis: An Application in Abnormal Event Detection

Most of the crowd abnormal event detection methods rely on complex hand-crafted features to represent the crowd motion and appearance. Convolutional Neural Networks (CNN) have shown to be a powerful instrument with excellent representational capacities, which can leverage the need for hand-crafted features. In this paper, we show that keeping track of the changes in the CNN feature across time can be used to effectively detect local anomalies. Specifically, we propose to measure local abnormality by combining semantic information (inherited from existing CNN models) with low-level optical-flow. One of the advantages of this method is that it can be used without the fine-tuning phase. The proposed method is validated on challenging abnormality detection datasets and the results show the superiority of our approach compared with the state-of-theart methods.
Minimizing dataset bias: Discriminative multi-task sparse coding through shared subspace learning for image classification

Sparse coding was shown to be able to find succinct representations of stimuli. Recently, it has been successfully applied to a variety of problems in image processing analysis. Sparse coding models data vectors as a linear combination of a few elements from a dictionary. However, most existing sparse coding methods are applied for a single task on a single dataset. The learned dictionary is then possibly biased towards the specific dataset and lacks of generalization abilities. In light of this, in this paper we propose a multitask sparse coding approach by uncovering a shared subspace among heterogeneous datasets. The proposed multi-task coding strategy leverages the commonality benefit from different datasets. Moreover, our multi-task coding framework is capable of direct classification by incorporating label information. Experimental results show that the dictionary learned by our approach has more generalization abilities and our model performs better classification compared to the model learned from only one dataset or the model learned from simply pooling different datasets together.
Multitask Linear Discriminant Analysis for View Invariant Action Recognition

Robust action recognition under viewpoint changes has received considerable attention recently. To this end, self-similarity matrices (SSMs) have been found to be effective view-invariant action descriptors. To enhance the performance of SSM-based methods, we propose multitask linear discriminant analysis (LDA), a novel multitask learning framework for multiview action recognition that allows for the sharing of discriminative SSM features among different views (i.e., tasks). Inspired by the mathematical connection between multivariate linear regression and LDA, we model multitask multiclass LDA as a single optimization problem by choosing an appropriate class indicator matrix. In particular, we propose two variants of graph-guided multitask LDA: 1) where the graph weights specifying view dependencies are fixed a priori and 2) where graph weights are flexibly learnt from the training data. We evaluate the proposed methods extensively on multiview RGB and RGBD video data sets, and experimental results confirm that the proposed approaches compare favorably with the state-of-the-art.
Brain and music: Music genre classification using brain signals

Genre classification can be considered as an essential part of music and movie recommender systems. So far, various automatic music genre classification methods have been proposed based on various audio features. However, such content-centric features are not capable of capturing the personal preferences of the listener. In this study, we provide preliminary experimental evidence for the possibility of the music genre classification based on the brain recorded signals of individuals. The brain decoding paradigm is employed to classify recorded brain signals into two broad genre classes: Pop and Rock. We compare the performance of our proposed paradigm on two neuroimaging datasets that contains the electroencephalographic (EEG) and the magnetoencephalographic (MEG) data of subjects who watched 40 music video clips. Our results indicate that the genre of the music clips can be retrieved significantly over the chancelevel using the brain signals. Our study provides a primary step towards user-centric music content retrieval by exploiting brain signals.
Cross-Paced Representation Learning With Partial Curricula for Sketch-Based Image Retrieval

In this paper, we address the problem of learning robust cross-domain representations for sketch-based image retrieval (SBIR). While, most SBIR approaches focus on extracting low- and mid-level descriptors for direct feature matching, recent works have shown the benefit of learning coupled feature representations to describe data from two related sources. However, cross-domain representation learning methods are typically cast into non-convex minimization problems that are difficult to optimize, leading to unsatisfactory performance. Inspired by self-paced learning (SPL), a learning methodology designed to overcome convergence issues related to local optima by exploiting the samples in a meaningful order (i.e., easy to hard), we introduce the cross-paced partial curriculum learning (CPPCL) framework. Compared with existing SPL methods which only consider a single modality and cannot deal with prior knowledge, CPPCL is specifically designed to assess the learning pace by jointly handling data from dual sources and modality-specific prior information provided in the form of partial curricula. In addition, thanks to the learned dictionaries, we demonstrate that the proposed CPPCL embeds robust coupled representations for SBIR. Our approach is extensively evaluated on four publicly available datasets (i.e., CUFS, Flickr15K, QueenMary SBIR, and TU-Berlin Extension datasets), showing superior performance over competing SBIR methods.
Recognizing Emotions from Abstract Paintings Using Non-Linear Matrix Completion

Advanced computer vision and machine learning techniques tried to automatically categorize the emotions elicited by abstract paintings with limited success. Since the annotation of the emotional content is highly resourceconsuming, datasets of abstract paintings are either constrained in size or partially annotated. Consequently, it is natural to address the targeted task within a transductive framework. Intuitively, the use of multi-label classification techniques is desirable so to synergically exploit the relations between multiple latent variables, such as emotional content, technique, author, etc. A very popular approach for transductive multi-label recognition under linear classification settings is matrix completion. In this study we introduce non-linear matrix completion (NLMC), thus extending classical linear matrix completion techniques to the non-linear case. Together with the theory grounding the model, we propose an efficient optimization solver. As shown by our extensive experimental validation on two publicly available datasets, NLMC outperforms state-of-the-art methods when recognizing emotions from abstract paintings.
Emotional Valence Recognition, Analysis of Salience and Eye Movements

This paper studies the performance of recorded eye movements and computational visual attention models (i.e. saliency models) in the recognition of emotional valence of an image. In the first part of this study, it employs eye movement data (fixation & saccade) to build image content descriptors and use them with support vector machines to classify the emotional valence. In the second part, it examines if the human saliency map can be substituted with the state-of-the-art computational visual attention models in the task of valence recognition. The results indicate that the eye movement based descriptors provide significantly better performance compared to the baselines, which apply low-level visual cues (e.g. color, texture and shape). Furthermore, it will be shown that the current computational models for visual attention are not able to capture the emotional information in similar extent as the real eye movements.
Multimodal Engagement Classification for Affective Cinema

This paper describes a multimodal approach to detect viewers' engagement through psycho-physiological affective signals. We investigate the individual contributions of the different modalities, and report experimental results obtained using several fusion strategies, in both per-clip and per-subject cross-validation settings. A sequence of clips from a short movie was showed to 15 participants, from whom we collected per-clip engagement self-assessments. Cues of the users' affective states were collected by means of (i) galvanic skin response (GSR), (ii) automatic facial tracking, and (iii) electroencephalogram(EEG) signals. The main findings of this study can be summarized as follows: (i) each individual modality significantly encodes the level of engagement of the viewers in response to movie clips, (ii) the GSR and EEG signals provide comparable contributions, and (iii) the best performance is obtained when the three modalities are used together.
Fast spatial pattern discovery integrating boosting with constellations of contextual descriptors

We present a novel approach for fast object class recognition incorporating contextual information into boosting. The object is represented as a constellation of generalized correlograms that integrate both information of local parts and their spatial relations. Incorporating the spatial relations into our constellation of descriptors, we show that an exhaustive search for the best matching can be avoided. Combining the contextual descriptors with boosting, the system simultaneously learns the information that characterize each part of the object along with their characteristic mutual spatial relations. The proposed framework includes a matching step between homologous parts in the training set, and learning the spatial pattern after matching. In the matching part two approaches are provided: a supervised algorithm and an unsupervised one. Our results are favorably compared against state-of-the-art results.
Beyond Bag-of-Words: Fast video classification with Fisher Kernel Vector of Locally Aggregated Descriptors

In this paper we introduce a new video description framework that replaces traditional Bag-of-Words with a combination of Fisher Kernels (FK) and Vector of Locally Aggregated Descriptors (VLAD). The main contributions are: (i) a fast algorithm to densely extract global frame features, easier and faster to compute than spatio-temporal local features; (ii) replacing the traditional k-means based vocabulary with a Random Forest approach that allows significant speedup; (iii) use of a modified VLAD and FK representation to replace the classic Bag-of-Words and obtaining better performance. We show that our framework is highly general and is not dependent on a particular type of descriptor. It achieves state-of-the-art results in several classification scenarios.
Flexible Manifold Learning With Optimal Graph for Image and Video Representation

Graph-based dimensionality reduction techniques have been widely and successfully applied to clustering and classification tasks. The basis of these algorithms is the constructed graph which dictates their performance. In general, the graph is defined by the input affinity matrix. However, the affinity matrix derived from the data is sometimes suboptimal for dimension reduction as the data used are very noisy. To address this issue, we propose the projective unsupervised flexible embedding models with optimal graph (PUFE-OG). We build an optimal graph by adjusting the affinity matrix. To tackle the out-of-sample problem, we employ a linear regression term to learn a projection matrix. The optimal graph and the projection matrix are jointly learned by integrating the manifold regularizer and regression residual into a unified model. The experimental results on the public benchmark datasets demonstrate that the proposed PUFE-OG outperforms state-of-the-art methods.
Sparse-coded cross-domain adaptation from the visual to the brain domain

Brain decoding (i.e., retrieving information from brain signals by employing machine learning algorithms) has recently received considerable attention across many communities. In a typical brain decoding paradigm, different types of stimuli are shown to the participant of the neuroimaging experiment, while his/her concurrent brain activity is captured using neuroimaging techniques. Then a machine learning algorithm is employed to categorize the measured brain signal into the target stimuli classes. Accurate prediction of the stimulus category by the algorithm is considered a positive evidence of the hypothesis of the existence of stimulus-related information in brain data. However, most of the brain decoding studies suffer from the constraint of having few and noisy samples. In order to overcome this limitation, in this paper, an adaptation paradigm is employed in order to transfer knowledge from visual domain to brain domain. We experimentally show that such adaptation procedure leads to improved results for the object recognition task in the brain domain, outperforming significantly the results achieved by the brain features alone. This is the first study in the direction of transferring knowledge by adapting representations learned on visual domain to the brain modality. We believe this paper opens up avenues for exploiting large-scale visual datasets to achieve performance gain in brain decoding.
No Matter Where You Are: Flexible Graph-Guided Multi-task Learning for Multi-view Head Pose Classification under Target Motion

We propose a novel Multi-Task Learning framework (FEGA-MTL) for classifying the head pose of a person who moves freely in an environment monitored by multiple, large field-of-view surveillance cameras. As the target (person) moves, distortions in facial appearance owing to camera perspective and scale severely impede performance of traditional head pose classification methods. FEGA-MTL operates on a dense uniform spatial grid and learns appearance relationships across partitions as well as partition-specific appearance variations for a given head pose to build region-specific classifiers. Guided by two graphs which a-priori model appearance similarity among (i) grid partitions based on camera geometry and (ii) head pose classes, the learner efficiently clusters appearance wise related grid partitions to derive the optimal partitioning. For pose classification, upon determining the target's position using a person tracker, the appropriate region specific classifier is invoked. Experiments confirm that FEGA-MTL achieves state-of-the-art classification with few training data.
Affective Meeting Video Analysis

In this paper, we examine the affective content of meeting videos. First, we asked five subjects to manually label three meeting videos using continuous response measurement (continuous-scale labeling in real-time) for energy and valence, (the two dimensions of the human affect space). Then we automatically extracted audio-visual features to characterize the affective content of the videos. We compare the results of manual labeling and low-level automatic audiovisual feature extraction. Our analysis yields promising results which suggest that affective meeting video analysis can lead to very interesting observations useful for automatic indexing
Extended performance graphs for cluster retrieval

Performance evaluations in probabilistic information retrieval are often presented as precision-recall or precision-scope graphs avoiding the otherwise dominating effect of the embedding irrelevant fraction. However, precision and recall values as such offer an incomplete overview of the information retrieval system under study: information about system parameters like generality (the embedding of the relevant fraction), random performance, and the effect of varying the scope is missed In this paper two cluster performance graphs are presented In those cases where complete ground truth is available (both cluster size and database size) the cluster precision-recall (Cluster PR) graph and the generality-precision=recall graph are proposed.
Compact Image Fingerprint Via Multiple Kernel Hashing

Image fingerprinting is regarded as an alternative approach to watermarking in terms of near-duplicate detection application. It consists of feature extraction and feature indexing. Generally, the former is mainly related to discrimination, robustness , and security while the latter closely focuses on the efficiency of fingerprints search. To enable fast fingerprints searching over a very large database, we propose a new kernelized multiple feature hashing method to convert the real-value fingerprints into compact binary-value fingerprints. During the process of converting, the proposed hashing method jointly utilizes the kernel trick and multiple feature fusion strategy to map the image represented by multiple features into a compact binary code. With the help of the kernel function, the hashing method can be applied to any format (such as string, graph, set, and so on) as long as there is an associated kernel function available for similarity measurement. In addition, taking multiple features into account aims at improving the discriminability since these multiple evidences are complementary to each other. The extensive experimental results show that the proposed algorithm outperforms state-of-the-art kernelized hashing methods by up to 10 percent.
Lonely but attractive: Sparse color salient points for object retrieval and categorization

Local image descriptors computed in areas around salient points in images are essential for many algorithms in computer vision. Recent work suggests using as many salient points as possible. While sophisticated classifiers have been proposed to cope with the resulting large number of descriptors, processing this large amount of data is computationally costly. In this paper, computational methods are proposed to compute salient points designed to allow a reduction in the number of salient points while maintaining state of the art performance in image retrieval and object recognition applications. To obtain a more sparse description, a color salient point and scale determination framework is proposed operating on color spaces that have useful perceptual and saliency properties. This allows for the necessary discriminative points to be located, allowing a significant reduction in the number of salient points and obtaining an invariant (repeatability) and discriminative (distinctiveness) image description. Experimental results on large image datasets show that the proposed method obtains state of the art results with the number of salient points reduced by half. This reduction in the number of points allows subsequent operations, such as feature extraction and clustering, to run more efficiently. It is shown that the method provides less ambiguous features, a more compact description of visual data, and therefore a faster classification of visual data.
Dynamic Affinity Graph Construction for Spectral Clustering Using Multiple Features

Spectral clustering (SC) has been widely applied to various computer vision tasks, where the key is to construct a robust affinity matrix for data partitioning. With the increase in visual features, conventional SC methods are facing two challenges: 1) how to effectively generate an affinity matrix based on multiple features? and 2) how to deal with high-dimensional visual features which could be redundant? To address these issues mentioned earlier, we present a new approach to: 1) learn a robust affinity matrix using multiple features, allowing us to simultaneously determine optimal weights for each feature; and 2) decide a set of optimal projection matrixes, one for each feature, that decide the lower dimensional space, as well as the optimal affinity weight of each data pair in the lower dimensional space. There are two major advantages of our new approach over the existing clustering techniques. First, our approach assigns affinity weights for data points on a per-data-pair basis. The learning procedure avoids the explicit specification of the size of the neighborhood in the affinity matrix, and the bandwidth parameter required to compute the Gaussian kernel, both of which are sensitive and yet difficult to determine beforehand. Second, the affinity weights are based on the distances in a lower dimensional space, while the low-dimensional space is inferred according to the optimized affinity weights. Both variables are jointly optimized so as to leverage mutual benefits. The experimental results outperform the compared alternatives, which indicate that the proposed method is effective in simultaneously learning the affinity graph and feature fusion, resulting in better clustering results.
Decoding affect in videos employing the MEG brain signal

This paper presents characterization of affect (valence and arousal) using the Magnetoencephalogram (MEG) brain signal. We attempt single-trial classification of movie and music videos with MEG responses extracted from seven participants. The main findings of this study are that: (i) the MEG signal effectively encodes affective viewer responses, (ii) clip arousal is better predicted than valence employing MEG and (iii) prediction performance is better for movie clips as compared to music videos.