nils-herrmann/clean_scirepeval huggingface dataset:
Context-Based Object-Class Recognition and Retrieval by Generalized Correlograms

We present a novel approach for retrieval of object categories based on a novel type of image representation: the generalized correlogram (GC). In our image representation, the object is described as a constellation of GCs, where each one encodes information about some local part and the spatial relations from this part to others (that is, the part's context). We show how such a representation can be used with fast procedures that learn the object category with weak supervision and efficiently match the model of the object against large collections of images. In the learning stage, we show that, by integrating our representation with Boosting, the system is able to obtain a compact model that is represented by very few features, where each feature conveys key properties about the object's parts and their spatial arrangement. In the matching step, we propose direct procedures that exploit our representation for efficiently considering spatial coherence between the matching of local parts. Combined with an appropriate data organization such as inverted files, we show that thousands of images can be evaluated efficiently. The framework has been applied to different standard databases, and we show that our results are favorably compared against state-of-the-art methods in both computational cost and accuracy.
Emotion Recognition Based on Joint Visual and Audio Cues

Recent technological advances have enabled human users to interact with computers in ways previously unimaginable. Beyond the confines of the keyboard and mouse, new modalities for human-computer interaction such as voice, gesture, and force-feedback are emerging. However, one necessary ingredient for natural interaction is still missing - emotions. This paper describes the problem of bimodal emotion recognition and advocates the use of probabilistic graphical models when fusing the different modalities. We test our audio-visual emotion recognition approach on 38 subjects with 11 HCI-related affect states. The experimental results show that the average person-dependent emotion recognition accuracy is greatly improved when both visual and audio information are used in classification
Discriminating Joint Feature Analysis for Multimedia Data Understanding

In this paper, we propose a novel semi-supervised feature analyzing framework for multimedia data understanding and apply it to three different applications: image annotation, video concept detection and 3-D motion data analysis. Our method is built upon two advancements of the state of the art: (1) l2, 1-norm regularized feature selection which can jointly select the most relevant features from all the data points. This feature selection approach was shown to be robust and efficient in literature as it considers the correlation between different features jointly when conducting feature selection; (2) manifold learning which analyzes the feature space by exploiting both labeled and unlabeled data. It is a widely used technique to extend many algorithms to semi-supervised scenarios for its capability of leveraging the manifold structure of multimedia data. The proposed method is able to learn a classifier for different applications by selecting the discriminating features closely related to the semantic concepts. The objective function of our method is non-smooth and difficult to solve, so we design an efficient iterative algorithm with fast convergence, thus making it applicable to practical applications. Extensive experiments on image annotation, video concept detection and 3-D motion data analysis are performed on different real-world data sets to demonstrate the effectiveness of our algorithm.
Weakly Supervised Photo Cropping

Photo cropping is widely used in the printing industry, photography, and cinematography. Conventional photo cropping methods suffer from three drawbacks: 1) the semantics used to describe photo aesthetics are determined by the experience of model designers and specific data sets, 2) image global configurations, an essential cue to capture photos aesthetics, are not well preserved in the cropped photo, and 3) multi-channel visual features from an image region contribute differently to human aesthetics, but state-of-the-art photo cropping methods cannot automatically weight them. Owing to the recent progress in image retrieval community, image-level semantics, i.e., photo labels obtained without much human supervision, can be efficiently and effectively acquired. Thus, we propose weakly supervised photo cropping, where a manifold embedding algorithm is developed to incorporate image-level semantics and image global configurations with graphlets, or, small-sized connected subgraph. After manifold embedding, a Bayesian Network (BN) is proposed. It incorporates the testing photo into the framework derived from the multi-channel post-embedding graphlets of the training data, the importance of which is determined automatically. Based on the BN, photo cropping can be casted as searching the candidate cropped photo that maximally preserves graphlets from the training photos, and the optimal cropping parameter is inferred by Gibbs sampling. Subjective evaluations demonstrate that: 1) our approach outperforms several representative photo cropping methods, including our previous cropping model that is guided by semantics-free graphlets, and 2) the visualized graphlets explicitly capture photo semantics and global spatial configurations.
Towards optimal ranking metrics

Euclidean metric is frequently used in computer vision, mostly ad-hoc without any justification. However we have found that other metrics like double exponential metric or Cauchy one provide better results, in accordance with the maximum likelihood approach. In this paper we experiment with different modeling functions for similarity noise and compute the accuracy of different methods using these modeling functions in three kinds of applications: content-based image retrieval from a large database, stereo matching and video sequences. We provide a way to determine the modeling distribution which fits best the similarity noise distribution according to the ground truth. In the optimum case, when one has chosen the best modeling distribution, its corresponding metric will give the best ranking results for the ground truth provided.
PET: An eye-tracking dataset for animal-centric Pascal object classes

We present PET- the Pascal animal classes Eye Tracking database. Our database comprises eye movement recordings compiled from forty users for the bird, cat, cow, dog, horse and sheep trainval sets from the VOC 2012 image set. Different from recent eye-tracking databases such as [1, 2], a salient aspect of PET is that it contains eye movements recorded for both the free-viewing and visual search task conditions. While some differences in terms of overall gaze behavior and scanning patterns are observed between the two conditions, a very similar number of fixations are observed on target objects for both conditions. As a utility application, we show how feature pooling around fixated locations enables enhanced (animal) object classification accuracy.
Clustered Multi-task Linear Discriminant Analysis for View Invariant Color-Depth Action Recognition

The widespread adoption of low-cost depth cameras has opened new opportunities to improve traditional action recognition systems. In this paper we focus on the specific problem of action recognition under view point changes and propose a novel approach for view-invariant action recognition operating jointly on visual data of color and depth camera channels. Our method is based on the unique combination of robust Self-Similarity Matrix (SSM) descriptors and multi-task learning. Indeed, multi-view action recognition is inherently a multi-task learning problem: images from a camera view can be modeled as visual data associated to the same task and it is reasonable to assume that the data of different tasks (camera views) are related to each other. In this work we propose a novel algorithm extending Multi-Task Linear Discriminant Analysis (MT-LDA) to enhance its flexibility by learning the dependencies between different views. Extensive experimental results on the publicly available ACT42 dataset demonstrate the effectiveness of the proposed method.
Emotional valence categorization using holistic image features

Can a machine learn to perceive emotions as evoked by an artwork? Here we propose an emotion categorization system, trained by ground truth from psychology studies. The training data contains emotional valences scored by human subjects on the International Affective Picture System (IAPS), a standard emotion evoking image set in psychology. Our approach is based on the assessment of local image statistics which are learned per emotional category using support vector machines. We show results for our system on the I APS dataset, and for a collection of masterpieces. Although the results are preliminary, they demonstrate the potential of machines to elicit realistic emotions when considering masterpieces.
Human-computer interaction: a Bayesian network approach

Human-computer interaction (HCI) lies at the crossroads of many scientific areas including artificial intelligence, computer vision, face recognition, motion tracking, etc. In this paper, we discuss training probabilistic classifiers with labeled and unlabeled data for human-computer interaction applications. We provide an analysis which shows under what conditions unlabeled data can be used in learning to improve classification performance and we investigate the implications of this analysis to a specific type of probabilistic classifiers, Bayesian networks. Finally, we show how the resulting algorithms are successfully employed in facial expression recognition, face detection, and skin detection.
Perceptual Attributes Optimization for Multivideo Summarization

Nowadays, many consumer videos are captured by portable devices such as iPhone. Different from constrained videos that are produced by professionals, e.g., those for broadcast, summarizing multiple handheld videos from a same scenery is a challenging task. This is because: 1) these videos have dramatic semantic and style variances, making it difficult to extract the representative key frames; 2) the handheld videos are with different degrees of shakiness, but existing summarization techniques cannot alleviate this problem adaptively; and 3) it is difficult to develop a quality model that evaluates a video summary, due to the subjectiveness of video quality assessment. To solve these problems, we propose perceptual multiattribute optimization which jointly refines multiple perceptual attributes (i.e., video aesthetics, coherence, and stability) in a multivideo summarization process. In particular, a weakly supervised learning framework is designed to discover the semantically important regions in each frame. Then, a few key frames are selected based on their contributions to cover the multivideo semantics. Thereafter, a probabilistic model is proposed to dynamically fit the key frames into an aesthetically pleasing video summary, wherein its frames are stabilized adaptively. Experiments on consumer videos taken from sceneries throughout the world demonstrate the descriptiveness, aesthetics, coherence, and stability of the generated summary.
Emotion recognition using a Cauchy Naive Bayes classifier

Recognizing human facial expression and emotion by computer is an interesting and challenging problem. In this paper we propose a method for recognizing emotions through facial expressions displayed in video sequences. We introduce the Cauchy Naive Bayes classifier which uses the Cauchy distribution as the model distribution, and provide a framework for choosing the best model distribution assumption. Our person-dependent and person-independent experiments show that the Cauchy distribution assumption typically provides better results than the Gaussian distribution assumption.
Egocentric Daily Activity Recognition via Multitask Clustering

Recognizing human activities from videos is a fundamental research problem in computer vision. Recently, there has been a growing interest in analyzing human behavior from data collected with wearable cameras. First-person cameras continuously record several hours of their wearers' life. To cope with this vast amount of unlabeled and heterogeneous data, novel algorithmic solutions are required. In this paper, we propose a multitask clustering framework for activity of daily living analysis from visual data gathered from wearable cameras. Our intuition is that, even if the data are not annotated, it is possible to exploit the fact that the tasks of recognizing everyday activities of multiple individuals are related, since typically people perform the same actions in similar environments, e.g., people working in an office often read and write documents). In our framework, rather than clustering data from different users separately, we propose to look for clustering partitions which are coherent among related tasks. In particular, two novel multitask clustering algorithms, derived from a common optimization problem, are introduced. Our experimental evaluation, conducted both on synthetic data and on publicly available first-person vision data sets, shows that the proposed approach outperforms several single-task and multitask learning methods.
Recurrent Face Aging

Modeling the aging process of human face is important for cross-age face verification and recognition. In this paper, we introduce a recurrent face aging (RFA) framework based on a recurrent neural network which can identify the ages of people from 0 to 80. Due to the lack of labeled face data of the same person captured in a long range of ages, traditional face aging models usually split the ages into discrete groups and learn a one-step face feature transformation for each pair of adjacent age groups. However, those methods neglect the in-between evolving states between the adjacent age groups and the synthesized faces often suffer from severe ghosting artifacts. Since human face aging is a smooth progression, it is more appropriate to age the face by going through smooth transition states. In this way, the ghosting artifacts can be effectively eliminated and the intermediate aged faces between two discrete age groups can also be obtained. Towards this target, we employ a twolayer gated recurrent unit as the basic recurrent module whose bottom layer encodes a young face to a latent representation and the top layer decodes the representation to a corresponding older face. The experimental results demonstrate our proposed RFA provides better aging faces over other state-of-the-art age progression methods.
Web Image Annotation Via Subspace-Sparsity Collaborated Feature Selection

The number of web images has been explosively growing due to the development of network and storage technology. These images make up a large amount of current multimedia data and are closely related to our daily life. To efficiently browse, retrieve and organize the web images, numerous approaches have been proposed. Since the semantic concepts of the images can be indicated by label information, automatic image annotation becomes one effective technique for image management tasks. Most existing annotation methods use image features that are often noisy and redundant. Hence, feature selection can be exploited for a more precise and compact representation of the images, thus improving the annotation performance. In this paper, we propose a novel feature selection method and apply it to automatic image annotation. There are two appealing properties of our method. First, it can jointly select the most relevant features from all the data points by using a sparsity-based model. Second, it can uncover the shared subspace of original features, which is beneficial for multi-label learning. To solve the objective function of our method, we propose an efficient iterative algorithm. Extensive experiments are performed on large image databases that are collected from the web. The experimental results together with the theoretical analysis have validated the effectiveness of our method for feature selection, thus demonstrating its feasibility of being applied to web image annotation.
Learning Cross-Modal Deep Representations for Robust Pedestrian Detection

This paper presents a novel method for detecting pedestrians under adverse illumination conditions. Our approach relies on a novel cross-modality learning framework and it is based on two main phases. First, given a multimodal dataset, a deep convolutional network is employed to learn a non-linear mapping, modeling the relations between RGB and thermal data. Then, the learned feature representations are transferred to a second deep network, which receives as input an RGB image and outputs the detection results. In this way, features which are both discriminative and robust to bad illumination conditions are learned. Importantly, at test time, only the second pipeline is considered and no thermal data are required. Our extensive evaluation demonstrates that the proposed approach outperforms the state-of-the-art on the challenging KAIST multispectral pedestrian dataset and it is competitive with previous methods on the popular Caltech dataset.
Skin detection: a Bayesian network approach

The automated detection and tracking of humans in computer vision necessitates improved modeling of the human skin appearance. We propose a Bayesian network approach for skin detection. We test several classifiers and propose a methodology for incorporating unlabeled data. We apply the semi-supervised approach to skin detection and we show that learning the structure of Bayesian network classifiers enables learning good classifiers with a small labeled set and a large unlabeled set.
Unsupervised Tube Extraction Using Transductive Learning and Dense Trajectories

We address the problem of automatic extraction of foreground objects from videos. The goal is to provide a method for unsupervised collection of samples which can be further used for object detection training without any human intervention. We use the well known Selective Search approach to produce an initial still-image based segmentation of the video frames. This initial set of proposals is pruned and temporally extended using optical flow and transductive learning. Specifically, we propose to use Dense Trajectories in order to robustly match and track candidate boxes over different frames. The obtained box tracks are used to collect samples for unsupervised training of track-specific detectors. Finally, the detectors are run on the videos to extract the final tubes. The combination of appearance-based static "objectness" (Selective Search), motion information (Dense Trajectories) and transductive learning (detectors are forced to "overfit" on the unsupervised data used for training) makes the proposed approach extremely robust. We outperform state-of-the-art systems by a large margin on common benchmarks used for tube proposal evaluation.
Faved! Biometrics: Tell Me Which Image You Like and I'll Tell You Who You Are

This paper builds upon the belief that every human being has a built-in image aesthetic evaluation system. This sort of personal aesthetics mostly follows certain aesthetic rules widely studied in image aesthetics (e.g., rules of thirds, colorfulness, etc.), though it likely contains some innate, unique preferences. This paper is a proof of concept of this intuition, presenting personal aesthetics as a novel behavioral biometrical trait. In our scenario, personal aesthetics activate when an individual is presented with a set of photos he may like or dislike. The goal is to distill and encode the uniqueness of his visual preferences into a compact template. To this aim, we extract a pool of low- and high-level state-of-the-art image features from a set of Flickr images preferred by a user, feeding them successively into a LASSO regressor. LASSO highlights the most discriminant cues for the individual, allowing authentication and recognition tasks. The results are surprising given only 1 image as test. We can match the user identity against a gallery of 200 individuals definitely much better than chance. Using 20 images (all preferred by a single user) as a biometrical trait, we reach an AUC of 96%, considering the cumulative matching characteristic curve. Extensive experiments also support the interpretability of our approach, effectively modeling what is the “what we like” that distinguishes us from others.
Evaluation of Intensity and Color Corner Detectors for Affine Invariant Salient Regions

Global features are commonly used to describe the image content. The problem with this approach is that these features cannot capture all parts of the image having different characteristics. Therefore, local computation of image information is necessary. By using salient points to represent local information, more discriminative features can be computed. This research is based on an existing affine invariant local feature detector, in which the features are assumed to be intensity corners. First, the existing algorithm is extended with the intensity based SUSAN corner detector which fundamentally differs from the original Harris corner detector. Second, the algorithm is extended to incorporate color information into the detection process. This results in a comparison between three different detection algorithms: the intensity based algorithm using the Harris or SUSAN detector and a color based algorithm that uses two color extended Harris detectors. The different algorithms are compared in terms of invariance and distinctiveness of the regions and computational complexity.
Multi-Paced Dictionary Learning for cross-domain retrieval and recognition

Several applications benefit from learning coupled representations able to describe data from multiple sources. For instance, cross-domain dictionary learning methods demonstrated to be particularly effective. In this paper we introduce Multi-Paced Dictionary Learning (MPDL) and propose an instantiation of it under the framework of cross-domain dictionary learning. MPDL is inspired by previous works on self-paced learning, a framework able to enhance the accuracy of conventional learning models by presenting the training data in a meaningful order, i.e. easy samples are provided first. However, most of existing self-paced learning methods only consider a single modality, while MPDL is specifically designed to assess the learning pace when data from multiple sources are available. We present the model and propose an efficient algorithm to learn the dictionaries and codes. The approach is validated via experiments on two different tasks, namely cross-media retrieval and sketch-to-photo face recognition, using publicly available datasets.
Collaborative Sparse Coding for Multiview Action Recognition

Multiview action recognition has received increasing attention over the past decade. Various approaches have been proposed to extract view-invariant features; among them, self-similarity matrices (SSMs) have shown outstanding performance. However, SSMs become sensitive when there's a very large view change. To make SSMs more robust to viewpoint changes, the authors propose a collaborative sparse coding framework. They integrate the classifier training process and sparse coding process into a unified collaborative filtering framework; this lets more discriminative sparse video representations and classifiers be learned by optimizing the dictionary and classifier jointly. Experimental results demonstrate the effectiveness of the framework.