nils-herrmann/clean_scirepeval huggingface dataset:
Barycentric distribution estimation for texture clustering based on information-geometry tools

The goal of the paper1 is to propose a new method for texture clustering based on the information-geometry tools. Considering textured images as a collection of heavy-tailed prior probability distributions related to some space/scale decomposition, an average of distributions, i.e. a barycentric distribution, is proposed for characterizing each cluster. We suggest the use of the Jeffrey divergence as a dissimilarity measure for the clustering of textured images. Taking into account the geometry of the probabilistic manifold associated to the prior family, we provide the steepest descent method used to estimate the barycentric distribution. The descent exploits the Fisher information matrix, which is the expected value of the Hessian matrix and the local metric to the manifold. The results of experimental evaluation conducted on well-known texture databases show that the Fisher information matrix approach provides a convergence speed significantly higher than the convergence speed of conventional methods of steepest descent.
Asymmetric gradient-based image alignment

A new method for template-based image alignment is presented in this paper. A gradient-based optimization of motion compensated error difference is addressed to solve the image alignment problem. Its novelty lies in the minimized error, which considers the bi-directional compensation of the reference template and the current image onto each other in a constrained asymmetrical fashion. The proposed approach is shown to be a generalization of previous state-of-the art gradient-based methods. An experimental evaluation is provided to show how the new method outperforms the former in presence of noisy images, and to give some insights into its properties.
Texture image classification with Riemannian fisher vectors issued from a Laplacian model

Many signal and image processing applications are based on the classification of covariance matrices. These latter are elements on a Riemannian manifold for which many generative models have been developed in the literature. Recently, the Riemannian Laplace distribution (RLD) has been proposed to model the within-class variability of images. In this context, the present paper proposes an application of RLDs to the definition of Riemannian Fisher vectors issued from this Laplacian model. The expression of these descriptors is derived for mixtures of RLDs and their relation with the Riemannian vectors of locally aggregated descriptors is shown. Some comparisons with the bag of Riemannian words model are also performed. All these aforementioned descriptors are applied to texture image classification to find the most discriminating one. Moreover, to determine the best model for fitting the data, the classification performances are compared to those given by the Riemannian Gaussian distribution.
A joint detection-estimation scheme for the analysis of noisy complex sinusoids

Classical high resolution spectral analysis methods for the estimation of complex sinusoids parameters require the a priori knowledge of the number of sinusoids. In most situations, the correctness of this knowledge is crucial. In this paper, we present a new high resolution sub-space method. Its novelty stems from the fact that the analysis of complex sinusoids is considered as a joint "detection-estimation" issue. Simulation results and an application on real radar signals are presented to illustrate the efficiency of this method.
K-centroids-based supervised classification of texture images: Handling the intra-class diversity

Natural texture images exhibit a high intra-class diversity due to different acquisition conditions (scene enlightenment, perspective angle, ... ). To handle with the diversity, a new supervised classification algorithm based on a parametric formalism is introduced: the K-centroids-based classifier (K-CB). A comparative study between various supervised classification algorithms on the VisTex and Brodatz image databases is conducted and reveals that the proposed K-CB classifier obtains relatively good classification accuracy with a low computational complexity.
Centroid-based texture classification using the SIRV representation

This paper introduces a centroid-based (CB) supervised classification algorithm of textured images. In the context of scale/orientation decomposition, we demonstrate the possibility to develop centroid approach based on multivariate stochastic modeling. The main interest of the multivariate modeling comparatively to the univariate case is to consider spatial dependency as additional features for characterizing texture content. The aim of this paper is twofold. Firstly, we introduce the Spherically Invariant Random Vector (SIRV) representation for the modeling of wavelet coefficients. Secondly, from the specific properties of the SIRV process, i.e. the independence between the two sub-processes of the compound model, we derive centroid estimation scheme. Experiments from various conventional texture databases are conducted and demonstrate the interest of the proposed classification algorithm.
Riemannian Gaussian Distributions on the Space of Symmetric Positive Definite Matrices

Data, which lie in the space <inline-formula> <tex-math notation="LaTeX">$\mathcal {P}_{m\,}$ </tex-math></inline-formula>, of <inline-formula> <tex-math notation="LaTeX">$m \times m$ </tex-math></inline-formula> symmetric positive definite matrices, (sometimes called <italic>tensor data</italic>), play a fundamental role in applications, including medical imaging, computer vision, and radar signal processing. An open challenge, for these applications, is to find a class of probability distributions, which is able to capture the statistical properties of data in <inline-formula> <tex-math notation="LaTeX">$\mathcal {P}_{m\,}$ </tex-math></inline-formula>, as they arise in real-world situations. The present paper meets this challenge by introducing <italic>Riemannian Gaussian distributions</italic> on <inline-formula> <tex-math notation="LaTeX">$\mathcal {P}_{m\,}$ </tex-math></inline-formula>. Distributions of this kind were first considered by Pennec in 2006. However, the present paper gives an exact expression of their probability density function for the first time in existing literature. This leads to two original contributions. First, a detailed study of statistical inference for Riemannian Gaussian distributions, uncovering the connection between the maximum likelihood estimation and the concept of Riemannian centre of mass, widely used in applications. Second, the derivation and the implementation of an expectation-maximisation algorithm, for the estimation of mixtures of Riemannian Gaussian distributions. The paper applies this new algorithm, to the classification of data in <inline-formula> <tex-math notation="LaTeX">$\mathcal {P}_{m\,}$ </tex-math></inline-formula>, (concretely, to the problem of texture classification, in computer vision), showing that it yields significantly better performance, in comparison to recent approaches.
Multivariate texture retrieval using the Kullback-Leibler divergence between bivariate generalized Gamma times an Uniform distribution

This paper presents a new multivariate elliptical distribution, namely the multivariate generalized Gamma times an Uniform (MGΓU) distribution. Because it generalizes the multivariate generalized Gaussian distribution (MGGD), the MGΓU distribution is able to fit a wider range of signals. For the bivariate case, we provide a closed-form of the KullbackLeibler divergence (KLD). We propose the MGΓU distribution for modeling chrominance wavelet coefficients and exercise it in a texture retrieval experiment. A comparative study between some multivariate models on the VisTex and Outex image database is conducted and reveals that the use of the MGΓU distribution of chromiance wavelet coefficient allows an indexing gain compared to other classical approaches such as MGGD and Copula based model).
Robust Scene Categorization by Learning Image Statistics in Context

We present a generic and robust approach for scene categorization. A complex scene is described by proto-concepts like vegetation, water, fire, sky etc. These proto-concepts are represented by low level features, where we use natural images statistics to compactly represent color invariant texture information by a Weibull distribution. We introduce the notion of contextures which preserve the context of textures in a visual scene with an occurrence histogram (context) of similarities to proto-concept descriptors (texture). In contrast to a codebook approach, we use the similarity to all vocabulary elements to generalize beyond the code words. Visual descriptors are attained by combining different types of contexts with different texture parameters. The visual scene descriptors are generalized to visual categories by training a support vector machine. We evaluate our approach on 3 different datasets: 1) 50 categories for the TRECVID video dataset; 2) the Caltech 101-object images; 3) 89 categories being the intersection of the Corel photo stock with the Art Explosion photo stock. Results show that our approach is robust over different datasets, while maintaining competitive performance.
COSTA: Co-Occurrence Statistics for Zero-Shot Classification

In this paper we aim for zero-shot classification, that is visual recognition of an unseen class by using knowledge transfer from known classes. Our main contribution is COSTA, which exploits co-occurrences of visual concepts in images for knowledge transfer. These inter-dependencies arise naturally between concepts, and are easy to obtain from existing annotations or web-search hit counts. We estimate a classifier for a new label, as a weighted combination of related classes, using the co-occurrences to define the weight. We propose various metrics to leverage these co-occurrences, and a regression model for learning a weight for each related class. We also show that our zero-shot classifiers can serve as priors for few-shot learning. Experiments on three multi-labeled datasets reveal that our proposed zero-shot methods, are approaching and occasionally outperforming fully supervised SVMs. We conclude that co-occurrence statistics suffice for zero-shot classification.
Are Concept Detector Lexicons Effective for Video Search?

Until now, systematic studies on the effectiveness of concept detectors for video search have been carried out using less than 20 detectors, or in combination with other retrieval techniques. We investigate whether video search using just large concept detector lexicons is a viable alternative for present day approaches. We demonstrate that increasing the number of concept detectors in a lexicon yields improved video retrieval performance indeed. In addition, we show that combining concept detectors at query time has the potential to boost performance further. We obtain the experimental evidence on the automatic video search task of TRECVID 2005 using 363 machine learned concept detectors.
Fisher and VLAD with FLAIR

A major computational bottleneck in many current algorithms is the evaluation of arbitrary boxes. Dense local analysis and powerful bag-of-word encodings, such as Fisher vectors and VLAD, lead to improved accuracy at the expense of increased computation time. Where a simplification in the representation is tempting, we exploit novel representations while maintaining accuracy. We start from state-of-the-art, fast selective search, but our method will apply to any initial box-partitioning. By representing the picture as sparse integral images, one per codeword, we achieve a Fast Local Area Independent Representation. FLAIR allows for very fast evaluation of any box encoding and still enables spatial pooling. In FLAIR we achieve exact VLAD's difference coding, even with L2 and power-norms. Finally, by multiple codeword assignments, we achieve exact and approximate Fisher vectors with FLAIR. The results are a 18x speedup, which enables us to set a new state-of-the-art on the challenging 2010 PASCAL VOC objects and the fine-grained categorization of the CUB-2011 200 bird species. Plus, we rank number one in the official ImageNet 2013 detection challenge.
Late fusion and calibration for multimedia event detection using few examples

The state-of-the-art in example-based multimedia event detection (MED) rests on heterogeneous classifiers whose scores are typically combined in a late-fusion scheme. Recent studies on this topic have failed to reach a clear consensus as to whether machine learning techniques can outperform rule-based fusion schemes with varying amount of training data. In this paper, we present two parametric approaches to late fusion: a normalization scheme for arithmetic mean fusion (logistic averaging) and a fusion scheme based on logistic regression, and compare them to widely used rule-based fusion schemes. We also describe how logistic regression can be used to calibrate the fused detection scores to predict an optimal threshold given a detection prior and costs on errors. We discuss the advantages and shortcomings of each approach when the amount of positives available for training varies from 10 positives (10Ex) to 100 positives (100Ex). Experiments were run using video data from the NIST TRECVID MED 2013 evaluation and results were reported in terms of a ranking metric: the mean average precision (mAP) and R0, a cost-based metric introduced in TRECVID MED 2013.
Multimedia event-based video indexing using time intervals

We propose the time interval multimedia event (TIME) framework as a robust approach for classification of semantic events in multimodal video documents. The representation used in TIME extends the Allen temporal interval relations and allows for proper inclusion of context and synchronization of the heterogeneous information sources involved in multimodal video analysis. To demonstrate the viability of our approach, it was evaluated on the domains of soccer and news broadcasts. For automatic classification of semantic events, we compare three different machine learning techniques, i.c. C4.5 decision tree, maximum entropy, and support vector machine. The results show that semantic video indexing results significantly benefit from using the TIME framework.
Deep Learning for Visual Understanding [From the Guest Editors]

In the past decade, there has been a transformative and permanent revolution in computer vision cultivated by the reinvigorated adoption of deep learning for visual understanding tasks. Driven by the increasing availability of large annotated data sets, efficient training techniques, and faster computational platforms, deep-learning-based solutions have been progressively employed in a broader spectrum of applications from image classification to activity recognition. Deep learning, in general, refers to a range of artificial neural networks that consist of multiple layers, mimicking the structure and cognitive process of the human brain. Instead of relying on handcrafted features, they allow the acquisition of knowledge directly from data. They regress intricate objective functions in a nested hierarchy, where more sophisticated representations with larger receptive fields computed in terms of less abstract ones with localized supports. Deep learning also makes it possible to incorporate explicit domain knowledge and replace a large variety of conventional algorithmic blocks with trainable differentiable modules. These all give deep learning an exceptional power and flexibility in modeling the relationship between the input data and target output.
What do 15,000 object categories tell us about classifying and localizing actions?

This paper contributes to automatic classification and localization of human actions in video. Whereas motion is the key ingredient in modern approaches, we assess the benefits of having objects in the video representation. Rather than considering a handful of carefully selected and localized objects, we conduct an empirical study on the benefit of encoding 15,000 object categories for action using 6 datasets totaling more than 200 hours of video and covering 180 action classes. Our key contributions are i) the first in-depth study of encoding objects for actions, ii) we show that objects matter for actions, and are often semantically relevant as well. iii) We establish that actions have object preferences. Rather than using all objects, selection is advantageous for action recognition. iv)We reveal that object-action relations are generic, which allows to transferring these relationships from the one domain to the other. And, v) objects, when combined with motion, improve the state-of-the-art for both action classification and localization.
Tag-based video retrieval by embedding semantic content in a continuous word space

Content-based event retrieval in unconstrained web videos, based on query tags, is a hard problem due to large intra-class variances, and limited vocabulary and accuracy of the video concept detectors, creating a "semantic query gap". We present a technique to overcome this gap by using continuous word space representations to explicitly compute query and detector concept similarity. This not only allows for fast query-video similarity computation with implicit query expansion, but leads to a compact video representation, which allows implementation of a real-time retrieval system that can fit several thousand videos in a few hundred megabytes of memory. We evaluate the effectiveness of our representation on the challenging NIST MEDTest 2014 dataset.
Time interval maximum entropy based event indexing in soccer video

Multimodal indexing of events in video documents poses problems with respect to representation, inclusion of contextual information, and synchronization of the heterogeneous information sources involved. In this paper, we present the time interval maximum entropy (TIME) framework that tackles aforementioned problems. To demonstrate the viability of TIME for event classification in multimodal video, an evaluation was performed on the domain of soccer broadcasts. It was found that by applying TIME, the amount of video a user has to watch in order to see almost all highlights is reduced considerably.
Learning Social Tag Relevance by Neighbor Voting

Social image analysis and retrieval is important for helping people organize and access the increasing amount of user tagged multimedia. Since user tagging is known to be uncontrolled, ambiguous, and overly personalized, a fundamental problem is how to interpret the relevance of a user-contributed tag with respect to the visual content the tag is describing. Intuitively, if different persons label visually similar images using the same tags, these tags are likely to reflect objective aspects of the visual content. Starting from this intuition, we propose in this paper a neighbor voting algorithm which accurately and efficiently learns tag relevance by accumulating votes from visual neighbors. Under a set of well-defined and realistic assumptions, we prove that our algorithm is a good tag relevance measurement for both image ranking and tag ranking. Three experiments on 3.5 million Flickr photos demonstrate the general applicability of our algorithm in both social image retrieval and image tag suggestion. Our tag relevance learning algorithm substantially improves upon baselines for all the experiments. The results suggest that the proposed algorithm is promising for real-world applications.
Annotating images by harnessing worldwide user-tagged photos

Automatic image tagging is important yet challenging due to the semantic gap and the lack of learning examples to model a tag's visual diversity. Meanwhile, social user tagging is creating rich multimedia content on the web. In this paper, we propose to combine the two tagging approaches in a search-based framework. For an unlabeled image, we first retrieve its visual neighbors from a large user-tagged image database. We then select relevant tags from the result images to annotate the unlabeled image. To tackle the unreliability and sparsity of user tagging, we introduce a joint-modality tag relevance estimation method which efficiently addresses both textual and visual clues. Experiments on 1.5 million Flickr photos and 10 000 Corel images verify the proposed method.
Video2vec Embeddings Recognize Events When Examples Are Scarce

This paper aims for event recognition when video examples are scarce or even completely absent. The key in such a challenging setting is a semantic video representation. Rather than building the representation from individual attribute detectors and their annotations, we propose to learn the entire representation from freely available web videos and their descriptions using an embedding between video features and term vectors. In our proposed embedding, which we call Video2vec, the correlations between the words are utilized to learn a more effective representation by optimizing a joint objective balancing descriptiveness and predictability. We show how learning the Video2vec embedding using a multimodal predictability loss, including appearance, motion and audio features, results in a better predictable representation. We also propose an event specific variant of Video2vec to learn a more accurate representation for the words, which are indicative of the event, by introducing a term sensitive descriptiveness loss. Our experiments on three challenging collections of web videos from the NIST TRECVID Multimedia Event Detection and Columbia Consumer Videos datasets demonstrate: i) the advantages of Video2vec over representations using attributes or alternative embeddings, ii) the benefit of fusing video modalities by an embedding over common strategies, iii) the complementarity of term sensitive descriptiveness and multimodal predictability for event recognition. By its ability to improve predictability of present day audio-visual video features, while at the same time maximizing their semantic descriptiveness, Video2vec leads to state-of-the-art accuracy for both few- and zero-example recognition of events in video.
VideOlympics: Real-Time Evaluation of Multimedia Retrieval Systems

The first VideOlympics brings content-based analysis to the archive and allows for many-to- many communication between video search engines and their audience It was a great Success. The VideOlympics provided the excitement of a competition without the associated stress on the participants. For the first time, the audience was able to compare different multimedia retrieval systems on the same tasks and see how they performed with unrehearsed topics. Many audience members felt they understood the technology's capabilities after seeing it in live action and in several system variations.
Harvesting Social Images for Bi-Concept Search

Searching for the co-occurrence of two visual concepts in unlabeled images is an important step towards answering complex user queries. Traditional visual search methods use combinations of the confidence scores of individual concept detectors to tackle such queries. In this paper we introduce the notion of bi-concepts, a new concept-based retrieval method that is directly learned from social-tagged images. As the number of potential bi-concepts is gigantic, manually collecting training examples is infeasible. Instead, we propose a multimedia framework to collect de-noised positive as well as informative negative training examples from the social web, to learn bi-concept detectors from these examples, and to apply them in a search engine for retrieving bi-concepts in unlabeled images. We study the behavior of our bi-concept search engine using 1.2 M social-tagged images as a data source. Our experiments indicate that harvesting examples for bi-concepts differs from traditional single-concept methods, yet the examples can be collected with high accuracy using a multi-modal approach. We find that directly learning bi-concepts is better than oracle linear fusion of single-concept detectors, with a relative improvement of 100%. This study reveals the potential of learning high-order semantics from social images, for free, suggesting promising new lines of research.