nils-herrmann/clean_scirepeval huggingface dataset:
A Kullback-Leibler Divergence Based Kernel for SVM Classification in Multimedia Applications

Over the last years significant efforts have been made to develop kernels that can be applied to sequence data such as DNA, text, speech, video and images. The Fisher Kernel and similar variants have been suggested as good ways to combine an underlying generative model in the feature space and discriminant classifiers such as SVM's. In this paper we suggest an alternative procedure to the Fisher kernel for systematically finding kernel functions that naturally handle variable length sequence data in multimedia domains. In particular for domains such as speech and images we explore the use of kernel functions that take full advantage of well known probabilistic models such as Gaussian Mixtures and single full covariance Gaussian models. We derive a kernel distance based on the Kullback-Leibler (KL) divergence between generative models. In effect our approach combines the best of both generative and discriminative methods and replaces the standard SVM kernels. We perform experiments on speaker identification/verification and image classification tasks and show that these new kernels have the best performance in speaker verification and mostly outperform the Fisher kernel based SVM's and the generative classifiers in speaker identification and image classification.
Decision-theoretic Image Retrieval with Embedded Multi-resolution Mixtures

The design of an effective architecture for image retrieval requires careful consideration of the interplay between the three major components of a retrieval system: feature transformation, feature representation, and similarity function. We introduce a decision theoretic formulation of the retrieval problem that enables the design of systems where all components are optimized with respect to the same end-to-end performance criteria: the minimization of the probability of retrieval error. The new formulation is shown to have two appealing properties. First, it leads to an optimal similarity function (the posterior probability of the query under the database image class) that generalizes many of its previously proposed counterparts. Second, it enables a theoretical characterization of the impact of the feature transformation and representation in the probability of error. In addition to exposing the major limitations of a large body of previous retrieval approaches, this characterization allows the derivation of a series of conditions for the optimal design of the feature transformation and representation. The search for a practical solution that can satisfy these conditions leads to the adoption of an embedded multi-resolution mixture representation and originates an efficient algorithm for optimal feature selection. The resulting retrieval architecture achieves a good compromise between retrieval accuracy, invariance, perceptual relevance of similarity judgments, and complexity. Extensive experimental results show that decision-theoretic retrieval performs well on color, texture, and generic image databases in terms of both retrieval accuracy and perceptual relevance of similarity judgments.  nuno.vasconcelos@compaq.com c Compaq Computer Corporation, 2002 This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of the Cambridge Research Laboratory of Compaq Computer Corporation in Cambridge, Massachusetts; an acknowledgment of the authors and individual contributors to the work; and all applicable portions of the copyright notice. Copying, reproducing, or republishing for any other purpose shall require a license with payment of fee to the Cambridge Research Laboratory. All rights reserved. CRL Technical reports are available on the CRL’s web page at http://crl.research.compaq.com.
Feature Selection by Maximum Marginal Diversity

We address the question of feature selection in the context of visual recognition. It is shown that, besides efficient from a computational standpoint, the infomax principle is nearly optimal in the minimum Bayes error sense. The concept of marginal diversity is introduced, leading to a generic principle for feature selection (the principle of maximum marginal diversity) of extreme computational simplicity. The relationships between infomax and the maximization of marginal diversity are identified, uncovering the existence of a family of classification procedures for which near optimal (in the Bayes error sense) feature selection does not require combinatorial search. Examination of this family in light of recent studies on the statistics of natural images suggests that visual recognition problems are a subset of it.
Decision-theoretic image retrieval

The design of an effective architecture for image retrieval requires careful consideration of the interplay between the three major components of a retrieval system: feature transformation, feature representation, and similarity function. We present a review of ongoing work on a decision theoretic formulation of the retrieval problem that enables the design of systems where all components are optimized with respect to the same end-to-end performance criteria: the minimization of the probability of retrieval error. In addition to some previously published results on the theoretical characterization of the impact of the feature transformation and representation in the probability of error, we present an efficient algorithm for optimal feature selection. Experimental results show that decision-theoretic retrieval performs well on color, texture, and generic image databases in terms of both retrieval accuracy and perceptual relevance of similarity judgments.
Exploiting group structure to improve retrieval accuracy and speed in image databases

Most image retrieval systems perform a linear search over the database to find the closest match to a query. However, databases usually exhibit a natural grouping structure into content classes that can be exploited to improve retrieval precision and speed. We investigate methods that enable search at both the class and image level. It is shown that, through the combination of Bayesian averaging and hierarchical density estimation, it is possible to achieve significant gains in retrieval accuracy and speed, at the cost of a marginal increase in training complexity. The technique is also shown to enable the efficient design of semantic classifiers.
What Is the Role of Independence for Visual Recognition?

Independent representations have recently attracted significant attention from the biological vision and cognitive science communities. It has been 1) argued that properties such as sparseness and independence play a major role in visual perception, and 2) shown that imposing such properties on visual representations originates receptive fields similar to those found in human vision. We present a study of the impact of feature independence in the performance of visual recognition architectures. The contributions of this study are of both theoretical and empirical natures, and support two main conclusions. The first is that the intrinsic complexity of the recognition problem (Bayes error) is higher for independent representations. The increase can be significant, close to 10% in the databases we considered. The second is that criteria commonly used in independent component analysis are not sufficient to eliminate all the dependencies that impact recognition. In fact, "independent components" can be less independent than previous representations, such as principal components or wavelet bases.
Empirical Bayesian Motion Segmentation

We introduce an empirical Bayesian procedure for the simultaneous segmentation of an observed motion field and estimation of the hyperparameters of a Markov random field prior. The new approach exhibits the Bayesian appeal of incorporating prior beliefs, but requires only a qualitative description of the prior, avoiding the requirement for a quantitative specification of its parameters. This eliminates the need for trial-and-error strategies for the determination of these parameters and leads to better segmentations.
On the complexity of probabilistic image retrieval

Probabilistic image retrieval approaches can lead to significant gains over standard retrieval techniques. However, this occurs at the cost of a significant increase in computational complexity. In fact, closed-form solutions for probabilistic retrieval are currently available only for simple representations such as the Gaussian and the histogram. We analyze the case of mixture densities and exploit the asymptotic equivalence between likelihood and Kullback-Leibler divergence to derive solutions for these models. In particular, (1) we show that the divergence can be computed exactly for vector quantizers and, (2) has an approximate solution for Gaussian mixtures that introduces no significant degradation of the resulting similarity judgments. In both cases, the new solutions have closed-form and computational complexity equivalent to that of standard retrieval approaches, but significantly better retrieval performance.
Content-based retrieval from image databases: current solutions and future directions

We review recent advances in image retrieval. The two fundamental components of a retrieval system, representation and learning, are analyzed. Each component is decomposed into its constituent building blocks: features, feature representation, and similarity function for the representation; short and long-term procedures for learning. We identify a series of requirements for each of the sub-areas, e.g. optimality, invariance, perceptual relevance, computational tractability, and point out various approaches proposed to satisfy them. Several open problems are also identified.
Image indexing with mixture hierarchies

We present an image indexing method based on a hierarchical description of the density of each of the image classes in a given database. The method is similar in spirit to traditional agglomerative clustering procedures but produces a complete mixture density, instead of a representative point, at each node of the indexing tree. Estimation of the density at a given node only requires knowledge of the mixture parameters of the children nodes, not the original data. The process is very flexible and efficient, therefore suited to problems involving large databases where existing groupings may have to be combined, or new groupings created, frequently. Experimental results show that the new indexing structure consistently outperforms a linear search when both efficiency and retrieval accuracy are taken into account.
Bayesian relevance feedback for content-based image retrieval

We present a Bayesian learning algorithm that relies on belief propagation to integrate feedback provided by the user over a retrieval session. Bayesian retrieval leads to a natural criteria for evaluating local image similarity without requiring any image segmentation. This allows the practical implementation of retrieval systems where users can provide image regions, or objects, as queries. Region-based queries are significantly less ambiguous than queries based on entire images leading to significant improvements in retrieval precision. When combined with local similarity, Bayesian belief propagation is a powerful paradigm for user interaction. Experimental results show that significant improvements in the frequency of convergence to the relevant images can be achieved by the inclusion of learning in the retrieval process.
Bayesian models for visual information retrieval

This thesis presents a unified solution to visual recognition and learning in the context of visual information retrieval. Realizing that the design of an effective recognition architecture requires careful consideration of the interplay between feature selection, feature representation, and similarity function, we start by searching for a performance criteria that can simultaneously guide the design of all three components. A natural solution is to formulate visual recognition as a decision theoretical problem, where the goal is to minimize the probability of retrieval error. This leads to a Bayesian architecture that is shown to generalize a significant number of previous recognition approaches, solving some of the most challenging problems faced by these: joint modeling of color and texture, objective guidelines for controlling the trade-off between feature transformation and feature representation, and unified support for local and global queries without requiring image segmentation. The new architecture is shown to perform well on color, texture, and generic image databases, providing a good trade-off between retrieval accuracy, invariance, perceptual relevance of similarity judgments, and complexity. 
Because all that is needed to perform optimal Bayesian decisions is the ability to evaluate beliefs on the different hypothesis under consideration, a Bayesian architecture is not restricted to visual recognition. On the contrary, it establishes a universal recognition language (the language of probabilities) that provides a computational basis for the integration of information from multiple content sources and modalities. In result, it becomes possible to build retrieval systems that can simultaneously account for text, audio, video, or any other content modalities. 
Since the ability to learn follows from the ability to integrate information over time, this language is also conducive to the design of learning algorithms. We show that learning is, indeed, an important asset for visual information retrieval by designing both short and long-term learning mechanisms. Over short time scales (within a retrieval session), learning is shown to assure faster convergence to the desired target images. Over long time scales (between retrieval sessions), it allows the retrieval system to tailor itself to the preferences of particular users. In both cases, all the necessary computations are carried out through Bayesian belief propagation algorithms that, although optimal in a decision-theoretic sense, are extremely simple, intuitive, and easy to implement.
A unifying view of image similarity

We study solutions to the problem of evaluating image similarity in the context of content-based image retrieval (CBIR). Retrieval is formulated as a classification problem, where the goal is to minimize probability of retrieval error. It is shown that this formulation establishes a common ground for comparing similarity functions, exposes assumptions hidden behind in most commonly used ones, enables a critical analysis of their relative merits, and determines the retrieval scenarios for which each may be most suited. We conclude that most of the current similarity functions are sub-optimal special cases of the Bayesian criteria that results from explicit minimization of error probability.
A probabilistic architecture for content-based image retrieval

The design of an effective architecture for content-based retrieval from visual libraries requires careful consideration of the interplay between feature selection, feature representation, and similarity metric. We present a solution where all the modules strive to optimize the same performance criteria: the probability of retrieval error. This solution consists of a Bayesian retrieval criteria (shown to generalize the most prevalent similarity metrics in current use) and an embedded mixture representation over a multiresolution feature space (shown to provide a good trade-off between retrieval accuracy, invariance, perceptual relevance of similarity, and complexity). The new representation extends standard models (histogram and Gaussian) by providing simultaneous support for high-dimensional features and multi-modal densities and performs well on color texture, and generic image databases.
Bayesian Video Shot Segmentation

Prior knowledge about video structure can be used both as a means to improve the performance of content analysis and to extract features that allow semantic classification. We introduce statistical models for two important components of this structure, shot duration and activity, and demonstrate the usefulness of these models by introducing a Bayesian formulation for the shot segmentation problem. The new formulations is shown to extend standard thresholding methods in an adaptive and intuitive way, leading to improved segmentation accuracy.
Feature representations for image retrieval: beyond the color histogram

We study solutions to the problem of feature representation in the context of content-based image retrieval (CBIR). Retrieval is formulated as a classification problem, where the goal is to minimize probability of retrieval error. Under this formulation, retrieval performance is directly related to the quality of density estimation which is, in turn, determined by properties of the feature representation. We show that most representations of interest for the retrieval problem are particular cases of the mixture model, and present detailed arguments for why this is the most appropriate representation for retrieval.
Learning Over Multiple Temporal Scales in Image Databases

The ability to learn from user interaction is an important asset for content-based image retrieval (CBIR) systems. Over short times scales, it enables the integration of information from successive queries assuring faster convergence to the desired target images. Over long time scales (retrieval sessions) it allows the retrieval system to tailor itself to the preferences of particular users. We address the issue of learning by formulating retrieval as a problem of Bayesian inference. The new formulation is shown to have various advantages over previous approaches: it leads to the minimization of the probability of retrieval error, enables region-based queries without prior image segmentation, and suggests elegant procedures for combining multiple user specifications. As a consequence of all this, it enables the design of short and long-term learning mechanisms that are simple, intuitive, and extremely efficient in terms of computational and storage requirements. We introduce two such algorithms and present experimental evidence illustrating the clear advantages of learning for CBIR.
Statistical models of video structure for content analysis and characterization

Content structure plays an important role in the understanding of video. In this paper, we argue that knowledge about structure can be used both as a means to improve the performance of content analysis and to extract features that convey semantic information about the content. We introduce statistical models for two important components of this structure, shot duration and activity, and demonstrate the usefulness of these models with two practical applications. First, we develop a Bayesian formulation for the shot segmentation problem that is shown to extend the standard thresholding model in an adaptive and intuitive way, leading to improved segmentation accuracy. Second, by applying the transformation into the shot duration/activity feature space to a database of movie clips, we also illustrate how the Bayesian model captures semantic properties of the content. We suggest ways in which these properties can be used as a basis for intuitive content-based access to movie libraries.
Learning from User Feedback in Image Retrieval Systems

We formulate the problem of retrieving images from visual databases as a problem of Bayesian inference. This leads to natural and effective solutions for two of the most challenging issues in the design of a retrieval system: providing support for region-based queries without requiring prior image segmentation, and accounting for user-feedback during a retrieval session. We present a new learning algorithm that relies on belief propagation to account for both positive and negative examples of the user's interests.
A Bayesian framework for semantic content characterization

Current systems for content filtering, browsing, and retrieval rely on low-level image descriptors which are unintuitive for most users. In this paper, we propose an alternative framework that exploits the structured nature of most content sources to achieve semantic content characterization, and lead to much more meaningful user interaction. Computationally, this framework is based on the principles of Bayesian inference and can be implemented efficiently with Bayesian networks. As an illustration of its potential we apply it to the domain of movie databases.
Learning Mixture Hierarchies

The hierarchical representation of data has various applications in domains such as data mining, machine vision, or information retrieval. In this paper we introduce an extension of the Expectation-Maximization (EM) algorithm that learns mixture hierarchies in a computationally efficient manner. Efficiency is achieved by progressing in a bottom-up fashion, i.e. by clustering the mixture components of a given level in the hierarchy to obtain those of the level above. This cl ustering requires only knowledge of the mixture parameters, there being no need to resort to intermediate samples. In addition to practical applications, the algorithm allows a new interpretation of EM that makes clear the relationship with non-parametric kernel-based estimation methods, provides explicit control over the trade-off between the bias and variance of EM estimates, and offers new insights about the behavior of deterministic annealing methods commonly used with EM to escape local minima of the likelihood.
A Bayesian framework for content-based indexing and retrieval

One of the important requirements for practical retrieval systems is the ability to jointly address the issues of indexing and compression. By formulating query by example as a problem of Bayesian inference and establishing a link between probability density estimation and vector quantization, we have previously introduced a representation that leads to very efficient procedures for indexing and retrieval directly in the compressed domain without compromise of the coding efficiency. In this paper, we build on the potential of the Bayesian formulation to support sophisticated inference, to incorporate this representation in a very flexible indexing and retrieval framework that (1) leads to intuitive retrieval procedures, (2) can integrate different content modalities to eliminate some of the strongest limitations of the query by example paradigm, and (3) supports statistical learning of all the model parameters and can, therefore, be trained automatically.
A spatiotemporal motion model for video summarization

The compact description of a video sequence through a single image map and a dominant motion has applications in several domains, including video browsing and retrieval, compression, mosaicing, and visual summarization. Building such a representation requires the capability to register all the frames with respect to the dominant object in the scene, a task which has been, in the past, addressed through temporally localized motion estimates. In this paper, we show how the lack of temporal consistency associated with such estimates can undermine the validity of the dominant motion assumption, leading to oscillation between different scene interpretations and poor registration. To avoid this oscillation, we augment the motion model with a generic temporal constraint which increases the robustness against competing interpretations, leading to more meaningful content summarization.