nils-herrmann/clean_scirepeval huggingface dataset:
Efficient Stochastic Source Coding and an Application to a Bayesian Network Source Model

In this paper, we introduce a new algorithm calledbits-back coding' that makes stochastic source codes ef"cient. For a given one-to-many source code, we show that this algorithm can actually be more ef"cient than the algorithm that always picks the shortest codeword. Optimal ef"ciency is achieved when codewords are chosen according to the Boltzmann distribution based on the codeword lengths. It turns out that a commonly used technique for determining parameters— maximum-likelihood estimation—actually minimizes the bits-back coding cost when codewords are chosen according to the Boltzmann distribution. A tractable approximation to maximum-likelihood estimation—the generalized expectation-maximization algorithm—minimizes the bits-back coding cost. After presenting a binary Bayesian network model that assigns exponentially many codewords to each symbol, we show how a tractable approximation to the Boltzmann distribution can be used for bits-back coding. We illustrate the performance of bits-back coding using non-synthetic data with a binary Bayesian network source model that produces 2 60 possible codewords for each input symbol. The rate for bits-back coding is nearly one half of that obtained by picking the shortest
Bayesian networks for pattern classification, data compression, and channel coding

Pattern classification, data compression, and channel coding are tasks that usually must deal with complex but structured natural or artificial systems. Patterns that we wish to classify are a consequence of a causal physical process. Images that we wish to compress are also a consequence of a causal physical process. Noisy outputs from a telephone line are corrupted versions of a signal produced by a structured man-made telephone modem. Not only are these tasks characterized by complex structure, but they also contain random elements. Graphical models such as Bayesian networks provide a way to describe the relationships between random variables in a stochastic system. 
In this thesis, I use Bayesian networks as an overarching framework to describe and solve problems in the areas of pattern classification, data compression, and channel coding. Results on the classification of handwritten digits show that Bayesian network pattern classifiers outperform other standard methods, such as the k-nearest neighbor method. When Bayesian networks are used as source models for data compression, an exponentially large number of codewords are associated with each input pattern. It turns out that the code can still be used efficiently, if a new technique called "bits-back coding" is used. Several new error-correcting decoding algorithms are instances of "probability propagation" in various Bayesian networks. These new schemes are rapidly closing the gap between the performances of practical channel coding systems and Shannon's 50-year-old channel coding limit. The Bayesian network framework exposes the similarities between these codes and leads the way to a new class of "trellis-constraint codes" which also operate close to Shannon's limit.
Variational inference for continuous sigmoidal Bayesian networks

Latent random variables can be useful for modelling covariance relationships between observed variables. The choice of whether specific latent variables ought to be continuous or discrete is often an arbitrary one. In a previous paper, I presented a "unit" that could adapt to be continuous or binary, as appropriate for the current problem, and showed how a Markov chain Monte Carlo method could be used for inference and parameter estimation in Bayesian networks of these units. In this paper, I develop a variational inference technique in the hope that it will prove to be more computationally efficient than Monte Carlo methods. After presenting promising \emph{inference} results on a toy problem, I discuss why the variational technique does not work well for \emph{parameter estimation} as compared to Monte Carlo.
Continuous Sigmoidal Belief Networks Trained using Slice Sampling

Real-valued random hidden variables can be useful for modelling latent structure that explains correlations among observed variables. I propose a simple unit that adds zero-mean Gaussian noise to its input before passing it through a sigmoidal squashing function. Such units can produce a variety of useful behaviors, ranging from deterministic to binary stochastic to continuous stochastic. I show how "slice sampling" can be used for inference and learning in top-down networks of these units and demonstrate learning on two simple problems.
Free energy coding

We introduce a new approach to the problem of optimal compression when a source code produces multiple codewords for a given symbol. It may seem that the most sensible codeword to use in this case is the shortest one. However, in the proposed free energy approach, random codeword selection yields an effective codeword length that can be less than the shortest codeword length. If the random choices are Boltzmann distributed, the effective length is optimal for the given source code. The expectation-maximization parameter estimation algorithms minimize this effective codeword length. We illustrate the performance of free energy coding on a simple problem where a compression factor of two is gained by using the new method.
Does the Wake-sleep Algorithm Produce Good Density Estimators?

The wake-sleep algorithm (Hinton, Dayan, Frey and Neal 1995) is a relatively efficient method of fitting a multilayer stochastic generative model to high-dimensional data. In addition to the top-down connections in the generative model, it makes use of bottom-up connections for approximating the probability distribution over the hidden units given the data, and it trains these bottom-up connections using a simple delta rule. We use a variety of synthetic and real data sets to compare the performance of the wake-sleep algorithm with Monte Carlo and mean field methods for fitting the same generative model and also compare it with other models that are less powerful but easier to fit.
Comparison of Classification Methods for Fingerprint Verification with Variable Numbers of Minutiae

The fingerprint verification task answers the question of whether or not two fingerprints belongs to the same finger. The paper focuses on the classification aspect of fingerprint verification. Classification is the third and final step after after the two earlier steps of feature extraction, where a known set of features (minutiae points) have been extracted from each fingerprint, and scoring, where a matcher has determined a degree of match between the two sets of features. Since this is a binary classification problem involving a single variable, the commonly used threshold method is related to the so-called receiver operating characteristics (ROC). In the ROC approach the optimal threshold on the score is determined so as to determine match or non-match. Such a method works well when there is a well-registered fingerprint image. On the other hand more sophisticated methods are needed when there exists a a partial imprint of a finger– as in the case of latent prints in forensics or due to limitations of the biometric device. In such situations it is useful to consider classification methods based on computing the likelihood ratio of match/non-match. Such methods are commonly used in some biometric and forensic domains such as speaker verification where there is a much higher degree of uncertainty. This paper compares the two approaches empiricially for the fingerprint classification task when the number of available minutiae are varied.
Signature verification using Kolmogorov-Smirnov statistic

Automatic signature verification of scanned documents are presented here. The strategy used for verification is applicable in scenarios where there are multiple knowns(genuine signature samples) from a writer. First the learning process invovles learning the variation and similarities from the known genuine samples from the given writer and then classification problem answers the question whether or not a given questioned sample belongs to the ensemble of known samples or not. The learning strategy discussed, compares pairs of signature samples from amongst the knwon samples, to obtain a distribution in distance space, that represents the distribution of the variation amongst samples, for that particular writer. The corresponding classification method involves comparing the questioned sample, with all the available knowns, to obtain another distribution in distance space. The classification task is now to compare the two distributions to obtain a probability of similarity of the two distributions, that represents, the probability of the questioned sample belonging to the ensemble of the knowns. The above strategies are applied to the problem of signature verification and performance results are presented.
A statistical model for writer verification

A statistical model for determining whether a pair of documents, a known and a questioned, were written by the same individual is proposed. The model has the following four components: (i) discriminating elements, e.g., global features and characters, are extracted from each document; (ii) differences between corresponding elements from each document are computed; (iii) using conditional probability estimates of each difference, the log-likelihood ratio (LLR) is computed for the hypotheses that the documents were written by the same or different writers; the conditional probability estimates themselves are determined from labeled samples using either Gaussian or gamma estimates for the differences assuming their statistical independence; and (iv) distributions of the LLRs for same and different writer LLRs are analyzed to calibrate the strength of evidence into a standard nine-point scale used by questioned document examiners. The model is illustrated with experimental results for a specific set of discriminating elements.
On Profiling Mobility and Predicting Locations of Campus-Wide Wireless Network Users

In this paper, we analyze a year long wireless network users’ mobility trace data collected on ETH Zurich campus. Unlike earlier work in , [21], [35], we profile the movement pattern of wireless users and predict their locations. More specifically, we show that each network user regularly visits a list of places, such as a building (also referred to as “hubs”) with some probability. The daily list of hubs, along with their corresponding visit probabilities, are referred to as a mobility profile. We also show that over a period of time (e.g., a week), a user may repeatedly follow a mixture of mobility profiles with certain probabilities associated with each of the profiles. Our analysis of the mobility trace data not only validate the existence of our so-called sociological orbits [13], but also demonstrate the advantages of exploiting it in performing hub-level location predictions. Moreover, such profile based location predictions are found not only to be more precise than a common statistical approach based on observed hub visitation frequencies, but also shown to incur a much lower overhead. We further illustrate the benefit of profiling users’ mobility by discussing relevant work and suggesting applications in different types of wireless networks, including mobile ad hoc networks.
Machine learning approaches for person identification and verification

New machine learning strategies are proposed for person identification which can be used in several biometric modalities such as friction ridges, handwriting, signatures and speech. The biometric or forensic performance task answers the question of whether or not a sample belongs to a known person. Two different learning paradigms are discussed: person-independent (or general learning) and person-dependent (or person-specific learning). In the first paradigm, learning is from a general population of ensemble of pairs, each of which is labelled as being from the same person or from different persons- the learning process determines the range of variations for given persons and between different persons. In the second paradigm the identity of a person is learnt when presented with multiple known samples of that person- where the variation and similarities within a particular person are learnt. The person-specific learning strategy is seen to perform better than general learning (5% higher performace with signatures). Improvement of person-specific performance with increasing number of samples is also observed.
Infinite Hidden Markov Models via the Hierarchical Dirichlet Process

In this presentation, we propose a new formalism under which we study the infinite hidden Markov model (iHMM) of Beal et al. . The iHMM is a hidden Markov model (HMM) in which the number of hidden states is allowed to be countably infinite. This is achieved using the formalism of the Dirichlet process. In particular, a two-level urn model is used to determine the transition probabilities of the HMM (an urn model is a particular aspect of the Dirichlet process.) At the first level, the probability of transitioning from a state u to a state v is proportional to the number of times the same transition is observed at other time steps, while with probability proportional to α0 an “oracle” process is invoked. At this second level, the probability of transitioning to state v is proportional to the number of times state v has been chosen by the oracle (regardless of the previous state), while the probability of transitioning to a previously unseen (i.e. novel) state is proportional to γ. The inclusion of the oracle was motivated by a need to tie together the transition models to have common destination states. Beal et al. presented an approximate Gibbs sampling algorithm for inference in this model, but without an explicit generative model the scope of the model was limited. A fresh aspect on the iHMM is provided by the recent development of the Hierarchical Dirichlet Process (HDP) of Teh et al. . The HDP framework considers problems involving related groups of data. In particular, each group of data is modelled by a Dirichlet process (DP) mixture model, with the common base measure of the DPs being itself distributed according to a global DP. This is a hierarchical Bayesian model, and is necessary to ensure that the different DP mixtures are able to share a common set of mixture components. Note that Beal et al. defined a notion of “hierarchical Dirichlet process”, but this is not hierarchical in the Bayesian sense—involving a distribution on the parameters of a Dirichlet process—but is instead a description of the two level urn model. Nevertheless, it turns out that it is possible to formulate the iHMM within the HDP framework of Teh et al.
Tree of Latent Mixtures for Bayesian Modelling and Classification of High Dimensional Data

Many domains of interest to machine learning, such as audio and video, computational biology, climate modelling, and quantitative finance, involve very high dimensional data. Such data are often characterized by statistical structure that includes correlations on multiple scales. Attempts to model those high dimensional data raise problems that are much less significant in low dimensions. This paper presents a novel approach to modelling and classification in high dimensions. The approach is based on a graphical model with a tree structure, which performs density estimation on multiple spatial scales. Exact inference in this model is computationally intractable; two algorithms for approximate inference using variational techniques are derived and analyzed. We discuss learning and classification using these algorithms, and demonstrate their performance on a handwritten digit recognition task.
Sharing Clusters among Related Groups: Hierarchical Dirichlet Processes

We propose the hierarchical Dirichlet process (HDP), a nonparametric Bayesian model for clustering problems involving multiple groups of data. Each group of data is modeled with a mixture, with the number of components being open-ended and inferred automatically by the model. Further, components can be shared across groups, allowing dependencies across groups to be modeled effectively as well as conferring generalization to new groups. Such grouped clustering problems occur often in practice, e.g. in the problem of topic discovery in document corpora. We report experimental results on three text corpora showing the effective and superior performance of the HDP over previous models.
Variational algorithms for approximate Bayesian inference

The Bayesian framework for machine learning allows for the incorporation of prior knowledge in a coherent way, avoids overfitting problems, and provides a principled basis for selecting between alternative models. Unfortunately the computations required are usually intractable. This thesis presents a unified variational Bayesian (VB) framework which approximates these computations in models with latent variables using a lower bound on the marginal likelihood. Chapter 1 presents background material on Bayesian inference, graphical models, and propagation algorithms. Chapter 2 forms the theoretical core of the thesis, generalising the expectationmaximisation (EM) algorithm for learning maximum likelihood parameters to the VB EM algorithm which integrates over model parameters. The algorithm is then specialised to the large family of conjugate-exponential (CE) graphical models, and several theorems are presented to pave the road for automated VB derivation procedures in both directed and undirected graphs (Bayesian and Markov networks, respectively). Chapters 3-5 derive and apply the VB EM algorithm to three commonly-used and important models: mixtures of factor analysers, linear dynamical systems, and hidden Markov models. It is shown how model selection tasks such as determining the dimensionality, cardinality, or number of variables are possible using VB approximations. Also explored are methods for combining sampling procedures with variational approximations, to estimate the tightness of VB bounds and to obtain more effective sampling algorithms. Chapter 6 applies VB learning to a long-standing problem of scoring discrete-variable directed acyclic graphs, and compares the performance to annealed importance sampling amongst other methods. Throughout, the VB approximation is compared to other methods including sampling, Cheeseman-Stutz, and asymptotic approximations such as BIC. The thesis concludes with a discussion of evolving directions for model selection including infinite models and alternative approximations to the marginal likelihood.
A Graphical Model for Audiovisual Object Tracking

We present a new approach to modeling and processing multimedia data. This approach is based on graphical models that combine audio and video variables. We demonstrate it by developing a new algorithm for tracking a moving object in a cluttered, noisy scene using two microphones and a camera. Our model uses unobserved variables to describe the data in terms of the process that generates them. It is therefore able to capture and exploit the statistical structure of the audio and video data separately, as well as their mutual dependencies. Model parameters are learned from data via an EM algorithm, and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location from data. We demonstrate successful performance on multimedia clips captured in real world scenarios using off-the-shelf equipment.
Inferring State Sequences for Non-linear Systems with Embedded Hidden Markov Models

We describe a Markov chain method for sampling from the distribution of the hidden state sequence in a non-linear dynamical system, given a sequence of observations. This method updates all states in the sequence simultaneously using an embedded Hidden Markov Model (HMM). An update begins with the creation of "pools" of candidate states at each time. We then define an embedded HMM whose states are indexes within these pools. Using a forward-backward dynamic programming algorithm, we can efficiently choose a state sequence with the appropriate probabilities from the exponentially large number of state sequences that pass through states in these pools. We illustrate the method in a simple one-dimensional example, and in an example showing how an embedded HMM can be used to in effect discretize the state space without any discretization error. We also compare the embedded HMM to a particle smoother on a more substantial problem of inferring human motion from 2D traces of markers.
A self-calibrating algorithm for speaker tracking based on audio-visual statistical models

We present a self-calibrating algorithm for audio-visual tracking using two microphones and a camera. The algorithm uses a parametrized statistical model which combines simple models of video and audio. Using unobserved variables, the model describes the process that generates the observed data. Hence, it is able to capture and exploit the statistical structure of the audio and video data, as well as their mutual dependencies, The model parameters are estimated by the EM algorithm; object templates are learned and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location using the model. Successful performance is demonstrated on real multimedia clips.
Audio-Video Sensor Fusion with Probabilistic Graphical Models

We present a new approach to modeling and processing multimedia data. This approach is based on graphical models that combine audio and video variables. We demonstrate it by developing a new algorithm for tracking a moving object in a cluttered, noisy scene using two microphones and a camera. Our model uses unobserved variables to describe the data in terms of the process that generates them. It is therefore able to capture and exploit the statistical structure of the audio and video data separately, as well as their mutual dependencies. Model parameters are learned from data via an EM algorithm, and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location from data. We demonstrate successful performance on multimedia clips captured in real world scenarios using off-the-shelf equipment.
A direct method for modeling non-rigid motion with thin plate spline

Thin plate spline (TPS) transformations have been applied to non-rigid shape matching with impressive results. However, existing methods often use a sparse set of point correspondences which are established prior to shape matching. A straightforward approach to finding point correspondences and computing TPS parameters imposes expensive computations, thereby motivating us to develop an efficient solution. In this paper, we present a direct method for recovering non-rigid object motion from its appearance in which the point correspondences are simultaneously established while estimating TPS parameters. The motion parameters are estimated in a stiff-to-flexible approach and the principal appearance deformations are learned that can be utilized for motion analysis and recognition. Numerous experiments demonstrate the efficiency and efficacy of the proposed algorithm in modeling the motion details of non-rigid objects undergoing shape deformation and pose variation.