nils-herrmann/clean_scirepeval huggingface dataset:
Learning Graphical Models of Images, Videos and Their Spatial Transformations

Mixtures of Gaussians, factor analyzers (probabilistic PCA) and hidden Markov models are staples of static and dynamic data modeling and image and video modeling in particular. We show how topographic transformations in the input, such as translation and shearing in images, can be accounted for in these models by including a discrete transformation variable. The resulting models perform clustering, dimensionality reduction and time-series analysis in a way that is invariant to transformations in the input. Using the EM algorithm, these transformation invariant models can be fit to static data and time series. We give results on filtering microscopy images, face and facial pose clustering, handwritten digit modeling and recognition, video clustering, object tracking, and removal of distractions from video sequences.
Transformed hidden Markov models: estimating mixture models of images and inferring spatial transformations in video sequences

In this paper we describe a novel generative model for video analysis called the transformed hidden Markov model (THMM). The video sequence is modeled as a set of frames generated by transforming a small number of class images that summarize the sequence. For each frame, the transformation and the class are discrete latent variables that depend on the previous class and transformation in the sequence. The set of possible transformations is defined in advance, and it can include a variety of transformation such as translation, rotation and shearing. In each stage of such a Markov model, a new frame is generated from a transformed Gaussian distribution based on the class/transformation combination generated by the Markov chain. This model can be viewed as an extension of a transformed mixture of Gaussians through time. We use this model to cluster unlabeled video segments and form a video summary in an unsupervised fashion. We also use the trained models to perform tracking, image stabilization and filtering. We demonstrate that the THMM is capable of combining long term dependencies in video sequences (repeating similar frames in remote parts of the sequence) with short term dependencies (such as short term image frame similarities and motion patterns) to better summarize and process a video sequence even in the presence of high levels of white or structured noise (such as foreground occlusion).
Time-series classification using mixed-state dynamic Bayesian networks

We present a novel mixed-state dynamic Bayesian network (DBN) framework for modeling and classifying time-series data such as object trajectories. A hidden Markov model (HMM) of discrete actions is coupled with a linear dynamical system (LDS) model of continuous trajectory motion. This combination allows us to model both the discrete and continuous causes of trajectories such as human gestures. The model is derived using a rich theoretical corpus from the Bayesian network literature. This allows us to use an approximate structured variational inference technique to solve the otherwise intractable inference of action and system states. Using the same DBN framework we show how to learn the mixed-state model parameters from data. Experiments show that with high statistical confidence the mixed-state DBNs perform favorably when compared to decoupled HMM/LDS models on the task of recognizing human gestures made with a computer mouse.
Local Probability Propagation for Factor Analysis

Ever since Pearl's probability propagation algorithm in graphs with cycles was shown to produce excellent results for error-correcting decoding a few years ago, we have been curious about whether local probability propagation could be used successfully for machine learning. One of the simplest adaptive models is the factor analyzer, which is a two-layer network that models bottom layer sensory inputs as a linear combination of top layer factors plus independent Gaussian sensor noise. We show that local probability propagation in the factor analyzer network usually takes just a few iterations to perform accurate inference, even in networks with 320 sensors and 80 factors. We derive an expression for the algorithm's fixed point and show that this fixed point matches the exact solution in a variety of networks, even when the fixed point is unstable. We also show that this method can be used successfully to perform inference for approximate EM and we give results on an online face recognition task.
Estimating mixture models of images and inferring spatial transformations using the EM algorithm

Mixture modeling and clustering algorithms are effective, simple ways to represent images using a set of data centers. However, in situations where the images include background clutter and transformations such as translation, rotation, shearing and warping, these methods extract data centers that include clutter and represent different transformations of essentially the same data. Taking face images as an example, it would be more useful for the different clusters to represent different poses and expressions, instead of cluttered versions of different translations, scales and rotations. By including clutter and transformation as unobserved, latent variables in a mixture model, we obtain a new "transformed mixture of Gaussians", which is invariant to a specified set of transformations. We show how a linear-time EM algorithm can be used to fit this model by jointly estimating a mixture model for the data and inferring the transformation for each image. We show that this algorithm can jointly align images of a human head and learn different poses. We also find that the algorithm performs better than k-nearest neighbors and mixtures of Gaussians on handwritten digit recognition.
Embedded face and facial expression recognition

A framework for embedded recognition of faces and facial expressions is described. Faces are modeled based on the appearances and positions of facial features. Hidden states are used to represent discrete facial expressions. A face model is constructed for each person in the database using video segments showing different facial expressions. Face recognition and facial expression recognition are carried out using Bayesian classification. In our current implementation, the face is divided into nine facial features grouped in four regions which are detected and tracked automatically in video segments. We report results on face and facial expression recognition using a video database of 18 people and six expressions.
Variational Learning in Nonlinear Gaussian Belief Networks

We view perceptual tasks such as vision and speech recognition as inference problems where the goal is to estimate the posterior distribution over latent variables (e.g., depth in stereo vision) given the sensory input. The recent flurry of research in independent component analysis exemplifies the importance of inferring the continuous-valued latent variables of input data. The latent variables found by this method are linearly related to the input, but perception requires nonlinear inferences such as classification and depth estimation. In this article, we present a unifying framework for stochastic neural networks with nonlinear latent variables. Nonlinear units are obtained by passing the outputs of linear gaussian units through various nonlinearities. We present a general variational method that maximizes a lower bound on the likelihood of a training set and give results on two visual feature extraction problems. We also show how the variational method can be used for pattern classification and compare the performance of these nonlinear networks with other methods on the problem of handwritten digit recognition.
Variational Learning in Mixed-State Dynamic Graphical Models

Many real-valued stochastic time-series are locally linear (Gassian), but globally non-linear. For example, the trajectory of a human hand gesture can be viewed as a linear dynamic system driven by a nonlinear dynamic system that represents muscle actions. We present a mixed-state dynamic graphical model in which a hidden Markov model drives a linear dynamic system. This combination allows us to model both the discrete and continuous causes of trajectories such as human gestures. The number of computations needed for exact inference is exponential in the sequence length, so we derive an approximate variational inference technique that can also be used to learn the parameters of the discrete and continuous models. We show how the mixed-state model and the variational technique can be used to classify human hand gestures made with a computer mouse.
Detection and tracking of faces and facial features

We describe a real-time system for face and facial feature detection and tracking in continuous video. The core of this system consists of a set of novel facial feature detectors based on our previously proposed information-based maximum discrimination learning technique. These classifiers are very fast and allow us to implement a fully automatic, real-time system for detection and tracking multiple faces. In addition to locking onto up to four target faces, this system locates and tracks nine facial features as they move under facial expression changes.
A probabilistic framework for embedded face and facial expression recognition

We present a Bayesian recognition framework in which a model of the whole face is enhanced by models of facial feature position and appearances. Face recognition and facial expression recognition are carried out using maximum likelihood decisions. The algorithm finds the model and facial expression that maximizes the likelihood of a test image. In this framework, facial appearance matching is improved by facial expression matching. Also, changes in facial features due to expressions are used together with facial deformation. Patterns to jointly perform expression recognition. In our current implementation, the face is divided into 9 facial features grouped in 4 regions which are detected and tracked automatically in video segments. The feature images are modeled using Gaussian distributions on a principal component sub-space. The training procedure is supervised; we use video segments of people in which the facial expressions have been segmented and labeled by hand. We report results on face and facial expression recognition using a video database of 18 people and 6 expressions.
Transformed component analysis: joint estimation of spatial transformations and image components

A simple, effective way to model images is to represent each input pattern by a linear combination of "component" vectors, where the amplitudes of the vectors are modulated to match the input. This approach includes principal component analysis, independent component analysis and factor analysis. In practice, images are subjected to randomly selected transformations of a known nature, such as translation and rotation. Direct use of the above methods will lead to severely blurred components that tend to ignore the more interesting and useful structure. In previous work, we introduced a clustering algorithm that is invariant to transformations. In this paper, we propose a method called transformed component analysis, which incorporates a discrete, hidden variable that accounts for transformations and uses the expectation maximization algorithm to jointly extract components and normalize for transformations. We illustrate the algorithm using a shading problem, facial expression modeling and written digit recognition.
Topographic Transformation as a Discrete Latent Variable

Invariance to topographic transformations such as translation and shearing in an image has been successfully incorporated into feed-forward mechanisms, >e.g., "convolutional neural networks", "tangent propagation". We describe a way to add transformation invariance to a generative density model by approximating the nonlinear transformation manifold by a discrete set of transformations. An EM algorithm for the original model can be extended to the new model by computing expectations over the set of transformations. We show how to add a discrete transformation variable to Gaussian mixture modeling, factor analysis and mixtures of factor analysis. We give results on filtering microscopy images, face and facial pose clustering, and handwritten digit modeling and recognition.
Factor Analysis Using Batch and Online EM

A simple way to encode input patterns is to suppose that each input can be well-approximated by a linear combination of component vectors, where the amplitudes of the vectors are modulated to match the input. For a given training set, the most appropriate set of component vectors will depend on how we expect the modulation levels to behave and how we measure the distance between the input and its approximation. These effects can be captured by a generative probability model that specifies a distribution p(z) over modulation levels z = (z1; : : : ; zK) and a distribution p(xjz) over sensors x = (x1; : : : ; xN) given the modulation levels. The linear combination constraint is given by E[xjz]= Λz; (1)
Turbo Factor Analysis

A few years ago, Pearl’s probability propagation algorithm in graphs with cycles was shown to produce excellent results for error-correcting “turbodecoding”. Ever since, we have wondered whether iterative probability propagation could be used successfully for machine learning. As a first step in this direction, we study iterative inference and learning in the simple factor analyzer network – a two-layer densely connected network that models bottom layer sensory inputs as a linear combination of top layer factors plus independent Gaussian sensor noise. The number of bottom-up/top-down iterations needed to exactly infer the factors given a network and an input scales with the number of factors in the top layer. In online learning, this iterative procedure must be reinitialized upon each pattern presentation and so learning becomes prohibitively slow in big networks, such as those used for face recognition and for large-scale models of the cortex. We show that probability propagation in a factor analyzer usually takes just a few iterations to achieve a low inference error, even in networks with 320 sensors and 80 factors. We derive an expression for the algorithm’s fixed point and provide an eigenvalue condition for global convergence. We also show how iterative inference can be used to do online learning and give results on using this method to do online dimensionality reduction for the purpose of recognizing 560-pixel images of faces using a 40-dimensional subspace. This work suggests that iterative probability propagation in densely connected networks may lead to a broad class of useful algorithms for machine learning. UW/CS Adaptive Computation TR-99-1,  Submitted to Neural Computation, May 25, 1999.
Probabilistic multimedia objects (multijects): a novel approach to video indexing and retrieval in multimedia systems

This paper proposes a novel scheme for bridging the gap between low level media features and high level semantics using a probabilistic framework. We propose a framework, in which scenes can be indexed at a semantic level. The fundamental components of the framework are sites, objects and events. Detection of presence of an instance of one of these influences the probability of the presence of instances within other classes. Detection of instances is done using probabilistic multimedia objects: multijects. Indexing using multijects can handle queries posed at semantic level. multijects are built in a Markovian framework. Two ways of building the multijects from low level features fusing features from multiple modalities are presented. A probabilistic framework is also envisioned to encode the higher level relationship between multijects, which enhances or reduces the probabilities of concurrent existence of various multijects. An actual implementation is presented by developing multijects representing the higher level concept of "explosion" and "waterfall". The models are evaluated by using the multijects to detect explosions and waterfalls in movies. Results reveal, that the multijects detect the aforementioned events with greater accuracy and are able to segment the video into scenes which have explosions and waterfalls.
Early Detection and Trellis Splicing: Reduced-Complexity Iterative Decoding

The bit-error rate (BER) performance of new iterative decoding algorithms (e,g,, turbodecoding) is achieved at the expense of a computationally burdensome decoding procedure. We present a method called early detection that can be used to reduce the computational complexity of a variety of iterative decoders. Using a confidence criterion, some information symbols, state variables, and codeword symbols are detected early on during decoding. In this way, the computational complexity of further processing is reduced with a controllable increase in the BER. We present an easily implemented instance of this algorithm, called trellis splicing, that can be used with turbodecoding. For a simulated system of this type, we obtain a reduction in the computational complexity of up to a factor of four, relative to a turbodecoder that obtains the same increase in the BER by performing fewer iterations.
Skewness and pseudocodewords in iterative decoding

Impressive performances by low-density parity-check codes and turbo codes have inspired researchers to determine how their iterative decoders depart from optimal decoding. Iterative decoding can be viewed as the application of the sum-product or min-sum algorithm in a graph that describes the constraints on a system of variables. The authors discuss here how iterative decoding maximizes correlation, and skewness and pseudocodewords in the decoding tree.
Mixtures of local linear subspaces for face recognition

Traditional subspace methods for face recognition compute a measure of similarity between images after projecting them onto a fixed linear subspace that is spanned by some principal component vectors (a.k.a. "eigenfaces") of a training set of images. By supposing a parametric Gaussian distribution over the subspace and a symmetric Gaussian noise model for the image given a point in the subspace, we can endow this framework with a probabilistic interpretation so that Bayes-optimal decisions can be made. However, we expect that different image clusters (corresponding, say, to different poses and expressions) will be best represented by different subspaces. In this paper, we study the recognition performance of a mixture of local linear subspaces model that can be fit to training data using the expectation maximization algorithm. The mixture model outperforms a nearest-neighbor classifier that operates in a PCA subspace.
Early Detection and Trellis Splicing Reduced Complexity Soft Iterative Decoding

The excellent bit error rate performance of new soft iter ative decoding algorithms eg turbo codes is achieved at the expense of a computationally burdensome itera tive decoding procedure In this paper we present a new method called early detection that can be used to reduce the computational complexity of a variety of soft iterative decoding methods Using a con dence criterion some in formation symbols state variables and codeword symbols are detected early on in the iterative decoding procedure leading to a reduction in the computational complexity of further processing After presenting a general Markov ran dom eld framework eg the Tanner graph for compound codes we use this framework to show how early detection leads to computational savings We then present an easily implemented instance of this algorithm called trellis splic ing that can be used with turbo codes For a simulated turbo code system at low BERs we obtain a reduction in computational complexity of over a factor of four relative to conventional turbo decoding without any increase in BER
Markovian inference in belief networks

Bayesian belief networks can represent the complicated probabilistic processes that form natural sensory inputs. Once the parameters of the network have been learned, nonlinear inferences about the input can be made by computing the posterior distribution over the hidden units (e.g., depth in stereo vision) given the input. Computing the posterior distribution exactly is not practical in richly-connected networks, but it turns out that by using a variational (a.k.a., mean field) method, it is easy to find a product-form distribution that approximates the true posterior distribution. This approximation assumes that the hidden variables are independent given the current input. In this paper, we explore a more powerful variational technique that models the posterior distribution using a Markov chain. We compare this method with inference using mean fields and mixtures of mean fields in randomly generated networks. Submitted to NIP98, Algorithms and Architectures, oral presentation.
Iterative Decoding of Compound Codes by Probability Propagation in Graphical Models

We present a unified graphical model framework for describing compound codes and deriving iterative decoding algorithms. After reviewing a variety of graphical models (Markov random fields, Tanner graphs, and Bayesian networks), we derive a general distributed marginalization algorithm for functions described by factor graphs. From this general algorithm, Pearl's (1986) belief propagation algorithm is easily derived as a special case. We point out that iterative decoding algorithms for various codes, including "turbo decoding" of parallel-concatenated convolutional codes, may be viewed as probability propagation in a graphical model of the code. We focus on Bayesian network descriptions of codes, which give a natural input/state/output/channel description of a code and channel, and we indicate how iterative decoders can be developed for parallel-and serially concatenated coding systems, product codes, and low-density parity-check codes.
A simple algorithm that discovers efficient perceptual codes

We describe the \wake-sleep" algorithm that allows a multilayer, unsupervised, neural network to build a hierarchy of representations of sensory input. The network has bottom-up \recognition" connections that are used to convert sensory input into underlying representations. Unlike most arti cial neural networks, it also has top-down \generative" connections that can be used to reconstruct the sensory input from the representations. In the \wake" phase of the learning algorithm, the network is driven by the bottom-up recognition connections and the top-down generative connections are trained to be better at reconstructing the sensory input from the representation chosen by the recognition process. In the \sleep" phase, the network is driven top-down by the generative connections to produce a fantasized representation and a fantasized sensory input. The recognition connections are then trained to be better at recovering the fantasized representation from the fantasized sensory input. In both phases, the synaptic learning rule is simple and local. The combined e ect of the two phases is to create representations of the sensory input that are e cient in the following sense: On average, it takes more bits to describe each sensory input vector directly than to rst describe the representation of the sensory input chosen by the recognition process and then describe the di erence between the sensory input and its reconstruction from the chosen representation.