the helmholtz machine discovering the structure inherent in a set of patterns is a fundamental aim of statistical inference or learning one fruitful approach is to build a parameterized stochastic generative model independent draws from which are likely to produce the patterns for all but the simplest generative models each pattern can be generated in exponentially many ways it is thus intractable to adjust the parameters to maximize the probability of the observed patterns we describe a way of finessing this combinatorial explosion by maximizing an easily computed lower bound on the probability of the observations our method can be viewed as a form of hierarchical self supervised learning that may relate to the function of bottom up and top down cortical processing pathways

message passing for maximum weight independent set in this paper we investigate the use of message passing algorithms for the problem of finding the max weight independent set mwis in a graph first we study the performance of the classical loopy max product belief propagation we show that each fixed point estimate of max product can be mapped in a natural way to an extreme point of the linear programming lp polytope associated with the mwis problem however this extreme point may not be the one that maximizes the value of node weights the particular extreme point at final convergence depends on the initialization of max product we then show that if max product is started from the natural initialization of uninformative messages it always solves the correct lp if it converges this result is obtained via a direct analysis of the iterative algorithm and cannot be obtained by looking only at fixed points the tightness of the lp relaxation is thus necessary for max product optimality but it is not sufficient motivated by this observation we show that a simple modification of max product becomes gradient descent on a smoothed version of the dual of the lp and converges to the dual optimum we also develop a message passing algorithm that recovers the primal mwis solution from the output of the descent algorithm we show that the mwis estimate obtained using these two algorithms in conjunction is correct when the graph is bipartite and the mwis is unique finally we show that any problem of maximum a posteriori map estimation for probability distributions over finite domains can be reduced to an mwis problem we believe this reduction will yield new insights and algorithms for map estimation

the generalized distributive law we discuss a general message passing algorithm which we call the generalized distributive law gdl the gdl is a synthesis of the work of many authors in information theory digital communications signal processing statistics and artificial intelligence it includes as special cases the baum welch algorithm the fast fourier transform fft on any finite abelian group the gallager tanner wiberg decoding algorithm viterbi s algorithm the bcjr algorithm pearl s belief propagation algorithm the shafer shenoy probability propagation algorithm and the turbo decoding algorithm although this algorithm is guaranteed to give exact answers only in certain cases the junction tree condition unfortunately not including the cases of gtw with cycles or turbo decoding there is much experimental evidence and a few theorems suggesting that it often works approximately even when it is not supposed to

improving variational methods via pairwise linear response identities inference methods are often formulated as variational approximations these approximations allow easy evaluation of statistics by marginalization or linear response but these estimates can be inconsistent we show that by introducing constraints on covariance one can ensure consistency of linear response with the variational parameters and in so doing inference of marginal probability distributions is improved for the bethe approximation and its generalizations improvements are achieved with simple choices of the constraints the approximations are presented as variational frameworks iterative procedures related to message passing are provided for finding the minima

simple deterministically constructed cycle reservoirs with regular jumps a new class of state space models reservoir models with a fixed state transition structure the reservoir and an adaptable readout from the state space has recently emerged as a way for time series processing and modeling echo state network esn is one of the simplest yet powerful reservoir models esn models are generally constructed in a randomized manner in our previous study rodan ti o 2011 we showed that a very simple cyclic deterministically generated reservoir can yield performance competitive with standard esn in this contribution we extend our previous study in three aspects first we introduce a novel simple deterministic reservoir model cycle reservoir with jumps crj with highly constrained weight values that has superior performance to standard esn on a variety of temporal tasks of different origin and characteristics second we elaborate on the possible link between reservoir characterizations such as eigenvalue distribution of the reservoir matrix or pseudo lyapunov exponent of the input driven reservoir dynamics and the model performance it has been suggested that a uniform coverage of the unit disk by such eigenvalues can lead to superior model performance we show that despite highly constrained eigenvalue distribution crj consistently outperforms esn which has much more uniform eigenvalue coverage of the unit disk also unlike in the case of esn pseudo lyapunov exponents of the selected optimal crj models are consistently negative third we present a new framework for determining the short term memory capacity of linear reservoir models to a high degree of precision using the framework we study the effect of shortcut connections in the crj reservoir topology on its memory capacity

improved belief propagation algorithm finds many bethe states in the random field ising model on random graphs we first present an empirical study of the belief propagation bp algorithm when run on the random field ising model defined on random regular graphs in the zero temperature limit we introduce the notion of extremal solutions for the bp equations and we use them to fix a fraction of spins in their ground state configuration at the phase transition point the fraction of unconstrained spins percolates and their number diverges with the system size this in turn makes the associated optimization problem highly non trivial in the critical region using the bounds on the bp messages provided by the extremal solutions we design a new and very easy to implement bp scheme which is able to output a large number of stable fixed points on one hand this new algorithm is able to provide the minimum energy configuration with high probability in a competitive time on the other hand we found that the number of fixed points of the bp algorithm grows with the system size in the critical region this unexpected feature poses new relevant questions about the physics of this class of models

belief propagation algorithm and the ising model on networks with arbitrary distributions of motifs we generalize the belief propagation algorithm to sparse random networks with arbitrary distributions of motifs triangles loops etc each vertex in these networks belongs to a given set of motifs generalization of the configuration model these networks can be treated as sparse uncorrelated hypergraphs in which hyperedges represent motifs here a hypergraph is a generalization of a graph where a hyperedge can connect any number of vertices these uncorrelated hypergraphs are treelike hypertrees which crucially simplifies the problem and allows us to apply the belief propagation algorithm to these loopy networks with arbitrary motifs as natural examples we consider motifs in the form of finite loops and cliques we apply the belief propagation algorithm to the ferromagnetic ising model with pairwise interactions on the resulting random networks and obtain an exact solution of this model we find an exact critical temperature of the ferromagnetic phase transition and demonstrate that with increasing the clustering coefficient and the loop size the critical temperature increases compared to ordinary treelike complex networks however weak clustering does not change the critical behavior qualitatively our solution also gives the birth point of the giant connected component in these loopy networks

inference by belief propagation in composite systems we devise a message passing algorithm for probabilistic inference in composite systems consisting of a large number of variables that exhibit weak random interactions among all variables and strong interactions with a small subset of randomly chosen variables the relative strength of the two interactions is controlled by a free parameter we examine the performance of the algorithm numerically on a number of systems of this type for varying mixing parameter values

cccp algorithms to minimize the bethe and kikuchi free energies convergent alternatives to belief propagation this article introduces a class of discrete iterative algorithms that are provably convergent alternatives to belief propagation bp and generalized belief propagation gbp our work builds on recent results by yedidia freeman and weiss 2000 who showed that the fixed points of bp and gbp algorithms correspond to extrema of the bethe and kikuchi free energies respectively we obtain two algorithms by applying cccp to the bethe and kikuchi free energies respectively cccp is a procedure introduced here for obtaining discrete iterative algorithms by decomposing a cost function into a concave and a convex part we implement our cccp algorithms on two and three dimensional spin glasses and compare their results to bp and gbp our simulations show that the cccp algorithms are stable and converge very quickly the speed of cccp is similar to that of bp and gbp unlike cccp bp will often not converge for these problems gbp usually but not always converges the results found by cccp applied to the bethe or kikuchi free energies are equivalent or slightly better than those found by bp or gbp respectively when bp and gbp converge note that for these and other problems bp and gbp give very accurate results see yedidia et al 2000 and failure to converge is their major error mode finally we point out that our algorithms have a large range of inference and learning applications

combinatorial resampling particle filter an effective and efficient method for articulated object tracking particle filter pf is a method dedicated to posterior density estimations using weighted samples whose elements are called particles in particular this approach can be applied to object tracking in video sequences in complex situations and in this paper we focus on articulated object tracking i e objects that can be decomposed as a set of subparts one of pf s crucial step is a resampling step in which particles are resampled to avoid degeneracy problems in this paper we propose to exploit mathematical properties of articulated objects to swap conditionally independent subparts of the particles in order to generate new particle sets we then introduce a new resampling method called combinatorial resampling that resamples over the particle set resulting from all the admissible swappings the so called combinatorial set in essence combinatorial resampling cr is quite similar to the combination of a crossover operator and a usual resampling but there exists a fundamental difference between cr and the use of crossover operators we prove that cr is sound i e in a bayesian framework it is guaranteed to represent without any bias the posterior densities of the states over time by construction the particle sets produced by cr better represent the density to estimate over the whole state space than the original set and therefore cr produces higher quality samples unfortunately the combinatorial set is generally of an exponential size and therefore to be scalable we show how it can be implicitly constructed and resampled from thus resulting in both an efficient and effective resampling scheme finally through experimentations both on challenging synthetic and real video sequences we also show that our resampling method outperforms all classical resampling methods both in terms of the quality of its results and in terms of computation times

log determinant relaxation for approximate inference in discrete markov random fields graphical models are well suited to capture the complex and non gaussian statistical dependencies that arise in many real world signals a fundamental problem common to any signal processing application of a graphical model is that of computing approximate marginal probabilities over subsets of nodes this paper proposes a novel method applicable to discrete valued markov random fields mrfs on arbitrary graphs for approximately solving this marginalization problem the foundation of our method is a reformulation of the marginalization problem as the solution of a low dimensional convex optimization problem over the marginal polytope exactly solving this problem for general graphs is intractable for binary markov random fields we describe how to relax it by using a gaussian bound on the discrete entropy and a semidefinite outer bound on the marginal polytope this combination leads to a log determinant maximization problem that can be solved efficiently by interior point methods thereby providing approximations to the exact marginals we show how a slightly weakened log determinant relaxation can be solved even more efficiently by a dual reformulation when applied to denoising problems in a coupled mixture of gaussian model defined on a binary mrf with cycles we find that the performance of this log determinant relaxation is comparable or superior to the widely used sum product algorithm over a range of experimental conditions

energy efficient distributed filtering with wireless sensor networks we consider a wireless sensor network wsn that monitors a physical field and communicates pertinent data to a distant fusion center fc we study the case of a binary valued hidden natural field observed in a significant amount of gaussian clutter which is relevant to applications like detection of plumes or oil slicks the considerable spatio temporal dependencies found in natural fields can be exploited to improve the reliability of the detection estimation of hidden phenomena while this problem has been previously treated using kernel regression techniques we formulate it as a task of delay free filtering on a process observed by the sensors we propose a distributed scalable implementation of the filter within the network this is achieved by i exploiting the localized spatio temporal dependencies to define a hidden markov model hmm in terms of an exponential family with o n parameters where n is the size of the wsn ii using a reduced state approximation of the propagated probability mass function and iii making a tractable approximation of model marginals by using iterated decoding algorithms like the gibbs sampler gs mean field decoding mfd iterated conditional modes icm and broadcast belief propagation bbp we compare the marginalization algorithms in terms of their information geometry performance complexity and communication load finally we analyze the energy efficiency of the proposed distributed filter relative to brute force data fusion it is demonstrated that when the fc is sufficiently far away from the sensor array distributed filtering is significantly more energy efficient and can increase the lifetime of the wsn by one to two orders of magnitude

finite alphabet iterative decoders part i decoding beyond belief propagation on the binary symmetric channel we introduce a new paradigm for finite precision iterative decoding on low density parity check codes over the binary symmetric channel the messages take values from a finite alphabet and unlike traditional quantized decoders which are quantized versions of the belief propagation bp decoder the proposed finite alphabet iterative decoders faids do not propagate quantized probabilities or log likelihoods and the variable node update functions do not mimic the bp decoder rather the update functions are maps designed using the knowledge of potentially harmful subgraphs that could be present in a given code thereby rendering these decoders capable of outperforming the bp in the error floor region on certain column weight three codes of practical interest we show that there exist faids that surpass the floating point bp decoder in the error floor region while requiring only three bits of precision for the representation of the messages hence faids are able to achieve a superior performance at much lower complexity we also provide a methodology for the selection of faids that is not code specific but gives a set of candidate faids containing potentially good decoders in the error floor region for any column weight three code we validate the code generality of our methodology by providing particularly good three bit precision faids for a variety of codes with different rates and lengths

p beyond move making algorithms for solving higher order functions in this paper we extend the class of energy functions for which the optimal alpha expansion and alphabeta swap moves can be computed in polynomial time specifically we introduce a novel family of higher order clique potentials and show that the expansion and swap moves for any energy function composed of these potentials can be found by minimizing a submodular function we also show that for a subset of these potentials the optimal move can be found by solving an st mincut problem we refer to this subset as the pn potts model our results enable the use of powerful alpha expansion and alphabeta swap move making algorithms for minimization of energy functions involving higher order cliques such functions have the capability of modeling the rich statistics of natural scenes and can be used for many applications in computer vision we demonstrate their use in one such application i e the texture based image or video segmentation problem

constructing free energy approximations and generalized belief propagation algorithms important inference problems in statistical physics computer vision error correcting coding theory and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs the belief propagation bp algorithm is an efficient way to solve these problems that is exact when the factor graph is a tree but only approximate when the factor graph has cycles we show that bp fixed points correspond to the stationary points of the bethe approximation of the free energy for a factor graph we explain how to obtain region based free energy approximations that improve the bethe approximation and corresponding generalized belief propagation gbp algorithms we emphasize the conditions a free energy approximation must satisfy in order to be a valid or maxent normal approximation we describe the relationship between four different methods that can be used to generate valid approximations the bethe method the junction graph method the cluster variation method and the region graph method finally we explain how to tell whether a region based approximation and its corresponding gbp algorithm is likely to be accurate and describe empirical results showing that gbp can significantly outperform bp

enumerators for protograph based ensembles of ldpc and generalized ldpc codes protograph based ldpc and generalized ldpc g ldpc codes have the advantages of a simple design procedure and highly structured encoders and decoders the design of such protograph based codes relies on what is effectively a computer based search as such following gallager it is prudent to restrict the search to a good ensemble for example an ensemble whose minimum distance grows linearly with codeword length a good ensemble can also mean one with good stopping set trapping set or pseudocodeword properties in this paper ensemble codeword weight enumerators for finite length ldpc and g ldpc codes based on protographs were derived and then the asymptotic case was considered the asymptotic results allow us to determine whether or not the typical relative minimum distance in the ensemble grows linearly with codeword length then the codeword weight enumerator technique is adapted to yield ensemble stopping set trapping set and pseudocodeword enumerators for protograph ldpc and g ldpc codes in this case the asymptotic results allow us to determine whether or not the typical relative smallest stopping set size trapping set size and pseudoweight grows linearly with codeword length trapping set enumerators for g ldpc code ensembles represent a more complex problem which we do not consider here

cortical circuitry implementing graphical models in this letter we develop and simulate a large scale network of spiking neurons that approximates the inference computations performed by graphical models unlike previous related schemes which used sum and product operations in either the log or linear domains the current model uses an inference scheme based on the sum and maximization operations in the log domain simulations show that using these operations a large scale circuit which combines populations of spiking neurons as basic building blocks is capable of finding close approximations to the full mathematical computations performed by graphical models within a few hundred milliseconds the circuit is general in the sense that it can be wired for any graph structure it supports multistate variables and it uses standard leaky integrate and fire neuronal units following previous work which proposed relations between graphical models and the large scale cortical anatomy we focus on the cortical microcircuitry and propose how anatomical and physiological aspects of the local circuitry may map onto elements of the graphical model implementation we discuss in particular the roles of three major types of inhibitory neurons small fast spiking basket cells large layer 2 3 basket cells and double bouquet neurons subpopulations of strongly interconnected neurons with their unique connectivity patterns in different cortical layers and the possible role of minicolumns in the realization of the population based maximum operation

a variational inference framework for soft in soft out detection in multiple access channels we propose a unified framework for deriving and studying soft in soft out siso detection in multiple access channels using the concept of variational inference the proposed framework may be used in multiple access interference mai intersymbol interference isi and multiple input multiple output mimo channels without loss of generality we will focus our attention on turbo multiuser detection to facilitate a more concrete discussion it is shown that with some loss of optimality variational inference avoids the exponential complexity of a posteriori probability app detection by optimizing a closely related but much more manageable objective function called variational free energy in addition to its systematic appeal there are several other advantages to this viewpoint first of all it provides unified and rigorous justifications for numerous detectors that were proposed on radically different grounds and facilitates convenient joint detection and decoding utilizing the turbo principle when error control codes are incorporated second efficient joint parameter estimation and data detection is possible via the variational expectation maximization em algorithm such that the detrimental effect of inaccurate channel knowledge at the receiver may be dealt with systematically we are also able to extend bpsk based siso detection schemes to arbitrary square qam constellations in a rigorous manner using a variational argument

dynamic hybrid algorithms for map inference in discrete mrfs in this paper we present novel techniques that improve the computational and memory efficiency of algorithms for solving multi label energy functions arising from discrete mrfs or crfs these methods are motivated by the observations that the performance of minimization algorithms depends on 1 the initialization used for the primal and dual variables and 2 the number of primal variables involved in the energy function our first method dynamic expansion works by recycling results from previous problem instances the second method simplifies the energy function by reducing the number of unknown variables present in the problem further we show that it can also be used to generate a good initialization for the dynamic expansion algorithm by reusing dual variables we test the performance of our methods on energy functions encountered in the problems of stereo matching and color and object based segmentation experimental results show that our methods achieve a substantial improvement in the performance of expansion as well as other popular algorithms such as sequential tree re weighted message passing and max product belief propagation we also demonstrate the applicability of our schemes for certain higher order energy functions such as the one described for interactive texture based image and video segmentation in most cases we achieve a 10 15 times speed up in the computation time our modified expansion algorithm provides similar performance to fast pd but is conceptually much simpler both expansion and fast pd can be made orders of magnitude faster when used in conjunction with the reduce scheme proposed in this paper

iterative decoding of serially concatenated cpm in fading channels with noisy channel state information we use the sum product algorithm spa to propose a reduced complexity detector for serially concatenated continuous phase modulated sccpm signals with noisy channel state information csi at the receiver two channel effects phase noise pn and flat fading are tackled using the statistical model and associated factor graph fg of the problem although optimal performance is not assured due to the loopy nature of the fg as well as simplifying assumptions made in the algorithm development simulations show that very significant performance gains are possible and the gap between systems with noiseless and noisy csi can be reduced to about 1 db using the proposed methods the increase in complexity compared to the conventional decoder which treats csi estimates as if they were equal to their true values is small

is sp bp the survey propagation sp algorithm for solving k sat problems has been shown recently as an instance of the belief propagation bp algorithm in this paper we show that for general constraint satisfaction problems sp may not be reducible from bp we also establish the conditions under which such a reduction is possible along our development we present a unification of the existing sp algorithms in terms of a probabilistically interpretable iterative procedure weighted probabilistic token passing

exact inferences in a neural implementation of a hidden markov model from first principles we derive a quadratic nonlinear first order dynamical system capable of performing exact bayes markov inferences for a wide class of biologically plausible stimulus dependent patterns of activity while simultaneously providing an online estimate of model performance this is accomplished by constructing a dynamical system that has solutions proportional to the probability distribution over the stimulus space but with a constant of proportionality adjusted to provide a local estimate of the probability of the recent observations of stimulus dependent activity given model parameters next we transform this exact equation to generate nonlinear equations for the exact evolution of log likelihood and log likelihood ratios and show that when the input has low amplitude linear rate models for both the likelihood and the log likelihood functions follow naturally from these equations we use these four explicit representations of the probability distribution to argue that in contrast to the arguments of previous work the dynamical system for the exact evolution of the likelihood as opposed to the log likelihood or log likelihood ratios not only can be mapped onto a biologically plausible network but is also more consistent with physiological observations

a topological characterization of classical brst cohomology the recent identification of classical brst cohomology with the vertical cohomology of a certain fibration is used to compute it in terms of the classical observables and the topology of the gauge orbits when the gauge orbits are compact and orientable a duality theorem is exhibited

samplesearch importance sampling in presence of determinism the paper focuses on developing effective importance sampling algorithms for mixed probabilistic and deterministic graphical models the use of importance sampling in such graphical models is problematic because it generates many useless zero weight samples which are rejected yielding an inefficient sampling process to address this rejection problem we propose the samplesearch scheme that augments sampling with systematic constraint based backtracking search we characterize the bias introduced by the combination of search with sampling and derive a weighting scheme which yields an unbiased estimate of the desired statistics e g probability of evidence when computing the weights exactly is too complex we propose an approximation which has a weaker guarantee of asymptotic unbiasedness we present results of an extensive empirical evaluation demonstrating that samplesearch outperforms other schemes in presence of significant amount of determinism

computing with continuous attractors stability and online aspects two issues concerning the application of continuous attractors in neural systems are investigated the computational robustness of continuous attractors with respect to input noises and the implementation of bayesian online decoding in a perfect mathematical model for continuous attractors decoding results for stimuli are highly sensitive to input noises and this sensitivity is the inevitable consequence of the system s neutral stability to overcome this shortcoming we modify the conventional network model by including extra dynamical interactions between neurons these interactions vary according to the biologically plausible hebbian learning rule and have the computational role of memorizing and propagating stimulus information accumulated with time as a result the new network model responds to the history of external inputs over a period of time and hence becomes insensitive to short term fluctuations also since dynamical interactions provide a mechanism to convey the prior knowledge of stimulus that is the information of the stimulus presented previously the network effectively implements online bayesian inference this study also reveals some interesting behavior in neural population coding such as the trade off between decoding stability and the speed of tracking time varying stimuli and the relationship between neural tuning width and the tracking speed

lp harmonic analysis and radon transforms on the heisenberg group in this paper harmonic analysis in the heisenberg group heisn cn r with group law z t z t z z t t 12im z z is taken to mean the joint spectral theory of the operators l heisenberg laplacian and t t the spectral decomposition of e l2 is given explicitly as e 0 e d where e 1 k 0 1 2 and e z t n 2 n 1 n 2k n 1 exp ie tn 2k exp z 24 n 2k lkn 1 z 22 n 2k where lkn 1 denotes the laguerre polynomial the eigenvalues are it e e n 2k e and l k e k e this decomposition is essentially well known but has not previously been described in this light it is also the decomposition into irreducibles of the representation of the heisenberg motion group semi direct product of heisn and u n on l2 heisn the main result is a summability in lp of the decomposition for e lp 1 p limr 1 rk 0 ed for p 1 only a weaker substitute is available and for p the decomposition is false there is also a plancherel formula for l2 2 2 k e n 2k 0 k e z 0 2dzd a number of explicit examples of the decomposition are computed including part of the schrodinger propagator eisl the heisenberg radon transform is defined by r z t n z t w 0 dw this operator was studied by geller and stein it is shown to be bounded from lp to lq if and only if p 2n 2 2n 1 and q 2n 2 some generalizations of these results to the free step two nilpotent lie groups are given but the results are not as complete

a restriction theorem for metivier groups abstract in the spirit of an earlier result of d muller on the heisenberg group we prove a restriction theorem on a certain class of two step nilpotent lie groups our result extends that of muller also in the framework of the heisenberg group

existence uniqueness and cohomology of the classical brst charge with ghosts of ghosts a complete canonical formulation of the brst theory of systems with redundant gauge symmetries is presented these systems includep form gauge fields the superparticle and the superstring we first define the koszul tate differential and explicitly show how the introduction of the momenta conjugate to the ghosts of ghosts makes it acyclic the global existence of the brst generator is then demonstrated and the brst charge is proved to be unique up to canonical transformations in the extended phase space which includes the ghosts finally the brst cohomology in classical mechanics is investigated and shown to be equal to the cohomology of the exterior derivative along the gauge orbits as in the irreducible case this is done by re expressing the exterior algebra along the gauge orbits as a free differential algebra containing generators of higher degree which are identified with the ghosts of ghosts the quantum cohomology is not dealt with

truncating the loop series expansion for belief propagation recently chertkov and chernyak 2006b derived an exact expression for the partition sum normalization constant corresponding to a graphical model which is an expansion around the belief propagation bp solution by adding correction terms to the bp free energy one for each generalized loop in the factor graph the exact partition sum is obtained however the usually enormous number of generalized loops generally prohibits summation over all correction terms in this article we introduce truncated loop series bp tlsbp a particular way of truncating the loop series of chertkov chernyak by considering generalized loops as compositions of simple loops we analyze the performance of tlsbp in different scenarios including the ising model on square grids and regular random graphs and on promedas a large probabilistic medical diagnostic system we show that tlsbp often improves upon the accuracy of the bp solution at the expense of increased computation time we also show that the performance of tlsbp strongly depends on the degree of interaction between the variables for weak interactions truncating the series leads to significant improvements whereas for strong interactions it can be ineffective even if a high number of terms is considered

on the decomposition method for linear programming decoding of ldpc codes in this paper we focus on solving the linear programming lp problem that arises in the decoding of low density parity check ldpc codes by means of the revised simplex method in order to take advantage of the structure of the lp problem we reformulate the dual lp and apply the idea of dantzig wolfe d w decomposition method to solve the problem each subproblem in the d w decomposition method is an lp over a convex polyhedral cone we define the convex polyhedral cone as local parity check cone and discuss its special structures then we enumerate its extreme rays and use these extreme rays to design an efficient method for the general lp decoding problem the proposed method exhibits advantages in reducing both the storage and computational requirements

randomly spread cdma asymptotics via statistical physics this paper studies randomly spread code division multiple access cdma and multiuser detection in the large system limit using the replica method developed in statistical physics arbitrary input distributions and flat fading are considered a generic multiuser detector in the form of the posterior mean estimator is applied before single user decoding the generic detector can be particularized to the matched filter decorrelator linear minimum mean square error mmse detector the jointly or the individually optimal detector and others it is found that the detection output for each user although in general asymptotically non gaussian conditioned on the transmitted symbol converges as the number of users go to infinity to a deterministic function of a hidden gaussian statistic independent of the interferers thus the multiuser channel can be decoupled each user experiences an equivalent single user gaussian channel whose signal to noise ratio snr suffers a degradation due to the multiple access interference mai the uncoded error performance e g symbol error rate and the mutual information can then be fully characterized using the degradation factor also known as the multiuser efficiency which can be obtained by solving a pair of coupled fixed point equations identified in this paper based on a general linear vector channel model the results are also applicable to multiple input multiple output mimo channels such as in multiantenna systems

blending learning and inference in conditional random fields conditional random fields maximize the log likelihood of training labels given the training data e g objects given images in many cases the training labels are structures that consist of a set of variables and the computational complexity for estimating their likelihood is exponential in the number of the variables learning algorithms relax this computational burden using approximate inference that is nested as a sub procedure in this paper we describe the objective function for nested learning and inference in conditional random fields the devised objective maximizes the log beliefs probability distributions over subsets of training variables that agree on their marginal probabilities this objective is concave and consists of two types of variables that are related to the learning and inference tasks respectively importantly we afterwards show how to blend the learning and inference procedure and effectively get to the identical optimum much faster the proposed algorithm currently achieves the state of the art in various computer vision applications

on decoding of the 73 37 13 quadratic residue code in this paper a method to search the set of syndromes indices needed in computing the unknown syndromes for the 73 37 13 quadratic residue qr code is proposed according to the resulting index sets one computes the unknown syndromes and thus finds the corresponding error locator polynomial by using an inverse free berlekamp massey bm algorithm based on the modified chase ii algorithm the performance of soft decision decoding for the 73 37 13 qr code is given this result is new moreover the error rate performance of linear programming lp decoding for the 73 37 13 qr code is also investigated and lp based decoding is shown to be significantly superior in performance to the algebraic soft decision decoding while requiring almost the same computational complexity in fact the algebraic hard decision and soft decision decoding of the 89 45 17 qr code outperforms that of the 73 37 13 qr code because the former has a larger minimal distance however experimental results indicate that the 73 37 13 qr code outperforms the 89 45 17 qr code with much fewer arithmetic operations when using the lp based decoding algorithms the pseudocodewords analysis partially explains this seemingly strange phenomenon

region graph partition function expansion and approximate free energy landscapes theory and some numerical results graphical models for finite dimensional spin glasses and real world combinatorial optimization and satisfaction problems usually have an abundant number of short loops the cluster variation method and its extension the region graph method are theoretical approaches for treating the complicated short loop induced local correlations for graphical models represented by non redundant or redundant region graphs approximate free energy landscapes are constructed in this paper through the mathematical framework of region graph partition function expansion several free energy functionals are obtained each of which use a set of probability distribution functions or functionals as order parameters these probability distribution function functionals are required to satisfy the region graph belief propagation equation or the region graph survey propagation equation to ensure vanishing correction contributions of region subgraphs with dangling edges as a simple application of the general theory we perform region graph belief propagation simulations on the square lattice ferromagnetic ising model and the edwards anderson model considerable improvements over the conventional bethe peierls approximation are achieved collective domains of different sizes in the disordered and frustrated square lattice are identified by the message passing procedure such collective domains and the frustrations among them are responsible for the low temperature glass like dynamical behaviors of the system

importance sampling based estimation over and or search spaces for graphical models it is well known that the accuracy of importance sampling can be improved by reducing the variance of its sample mean and therefore variance reduction schemes have been the subject of much research in this paper we introduce a family of variance reduction schemes that generalize the sample mean from the conventional or search space to the and or search space for graphical models the new and or sample means allow trading time and space with variance at one end is the and or sample tree mean which has the same time and space complexity as the conventional or sample tree mean but has smaller variance at other end is the and or sample graph mean which requires more time and space to compute but has the smallest variance theoretically we show that the variance is smaller in the and or space because the and or sample mean is defined over a larger virtual sample size compared with the or sample mean empirically we demonstrate that the and or sample mean is far closer to the true mean than the or sample mean

decomposition methods for large scale lp decoding when binary linear error correcting codes are used over symmetric channels a relaxed version of the maximum likelihood decoding problem can be stated as a linear program lp this lp decoder can be used to decode error correcting codes at bit error rates comparable to state of the art belief propagation bp decoders but with significantly stronger theoretical guarantees however lp decoding when implemented with standard lp solvers does not easily scale to the block lengths of modern error correcting codes in this paper we draw on decomposition methods from optimization theory specifically the alternating direction method of multipliers admm to develop efficient distributed algorithms for lp decoding the key enabling technical result is a two slice characterization of the parity polytope the polytope formed by taking the convex hull of all codewords of the single parity check code this new characterization simplifies the representation of points in the polytope using this simplification we develop an efficient algorithm for euclidean norm projection onto the parity polytope this projection is required by the admm decoder and its solution allows us to use lp decoding with all its theoretical guarantees to decode large scale error correcting codes efficiently we present numerical results for ldpc codes of lengths more than 1000 the waterfall region of lp decoding is seen to initiate at a slightly higher snr than for sum product bp however an error floor is not observed for lp decoding which is not the case for bp our implementation of lp decoding using the admm executes as fast as our baseline sum product bp decoder is fully parallelizable and can be seen to implement a type of message passing with a particularly simple schedule

optimum estimation via gradients of partition functions and information measures a statistical mechanical perspective in continuation to a recent work on the statistical mechanical analysis of minimum mean square error mmse estimation in gaussian noise via its relation to the mutual information the i mmse relation here we propose a simple and more direct relationship between optimum estimation and certain information measures e g the information density and the fisher information which can be viewed as partition functions and hence are amenable to analysis using statistical mechanical techniques the proposed approach has several advantages most notably its applicability to general sources and channels as opposed to the i mmse relation and its variants which hold only for certain classes of channels e g additive white gaussian noise channels we then demonstrate the derivation of the conditional mean estimator and the mmse in a few examples two of these examples turn out to be generalizable to a fairly wide class of sources and channels for this class the proposed approach is shown to yield an approximate conditional mean estimator and an mmse formula that has the flavor of a single letter expression we also show how our approach can easily be generalized to situations of mismatched estimation

bayesian inference and online learning in poisson neuronal networks motivated by the growing evidence for bayesian computation in the brain we show how a two layer recurrent network of poisson neurons can perform both approximate bayesian inference and learning for any hidden markov model the lower layer sensory neurons receive noisy measurements of hidden world states the higher layer neurons infer a posterior distribution over world states via bayesian inference from inputs generated by sensory neurons we demonstrate how such a neuronal network with synaptic plasticity can implement a form of bayesian inference similar to monte carlo methods such as particle filtering each spike in a higher layer neuron represents a sample of a particular hidden world state the spiking activity across the neural population approximates the posterior distribution over hidden states in this model variability in spiking is regarded not as a nuisance but as an integral feature that provides the variability necessary for sampling during inference we demonstrate how the network can learn the likelihood model as well as the transition probabilities underlying the dynamics using a hebbian learning rule we present results illustrating the ability of the network to perform inference and learning for arbitrary hidden markov models

brst cohomology in classical mechanics the classical non quantum cohomology of the becchi rouet stora tyutin brst symmetry in phase space is defined and worked out no group action for the gauge transformations is assumed the results cover therefore the general case of an open algebra and are valid off shell each cohomology class contains all brst invariant functions with fixed ghost number an integer which differ from each other by a brst variation if the ghost number is negative there is only the trivial class whose elements are equivalent to zero if the ghost number is positive or zero there is a bijective correspondence between the brst classes and those of the exterior derivative along the gauge orbits these gauge orbits lie in the phase space surface on which the gauge generators are constrained to vanish the brst invariant functions of ghost numberp are then related to closedp forms along the orbits the addition of a brst variation corresponds to the addition of an exact form some comments about the quantum case are included the physical meaning of the classes with ghost number greater than zero is not discussed

correctness of local probability propagation in graphical models with loops graphical models such as bayesian networks and markov networks represent joint distributions over a set of variables by means of a graph when the graph is singly connected local propagation rules of the sort proposed by pearl 1988 are guaranteed to converge to the correct posterior probabilities recently a number of researchers have empirically demonstrated good performance of these same local propagation schemes on graphs with loops but a theoretical understanding of this performance has yet to be achieved for graphical models with a single loop we derive an analytical relationship between the probabilities computed using local propagation and the correct marginals using this relationship we show a category of graphical models with loops for which local propagation gives rise to provably optimal maximum a posteriori assignments although the computed marginals will be incorrect we also show how nodes can use local information in the messages they receive in order to correct their computed marginals we discuss how these results can be extended to graphical models with multiple loops and show simulation results suggesting that some properties of propagation on single loop graphs may hold for a larger class of graphs specifically we discuss the implication of our results for understanding a class of recently proposed error correcting codes known as turbo codes

bounds for spectral multipliers on nilpotent groups a criterion is given for the lp boundedness of a class of spec tral multiplier operators associated to left invariant homogeneous subelliptic second order differential operators on nilpotent lie groups generalizing a theo rem of hormander for radial fourier multipliers on euclidean space the order of differentiability required is half the homogeneous dimension of the group improving previous results in the same direction

tight bounds for ldpc and ldgm codes under map decoding a new method for analyzing low density parity check ldpc codes and low density generator matrix ldgm codes under bit maximum a posteriori probability map decoding is introduced the method is based on a rigorous approach to spin glasses developed by francesco guerra it allows one to construct lower bounds on the entropy of the transmitted message conditional to the received one based on heuristic statistical mechanics calculations we conjecture such bounds to be tight the result holds for standard irregular ensembles when used over binary input output symmetric bios channels the method is first developed for tanner graph ensembles with poisson left degree distribution it is then generalized to multi poisson graphs and by a completion procedure to arbitrary degree distribution

approximating the permanent with fractional belief propagation we discuss schemes for exact and approximate computations of permanents and compare them with each other specifically we analyze the belief propagation bp approach and its fractional belief propagation fbp generalization for computing the permanent of a non negative matrix known bounds and conjectures are verified in experiments and some new theoretical relations bounds and conjectures are proposed the fractional free energy ffe function is parameterized by a scalar parameter 1 1 where 1 corresponds to the bp limit and 1 corresponds to the exclusion principle but ignoring perfect matching constraints mean field mf limit ffe shows monotonicity and continuity with respect to for every non negative matrix we define its special value 1 0 to be the for which the minimum of the parameterized ffe function is equal to the permanent of the matrix where the lower and upper bounds of the interval corresponds to respective bounds for the permanent our experimental analysis suggests that the distribution of varies for different ensembles but always lies within the 1 1 2 interval moreover for all ensembles considered the behavior of is highly distinctive offering an empirical practical guidance for estimating permanents of non negative matrices via the ffe approach

estimation and marginalization using the kikuchi approximation methods in this letter we examine a general method of approximation known as the kikuchi approximation method for finding the marginals of a product distribution as well as the corresponding partition function the kikuchi approximation method defines a certain constrained optimization problem called the kikuchi problem and treats its stationary points as approximations to the desired marginals we show how to associate a graph to any kikuchi problem and describe a class of local message passing algorithms along the edges of any such graph which attempt to find the solutions to the problem implementation of these algorithms on graphs with fewer edges requires fewer operations in each iteration we therefore characterize minimal graphs for a kikuchi problem which are those with the minimum number of edges we show with empirical results that these simpler algorithms often offer significant savings in computational complexity without suffering a loss in the convergence rate we give conditions for the convexity of a given kikuchi problem and the exactness of the approximations in terms of the loops of the minimal graph more precisely we show that if the minimal graph is cycle free then the kikuchi approximation method is exact and the converse is also true generically together with the fact that in the cycle free case the iterative algorithms are equivalent to the well known belief propagation algorithm our results imply that generically the kikuchi approximation method can be exact if and only if traditional junction tree methods could also solve the problem exactly

algebraic structure of the brst symmetry an explicit construction of the brst symmetry is presented in the hamiltonian approach the construction is based on the splitting homotopy and the transference problem

survey propagation at finite temperature application to a sourlas code as a toy model in this paper we investigate a finite temperature generalization of survey propagation by applying it to the problem of finite temperature decoding of a biased finite connectivity sourlas code for temperatures lower than the nishimori temperature we observe that the result is a shift of the location of the dynamical critical channel noise to larger values than the corresponding dynamical transition for belief propagation as suggested recently by migliorini and saad for ldpc codes we show how the finite temperature 1rsb sp gives accurate results in the regime where competing approaches fail to converge or fail to recover the retrieval state

bayesian computation in recurrent neural circuits a large number of human psychophysical results have been successfully explained in recent years using bayesian models however the neural implementation of such models remains largely unclear in this article we show that a network architecture commonly used to model the cerebral cortex can implement bayesian inference for an arbitrary hidden markov model we illustrate the approach using an orientation discrimination task and a visual motion detection task in the case of orientation discrimination we show that the model network can infer the posterior distribution over orientations and correctly estimate stimulus orientation in the presence of significant noise in the case of motion detection we show that the resulting model network exhibits direction selectivity and correctly computes the posterior probabilities over motion direction and position when used to solve the well known random dots motion discrimination task the model generates responses that mimic the activities of evidence accumulating neurons in cortical areas lip and fef the framework we introduce posits a new interpretation of cortical activities in terms of log posterior probabilities of stimuli occurring in the natural world

on the uniqueness of loopy belief propagation fixed points we derive sufficient conditions for the uniqueness of loopy belief propagation fixed points these conditions depend on both the structure of the graph and the strength of the potentials and naturally extend those for convexity of the bethe free energy we compare them with a strengthened version of conditions derived elsewhere for pairwise potentials we discuss possible implications for convergent algorithms as well as for other approximate free energies

training recurrent networks by evolino in recent years gradient based lstm recurrent neural networks rnns solved many previously rnn unlearnable tasks sometimes however gradient information is of little use for training rnns due to numerous local minima for such cases we present a novel method evolution of systems with linear outputs evolino evolino evolves weights to the nonlinear hidden nodes of rnns while computing optimal linear mappings from hidden state to output using methods such as pseudo inverse based linear regression if we instead use quadratic programming to maximize the margin we obtain the first evolutionary recurrent support vector machines we show that evolino based lstm can solve tasks that echo state nets jaeger 2004a cannot and achieves higher accuracy in certain continuous function generation tasks than conventional gradient descent rnns including gradient based lstm

belief propagation in networks of spiking neurons from a theoretical point of view statistical inference is an attractive model of brain operation however it is unclear how to implement these inferential processes in neuronal networks we offer a solution to this problem by showing in detailed simulations how the belief propagation algorithm on a factor graph can be embedded in a network of spiking neurons we use pools of spiking neurons as the function nodes of the factor graph each pool gathers messages in the form of population activities from its input nodes and combines them through its network dynamics each of the various output messages to be transmitted over the edges of the graph is computed by a group of readout neurons that feed in their respective destination pools we use this approach to implement two examples of factor graphs the first example drawn from coding theory models the transmission of signals through an unreliable channel and demonstrates the principles and generality of our network approach the second more applied example is of a psychophysical mechanism in which visual cues are used to resolve hypotheses about the interpretation of an object s shape and illumination these two examples and also a statistical analysis demonstrate good agreement between the performance of our networks and the direct numerical evaluation of belief propagation

applicability of n vicinity method for calculation of free energy of ising model abstract here we apply the n vicinity method of approximate calculation of the partition function to an ising model with the nearest neighbor interaction on d dimensional hypercube lattice we solve the equation of state for an arbitrary dimension d and analyze the behavior of the free energy as expected for large dimensions d 3 the system demonstrates a phase transition of the second kind in this case we obtain a simple analytical expression for the critical value of the inverse temperature when 3 d 7 this expression is in a very good agreement with the results of computer simulations in the case of small dimensions d 1 2 there is a noticeable discrepancy with the known exact results

convergence analysis of reweighted sum product algorithms markov random fields are designed to represent structured dependencies among large collections of random variables and are well suited to capture the structure of real world signals many fundamental tasks in signal processing e g smoothing denoising segmentation etc require efficient methods for computing approximate marginal probabilities over subsets of nodes in the graph the marginalization problem though solvable in linear time for graphs without cycles is computationally intractable for general graphs with cycles this intractability motivates the use of approximate ldquomessage passingrdquo algorithms this paper studies the convergence and stability properties of the family of reweighted sum product algorithms a generalization of the widely used sum product or belief propagation algorithm in which messages are adjusted with graph dependent weights for pairwise markov random fields we derive various conditions that are sufficient to ensure convergence and also provide bounds on the geometric convergence rates when specialized to the ordinary sum product algorithm these results provide strengthening of previous analyses we prove that some of our conditions are necessary and sufficient for subclasses of homogeneous models but not for general models the experimental simulations on various classes of graphs validate our theoretical results

random k satisfiability problem from an analytic solution to an efficient algorithm we study the problem of satisfiability of randomly chosen clauses each with k boolean variables using the cavity method at zero temperature we find the phase diagram for the k 3 case we show the existence of an intermediate phase in the satisfiable region where the proliferation of metastable states is at the origin of the slowdown of search algorithms the fundamental order parameter introduced in the cavity method which consists of surveys of local magnetic fields in the various possible states of the system can be computed for one given sample these surveys can be used to invent new types of algorithms for solving hard combinatorial optimizations problems one such algorithm is shown here for the k 3 satisfiability problem with very good performances

tree based reparameterization framework for analysis of sum product and related algorithms we present a tree based reparameterization trp framework that provides a new conceptual view of a large class of algorithms for computing approximate marginals in graphs with cycles this class includes the belief propagation bp or sum product algorithm as well as variations and extensions of bp algorithms in this class can be formulated as a sequence of reparameterization updates each of which entails refactorizing a portion of the distribution corresponding to an acyclic subgraph i e a tree or more generally a hypertree the ultimate goal is to obtain an alternative but equivalent factorization using functions that represent exact or approximate marginal distributions on cliques of the graph our framework highlights an important property of the sum product algorithm and the larger class of reparameterization algorithms the original distribution on the graph with cycles is not changed the perspective of tree based updates gives rise to a simple and intuitive characterization of the fixed points in terms of tree consistency we develop interpretations of these results in terms of information geometry the invariance of the distribution in conjunction with the fixed point characterization enables us to derive an exact expression for the difference between the true marginals on an arbitrary graph with cycles and the approximations provided by belief propagation more broadly our analysis applies to any algorithm that minimizes the bethe free energy we also develop bounds on the approximation error which illuminate the conditions that govern their accuracy finally we show how the reparameterization perspective extends naturally to generalizations of bp e g kikuchi 1951 approximations and variants via the notion of hypertree reparameterization

on riesz means of eigenfunction expansions for the kohn laplacian in a recent article c sogge improved some classical results by l hormander about the l p convergence of the riesz means of eigenfunction expansions associated to elliptic differential operators on compact manifolds in this paper we shall deal with the analogous problem for the model operator of a non elliptic but subelliptic differential operator the kohn laplacian on the heisenberg group

learning bayesian network parameters under equivalence constraints we propose a principled approach for learning parameters in bayesian networks from incomplete datasets where the examples of a dataset are subject to equivalence constraints these equivalence constraints arise from datasets where examples are tied together in that we may not know the value of a particular variable but whatever that value is we know it must be the same across different examples we formalize the problem by defining the notion of a constrained dataset and a corresponding constrained likelihood that we seek to optimize we further propose a new learning algorithm that can effectively learn more accurate bayesian networks using equivalence constraints which we demonstrate empirically moreover we highlight how our general approach can be brought to bear on more specialized learning tasks such as those in semi supervised clustering and topic modeling where more domain specific approaches were previously developed

geometric brst quantization i prequantization this is the first part of a two part paper dedicated to the definition of brst quantization in the framework of geometric quantization after recognizing prequantization as a manifestation of the poisson module structure of the sections of the prequantum line bundle we define brst prequantization and show that it is the homological analog of the symplectic reduction of prequantum data we define a prequantum brst cohomology theory and interpret it in terms of geometric objects we then show that all poisson structures correspond under homological reduction this allows to prove in the brst context that prequantization and reduction commute

code aided maximum likelihood ambiguity resolution through free energy minimization in digital communication receivers ambiguities in terms of timing and phase need to be resolved prior to data detection in the presence of powerful error correcting codes which operate in low signal to noise ratios snr long training sequences are needed to achieve good performance in this contribution we develop a new class of code aided ambiguity resolution algorithms which require no training sequence and achieve good performance with reasonable complexity in particular we focus on algorithms that compute the maximum likelihood ml solution exactly or in good approximation with a tractable complexity using a factor graph representation the complexity of the proposed algorithm is discussed and reduced complexity variations including stopping criteria and sequential implementation are developed

a new class of upper bounds on the log partition function we introduce a new class of upper bounds on the log partition function of a markov random field mrf this quantity plays an important role in various contexts including approximating marginal distributions parameter estimation combinatorial enumeration statistical decision theory and large deviations bounds our derivation is based on concepts from convex duality and information geometry in particular it exploits mixtures of distributions in the exponential domain and the legendre mapping between exponential and mean parameters in the special case of convex combinations of tree structured distributions we obtain a family of variational problems similar to the bethe variational problem but distinguished by the following desirable properties i they are convex and have a unique global optimum and ii the optimum gives an upper bound on the log partition function this optimum is defined by stationary conditions very similar to those defining fixed points of the sum product algorithm or more generally any local optimum of the bethe variational problem as with sum product fixed points the elements of the optimizing argument can be used as approximations to the marginals of the original model the analysis extends naturally to convex combinations of hypertree structured distributions thereby establishing links to kikuchi approximations and variants

discrete input two dimensional gaussian channels with memory estimation and information rates via graphical models and statistical mechanics discrete input two dimensional 2d gaussian channels with memory represent an important class of systems which appears extensively in communications and storage in spite of their widespread use the workings of 2d channels are still very much unknown in this work we try to explore their properties from the perspective of estimation theory and information theory at the heart of our approach is a mapping of a 2d channel to an undirected graphical model and inferring its a posteriori probabilities apps using generalized belief propagation gbp the derived probabilities are shown to be practically accurate thus enabling optimal maximum a posteriori map estimation of the transmitted symbols also the shannon theoretic information rates are deduced either via the vector wise shannon mcmillan breiman smb theorem or via the recently derived symbol wise guo shamai verdu gsv theorem our approach is also described from the perspective of statistical mechanics as the graphical model and inference algorithm have their analogues in physics our experimental study based on common channel settings taken from cellular networks and magnetic recording devices demonstrates that under nontrivial memory conditions the performance of this fully tractable gbp estimator is almost identical to the performance of the optimal map estimator it also enables a practically accurate simulation based estimate of the information rate rationalization of this excellent performance of gbp in the 2 d gaussian channel setting is addressed