decreasing the size of the restricted boltzmann machine in this letter we propose a method to decrease the number of hidden units of the restricted boltzmann machine while avoiding a decrease in the performance quantified by the kullback leibler divergence our algorithm is then demonstrated by numerical simulations

learning maximum entropy models from finite size data sets a fast data driven algorithm allows sampling from the posterior distribution maximum entropy models provide the least constrained probability distributions that reproduce statistical properties of experimental datasets in this work we characterize the learning dynamics that maximizes the log likelihood in the case of large but finite datasets we first show how the steepest descent dynamics is not optimal as it is slowed down by the inhomogeneous curvature of the model parameters space we then provide a way for rectifying this space which relies only on dataset properties and does not require large computational efforts we conclude by solving the long time limit of the parameters dynamics including the randomness generated by the systematic use of gibbs sampling in this stochastic framework rather than converging to a fixed point the dynamics reaches a stationary distribution which for the rectified dynamics reproduces the posterior distribution of the parameters we sum up all these insights in a rectified data driven algorithm that is fast and by sampling from the parameters posterior avoids both under and overfitting along all the directions of the parameters space through the learning of pairwise ising models from the recording of a large population of retina neurons we show how our algorithm outperforms the steepest descent method

probabilistic joint image segmentation and labeling by figure ground composition we propose a layered statistical model for image segmentation and labeling obtained by combining independently extracted possibly overlapping sets of figure ground fg segmentations the process of constructing consistent image segmentations called tilings is cast as optimization over sets of maximal cliques sampled from a graph connecting all non overlapping figure ground segment hypotheses potential functions over cliques combine unary gestalt based figure qualities and pairwise compatibilities among spatially neighboring segments constrained by t junctions and the boundary interface statistics of real scenes building on the segmentation layer we further derive a joint image segmentation and labeling model jsl which given a bag of fgs constructs a joint probability distribution over both the compatible image interpretations tilings composed from those segments and over their labeling into categories the process of drawing samples from the joint distribution can be interpreted as first sampling tilings followed by sampling labelings conditioned on the choice of a particular tiling we learn the segmentation and labeling parameters jointly based on maximum likelihood with a novel estimation procedure we refer to as incremental saddle point approximation the partition function over tilings and labelings is increasingly more accurately approximated by including incorrect configurations that are rated as probable by candidate models during learning state of the art results are reported on the berkeley stanford and pascal voc datasets where an improvement of 28 was achieved for the segmentation task only tiling and an accuracy of 47 8 was obtained on the test set of voc12 for semantic labeling jsl

connections between score matching contrastive divergence and pseudolikelihood for continuous valued variables score matching sm and contrastive divergence cd are two recently proposed methods for estimation of nonnormalized statistical methods without computation of the normalization constant partition function although they are based on very different approaches we show in this letter that they are equivalent in a special case in the limit of infinitesimal noise in a specific monte carlo method further we show how these methods can be interpreted as approximations of pseudolikelihood

on nonlinear conservation laws with a nonlocal diffusion term abstract scalar one dimensional conservation laws with a nonlocal diffusion term corresponding to a riesz feller differential operator are considered solvability results for the cauchy problem in l are adapted from the case of a fractional derivative with homogeneous symbol the main interest of this work is the investigation of smooth shock profiles in the case of a genuinely nonlinear smooth flux function we prove the existence of such travelling waves which are monotone and satisfy the standard entropy condition moreover the dynamic nonlinear stability of the travelling waves under small perturbations is proven similarly to the case of the standard diffusive regularisation by constructing a lyapunov functional

comment on a new integrable equation with cuspons and w m shape peaks solitons j math phys 47 112701 2006 in a recent paper j math phys 47 112701 2006 qiao proposed a new completely integrable wave equation the equation was derived from the two dimensional euler equations and was proven to posses lax pair bi hamiltonian structure and new types of soliton solutions in this comment we show that qiao s equation is a subcase of an integrable equation derived by a method that was proposed by fuchssteiner and fokas physica d 4 47 1981 in the early 1980s we also discuss the existence of the new types of soliton solutions

distances and means of direct similarities the non euclidean nature of direct isometries in a euclidean space i e transformations consisting of a rotation and a translation creates difficulties when computing distances means and distributions over them which have been well studied in the literature direct similarities transformations consisting of a direct isometry and a positive uniform scaling present even more of a challenge one which we demonstrate and address here in this article we investigate divergences a superset of distances without constraints on symmetry and sub additivity for comparing direct similarities and means induced by them via minimizing a sum of squared divergences we analyze several standard divergences the euclidean distance using the matrix representation of direct similarities a divergence from lie group theory and the family of all left invariant distances derived from riemannian geometry we derive their properties and those of their induced means highlighting several shortcomings in addition we introduce a novel family of left invariant divergences called srt divergences which resolve several issues associated with the standard divergences in our evaluation we empirically demonstrate the derived properties of the divergences and means both qualitatively and quantitatively on synthetic data finally we compare the divergences in a real world application vote based scale invariant object recognition our results show that the new divergences presented here and their means are both more effective and faster to compute for this task

efficient robust conditional random fields conditional random fields crfs are a flexible yet powerful probabilistic approach and have shown advantages for popular applications in various areas including text analysis bioinformatics and computer vision traditional crf models however are incapable of selecting relevant features as well as suppressing noise from noisy original features moreover conventional optimization methods often converge slowly in solving the training procedure of crfs and will degrade significantly for tasks with a large number of samples and features in this paper we propose robust crfs rcrfs to simultaneously select relevant features an optimal gradient method ogm is further designed to train rcrfs efficiently specifically the proposed rcrfs employ the ell 1 norm of the model parameters to regularize the objective used by traditional crfs therefore enabling discovery of the relevant unary features and pairwise features of crfs in each iteration of ogm the gradient direction is determined jointly by the current gradient together with the historical gradients and the lipschitz constant is leveraged to specify the proper step size we show that an ogm can tackle the rcrf model training very efficiently achieving the optimal convergence rate o 1 k vphantom r r 2 where k is the number of iterations this convergence rate is theoretically superior to the convergence rate o 1 k of previous first order optimization methods extensive experiments performed on three practical image segmentation tasks demonstrate the efficacy of ogm in training our proposed rcrfs

functional clones and expressibility of partition functions abstract we study functional clones which are sets of non negative pseudo boolean functions functions 0 1 k r 0 closed under essentially multiplication summation and limits functional clones naturally form a lattice under set inclusion and are closely related to counting constraint satisfaction problems csps we identify a sublattice of interesting functional clones and investigate the relationships and properties of the functional clones in this sublattice

mesh convolutional restricted boltzmann machines for unsupervised learning of features with structure preservation on 3 d meshes discriminative features of 3 d meshes are significant to many 3 d shape analysis tasks however handcrafted descriptors and traditional unsupervised 3 d feature learning methods suffer from several significant weaknesses 1 the extensive human intervention is involved 2 the local and global structure information of 3 d meshes cannot be preserved which is in fact an important source of discriminability 3 the irregular vertex topology and arbitrary resolution of 3 d meshes do not allow the direct application of the popular deep learning models 4 the orientation is ambiguous on the mesh surface and 5 the effect of rigid and nonrigid transformations on 3 d meshes cannot be eliminated as a remedy we propose a deep learning model with a novel irregular model structure called mesh convolutional restricted boltzmann machines mcrbms mcrbm aims to simultaneously learn structure preserving local and global features from a novel raw representation local function energy distribution in addition multiple mcrbms can be stacked into a deeper model called mesh convolutional deep belief networks mcdbns mcdbn employs a novel local structure preserving convolution lspc strategy to convolve the geometry and the local structure learned by the lower mcrbm to the upper mcrbm lspc facilitates resolving the challenging issue of the orientation ambiguity on the mesh surface in mcdbn experiments using the proposed mcrbm and mcdbn were conducted on three common aspects global shape retrieval partial shape retrieval and shape correspondence results show that the features learned by the proposed methods outperform the other state of the art 3 d shape features

a multiclass classification method based on decoding of binary classifiers in this letter we present new methods of multiclass classification that combine multiple binary classifiers misclassification of each binary classifier is formulated as a bit inversion error with probabilistic models by making an analogy to the context of information transmission theory dependence between binary classifiers is incorporated into our model which makes a decoder a type of boltzmann machine we performed experimental studies using a synthetic data set data sets from the uci repository and bioinformatics data sets and the results show that the proposed methods are superior to the existing multiclass classification methods

graph regularized restricted boltzmann machine the restricted boltzmann machine rbm has received an increasing amount of interest in recent years it determines good mapping weights that capture useful latent features in an unsupervised manner the rbm and its generalizations have been successfully applied to a variety of image classification and speech recognition tasks however most of the existing rbm based models disregard the preservation of the data manifold structure in many real applications the data generally reside on a low dimensional manifold embedded in high dimensional ambient space in this brief we propose a novel graph regularized rbm to capture features and learning representations explicitly considering the local manifold structure of the data by imposing manifold based locality that preserves constraints on the hidden layer of the rbm the model ultimately learns sparse and discriminative representations the representations can reflect data distributions while simultaneously preserving the local manifold structure of data we test our model using several benchmark image data sets for unsupervised clustering and supervised classification problem the results demonstrate that the performance of our method exceeds the state of the art alternatives

the shape boltzmann machine a strong model of object shape a good model of object shape is essential in applications such as segmentation detection inpainting and graphics for example when performing segmentation local constraints on the shapes can help where object boundaries are noisy or unclear and global constraints can resolve ambiguities where background clutter looks similar to parts of the objects in general the stronger the model of shape the more performance is improved in this paper we use a type of deep boltzmann machine salakhutdinov and hinton international conference on artificial intelligence and statistics 2009 that we call a shape boltzmann machine sbm for the task of modeling foreground background binary and parts based categorical shape images we show that the sbm characterizes a strong model of shape in that samples from the model look realistic and it can generalize to generate samples that differ from training examples we find that the sbm learns distributions that are qualitatively and quantitatively better than existing models for this task

consistency of pseudolikelihood estimation of fully visible boltzmann machines a boltzmann machine is a classic model of neural computation and a number of methods have been proposed for its estimation most methods are plagued by either very slow convergence or asymptotic bias in the resulting estimates here we consider estimation in the basic case of fully visible boltzmann machines we show that the old principle of pseudolikelihood estimation provides an estimator that is computationally very simple yet statistically consistent

comparing classification methods for longitudinal fmri studies we compare 10 methods of classifying fmri volumes by applying them to data from a longitudinal study of stroke recovery adaptive fisher s linear and quadratic discriminant gaussian naive bayes support vector machines with linear quadratic and radial basis function rbf kernels logistic regression two novel methods based on pairs of restricted boltzmann machines rbm and k nearest neighbors all methods were tested on three binary classification tasks and their out of sample classification accuracies are compared the relative performance of the methods varies considerably across subjects and classification tasks the best overall performers were adaptive quadratic discriminant support vector machines with rbf kernels and generatively trained pairs of rbms

the cauchy problem for the fokas olver rosenau qiao equation abstract it is shown that the initial value problem for the fokas olver rosenau qiao equation forq is well posed in sobolev spaces h s s 5 2 in the sense of hadamard furthermore it is proved that the dependence on initial data is sharp i e the data to solution map is continuous but not uniformly continuous also peakon travelling wave solutions are derived on both the circle and the line and are used to prove that the solution map is not uniformly continuous in h s for s 3 2

discriminative deep face shape model for facial point detection facial point detection is an active area in computer vision due to its relevance to many applications it is a nontrivial task since facial shapes vary significantly with facial expressions poses or occlusion in this paper we address this problem by proposing a discriminative deep face shape model that is constructed based on an augmented factorized three way restricted boltzmann machines model specifically the discriminative deep model combines the top down information from the embedded face shape patterns and the bottom up measurements from local point detectors in a unified framework in addition along with the model effective algorithms are proposed to perform model learning and to infer the true facial point locations from their measurements based on the discriminative deep face shape model 68 facial points are detected on facial images in both controlled and in the wild conditions experiments on benchmark data sets show the effectiveness of the proposed facial point detection algorithm against state of the art methods

feature extraction with deep neural networks by a generalized discriminant analysis we present an approach to feature extraction that is a generalization of the classical linear discriminant analysis lda on the basis of deep neural networks dnns as for lda discriminative features generated from independent gaussian class conditionals are assumed this modeling has the advantages that the intrinsic dimensionality of the feature space is bounded by the number of classes and that the optimal discriminant function is linear unfortunately linear transformations are insufficient to extract optimal discriminative features from arbitrarily distributed raw measurements the generalized discriminant analysis gerda proposed in this paper uses nonlinear transformations that are learnt by dnns in a semisupervised fashion we show that the feature extraction based on our approach displays excellent performance on real world recognition and detection tasks such as handwritten digit recognition and face detection in a series of experiments we evaluate gerda features with respect to dimensionality reduction visualization classification and detection moreover we show that gerda dnns can preprocess truly high dimensional input data to low dimensional representations that facilitate accurate predictions even if simple linear predictors or measures of similarity are used

travelling waves for a non local korteweg de vries burgers equation abstract we study travelling wave solutions of a korteweg de vries burgers equation with a non local diffusion term this model equation arises in the analysis of a shallow water flow by performing formal asymptotic expansions associated to the triple deck regularisation which is an extension of classical boundary layer theory the resulting non local operator is a fractional derivative of order between 1 and 2 travelling wave solutions are typically analysed in relation to shock formation in the full shallow water problem we show rigorously the existence of these waves in absence of the dispersive term the existence of travelling waves and their monotonicity was established previously by two of the authors in contrast travelling waves of the non local kdv burgers equation are not in general monotone as is the case for the corresponding classical kdv burgers equation this requires a more complicated existence proof compared to the previous work moreover the travelling wave problem for the classical kdv burgers equation is usually analysed via a phase plane analysis which is not applicable here due to the presence of the non local diffusion operator instead we apply fractional calculus results available in the literature and a lyapunov functional in addition we discuss the monotonicity of the waves in terms of a control parameter and prove their dynamic stability in case they are monotone

remarks on the stability of shock profiles for conservation laws with dissipation two remarks are made the first is to establish the stability of monotone shock profiles of the kdv burgers equation based on an energy method of goodman the second remark illustrates specifically in burgers equation that uniform rates of decay are not to be expected for perturbations of shock profiles in typical norms introduction we will prove two results the first is to establish the orbital stability of monotone shock profile solutions 4 x st of the kdv burgers equation 1 1 ut uux uxx uxxx 0 0 the second is to show that for shock profile solutions of viscous conservation laws and specifically for burgers equation 1 2 ut uux uxx one cannot expect perturbations to decay at any uniform rate algebraic or otherwise in ordinary norms such as lp the first result is related to goodman s conditional stability result in 5 for small amplitude shock profiles x st in systems of genuinely nonlinear conservation laws with viscosity u f u x buxx u e rm the second is intended to be relevant to the question of full stability for shock profiles of such systems the nonlinear stability to small perturbations of constant state solutions u0 x c of the system u f u x b u ux x where b c is a strictly stable viscosity matrix see 9 can be demonstrated based on a uniform rate of decay of solutions to the linearized equation the method used is similar to that developed in 8 and 10 for systems in n 1 space dimensions faster decay rates in higher space dimensions make the method easier to apply when n is large but it does just work for conservation laws with stable viscosities in one space dimension see 13 thus for example assuming c 0 if iiuo iih6 ll is sufficiently small then iiu t llh4 ct 1 4iiuoii h4 n la in this note we want to point out that such uniform decay rates cannot be valid for perturbations of shock profiles in particular for the shock profile solution of burgers equations we construct a family of perturbations which decay to zero but do so at no uniform rate in most ordinary norms including all lp norms thus for received by the editors october 15 1984 and in revised form december 19 1984 1980 mathematics subject classification primary 35q20 35l65 1985 american mathematical society 0002 9947 85 1 00 25 per page

deep adaptive feature embedding with local sample distributions for person re identification person re identification re id aims to match pedestrians observed by disjoint camera views it attracts increasing attention in computer vision due to its importance to surveillance systems to combat the major challenge of cross view visual variations deep embedding approaches are proposed by learning a compact feature space from images such that the euclidean distances correspond to their cross view similarity metric however the global euclidean distance cannot faithfully characterize the ideal similarity in a complex visual feature space because features of pedestrian images exhibit unknown distributions due to large variations in poses illumination and occlusion moreover intra personal training samples within a local range which are robust to guide deep embedding against uncontrolled variations cannot be captured by a global euclidean distance in this paper we study the problem of person re id by proposing a novel sampling to mine suitable positives i e intra class within a local range to improve the deep embedding in the context of large intra class variations our method is capable of learning a deep similarity metric adaptive to local sample structure by minimizing each sample s local distances while propagating through the relationship between samples to attain the whole intra class minimization to this end a novel objective function is proposed to jointly optimize similarity metric learning local positive mining and robust deep feature embedding this attains local discriminations by selecting local ranged positive samples and the learned features are robust to dramatic intra class variations experiments on benchmarks show state of the art results achieved by our method c 2017 elsevier ltd all rights reserved

universal approximation depth and errors of narrow belief networks with discrete units we generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well we relax the setting of binary units sutskever hinton 2008 le roux bengio 2008 2010 montufar ay 2011 to units with arbitrary finite state spaces and the vanishing approximation error to an arbitrary approximation error tolerance for example we show that a q ary deep belief network with layers of width for some can approximate any probability distribution on without exceeding a kullback leibler divergence of our analysis covers discrete restricted boltzmann machines and naive bayes models as special cases

oblique random forest ensemble via least square estimation for time series forecasting abstract recent studies in machine learning indicates that the classifiers most likely to be the bests are the random forests as an ensemble classifier random forest combines multiple decision trees to significant decrease the overall variances conventional random forest employs orthogonal decision tree which selects one optimal feature to split the data instances within a non leaf node according to impurity criteria such as gini impurity information gain and so on however orthogonal decision tree may fail to capture the geometrical structure of the data samples motivated by this we make the first attempt to study the oblique random forest in the context of time series forecasting in each node of the decision tree instead of the single optimal feature based orthogonal classification algorithms used by standard random forest a least square classifier is employed to perform partition the proposed method is advantageous with respect to both efficiency and accuracy we empirically evaluate the proposed method on eight generic time series datasets and five electricity load demand time series datasets from the australian energy market operator and compare with several other benchmark methods

travelling wave solutions to the korteweg de vries burgers equation the existence and certain qualitative properties of travelling wave solutions to the korteweg de vries burgers equation are established the limiting behaviour of these waves when e tends to zero and when tends to zero is examined together with a singular limit wherein both e and tend to zero

a fuzzy restricted boltzmann machine novel learning algorithms based on the crisp possibilistic mean value of fuzzy numbers a fuzzy restricted boltzmann machine frbm is extended from a restricted boltzmann machine rbm by replacing all the real valued parameters with fuzzy numbers a new frbm that employs the crisp possibilistic mean value of a fuzzy number to defuzzify the fuzzy free energy function is presented this approach is much clearer and easier to obtain the expression of the defuzzified free energy function and its approximation than the centroid method several theorems that discuss the error bounds of the approximation to ensure the rationality and validity are also investigated learning algorithms are given for the designed frbm with symmetric triangular fuzzy numbers stfns asymmetric triangular fuzzy numbers and gaussian fuzzy numbers by appropriately choosing the parameters a theorem is concluded that all frbms with symmetric fuzzy numbers will have identical learning algorithm to that of frbms with stfns this is illustrated by a case of frbm with gaussian fuzzy numbers two experiments including the mnist handwriting recognition and the bars and stripes benchmark are carried out the results show that the proposed frbms significantly outperform rbms in learning accuracy and generalization ability especially when encountering unlearned samples and recovering incomplete images

approximate learning algorithm in boltzmann machines boltzmann machines can be regarded as markov random fields for binary cases they are equivalent to the ising spin model in statistical mechanics learning systems in boltzmann machines are one of the np hard problems thus in general we have to use approximate methods to construct practical learning algorithms in this context in this letter we propose new and practical learning algorithms for boltzmann machines by using the belief propagation algorithm and the linear response approximation which are often referred as advanced mean field methods finally we show the validity of our algorithm using numerical experiments

estimating 3d trajectories from 2d projections via disjunctive factored four way conditional restricted boltzmann machines estimation recognition and near future prediction of 3d trajectories based on their two dimensional projections available from one camera source is an exceptionally difficult problem due to uncertainty in the trajectories and environment high dimensionality of the specific trajectory states lack of enough labeled data and so on in this article we propose a solution to solve this problem based on a novel deep learning model dubbed disjunctive factored four way conditional restricted boltzmann machine dffw crbm our method improves state of the art deep learning techniques for high dimensional time series modeling by introducing a novel tensor factorization capable of driving forth order boltzmann machines to considerably lower energy levels at no computational costs dffw crbms are capable of accurately estimating recognizing and performing near future prediction of three dimensional trajectories from their 2d projections while requiring limited amount of labeled data we evaluate our method on both simulated and real world data showing its effectiveness in predicting and classifying complex ball trajectories and human activities

deep narrow sigmoid belief networks are universal approximators in this note we show that exponentially deep belief networks can approximate any distribution over binary vectors to arbitrary accuracy even when the width of each layer is limited to the dimensionality of the data we further show that such networks can be greedily learned in an easy yet impractical way

discrete restricted boltzmann machines we describe discrete restricted boltzmann machines probabilistic graphical models with bipartite interactions between visible and hidden discrete variables examples are binary restricted boltzmann machines and discrete naive bayes models we detail the inference functions and distributed representations arising in these models in terms of configurations of projected products of simplices and normal fans of products of simplices we bound the number of hidden variables depending on the cardinalities of their state spaces for which these models can approximate any probability distribution on their visible states to any given accuracy in addition we use algebraic methods and coding theory to compute their dimension

attractor dynamics in feedforward neural networks we study the probabilistic generative models parameterized by feedforward neural networks an attractor dynamics for probabilistic inference in these models is derived from a mean field approximation for large layered sigmoidal networks fixed points of the dynamics correspond to solutions of the mean field equations which relate the statistics of each unit to those of its markov blanket we establish global convergence of the dynamics by providing a lyapunov function and show that the dynamics generate the signals required for unsupervised learning our results for feedforward networks provide a counterpart to those of cohen grossberg and hopfield for symmetric networks

on a class of physically important integrable equations abstract a methodology introduced by fuchssteiner and the author is used to derive a class of physically important integrable evolution equations among these equations are integrable generalizations of the korteweg devries kdv of the modified kdv of the nonlinear schrodinger nls and of the sine gordon equations the modeling of water waves as well as general asymptotic considerations are used to illustrate the occurrence of the generalized modified kdv and nls equations respectively

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

learning to represent spatial transformations with factored higher order boltzmann machines to allow the hidden units of a restricted boltzmann machine to model the transformation between two successive images memisevic and hinton 2007 introduced three way multiplicative interactions that use the intensity of a pixel in the first image as a multiplicative gain on a learned symmetric weight between a pixel in the second image and a hidden unit this creates cubically many parameters which form a three dimensional interaction tensor we describe a low rank approximation to this interaction tensor that uses a sum of factors each of which is a three way outer product this approximation allows efficient learning of transformations between larger image patches since each factor can be viewed as an image filter the model as a whole learns optimal filter pairs for efficiently representing transformations we demonstrate the learning of optimal filter pairs from various synthetic and real image sequences we also show how learning about image transformations allows the model to perform a simple visual analogy task and we show how a completely unsupervised network trained on transformations perceives multiple motions of transparent dot patterns in the same way as humans

a fast learning algorithm for deep belief nets we show how to use complementary priors to eliminate the explaining away effects that make inference difficult in densely connected belief nets that have many hidden layers using complementary priors we derive a fast greedy algorithm that can learn deep directed belief networks one layer at a time provided the top two layers form an undirected associative memory the fast greedy algorithm is used to initialize a slower learning procedure that fine tunes the weights using a contrastive version of the wake sleep algorithm after fine tuning a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels this generative model gives better digit classification than the best discriminative learning algorithms the low dimensional manifolds on which the digits lie are modeled by long ravines in the free energy landscape of the top level associative memory and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind

energy based models for sparse overcomplete representations we present a new way of extending independent components analysis ica to overcomplete representations in contrast to the causal generative extensions of ica which maintain marginal independence of sources we define features as deterministic linear functions of the inputs this assumption results in marginal dependencies among the features but conditional independence of the features given the inputs by assigning energies to the features a probability distribution over the input states is defined through the boltzmann distribution free parameters of this model are trained using the contrastive divergence objective hinton 2002 when the number of features is equal to the number of input dimensions this energy based model reduces to noiseless ica and we show experimentally that the proposed learning algorithm is able to perform blind source separation on speech data in additional experiments we train overcomplete energy based models to extract features from various standard data sets containing speech natural images hand written digits and faces

on the representational power of restricted boltzmann machines for symmetric functions and boolean functions restricted boltzmann machines rbms are used to build deep belief networks that are widely thought to be one of the first effective deep learning neural networks this paper studies the ability of rbms to represent distributions over 0 1 n via softplus hardplus rbm networks it is shown that any distribution whose density depends on the number of 1 s in their input can be approximated with arbitrarily high accuracy by an rbm of size 2n 1 which improves the result of a previous study by reducing the size from n 2 to 2n 1 a theorem for representing partially symmetric boolean functions by softplus rbm networks is established accordingly the representational power of rbms for distributions whose mass represents the boolean functions is investigated in comparison with that of threshold circuits and polynomial threshold functions it is shown that a distribution over 0 1 n whose mass represents a boolean function can be computed with a given margin delta by an rbm of size and parameters bounded by polynomials in n if and only if it can be computed by a depth 2 threshold circuit with size and parameters bounded by polynomials in n

universal approximation results for the temporal restricted boltzmann machine and the recurrent temporal restricted boltzmann machine the restricted boltzmann machine rbm has proved to be a powerful tool in machine learning both on its own and as the building block for deep belief networks multi layer generative graphical models the rbm and deep belief network have been shown to be universal approximators for probability distributions on binary vectors in this paper we prove several similar universal approximation results for two variations of the restricted boltzmann machine with time dependence the temporal restricted boltzmann machine trbm and the recurrent temporal restricted boltzmann machine rtrbm we show that the trbm is a universal approximator for markov chains and generalize the theorem to sequences with longer time dependence we then prove that the rtrbm is a universal approximator for stochastic processes with nite time dependence we conclude with a discussion on efficiency and how the constructions developed could explain some previous experimental results

learning where to attend with deep architectures for image tracking we discuss an attentional model for simultaneous object tracking and recognition that is driven by gaze data motivated by theories of perception the model consists of two interacting pathways identity and control intended to mirror the what and where pathways in neuroscience models the identity pathway models object appearance and performs classification using deep factored restricted boltzmann machines at each point in time the observations consist of foveated images with decaying resolution toward the periphery of the gaze the control pathway models the location orientation scale and speed of the attended object the posterior distribution of these states is estimated with particle filtering deeper in the control pathway we encounter an attentional mechanism that learns to select gazes so as to minimize tracking uncertainty unlike in our previous work we introduce gaze selection strategies that operate in the presence of partial information and on a continuous action space we show that a straightforward extension of the existing approach to the partial information setting results in poor performance and we propose an alternative method based on modeling the reward surface as a gaussian process this approach gives good performance in the presence of partial information and allows us to expand the action space from a small discrete set of fixation points to a continuous domain

a study of parts based object class detection using complete graphs object detection is one of the key components in modern computer vision systems while the detection of a specific rigid object under changing viewpoints was considered hard just a few years ago current research strives to detect and recognize classes of non rigid articulated objects hampered by the omnipresent confusing information due to clutter and occlusion the focus has shifted from holistic approaches for object detection to representations of individual object parts linked by structural information along with richer contextual descriptions of object configurations along this line of research we present a practicable and expandable probabilistic framework for parts based object class representation enabling the detection of rigid and articulated object classes in arbitrary views we investigate learning of this representation from labelled training images and infer globally optimal solutions to the contextual map detection problem using a search with a novel lower bound as admissible heuristic an assessment of the inference performance of belief propagation and tree reweighted belief propagation is obtained as a by product the generality of our approach is demonstrated on four different datasets utilizing domain dependent information cues

a novel parameter estimation method for boltzmann machines we propose a novel estimator for a specific class of probabilistic models on discrete spaces such as the boltzmann machine the proposed estimator is derived from minimization of a convex risk function and can be constructed without calculating the normalization constant whose computational cost is exponential order we investigate statistical properties of the proposed estimator such as consistency and asymptotic normality in the framework of the estimating function small experiments show that the proposed estimator can attain comparable performance to the maximum likelihood expectation at a much lower computational cost and is applicable to high dimensional data

addendum to travelling waves for a non local korteweg de vries burgers equation j differential equations 257 3 2014 720 758 abstract we add a theorem to f achleitner c m cuesta and s hittmeir 2014 1 in that paper we studied travelling wave solutions of a korteweg de vries burgers type equation with a non local diffusion term in particular the proof of existence and uniqueness of these waves relies on the assumption that the exponentially decaying functions are the only bounded solutions of the linearised equation in this addendum we prove this assumption and thus close the existence and uniqueness proof of travelling wave solutions

reinforcement learning with factored states and actions a novel approximation method is presented for approximating the value function and selecting good actions for markov decision processes with large state and action spaces the method approximates state action values as negative free energies in an undirected graphical model called a product of experts the model parameters can be learned efficiently because values and derivatives can be efficiently computed for a product of experts actions can be found even in large factored action spaces by the use of markov chain monte carlo sampling simulation results show that the product of experts approximation can be used to solve large problems in one simulation it is used to find actions in action spaces of size 240

algorithms for estimating the partition function of restricted boltzmann machines abstract accurate estimates of the normalization constants partition functions of energy based probabilistic models markov random fields are highly important for example for assessing the performance of models monitoring training progress and conducting likelihood ratio tests several algorithms for estimating the partition function in relation to a reference distribution have been introduced including annealed importance sampling ais and bennett s acceptance ratio method bar however their conceptual similarities and differences have not been worked out so far and systematic comparisons of their behavior in practice have been missing we devise a unifying theoretical framework for these algorithms which comprises existing variants and suggests new approaches it is based on a generalized form of crooks equality linking the expectation over a distribution of samples generated by a transition operator to the expectation over the distribution induced by the reversed operator the framework covers different ways of generating samples such as parallel tempering and path sampling an empirical comparison revealed the differences between the methods when estimating the partition function of restricted boltzmann machines and ising models in our experiments bar using parallel tempering worked well with a small number of bridging distributions while path sampling based ais performed best when many bridging distributions were available because bar gave the overall best results we favor it over ais furthermore the experiments showed the importance of choosing a proper reference distribution

deep belief networks are compact universal approximators deep belief networks dbn are generative models with many layers of hidden causal variables recently introduced by hinton osindero and teh 2006 along with a greedy layer wise unsupervised learning algorithm building on le roux and bengio 2008 and sutskever and hinton 2008 we show that deep but narrow generative networks do not require more parameters than shallow ones to achieve universal approximation exploiting the proof technique we prove that deep but narrow feedforward neural networks with sigmoidal units can represent any boolean expression

bias variance decompositions for likelihood based estimators the bias variance decomposition of mean squared error is well understood and relatively straightforward in this note a similar simple decomposition is derived valid for any kind of error measure that when using the appropriate probability model can be derived from a kullback leibler divergence or log likelihood

contrastive divergence in gaussian diffusions this letter presents an analysis of the contrastive divergence cd learning algorithm when applied to continuous time linear stochastic neural networks for this case powerful techniques exist that allow a detailed analysis of the behavior of cd the analysis shows that cd converges to maximum likelihood solutions only when the network structure is such that it can match the first moments of the desired distribution otherwise cd can converge to solutions arbitrarily different from the log likelihood solutions or they can even diverge this result suggests the need to improve our theoretical understanding of the conditions under which cd is expected to be well behaved and the conditions under which it may fail in addition the results point to practical ideas on how to improve the performance of cd

training products of experts by minimizing contrastive divergence it is possible to combine multiple latent variable models of the same data by multiplying their probability distributions together and then renormalizing this way of combining individual expert models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data a product of experts poe is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary training a poe by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule fortunately a poe can be trained using a different objective function called contrastive divergence whose derivatives with regard to the parameters can be approximated accurately and efficiently examples are presented of contrastive divergence learning using several types of expert on several types of data

refinements of universal approximation results for deep belief networks and restricted boltzmann machines we improve recently published results about resources of restricted boltzmann machines rbm and deep belief networks dbn required to make them universal approximators we show that any distribution on the set of binary vectors of length can be arbitrarily well approximated by an rbm with hidden units where is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of in important cases this number is half the cardinality of the support set of given in le roux bengio 2008 we construct a dbn with hidden layers of width that is capable of approximating any distribution on arbitrarily well this confirms a conjecture presented in le roux and bengio 2010

a pseudo spectral method for a non local kdv burgers equation posed on r abstract in this paper we present a new pseudo spectral method to solve the initial value problem associated to a non local kdv burgers equation involving a caputo type fractional derivative the basic idea is using an algebraic map to transform the whole real line into a bounded interval where we can apply a fourier expansion special attention is given to the correct computation of the fractional derivative in this setting

the modified camassa holm equation backlund transformation and nonlinear superposition formula this paper is concerned with the modified camassa holm equation and its backlund transformation with the aid of reciprocal transformation and the associated modified camassa holm equation a backlund transformation which involves both dependent and independent variables is constructed for the modified camassa holm equation the related nonlinear superposition formula is worked out and applications of both backlund transformation and nonlinear superposition formula are considered

discriminant deep belief network for high resolution sar image classification classification plays an important role in many fields of synthetic aperture radar sar image understanding and interpretation many scholars have devoted to design features to characterize the content of sar images however it is still a challenge to design discriminative and robust features for sar image classification recently the deep learning has attracted much attention and has been successfully applied in many fields of computer vision in this paper a novel feature learning approach that is called discriminant deep belief network disdbn is proposed to learning high level features for sar image classification in which the discriminant features are learned by combining ensemble learning with a deep belief network in an unsupervised manner firstly some subsets of sar image patches are selected and marked with pseudo labels to train weak classifiers secondly the specific sar image patch is characterized by a set of projection vectors that are obtained by projecting the sar image patch onto each weak decision space spanned by each weak classifier finally the discriminant features are generated by feeding the projection vectors to a dbn for sar image classification experimental results demonstrate that better classification performance can be achieved by the proposed approach than the other state of the art approaches highlightsa disdbn is proposed to characterize sar image patches in an unsupervised manner both the cpl and ipl are investigated to produce prototypes of sar image patches some weak decision spaces are constructed based on the learned prototypes a high level feature is learned for the sar image patch in a hierarchy manner we show that our method can achieve a better classification performance

symplectic structures their backlund transformations and hereditary symmetries abstract it is shown that compatible symplectic structures lead in a natural way to hereditary symmetries we recall that a hereditary symmetry is an operator valued function which immediately yields a hierarchy of evolution equations each having infinitely many commuting symmetries all generated by this hereditary symmetry furthermore this hereditary symmetry usually describes completely the soliton structure and the conservation laws of these equations this result then provide us with a method for constructing hereditary symmetries and hence exactly solvable evolution equations in addition we show how symplectic structures transform under backlund transformations this leads to a method for generating a whole class of symplectic structures from a given one several examples and applications are given illustrating the above results also the connection of our results with those of gelfand and dikii and of magri is briefly pointed out

representational power of restricted boltzmann machines and deep belief networks deep belief networks dbn are generative neural network models with many layers of hidden explanatory factors recently introduced by hinton osindero and teh 2006 along with a greedy layer wise unsupervised learning algorithm the building block of a dbn is a probabilistic model called a restricted boltzmann machine rbm used to represent one layer of the model restricted boltzmann machines are interesting because inference is easy in them and because they have been successfully used as building blocks for training deeper models we first prove that adding hidden units yields strictly improved modeling power while a second theorem shows that rbms are universal approximators of discrete distributions we then study the question of whether dbns with more layers are strictly more powerful in terms of representational power this suggests a new and less greedy criterion for training rbms within dbns

information geometric optimization algorithms a unifying picture via invariance principles we present a canonical way to turn any smooth parametric family of probability distributions on an arbitrary search space x into a continuous time black box optimization method on x the information geometric optimization igo method invariance as a major design principle keeps the number of arbitrary choices to a minimum the resulting igo flow is the flow of an ordinary differential equation conducting the natural gradient ascent of an adaptive time dependent transformation of the objective function it makes no particular assumptions on the objective function to be optimized the igo method produces explicit igo algorithms through time discretization it naturally recovers versions of known algorithms and offers a systematic way to derive new ones in continuous search spaces igo algorithms take a form related to natural evolution strategies nes the cross entropy method is recovered in a particular case with a large time step and can be extended into a smoothed parametrization independent maximum likelihood update igo ml when applied to the family of gaussian distributions on rd the igo framework recovers a version of the well known cma es algorithm and of xnes for the family of bernoulli distributions on 0 1 d we recover the seminal pbil algorithm and cga for the distributions of restricted boltzmann machines we naturally obtain a novel algorithm for discrete optimization on 0 1 d all these algorithms are natural instances of and unified under the single information geometric optimization framework the igo method achieves thanks to its intrinsic formulation maximal invariance properties invariance under reparametrization of the search space x under a change of parameters of the probability distribution and under increasing transformation of the function to be optimized the latter is achieved through an adaptive quantile based formulation of the objective theoretical considerations strongly suggest that igo algorithms are essentially characterized by a minimal change of the distribution over time therefore they have minimal loss in diversity through the course of optimization provided the initial diversity is high first experiments using restricted boltzmann machines confirm this insight as a simple consequence igo seems to provide from information theory an elegant way to simultaneously explore several valleys of a fitness landscape in a single run

a maximum likelihood interpretation for slow feature analysis the brain extracts useful features from a maelstrom of sensory information and a fundamental goal of theoretical neuroscience is to work out how it does so one proposed feature extraction strategy is motivated by the observation that the meaning of sensory data such as the identity of a moving visual object is often more persistent than the activation of any single sensory receptor this notion is embodied in the slow feature analysis sfa algorithm which uses slowness as a heuristic by which to extract semantic information from multidimensional time series here we develop a probabilistic interpretation of this algorithm showing that inference and learning in the limiting case of a suitable probabilistic model yield exactly the results of sfa similar equivalences have proved useful in interpreting and extending comparable algorithms such as independent component analysis for sfa we use the equivalent probabilistic model as a conceptual springboard with which to motivate several novel extensions to the algorithm

unsupervised 3d local feature learning by circle convolutional restricted boltzmann machine extracting local features from 3d shapes is an important and challenging task that usually requires carefully designed 3d shape descriptors however these descriptors are hand crafted and require intensive human intervention with prior knowledge to tackle this issue we propose a novel deep learning model namely circle convolutional restricted boltzmann machine ccrbm for unsupervised 3d local feature learning ccrbm is specially designed to learn from raw 3d representations it effectively overcomes obstacles such as irregular vertex topology orientation ambiguity on the 3d surface and rigid or slightly non rigid transformation invariance in the hierarchical learning of 3d data that cannot be resolved by the existing deep learning models specifically by introducing the novel circle convolution ccrbm holds a novel ring like multi layer structure to learn 3d local features in a structure preserving manner circle convolution convolves across 3d local regions via rotating a novel circular sector convolution window in a consistent circular direction in the process of circle convolution extra points are sampled in each 3d local region and projected onto the tangent plane of the center of the region in this way the projection distances in each sector window are employed to constitute a novel local raw 3d representation called projection distance distribution pdd in addition to eliminate the initial location ambiguity of a sector window the fourier transform modulus is used to transform the pdd into the fourier domain which is then conveyed to ccrbm experiments using the learned local features are conducted on three aspects global shape retrieval partial shape retrieval and shape correspondence the experimental results show that the learned local features outperform other state of the art 3d shape descriptors

solving a system of nonlinear fractional differential equations using adomian decomposition adomian decomposition method has been employed to obtain solutions of a system of nonlinear fractional differential equations dxiyi x ni x y1 yn yi k 0 cki 0 k i 1 i n and dxi denotes caputo fractional derivative some examples are solved as illustrations using symbolic computation

a framework for efficient structured max margin learning of high order mrf models we present a very general algorithm for structured prediction learning that is able to efficiently handle discrete mrfs crfs including both pairwise and higher order models so long as they can admit a decomposition into tractable subproblems at its core it relies on a dual decomposition principle that has been recently employed in the task of mrf optimization by properly combining such an approach with a max margin learning method the proposed framework manages to reduce the training of a complex high order mrf to the parallel training of a series of simple slave mrfs that are much easier to handle this leads to a very efficient and general learning scheme that relies on solid mathematical principles we thoroughly analyze its theoretical properties and also show that it can yield learning algorithms of increasing accuracy since it naturally allows a hierarchy of convex relaxations to be used for loss augmented map mrf inference within a max margin learning approach furthermore it can be easily adapted to take advantage of the special structure that may be present in a given class of mrfs we demonstrate the generality and flexibility of our approach by testing it on a variety of scenarios including training of pairwise and higher order mrfs training by using different types of regularizers and or different types of dissimilarity loss functions as well as by learning of appropriate models for a variety of vision tasks including high order models for compact pose invariant shape priors knowledge based segmentation image denoising stereo matching as well as high order potts mrfs

asymptotic normality of the maximum pseudolikelihood estimator for fully visible boltzmann machines boltzmann machines bms are a class of binary neural networks for which there have been numerous proposed methods of estimation recently it has been shown that in the fully visible case of the bm the method of maximum pseudolikelihood estimation mple results in parameter estimates which are consistent in the probabilistic sense in this brief we investigate the properties of mple for the fully visible bms further and prove that mple also yields an asymptotically normal parameter estimator these results can be used to construct confidence intervals and to test statistical hypotheses these constructions provide a closed form alternative to the current methods that require monte carlo simulation or resampling we support our theoretical results by showing that the estimator behaves as expected in simulation studies

integration of stochastic models by minimizing α divergence when there are a number of stochastic models in the form of probability distributions one needs to integrate them mixtures of distributions are frequently used but exponential mixtures also provide a good means of integration this letter proposes a one parameter family of integration called integration which includes all of these well known integrations these are generalizations of various averages of numbers such as arithmetic geometric and harmonic averages there are psychophysical experiments that suggest that integrations are used in the brain the divergence between two distributions is defined which is a natural generalization of kullback leibler divergence and hellinger distance and it is proved that integration is optimal in the sense of minimizing divergence the theory is applied to generalize the mixture of experts and the product of experts to the mixture of experts the predictive distribution is also stated in the bayesian framework

a new integrable equation with cuspons and w m shape peaks solitons in this paper we propose a new completely integrable wave equation mt mx u2 ux2 2m2ux 0 m u uxx the equation is derived from the two dimensional euler equation and is proven to have lax pair and bi hamiltonian structures this equation possesses new cusp solitons cuspons instead of regular peakons ce x ct with speed c through investigating the equation we develop a new kind of soliton solutions w m shape peaks solitons there exist no smooth solitons for this integrable water wave equation

learning harmonium models with infinite latent features undirected latent variable models represent an important class of graphical models that have been successfully developed to deal with various tasks one common challenge in learning such models is to determine the number of hidden units that are unknown a priori although bayesian nonparametrics have provided promising results in bypassing the model selection problem in learning directed bayesian networks very little effort has been made toward applying bayesian nonparametrics to learn undirected latent variable models in this paper we present the infinite exponential family harmonium iefh a bipartite undirected latent variable model that automatically determines the number of latent units from an unbounded pool we also present two important extensions of iefh to 1 multiview iefh for dealing with heterogeneous data and 2 infinite maximum margin harmonium immh for incorporating supervising side information to learn predictive latent features we develop variational inference algorithms to learn model parameters our methods are computationally competitive because of the avoidance of selecting the number of latent units our extensive experiments on real image datasets and text datasets appear to demonstrate the benefits of iefh and immh inherited from bayesian nonparametrics and max margin learning such results were not available until now and contribute to expanding the scope of bayesian nonparametrics to learn the structures of undirected latent variable models