connections between score matching contrastive divergence and pseudolikelihood for continuous valued variables score matching sm and contrastive divergence cd are two recently proposed methods for estimation of nonnormalized statistical methods without computation of the normalization constant partition function although they are based on very different approaches we show in this letter that they are equivalent in a special case in the limit of infinitesimal noise in a specific monte carlo method further we show how these methods can be interpreted as approximations of pseudolikelihood

efficiency of a kernel density estimator under an autoregressive dependence model abstract the problem of estimating the probability density function of a strictly stationary process is considered to study the effect of a dependence structure on the efficiency of a kernel density estimator the mean integrated squared error mise of the fourier integral estimator fie is derived on the assumption that the observed data are generated by a first order autoregressive process numerical results for the normal and cauchy densities show that even moderate departures from independence can lead to a considerable loss in efficiency of the fie in addition to efficiency considerations the issue of determining an optimal smoothing parameter for the fie under the autoregressive model is addressed

a data driven stochastic collocation approach for uncertainty quantification in mems this work presents a data driven stochastic collocation approach to include the effect of uncertain design parameters during complex multi physics simulation of micro electromechanical systems mems the proposed framework comprises of two key steps first probabilistic characterization of the input uncertain parameters based on available experimental information and second propagation of these uncertainties through the predictive model to relevant quantities of interest the uncertain input parameters are modeled as independent random variables for which the distributions are estimated based on available experimental observations using a nonparametric diffusion mixing based estimator botev nonparametric density estimation via diffusion mixing technical report 2007 the diffusion based estimator derives from the analogy between the kernel density estimation kde procedure and the heat dissipation equation and constructs density estimates that are smooth and asymptotically consistent the diffusion model allows for the incorporation of the prior density and leads to an improved density estimate in comparison with the standard kde approach as demonstrated through several numerical examples following the characterization step the uncertainties are propagated to the output variables using the stochastic collocation approach based on sparse grid interpolation smolyak soviet math dokl 1963 4 240 243 the developed framework is used to study the effect of variations in young s modulus induced as a result of variations in manufacturing process parameters or heterogeneous measurements on the performance of a mems switch copyright 2010 john wiley sons ltd

clustering methods based on variational analysis in the space of measures we formulate clustering as a minimisation problem in the space of measures by modelling the cluster centres as a poisson process with unknown intensity function we derive a ward type clustering criterion which under the poisson assumption can easily be evaluated explicitly in terms of the intensity function we show that asymptotically i e for increasing total intensity the optimal intensity function is proportional to a dimension dependent power of the density of the observations for fixed finite total intensity no explicit solution seems available however the ward type criterion to be minimised is convex in the intensity function so that the steepest descent method of molchanov and zuyev 2001 can be used to approximate the global minimum it turns out that the gradient is similar in form to the functional to be optimised if we discretise over a grid the steepest descent algorithm at each iteration step increases the current intensity function at those points where the gradient is minimal at the expense of regions with a large gradient value the algorithm is applied to a toy one dimensional example a simulation from a popular spatial cluster model and a real life dataset from strauss 1975 concerning the positions of redwood seedlings finally we discuss the relative merits of our approach compared to classical hierarchical and partition clustering techniques as well as to modern model based clustering methods using markov point processes and mixture distributions

exact mean integrated squared error an exact and easily computable expression for the mean integrated squared error mise for the kernel estimator of a general normal mixture density is given for gaussian kernels of arbitrary order this provides a powerful new way of understanding density estimation which complements the usual tools of simulation and asymptotic analysis the family of normal mixture densities is very flexible and the formulae derived allow simple exact analysis for a wide variety of density shapes a number of applications of this method giving important new insights into kernel density estimation are presented among these is the discovery that the usual asymptotic approximations to the mise can be quite inaccurate especially when the underlying density contains substantial fine structure and also strong evidence that the practical importance of higher order kernels is surprisingly small for moderate sample sizes 1 introduction substantial research has been devoted to kernel density estimation this is because it provides a simple yet appealing context in which to study problems and issues that arise in all types of nonparametric curve estimation this includes regression spectral density and hazard estimation and also a variety of other estimators including histograms splines and orthogonal series three important and useful tools for understanding the behavior of nonparametric curve estimators are asymptotic analysis simulation and numerical calculation of error criteria each of these methods provides many useful insights into the complicated structure present in the study of curve estimation however each has its limitations as well rhe strength of asymptotic analysis is that it frequently allows simultaneous study of many different specific examples through general results applying to entire classes of settings the weakness of asymptotics is that they only describe behavior in the limit this is still very useful in many situations because the asymptotics describe the actual situation quite well however it is less useful when the asymptotics have not yet kicked in that is in studying situations where the asymptotically dominant effect has not taken over yet perhaps the biggest drawback to asymptotics is that it is very difficult to determine in a given situation which of these possibilities is occurring

estimation of the scaling parameter for a kernel type density estimate abstract an algorithm is given for estimation of the scaling parameter in the kernel type density estimate derived from the fourier integral a monte carlo study of five different densities compares the integrated square error ise for the resulting estimate to the ise for the fourier integral estimate fie with mean integrated squared error mise optimal and to the ise for the normal kernel with asymptotically mise optimal the study shows the fie with estimated compares favorably with the other estimates and demonstrates the asymptotic optimality of the fie for smooth densities

optimal approximation and quantisation abstract a number of interesting functionals f x of a finite point set x in the euclidean space can be represented as integrals of a function that depends on the integration variable y and x restricted onto a certain set c y x that is determined by y and x and satisfies separation and uniform boundedness conditions for instance c y x can be the voronoi cell generated by x that contains y we single out the general properties of c and that ensure that the normalised infimum of f x over all sets x with cardinality n converges to a limit that can be identified using a particular form of the considered functionals include those that appear in quantisation problems for probability measures and in finding the optimal approximation of a function by splines tangent planes and triangulated surfaces for instance it is shown that the minimum l approximation error normalised by n of a sufficiently smooth bivariate convex function f using the convex triangulation converges to the l 1 1 norm of the determinant of the second derivative of f times a certain absolute constant

oversmoothed nonparametric density estimates abstract the optimal histogram for a sample of size n from a density defined on the entire real line requires at least 2n 1 3 bins under mild smoothness conditions similar bounds exist for the frequency polygon and kernel estimators values near these bounds give nearly optimal results for a variety of smooth densities providing good quick density estimates

variational analysis of functionals of poisson processes letf ii be a functional of a generally nonhomogeneous poisson process ii with intensity measure considering the expectatione f ii as a functional of from the cone m of positive finite measures we derive closed form expressions for its fra chet derivatives of all orders that generalize the perturbation analysis formulae for poisson processes variational methods developed for the space m allow us to obtain first and second order sufficient conditions for various types of constrained optimization problems fore f we study in detail optimization over the class of measures with a fixed total massa and develop a technique that often allows us to obtain the asymptotic behavior of the optimal intensity measure in the high intensity setting whena grows to infinity as a particular application we consider the problem of optimal placement of stations in the poisson model of a two layer telecommunication network

the essential histogram the histogram is widely used as a simple exploratory display of data but it is usually not clear how to choose the number and size of bins we construct a confidence set of distribution functions that optimally address the two main tasks of the histogram estimating probabilities and detecting features such as increases and modes in the distribution we define the essential histogram as the histogram in the confidence set with the fewest bins thus the essential histogram is the simplest visualization of the data that optimally achieves the main tasks of the histogram the only assumption we make is that the data are independent and identically distributed we provide a fast algorithm for the essential histogram and illustrate our methodology with examples an r package is available on cran

densities spectral densities and modality this paper considers the problem of specifying a simple approximating density function for a given data set x 1 x n simplicity is measured by the number of modes but several different definitions of approximation are introduced the taut string method is used to control the numbers of modes and to produce candidate approximating densities refinements are introduced that improve the local adaptivity of the procedures and the method is extended to spectral densities

kernels for nonparametric curve estimation summary the choice of kernels for the nonparametric estimation of regression functions and of their derivatives is investigated explicit expressions are obtained for kernels minimizing the asymptotic variance or the asymptotic integrated mean square error imse the present proof of the optimality of the latter kernels is restricted up to order k 5 these kernels are also of interest for the nonparametric estimation of probability densities and spectral densities a finite sample study indicates that higher order kernels asymptotically improving the rate of convergence may become attractive for realistic finite sample size suitably modified kernels are considered for estimating at the extremities of the data in a way which allows to retain the order of the bias found for interior points

relative efficiencies of kernel and local likelihood density estimators summary local likelihood methods enjoy advantageous properties such as good performance in the presence of edge effects that are rarely found in other approaches to nonparametric density estimation however as we argue in this paper standard kernel methods can have distinct advantages when edge effects are not present we show that whereas the integrated variances of the two methods are virtually identical the integrated squared bias of a conventional kernel estimator is less than that of a local log linear estimator by as much as a factor of 4 moreover the greatest bias improvements offered by kernel methods occur when they are needed most i e when the effect of bias is particularly high similar comparisons can also be made when high degree local log polynomial fits are assessed against high order kernel methods for example although as is well known high degree local polynomial fits offer potentially infinite efficiency gains relative to their kernel competitors the converse is also true indeed the asymptotic value of the integrated squared bias of a local log quadratic estimator can exceed any given constant multiple of that for the competing kernel method in all cases the densities that suffer problems in the context of local log likelihood methods can be chosen to be symmetric either unimodal or bimodal either infinitely or compactly supported and to have arbitrarily many derivatives as functions on the real line they are not pathological however our results reveal quantitative differences between global performances of local log polynomial estimators applied to unimodal or multimodal distributions

contrastive divergence in gaussian diffusions this letter presents an analysis of the contrastive divergence cd learning algorithm when applied to continuous time linear stochastic neural networks for this case powerful techniques exist that allow a detailed analysis of the behavior of cd the analysis shows that cd converges to maximum likelihood solutions only when the network structure is such that it can match the first moments of the desired distribution otherwise cd can converge to solutions arbitrarily different from the log likelihood solutions or they can even diverge this result suggests the need to improve our theoretical understanding of the conditions under which cd is expected to be well behaved and the conditions under which it may fail in addition the results point to practical ideas on how to improve the performance of cd

training products of experts by minimizing contrastive divergence it is possible to combine multiple latent variable models of the same data by multiplying their probability distributions together and then renormalizing this way of combining individual expert models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data a product of experts poe is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary training a poe by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule fortunately a poe can be trained using a different objective function called contrastive divergence whose derivatives with regard to the parameters can be approximated accurately and efficiently examples are presented of contrastive divergence learning using several types of expert on several types of data

practical bayesian density estimation using mixtures of normals abstract mixtures of normals provide a flexible model for estimating densities in a bayesian framework there are some difficulties with this model however first standard reference priors yield improper posteriors second the posterior for the number of components in the mixture is not well defined if the reference prior is used third posterior simulation does not provide a direct estimate of the posterior for the number of components we present some practical methods for coping with these problems finally we give some results on the consistency of the method when the maximum number of components is allowed to grow with the sample size

tangent sets in the space of measures with applications to variational analysis we characterise the first and second order tangent sets in the linear space of signed measures with finite total variation and apply the obtained results to derive necessary optimality conditions for functionals on measures

analysis and geometry on configuration spaces abstract in this paper foundations are presented to a new systematic approach to analysis and geometry for an important class of infinite dimensional manifolds namely configuration spaces more precisely a differential geometry is introduced on the configuration space x over a riemannian manifold x this geometry is non flat even if x r d it is obtained as a natural lifting of the riemannian structure on x in particular a corresponding gradient divergence div and laplace beltrami operator h div gamma are constructed the associated volume elements i e all measures on x w r t which and div gamma become dual operators on l 2 x are identified as exactly the mixed poisson measures with mean measure equal to a multiple of the volume element dx on x in particular all these measures obey an integration by parts formula w r t vector fields on x the corresponding dirichlet forms e on l 2 x are therefore defined each is shown to be associated with a diffusion process which is thus the brownian motion on x and which is sub sequently identified as the usual independent infinite particle process on x the associated heat semigroup t t t 0 is calculated explicitly it is also proved that the diffusion process when started with is time ergodic or equivalently e gamma is irreducible or equivalently t t t 0 is ergodic if and only if is poisson measure z dx with intensity z dx for some z 0 furthermore it is shown that the laplace beltrami operator h div gamma gamma on l 2 x z dx is unitary equivalent to the second quantization of the laplacian x on x on the corresponding fock space n 0 l 2 x z dx n as another direct consequence of our results we obtain a representation of the lie algebra of compactly supported vector fields on x on poisson space finally generalizations to the case where dx is replaced by an absolutely continuous measure and also to interacting particle systems on x are described in particular the case where the mixed poisson measures are replaced by gibbs measures of ruelle type on x

poisson process fock space representation chaos expansion and covariance inequalities we consider a poisson process on an arbitrary measurable space with an arbitrary sigma finite intensity measure we establish an explicit fock space representation of square integrable functions of as a consequence we identify explicitly in terms of iterated difference operators the integrands in the wiener ito chaos expansion we apply these results to extend well known variance inequalities for homogeneous poisson processes on the line to the general poisson case the poincare inequality is a special case further applications are covariance identities for poisson processes on strictly ordered spaces and harris fkg inequalities for monotone functions of

a connection between score matching and denoising autoencoders denoising autoencoders have been previously shown to be competitive alternatives to restricted boltzmann machines for unsupervised pretraining of each layer of a deep architecture we show that a simple denoising autoencoder training criterion is equivalent to matching the score with respect to the data of a specific energy based model to that of a nonparametric parzen density estimator of the data this yields several useful insights it defines a proper probabilistic model for the denoising autoencoder technique which makes it in principle possible to sample from them or rank examples by their energy it suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives it justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy based models

central limit theorems for u statistics of poisson point processes a u statistic of a poisson point process is defined as the sum f x1 xk over all possibly infinitely many k tuples of distinct points of the point process using the malliavin calculus the wiener ito chaos expansion of such a functional is computed and used to derive a formula for the variance central limit theorems for u statistics of poisson point processes are shown with explicit bounds for the wasserstein distance to a gaussian random variable as applications the intersection process of poisson hyperplanes and the length of a random geometric graph are investigated

a note on jackknifing kernel regression function estimators corresp estimation of the value of a regression function at a point of continuity using a kernel type estimator is considered and improvements by a jackknife technique are discussed it is seen that a so called generalized jackknife estimator asymptotically improves upon an ordinary kernel type estimator however for a fixed sample size the generalized jackknife method may inflate the mean square error