the helmholtz machine discovering the structure inherent in a set of patterns is a fundamental aim of statistical inference or learning one fruitful approach is to build a parameterized stochastic generative model independent draws from which are likely to produce the patterns for all but the simplest generative models each pattern can be generated in exponentially many ways it is thus intractable to adjust the parameters to maximize the probability of the observed patterns we describe a way of finessing this combinatorial explosion by maximizing an easily computed lower bound on the probability of the observations our method can be viewed as a form of hierarchical self supervised learning that may relate to the function of bottom up and top down cortical processing pathways

advances on byy harmony learning information theoretic perspective generalized projection geometry and independent factor autodetermination the nature of bayesian ying yang harmony learning is reexamined from an information theoretic perspective not only its ability for model selection and regularization is explained with new insights but also discussions are made on its relations and differences from the studies of minimum description length mdl bayesian approach the bit back based mdl akaike information criterion aic maximum likelihood information geometry helmholtz machines and variational approximation moreover a generalized projection geometry is introduced for further understanding such a new mechanism furthermore new algorithms are also developed for implementing gaussian factor analysis fa and non gaussian factor analysis nfa such that selecting appropriate factors is automatically made during parameter learning

minimax entropy principle and its application to texture modeling this article proposes a general theory and methodology called the minimax entropy principle for building statistical models for images or signals in a variety of applications this principle consists of two parts the first is the maximum entropy principle for feature binding or fusion for a given set of observed feature statistics a distribution can be built to bind these feature statistics together by maximizing the entropy over all distributions that reproduce them the second part is the minimum entropy principle for feature selection among all plausible sets of feature statistics we choose the set whose maximum entropy distribution has the minimum entropy computational and inferential issues in both parts are addressed in particular a feature pursuit procedure is proposed for approximately selecting the optimal set of features the minimax entropy principle is then corrected by considering the sample variation in the observed feature statistics and an information criterion for feature pursuit is derived the minimax entropy principle is applied to texture modeling where a novel markov random field mrf model called frame filter random field and minimax entropy is derived and encouraging results are obtained in experiments on a variety of texture images the relationship between our theory and the mechanisms of neural computation is also discussed

ridge regression biased estimation for nonorthogonal problems in multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory if not incorrect if the prediction vectors are not orthogonal proposed is an estimation procedure based on adding small positive quantities to the diagonal of x x introduced is the ridge trace a method for showing in two dimensions the effects of nonorthogonality it is then shown how to augment x x to obtain biased estimates with smaller mean square error

bayesian prediction of deterministic functions with applications to the design and analysis of computer experiments abstract this article is concerned with prediction of a function y t over a multidimensional domain t given the function values at a set of sites t 1 t 2 t n in t and with the design that is with the selection of those sites the motivating application is the design and analysis of computer experiments where t determines the input to a computer model of a physical or behavioral system and y t is a response that is part of the output or is calculated from it following a bayesian formulation prior uncertainty about the function y is expressed by means of a random function y which is taken here to be a gaussian stochastic process the mean of the posterior process can be used as the prediction function t and the variance can be used as a measure of uncertainty this kind of approach has been used previously in bayesian interpolation and is strongly related to the kriging methods used in geostatistics here emphasis is placed on product linear and product cubic correlation func

gaussian process surrogates for failure detection an important task of uncertainty quantification is to identify the probability of undesired events in particular system failures caused by various sources of uncertainties in this work we consider the construction of gaussian process surrogates for failure detection and failure probability estimation in particular we consider the situation that the underlying computer models are extremely expensive and in this setting determining the sampling points in the state space is of essential importance we formulate the problem as an optimal experimental design for bayesian inferences of the limit state i e the failure boundary and propose an efficient numerical scheme to solve the resulting optimization problem in particular the proposed limit state inference method is capable of determining multiple sampling points at a time and thus it is well suited for problems where multiple computer simulations can be performed in parallel the accuracy and performance of the proposed method is demonstrated by both academic and practical examples

image denoising via sparse and redundant representations over learned dictionaries we address the image denoising problem where zero mean white and homogeneous gaussian additive noise is to be removed from a given image the approach taken is based on sparse and redundant representations over trained dictionaries using the k svd algorithm we obtain a dictionary that describes the image content effectively two training options are considered using the corrupted image itself or training on a corpus of high quality image database since the k svd is limited in handling small image patches we extend its deployment to arbitrary image sizes by defining a global image prior that forces sparsity over patches in every location in the image we show how such bayesian treatment leads to a simple and effective denoising algorithm this leads to a state of the art denoising performance equivalent and sometimes surpassing recently published leading alternative denoising methods

bayesian sequential parameter estimation by cognitive radar with multiantenna arrays in this paper we consider the problem of bayesian sequential parameter estimation of extended targets for cognitive radar with multi antenna arrays using adaptive waveforms the target is modeled as a complex gaussian random process using iterative waveform transmission the cognitive radar estimates the target s characteristic parameters and updates its probabilistic model based on new measurements the adaptive waveform is designed by minimizing the conditional entropy from the posterior density of the model parameters we analyze the performance of the developed bayesian sequential estimation algorithm and derive expressions for the signal to noise ratio gain the asymp totical posterior cramer rao bound and the mutual information gain the analysis and numerical simulations demonstrate that the adaptive sequential bayesian estimator yields accelerated convergence of the estimate towards its true value and a smaller estimation error compared with the conventional bayesian esti mator that uses fixed waveform transmission under gaussian or non gaussian noise

improving the sample complexity using global data we study the sample complexity of proper and improper learning problems with respect to different q loss functions we improve the known estimates for classes which have relatively small covering numbers in empirical l sub 2 spaces e g log covering numbers which are polynomial with exponent p 2 we present several examples of relevant classes which have a small fat shattering dimension and hence fit our setup the most important of which are kernel machines

competition and multiple cause models if different causes can interact on any occasion to generate a set of patterns then systems modeling the generation have to model the interaction too we discuss a way of combining multiple causes that is based on the integrated segmentation and recognition architecture of keeler et al 1991 it is more cooperative than the scheme embodied in the mixture of experts architecture which insists that just one cause generate each output and more competitive than the noisy or combination function which was recently suggested by saund 1994a b simulations confirm its efficacy

leave one out bounds for kernel methods in this article we study leave one out style cross validation bounds for kernel methods the essential element in our analysis is a bound on the parameter estimation stability for regularized kernel formulations using this result we derive bounds on expected leave one out cross validation errors which lead to expected generalization bounds for various kernel algorithms in addition we also obtain variance bounds for leave one out errors we apply our analysis to some classification and regression problems and compare them with previous results

optimal bayesian design by inhomogeneous markov chain simulation we consider decision problems defined by a utility function and an underlying probability model for all unknowns the utility function quantifies the decision maker s preferences over consequences the optimal decision maximizes the expected utility function where the expectation is taken with respect to all unknowns that is future data and parameters in many problems the solution is not analytically tractable for example the utility function might involve moments that can be computed only by numerical integration or simulation also the nature of the decision space i e the set of all possible actions might have a shape or dimension that complicates the maximization the motivating application for this discussion is the choice of a monitoring network when the optimization is performed over the high dimensional set of all possible locations of monitoring stations possibly including choice of the number of locations we propose an approach to optimal bayesian design based on inhomogeneous markov

sketched ridge regression optimization perspective statistical perspective and model averaging we address the statistical and optimization impacts of the classical sketch and hessian sketch used to approximately solve the matrix ridge regression mrr problem prior research has quantified the effects of classical sketch on the strictly simpler least squares regression lsr problem we establish that classical sketch has a similar effect upon the optimization properties of mrr as it does on those of lsr namely it recovers nearly optimal solutions by contrast hessian sketch does not have this guarantee instead the approximation error is governed by a subtle interplay between the mass in the responses and the optimal objective value for both types of approximation the regularization in the sketched mrr problem results in significantly different statistical properties from those of the sketched lsr problem in particular there is a bias variance trade off in sketched mrr that is not present in sketched lsr we provide upper and lower bounds on the bias and variance of sketched mrr these bounds show that classical sketch significantly increases the variance while hessian sketch significantly increases the bias empirically sketched mrr solutions can have risks that are higher by an order of magnitude than those of the optimal mrr solutions we establish theoretically and empirically that model averaging greatly decreases the gap between the risks of the true and sketched solutions to the mrr problem thus in parallel or distributed settings sketching combined with model averaging is a powerful technique that quickly obtains near optimal solutions to the mrr problem while greatly mitigating the increased statistical risk incurred by sketching

another look at distance weighted discrimination summary distance weighted discrimination dwd is a modern margin based classifier with an interesting geometric motivation it was proposed as a competitor to the support vector machine svm despite many recent references on dwd dwd is far less popular than the svm mainly because of computational and theoretical reasons we greatly advance the current dwd methodology and its learning theory we propose a novel thrifty algorithm for solving standard dwd and generalized dwd and our algorithm can be several hundred times faster than the existing state of the art algorithm based on second order cone programming in addition we exploit the new algorithm to design an efficient scheme to tune generalized dwd furthermore we formulate a natural kernel dwd approach in a reproducing kernel hilbert space and then establish the bayes risk consistency of the kernel dwd by using a universal kernel such as the gaussian kernel this result solves an open theoretical problem in the dwd literature a comparison study on 16 benchmark data sets shows that data driven generalized dwd consistently delivers higher classification accuracy with less computation time than the svm

decision theoretic sensitivity analysis for complex computer models when using a computer model to inform a decision it is important to investigate any uncertainty in the model and determine how that uncertainty may impact on the decision in probabilistic sensitivity analysis model users can investigate how various uncertain model inputs contribute to the uncertainty in the model output however much of the literature focuses only on output uncertainty as measured by variance the decision problem itself often is ignored even though uncertainty as measured by variance may not equate to uncertainty about the optimum decision consequently traditional variance based measures of input parameter importance may not correctly describe the importance of each input we review a decision theoretic framework for conducting sensitivity analysis that addresses this problem because computation of these decision theoretic measures can be impractical for complex computer models we provide efficient computational tools using gaussian processes we give an illustration in the fiel

traveling waves in diffusive predator prey equations periodic orbits and point to periodic heteroclinic orbits we demonstrate the existence of periodic traveling wave train and traveling front solutions for a diffusive predator prey system the analysis is in the three dimensional phase space of the nonlinear ordinary differential equation system given by the diffusive predator prey system in the traveling wave variable the analysis shows the existence of periodic orbits heteroclinic orbits and a heteroclinic connection of a point and a periodic orbit the proof uses shooting techniques invariant manifold theory and the qualitative theory of ordinary differential equations

a multiple cause mixture model for unsupervised learning this paper presents a formulation for unsupervised learning of clusters reflecting multiple causal structure in binary data unlike the hard k means clustering algorithm and the soft mixture model each of which assumes that a single hidden event generates each data point a multiple cause model accounts for observed data by combining assertions from many hidden causes each of which can pertain to varying degree to any subset of the observable dimensions we employ an objective function and iterative gradient descent learning algorithm resembling the conventional mixture model a crucial issue is the mixing function for combining beliefs from different cluster centers in order to generate data predictions whose errors are minimized both during recognition and learning the mixing function constitutes a prior assumption about underlying structural regularities of the data domain we demonstrate a weakness inherent to the popular weighted sum followed by sigmoid squashing and offer alternative forms of the nonlinearity for two types of data domain results are presented demonstrating the algorithm s ability successfully to discover coherent multiple causal representations in several experimental data sets

learning theory for distribution regression we focus on the distribution regression problem regressing to vector valued outputs from probability measures many important machine learning and statistical tasks fit into this framework including multi instance learning and point estimation problems without analytical solution such as hyperparameter or entropy estimation despite the large number of available heuristics in the literature the inherent two stage sampled nature of the problem makes the theoretical analysis quite challenging since in practice only samples from sampled distributions are observable and the estimates have to rely on similarities computed between sets of points to the best of our knowledge the only existing technique with consistency guarantees for distribution regression requires kernel density estimation as an intermediate step which often performs poorly in practice and the domain of the distributions to be compact euclidean in this paper we study a simple analytically computable ridge regression based alternative to distribution regression where we embed the distributions to a reproducing kernel hilbert space and learn the regressor from the embeddings to the outputs our main contribution is to prove that this scheme is consistent in the two stage sampled setup under mild conditions on separable topological domains enriched with kernels we present an exact computational statistical efficiency trade off analysis showing that our estimator is able to match the one stage sampled minimax optimal rate caponnetto and de vito 2007 steinwart et al 2009 this result answers a 17 year old open question establishing the consistency of the classical set kernel haussler 1999 gartner et al 2002 in regression we also cover consistency for more recent kernels on distributions including those due to christmann and steinwart 2010

learning bounds for kernel regression using effective data dimensionality kernel methods can embed finite dimensional data into infinite dimensional feature spaces in spite of the large underlying feature dimensionality kernel methods can achieve good generalization ability this observation is often wrongly interpreted and it has been used to argue that kernel learning can magically avoid the curse of dimensionality phenomenon encountered in statistical estimation problems this letter shows that although using kernel representation one can embed data into an infinite dimensional feature space the effective dimensionality of this embedding which determines the learning complexity of the underlying kernel machine is usually small in particular we introduce an algebraic definition of a scale sensitive effective dimension associated with a kernel representation based on this quantity we derive upper bounds on the generalization performance of some kernel regression methods moreover we show that the resulting convergent rates are optimal under various circumstances

emergence of optimal decoding of population codes through stdp the brain faces the problem of inferring reliable hidden causes from large populations of noisy neurons for example the direction of a moving object from spikes in area mt it is known that a theoretically optimal likelihood decoding could be carried out by simple linear readout neurons if weights of synaptic connections were set to certain values that depend on the tuning functions of sensory neurons we show here that such theoretically optimal readout weights emerge autonomously through stdp in conjunction with lateral inhibition between readout neurons in particular we identify a class of optimal stdp learning rules with homeostatic plasticity for which the autonomous emergence of optimal readouts can be explained on the basis of a rigorous learning theory this theory shows that the network motif we consider approximates expectation maximization for creating internal generative models for hidden causes of high dimensional spike inputs notably we find that this optimal functionality can be well approximated by a variety of stdp rules beyond those predicted by theory furthermore we show that this learning process is very stable and automatically adjusts weights to changes in the number of readout neurons the tuning functions of sensory neurons and the statistics of external stimuli

gp select accelerating em using adaptive subspace preselection we propose a nonparametric procedure to achieve fast inference in generative graphical models when the number of latent states is very large the approach is based on iterative latent variable preselection where we alternate between learning a selection function to reveal the relevant latent variables and using this to obtain a compact approximation of the posterior distribution for em this can make inference possible where the number of possible latent states is for example exponential in the number of latent variables whereas an exact approach would be computationally infeasible we learn the selection function entirely from the observed data and current expectation maximization state via gaussian process regression this is in contrast to earlier approaches where selection functions were manually designed for each problem setting we show that our approach performs as well as these bespoke selection functions on a wide variety of inference problems in particular for the challenging case of a hierarchical model for object localization with occlusion we achieve results that match a customized state of the art selection method at a far lower computational cost

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

divide and conquer kernel ridge regression a distributed algorithm with minimax optimal rates we study a decomposition based scalable approach to kernel ridge regression and show that it achieves minimax optimal convergence rates under relatively mild conditions the method is simple to describe it randomly partitions a dataset of size n into m subsets of equal size computes an independent kernel ridge regression estimator for each subset using a careful choice of the regularization parameter then averages the local solutions into a global predictor this partitioning leads to a substantial reduction in computation time versus the standard approach of performing kernel ridge regression on all n samples our two main theorems establish that despite the computational speed up statistical optimality is retained as long as m is not too large the partition based estimator achieves the statistical minimax rate over all estimators using the set of n samples as concrete examples our theory guarantees that the number of subsets m may grow nearly linearly for finite rank or gaussian kernels and polynomially in n for sobolev spaces which in turn allows for substantial reductions in computational cost we conclude with experiments on both simulated data and a music prediction task that complement our theoretical results exhibiting the computational and statistical benefits of our approach

a bayesian analysis of self organizing maps in this paper bayesian methods are used to analyze some of the properties of a special type of markov chain the forward transitions through the chain are followed by inverse transitions using bayes theorem backward through a copy of the same chain this will be called a folded markov chain if an appropriately defined euclidean error between the original input and its reconstruction via bayes theorem is minimized with respect to the choice of markov chain transition probabilities then the familiar theories of both vector quantizers and self organizing maps emerge this approach is also used to derive the theory of self supervision in which the higher layers of a multilayer network supervise the lower layers even though overall there is no external teacher

optimal learning rates for localized svms one of the limiting factors of using support vector machines svms in large scale applications are their super linear computational requirements in terms of the number of training samples to address this issue several approaches that train svms on many small chunks separately have been proposed in the literature with the exception of random chunks which is also known as divide and conquer kernel ridge regression however these approaches have only been empirically investigated in this work we investigate a spatially oriented method to generate the chunks for the resulting localized svm that uses gaussian kernels and the least squares loss we derive an oracle inequality which in turn is used to deduce learning rates that are essentially minimax optimal under some standard smoothness assumptions on the regression function in addition we derive local learning rates that are based on the local smoothness of the regression function we further introduce a data dependent parameter selection method for our local svm approach and show that this method achieves the same almost optimal learning rates finally we present a few larger scale experiments for our localized svm showing that it achieves essentially the same test error as a global svm for a fraction of the computational requirements in addition it turns out that the computational requirements for the local svms are similar to those of a vanilla random chunk approach while the achieved test errors are significantly better

computing with continuous attractors stability and online aspects two issues concerning the application of continuous attractors in neural systems are investigated the computational robustness of continuous attractors with respect to input noises and the implementation of bayesian online decoding in a perfect mathematical model for continuous attractors decoding results for stimuli are highly sensitive to input noises and this sensitivity is the inevitable consequence of the system s neutral stability to overcome this shortcoming we modify the conventional network model by including extra dynamical interactions between neurons these interactions vary according to the biologically plausible hebbian learning rule and have the computational role of memorizing and propagating stimulus information accumulated with time as a result the new network model responds to the history of external inputs over a period of time and hence becomes insensitive to short term fluctuations also since dynamical interactions provide a mechanism to convey the prior knowledge of stimulus that is the information of the stimulus presented previously the network effectively implements online bayesian inference this study also reveals some interesting behavior in neural population coding such as the trade off between decoding stability and the speed of tracking time varying stimuli and the relationship between neural tuning width and the tracking speed

synergy between object recognition and image segmentation using the expectation maximization algorithm in this work we formulate the interaction between image segmentation and object recognition in the framework of the expectation maximization em algorithm we consider segmentation as the assignment of image observations to object hypotheses and phrase it as the e step while the m step amounts to fitting the object models to the observations these two tasks are performed iteratively thereby simultaneously segmenting an image and reconstructing it in terms of objects we model objects using active appearance models aams as they capture both shape and appearance variation during the e step the fidelity of the aam predictions to the image is used to decide about assigning observations to the object for this we propose two top down segmentation algorithms the first starts with an oversegmentation of the image and then softly assigns image segments to objects as in the common setting of em the second uses curve evolution to minimize a criterion derived from the variational interpretation of em and introduces aams as shape priors for the m step we derive aam fitting equations that accommodate segmentation information thereby allowing for the automated treatment of occlusions apart from top down segmentation results we provide systematic experiments on object detection that validate the merits of our joint segmentation and recognition approach

images frames and connectionist hierarchies the representation of hierarchically structured knowledge in systems using distributed patterns of activity is an abiding concern for the connectionist solution of cognitively rich problems here we use statistical unsupervised learning to consider semantic aspects of structured knowledge representation we meld unsupervised learning notions formulated for multilinear models with tensor product ideas for representing rich information we apply the model to images of faces

early stopping and non parametric regression an optimal data dependent stopping rule early stopping is a form of regularization based on choosing when to stop running an iterative algorithm focusing on non parametric regression in a reproducing kernel hilbert space we analyze the early stopping strategy for a form of gradient descent applied to the least squares loss function we propose a data dependent stopping rule that does not involve hold out or cross validation data and we prove upper bounds on the squared error of the resulting function estimate measured in either the l2 p and l2 pn norm these upper bounds lead to minimax optimal rates for various kernel classes including sobolev smoothness classes and other forms of reproducing kernel hilbert spaces we show through simulation that our stopping rule compares favorably to two other stopping rules one based on hold out data and the other based on stein s unbiased risk estimate we also establish a tight connection between our early stopping strategy and the solution path of a kernel ridge regression estimator

bayesian optimal design via interacting particle systems we propose a new stochastic algorithm for bayesian optimal design in nonlinear and high dimensional contexts following peter muller we solve an optimization problem by exploring the expected utility surface through markov chain monte carlo simulations the optimal design is the mode of this surface considered a probability distribution our algorithm relies on a particle method to efficiently explore high dimensional multimodal surfaces with simulated annealing to concentrate the samples near the modes we first test the method on an optimal allocation problem for which the explicit solution is available to compare its efficiency with a simpler algorithm we then apply our method to a challenging medical case study in which an optimal protocol treatment needs to be determined for this case we propose a formalization of the problem in the framework of bayesian decision theory taking into account physicians knowledge and motivations we also briefly review further improvements and alternatives

probabilistic sensitivity analysis of complex models a bayesian approach summary in many areas of science and technology mathematical models are built to simulate complex real world phenomena such models are typically implemented in large computer programs and are also very complex such that the way that the model responds to changes in its inputs is not transparent sensitivity analysis is concerned with understanding how changes in the model inputs influence the outputs this may be motivated simply by a wish to understand the implications of a complex model but often arises because there is uncertainty about the true values of the inputs that should be used for a particular application a broad range of measures have been advocated in the literature to quantify and describe the sensitivity of a model s output to variation in its inputs in practice the most commonly used measures are those that are based on formulating uncertainty in the model inputs by a joint probability distribution and then analysing the induced uncertainty in outputs an approach which is known as probabilistic sensitivity analysis we present a bayesian framework which unifies the various tools of prob abilistic sensitivity analysis the bayesian approach is computationally highly efficient it allows effective sensitivity analysis to be achieved by using far smaller numbers of model runs than standard monte carlo methods furthermore all measures of interest may be computed from a single set of runs

distributed testing and estimation under sparse high dimensional models this paper studies hypothesis testing and parameter estimation in the context of the divide and conquer algorithm in a unified likelihood based framework we propose new test statistics and point estimators obtained by aggregating various statistics from k subsamples of size n k where n is the sample size in both low dimensional and sparse high dimensional settings we address the important question of how large k can be as n grows large such that the loss of efficiency due to the divide and conquer algorithm is negligible in other words the resulting estimators have the same inferential efficiencies and estimation rates as an oracle with access to the full sample thorough numerical results are provided to back up the theory

maximal causes for non linear component extraction we study a generative model in which hidden causes combine competitively to produce observations multiple active causes combine to determine the value of an observed variable through a max function in the place where algorithms such as sparse coding independent component analysis or non negative matrix factorization would use a sum this max rule can represent a more realistic model of non linear interaction between basic components in many settings including acoustic and image data while exact maximum likelihood learning of the parameters of this model proves to be intractable we show that efficient approximations to expectation maximization em can be found in the case of sparsely active hidden causes one of these approximations can be formulated as a neural network model with a generalized softmax activation function and hebbian learning thus we show that learning in recent softmax like neural networks may be interpreted as approximate maximization of a data likelihood we use the bars benchmark test to numerically verify our analytical results and to demonstrate the competitiveness of the resulting algorithms finally we show results of learning model parameters to fit acoustic and visual data sets in which max like component combinations arise naturally

distributed regression learning with coefficient regularization abstract we study distributed regression learning with coefficient regularization scheme in a reproducing kernel hilbert space rkhs the algorithm randomly partitions the sample set z i i 1 n into m disjoint sample subsets of equal size applies the coefficient regularization scheme to each sample subset to produce an output function and averages the individual output functions to get the final global estimator we deduce the error bound in expectation in the l 2 metric and prove the asymptotic convergence for this distributed coefficient regularization learning satisfactory learning rates are then derived under a standard regularity condition on the regression function which reveals an interesting phenomenon that when m n s and s is small enough this distributed learning has the same convergence rate compared with the algorithm processing the whole data in one single machine

efficient svm training using low rank kernel representations svm training is a convex optimization problem which scales with the training set size rather than the feature space dimension while this is usually considered to be a desired quality in large scale problems it may cause training to be impractical the common techniques to handle this difficulty basically build a solution by solving a sequence of small scale subproblems our current effort is concentrated on the rank of the kernel matrix as a source for further enhancement of the training procedure we first show that for a low rank kernel matrix it is possible to design a better interior point method ipm in terms of storage requirements as well as computational complexity we then suggest an efficient use of a known factorization technique to approximate a given kernel matrix by a low rank matrix which in turn will be used to feed the optimizer finally we derive an upper bound on the change in the objective function value based on the approximation error and the number of active constraints support vectors this bound is general in the sense that it holds regardless of the approximation method

invariant probabilistic sensitivity analysis in evaluating opportunities investors wish to identify key sources of uncertainty we propose a new way to measure how sensitive model outputs are to each probabilistic input e g revenues growth idiosyncratic risk parameters we base our approach on measuring the distance between cumulative distributions risk profiles using a metric that is invariant to monotonic transformations thus the sensitivity measure will not vary by alternative specifications of the utility function over the output to measure separation we propose using either kuiper s metric or kolmogorov smirnov s metric we illustrate the advantages of our proposed sensitivity measure by comparing it with others most notably the contribution to variance measures our measure can be obtained as a by product of a monte carlo simulation we illustrate our approach in several examples focusing on investment analysis situations this paper was accepted by peter wakker decision analysis

bayesian inference for the uncertainty distribution of computer model outputs we consider a problem of inference for the output of a computationally expensive computer model we suppose that the model is to be used in a context where the values of one or more inputs are uncertain so that the input configuration is a random variable we require to make inference about the induced distribution of the output this distribution is called the uncertainty distribution and the general problem is known to users of computer models as uncertainty analysis to be specific we develop bayesian inference for the distribution and density functions of the model output modelling the output as a function of its inputs as a gaussian process we derive expressions for the posterior mean and variance of the distribution and density functions based on data comprising observed outputs at a sample of input configurations we show that direct computation of these expressions may encounter numerical difficulties we develop an alternative approach based on simulating approximate realisations from the posterior distribution of the output function two examples are given to illustrate our methods

local rademacher complexities and oracle inequalities in risk minimization let be a class of measurable functions f s 0 1 defined on a probability space s mathcal a p given a sample x1 xn of i i d random variables taking values in s with common distribution p let pn denote the empirical measure based on x1 xn we study an empirical risk minimization problem pnf min f given a solution fn of this problem the goal is to obtain very general upper bounds on its excess risk mathcal e p hat f n p hat f n inf f in mathcal f pf expressed in terms of relevant geometric parameters of the class using concentration inequalities and other empirical processes tools we obtain both distribution dependent and data dependent upper bounds on the excess risk that are of asymptotically correct order in many examples the bounds involve localized sup norms of empirical and rademacher processes indexed by functions from the class we use these bounds to develop model selection techniques in abstract risk minimization problems that can be applied to more specialized frameworks of regression and classification

bayesian computation in recurrent neural circuits a large number of human psychophysical results have been successfully explained in recent years using bayesian models however the neural implementation of such models remains largely unclear in this article we show that a network architecture commonly used to model the cerebral cortex can implement bayesian inference for an arbitrary hidden markov model we illustrate the approach using an orientation discrimination task and a visual motion detection task in the case of orientation discrimination we show that the model network can infer the posterior distribution over orientations and correctly estimate stimulus orientation in the presence of significant noise in the case of motion detection we show that the resulting model network exhibits direction selectivity and correctly computes the posterior probabilities over motion direction and position when used to solve the well known random dots motion discrimination task the model generates responses that mimic the activities of evidence accumulating neurons in cortical areas lip and fef the framework we introduce posits a new interpretation of cortical activities in terms of log posterior probabilities of stimuli occurring in the natural world

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

existence of travelling wave solutions of predator prey systems via the connection index we consider a reaction diffusion system which arises as a model of predator prey interactions in mathematical ecology the system admits four constant equilibria two of which are stable the main result is an existence theorem for travelling wave solutions which connect the two stable equilibria most previous efforts with regard to travelling waves for systems employ singular perturbations to locate the connecting orbit the new aspect of the present discussion lies in devising some novel methods for locating the connecting solution in phase space which do not require that the equations be singularly perturbed in certain cases the main result holds for an arbitrary choice of diffusion coefficients

general conditions for the existence of a critical point periodic wave front connection for reaction diffusion systems on etudie l existence de fronts d onde plane heteroclines pour les equations u t d u f u et l equation des etats homogenes du dt f u u r n r r m x t u x t u i x t 1 i m d est une matrice symetrique definie positive

expectation truncation and the benefits of preselection in training generative models we show how a preselection of hidden variables can be used to efficiently train generative models with binary hidden variables the approach is based on expectation maximization em and uses an efficiently computable approximation to the sufficient statistics of a given model the computational cost to compute the sufficient statistics is strongly reduced by selecting for each data point the relevant hidden causes the approximation is applicable to a wide range of generative models and provides an interpretation of the benefits of preselection in terms of a variational em approximation to empirically show that the method maximizes the data likelihood it is applied to different types of generative models including a version of non negative matrix factorization nmf a model for non linear component extraction mca and a linear generative model similar to sparse coding the derived algorithms are applied to both artificial and realistic data and are compared to other models in the literature we find that the training scheme can reduce computational costs by orders of magnitude and allows for a reliable extraction of hidden causes

debiased magnitude preserving ranking learning rate and bias characterization abstract magnitude preserving ranking mprank under tikhonov regularization framework has shown competitive performance on information retrial besides theoretical advantages for computation feasibility and statistical guarantees in this paper we further characterize the learning rate and asymptotic bias of mprank and then propose a new debiased ranking algorithm in terms of the operator representation and approximation techniques we establish their convergence rates and bias characterizations these theoretical results demonstrate that the new model has smaller asymptotic bias than mprank and can achieve the satisfactory convergence rate under appropriate conditions in addition some empirical examples are provided to verify the effectiveness of our debiased strategy

a truncated em approach for spike and slab sparse coding we study inference and learning based on a sparse coding model with spike and slab prior as in standard sparse coding the model used assumes independent latent sources that linearly combine to generate data points however instead of using a standard sparse prior such as a laplace distribution we study the application of a more flexible spike and slab distribution which models the absence or presence of a source s contribution independently of its strength if it contributes we investigate two approaches to optimize the parameters of spike and slab sparse coding a novel truncated em approach and for comparison an approach based on standard factored variational distributions the truncated approach can be regarded as a variational approach with truncated posteriors as variational distributions in applications to source separation we find that both approaches improve the state of the art in a number of standard benchmarks which argues for the use of spike and slab priors for the corresponding data domains furthermore we find that the truncated em approach improves on the standard factored approach in source separation tasks which hints to biases introduced by assuming posterior independence in the factored variational approach likewise on a standard benchmark for image denoising we find that the truncated em approach improves on the factored variational approach while the performance of the factored approach saturates with increasing numbers of hidden dimensions the performance of the truncated approach improves the state of the art for higher noise levels

transformations and invariance in the sensitivity analysis of computer experiments summary monotonic transformations are widely employed in statistics and data analysis in computer experiments they are often used to gain accuracy in the estimation of global sensitivity statistics however one faces the question of interpreting results that are obtained on the transformed data back on the original data the situation is even more complex in computer experiments because transformations alter the model input output mapping and distort the estimators this work demonstrates that the problem can be solved by utilizing statistics which are monotonic transformation invariant to do so we offer an investigation into the families of metrics either based on densities or on cumulative distribution functions that are monotonic transformation invariant and we introduce a new generalized family of metrics numerical experiments show that transformations allow numerical convergence in the estimates of global sensitivity statistics both invariant and not in cases in which it would otherwise be impossible to obtain convergence however one fully exploits the increased numerical accuracy if the global sensitivity statistic is monotonic transformation invariant conversely estimators of measures that do not have this invariance property might lead to misleading deductions

local rademacher complexities we propose new bounds on the error of learning algorithms in terms of a data dependent notion of complexity the estimates we establish give optimal rates and are based on a local and empirical version of rademacher averages in the sense that the rademacher averages are computed from the data on a subset of functions with small empirical error we present some applications to classification and prediction with convex function classes and with kernel classes in particular

compressed gaussian process for manifold regression nonparametric regression for large numbers of features p is an increasingly important problem if the sample size n is massive a common strategy is to partition the feature space and then separately apply simple models to each partition set this is not ideal when n is modest relative to p and we propose an alternative approach relying on random compression of the feature vector combined with gaussian process regression the proposed approach is particularly motivated by the setting in which the response is conditionally independent of the features given the projection to a low dimensional manifold conditionally on the random compression matrix and a smoothness parameter the posterior distribution for the regression surface and posterior predictive distributions are available analytically running the analysis in parallel for many random compression matrices and smoothness parameters model averaging is used to combine the results the algorithm can be implemented rapidly even in very large p and moderately large n nonparametric regression has strong theoretical justification and is found to yield state of the art predictive performance

on the relative importance of input factors in mathematical models safety assessment for nuclear waste disposal this article deals with global quantitative sensitivity analysis of the level e model a computer code used in safety assessment for nuclear waste disposal the level e code has been the subject of two international benchmarks of risk assessment codes and monte carlo methods and is well known in the literature we discuss the level e model with reference to two different settings in the first setting the objective is to find the input factor that drives most of the output variance in the second setting we strive to achieve a preestablished reduction in the variance of the model output by fixing the smallest number of factors the emphasis of this work is on how to define the concept of importance in an unambiguous way and how to assess it in the simultaneous occurrence of correlated input factors and non additive models

modeling the manifolds of images of handwritten digits this paper describes two new methods for modeling the manifolds of digitized images of handwritten digits the models allow a priori information about the structure of the manifolds to be combined with empirical data accurate modeling of the manifolds allows digits to be discriminated using the relative probability densities under the alternative models one of the methods is grounded in principal components analysis the other in factor analysis both methods are based on locally linear low dimensional approximations to the underlying data manifold links with other methods that model the manifold are discussed

integration of stochastic models by minimizing α divergence when there are a number of stochastic models in the form of probability distributions one needs to integrate them mixtures of distributions are frequently used but exponential mixtures also provide a good means of integration this letter proposes a one parameter family of integration called integration which includes all of these well known integrations these are generalizations of various averages of numbers such as arithmetic geometric and harmonic averages there are psychophysical experiments that suggest that integrations are used in the brain the divergence between two distributions is defined which is a natural generalization of kullback leibler divergence and hellinger distance and it is proved that integration is optimal in the sense of minimizing divergence the theory is applied to generalize the mixture of experts and the product of experts to the mixture of experts the predictive distribution is also stated in the bayesian framework