divisive gaussian processes for nonstationary regression standard gaussian process regression gpr assumes constant noise power throughout the input space and stationarity when combined with the squared exponential covariance function this can be unrealistic and too restrictive for many real world problems nonstationarity can be achieved by specific covariance functions though prior knowledge about this nonstationarity can be difficult to obtain on the other hand the homoscedastic assumption is needed to allow gpr inference to be tractable in this paper we present a divisive gpr model which performs nonstationary regression under heteroscedastic noise using the pointwise division of two nonparametric latent functions as the inference on the model is not analytically tractable we propose a variational posterior approximation using expectation propagation ep which allows for accurate inference at reduced cost we have also made a markov chain monte carlo implementation with elliptical slice sampling to assess the quality of the ep approximation experiments support the usefulness of the proposed approach

some aspects of the spline smoothing approach to non parametric regression curve fitting non parametric regression using cubic splines is an attractive flexible and widely applicable approach to curve estimation although the basic idea was formulated many years ago the method is not as widely known or adopted as perhaps it should be the topics and examples discussed in this paper are intended to promote the understanding and extend the practicability of the spline smoothing methodology particular subjects covered include the basic principles of the method the relation with moving average and other smoothing methods the automatic choice of the amount of smoothing and the use of residuals for diagnostic checking and model adaptation the question of providing inference regions for curves and for relevant properties of curves is approached via a finite dimensional bayesian formulation

infinite dimensional bayesian filtering for detection of quasiperiodic phenomena in spatiotemporal data this paper introduces a spatiotemporal resonator model and an inference method for detection and estimation of nearly periodic temporal phenomena in spatiotemporal data the model is derived as a spatial extension of a stochastic harmonic resonator model which can be formulated in terms of a stochastic differential equation the spatial structure is included by introducing linear operators which affect both the oscillations and damping and by choosing the appropriate spatial covariance structure of the driving time white noise process with the choice of the linear operators as partial differential operators the resonator model becomes a stochastic partial differential equation which is compatible with infinite dimensional kalman filtering the resulting infinite dimensional kalman filtering problem allows for a computationally efficient solution as the computational cost scales linearly with measurements in the temporal dimension this framework is applied to weather prediction and to physiological noise elimination in functional magnetic resonance imaging brain data

best approximation of gaussian neural networks with nodes uniformly spaced this paper is aimed at exposing the reader to certain aspects in the design of the best approximants with gaussian radial basis functions rbfs the class of functions to which this approach applies consists of those compactly supported in frequency the approximative properties of uniqueness and existence are restricted to this class functions which are smooth enough can be expanded in gaussian series converging uniformly to the objective function the uniqueness of these series is demonstrated by the context of the orthonormal basis in a hilbert space furthermore the best approximation to a given band limited function from a truncated gaussian series is analyzed by an energy based argument this analysis not only gives a theoretical proof concerned with the existence of best approximations but addresses the problems of architectural selection specifically guidance for selecting the variance and the oversampling parameters is provided for practitioners

online learning of single and multivalued functions with an infinite mixture of linear experts we present a supervised learning algorithm for estimation of generic input output relations in a real time online fashion the proposed method is based on a generalized expectation maximization approach to fit an infinite mixture of linear experts imle to an online stream of data samples this probabilistic model while not fully bayesian can efficiently choose the number of experts that are allocated to the mixture this way effectively controlling the complexity of the resulting model the result is an incremental online and localized learning algorithm that performs nonlinear multivariate regression on multivariate outputs by approximating the target function by a linear relation within each expert input domain and that can allocate new experts as needed a distinctive feature of the proposed method is the ability to learn multivalued functions one to many mappings that naturally arise in some robotic and computer vision learning domains using an approach based on a bayesian generative model for the predictions provided by each of the mixture experts as a consequence it is able to directly provide forward and inverse relations from the same learned mixture model we conduct an extensive set of experiments to evaluate the proposed algorithm performance and the results show that it can outperform state of the art online function approximation algorithms in single valued regression while demonstrating good estimation capabilities in a multivalued function approximation context

efficient optimization for sparse gaussian process regression we propose an efficient optimization algorithm to select a subset of training data as the inducing set for sparse gaussian process regression previous methods either use different objective functions for inducing set and hyperparameter selection or else optimize the inducing set by gradient based continuous optimization the former approaches are harder to interpret and suboptimal whereas the latter cannot be applied to discrete input domains or to kernel functions that are not differentiable with respect to the input the algorithm proposed in this work estimates an inducing set and the hyperparameters using a single objective it can be used to optimize either the marginal likelihood or a variational free energy space and time complexity are linear in training set size and the algorithm can be applied to large regression problems on discrete or continuous domains empirical evaluation shows state of art performance in discrete cases competitive prediction results as well as a favorable trade off between training and test time in continuous cases

sparse spectrum gaussian process regression we present a new sparse gaussian process gp model for regression the key novel idea is to sparsify the spectral representation of the gp this leads to a simple practical algorithm for regression tasks we compare the achievable trade offs between predictive accuracy and computational requirements and show that these are typically superior to existing state of the art sparse approximations we discuss both the weight space and function space representations and note that the new construction implies priors over functions which are always stationary and can approximate any covariance function in this class

3d human pose tracking priors using geodesic mixture models we present a novel approach for learning a finite mixture model on a riemannian manifold in which euclidean metrics are not applicable and one needs to resort to geodesic distances consistent with the manifold geometry for this purpose we draw inspiration on a variant of the expectation maximization algorithm that uses a minimum message length criterion to automatically estimate the optimal number of components from multivariate data lying on an euclidean space in order to use this approach on riemannian manifolds we propose a formulation in which each component is defined on a different tangent space thus avoiding the problems associated with the loss of accuracy produced when linearizing the manifold with a single tangent space our approach can be applied to any type of manifold for which it is possible to estimate its tangent space additionally we consider using shrinkage covariance estimation to improve the robustness of the method especially when dealing with very sparsely distributed samples we evaluate the approach on a number of situations going from data clustering on manifolds to combining pose and kinematics of articulated bodies for 3d human pose tracking in all cases we demonstrate remarkable improvement compared to several chosen baselines

domain decomposition approach for fast gaussian process regression of large spatial data sets gaussian process regression is a flexible and powerful tool for machine learning but the high computational complexity hinders its broader applications in this paper we propose a new approach for fast computation of gaussian process regression with a focus on large spatial data sets the approach decomposes the domain of a regression function into small subdomains and infers a local piece of the regression function for each subdomain we explicitly address the mismatch problem of the local pieces on the boundaries of neighboring subdomains by imposing continuity constraints the new approach has comparable or better computation complexity as other competing methods but it is easier to be parallelized for faster computation moreover the method can be adaptive to non stationary features because of its local nature and in particular its use of different hyperparameters of the covariance function for different local regions we illustrate application of the method and demonstrate its advantages over existing methods using two synthetic data sets and two real spatial data sets

universal approximation using radial basis function networks there have been several recent studies concerning feedforward networks and the problem of approximating arbitrary functionals of a finite number of real variables some of these studies deal with cases in which the hidden layer nonlinearity is not a sigmoid this was motivated by successful applications of feedforward networks with nonsigmoidal hidden layer units this paper reports on a related study of radial basis function rbf networks and it is proved that rbf networks having one hidden layer are capable of universal approximation here the emphasis is on the case of typical rbf networks and the results show that a certain class of rbf networks with the same smoothing factor in each kernel node is broad enough for universal approximation

constructive approximate interpolation by neural networks we present a type of single hidden layer feedforward neural networks with sigmoidal nondecreasing activation function we call them ai nets they can approximately interpolate with arbitrary precision any set of distinct data in one or several dimensions they can uniformly approximate any continuous function of one variable and can be used for constructing uniform approximants of continuous functions of several variables all these capabilities are based on a closed expression of the networks

constructive approximation to multivariate function by decay rbf neural network it is well known that single hidden layer feedforward networks with radial basis function rbf kernels are universal approximators when all the parameters of the networks are obtained through all kinds of algorithms however as observed in most neural network implementations tuning all the parameters of the network may cause learning complicated poor generalization overtraining and unstable unlike conventional neural network theories this brief gives a constructive proof for the fact that a decay rbf neural network with n 1 hidden neurons can interpolate n 1 multivariate samples with zero error then we prove that the given decay rbfs can uniformly approximate any continuous multivariate functions with arbitrary precision without training the faster convergence and better generalization performance than conventional rbf algorithm bp algorithm extreme learning machine and support vector machines are shown by means of two numerical experiments

distinguishing cause from effect using observational data methods and benchmarks the discovery of causal relationships from purely observational data is a fundamental problem in science the most elementary form of such a causal discovery problem is to decide whether x causes y or alternatively y causes x given joint observations of two variables x y an example is to decide whether altitude causes temperature or vice versa given only joint measurements of both variables even under the simplifying assumptions of no confounding no feedback loops and no selection bias such bivariate causal discovery problems are challenging nevertheless several approaches for addressing those problems have been proposed in recent years we review two families of such methods methods based on additive noise models anms and information geometric causal inference igci we present the benchmark causeeffectpairs that consists of data for 100 different causee ffect pairs selected from 37 data sets from various domains e g meteorology biology medicine engineering economy etc and motivate our decisions regarding the ground truth causal directions of all pairs we evaluate the performance of several bivariate causal discovery methods on these real world benchmark data and in addition on artificially simulated data our empirical results on real world data indicate that certain methods are indeed able to distinguish cause from effect using only purely observational data although more benchmark data would be needed to obtain statistically significant conclusions one of the best performing methods overall is the method based on additive noise models that has originally been proposed by hoyer et al 2009 which obtains an accuracy of 63 10 and an auc of 0 74 0 05 on the real world benchmark as the main theoretical contribution of this work we prove the consistency of that method

a multiresolution stochastic process model for predicting basketball possession outcomes abstractbasketball games evolve continuously in space and time as players constantly interact with their teammates the opposing team and the ball however current analyses of basketball outcomes rely on discretized summaries of the game that reduce such interactions to tallies of points assists and similar events in this article we propose a framework for using optical player tracking data to estimate in real time the expected number of points obtained by the end of a possession this quantity called expected possession value epv derives from a stochastic process model for the evolution of a basketball possession we model this process at multiple levels of resolution differentiating between continuous infinitesimal movements of players and discrete events such as shot attempts and turnovers transition kernels are estimated using hierarchical spatiotemporal models that share information across players while remaining computationally tractable on very large data sets in addition to estima

the connection between bayesian estimation of a gaussian random field and rkhs reconstruction of a function from noisy data is key in machine learning and is often formulated as a regularized optimization problem over an infinite dimensional reproducing kernel hilbert space rkhs the solution suitably balances adherence to the observed data and the corresponding rkhs norm when the data fit is measured using a quadratic loss this estimator has a known statistical interpretation given the noisy measurements the rkhs estimate represents the posterior mean minimum variance estimate of a gaussian random field with covariance proportional to the kernel associated with the rkhs in this brief we provide a statistical interpretation when more general losses are used such as absolute value vapnik or huber specifically for any finite set of sampling locations that includes where the data were collected the maximum a posteriori estimate for the signal samples is given by the rkhs estimate evaluated at the sampling locations this connection establishes a firm statistical foundation for several stochastic approaches used to estimate unknown regularization parameters to illustrate this we develop a numerical scheme that implements a bayesian estimator with an absolute value loss this estimator is used to learn a function from measurements contaminated by outliers

dirichlet process mixtures of generalized linear models we propose dirichlet process mixtures of generalized linear models dp glm a new class of methods for nonparametric regression given a data set of input response pairs the dp glm produces a global model of the joint distribution through a mixture of local generalized linear models dp glms allow both continuous and categorical inputs and can model the same class of responses that can be modeled with a generalized linear model we study the properties of the dp glm and show why it provides better predictions and density estimates than existing dirichlet process mixture regression models we give conditions for weak consistency of the joint distribution and pointwise consistency of the regression estimate

an efficient configuration for probabilistic fuzzy logic system a novel inference configuration is proposed to improve the computational efficiency and information loss in the probabilistic fuzzy inference process the probabilistic inference and the fuzzy inference are unified in one operation based on the continuous form of the probabilistic fuzzy set besides the faster inference operation it is able to produce fuzzy outputs in a complete probabilistic distribution that in turn will provide information about the approximation bound the computational analyses of six different fuzzy systems demonstrate the inference efficiency of the proposed method its effectiveness can be further demonstrated on the application to modeling of an industrial curing process the robust modeling performance discloses its potential in process modeling under complex environment

variational fourier features for gaussian processes this work brings together two powerful concepts in gaussian processes the variational approach to sparse approximation and the spectral representation of gaussian processes this gives rise to an approximation that inherits the benefits of the variational approach but with the representational power and computational scalability of spectral representations the work hinges on a key result that there exist spectral features related to a finite domain of the gaussian process which exhibit almost independent covariances we derive these expressions for matern kernels in one dimension and generalize to more dimensions using kernels with specific structures under the assumption of additive gaussian noise our method requires only a single pass through the data set making for very fast and accurate computation we fit a model to 4 million training points in just a few minutes on a standard laptop with non conjugate likelihoods our mcmc scheme reduces the cost of computation from o nm2 for a sparse gaussian process to o nm per iteration where n is the number of data and m is the number of features

bayesian prediction of deterministic functions with applications to the design and analysis of computer experiments abstract this article is concerned with prediction of a function y t over a multidimensional domain t given the function values at a set of sites t 1 t 2 t n in t and with the design that is with the selection of those sites the motivating application is the design and analysis of computer experiments where t determines the input to a computer model of a physical or behavioral system and y t is a response that is part of the output or is calculated from it following a bayesian formulation prior uncertainty about the function y is expressed by means of a random function y which is taken here to be a gaussian stochastic process the mean of the posterior process can be used as the prediction function t and the variance can be used as a measure of uncertainty this kind of approach has been used previously in bayesian interpolation and is strongly related to the kriging methods used in geostatistics here emphasis is placed on product linear and product cubic correlation func

an information theoretic approach of designing sparse kernel adaptive filters this paper discusses an information theoretic approach of designing sparse kernel adaptive filters to determine useful data to be learned and remove redundant ones a subjective information measure called surprise is introduced surprise captures the amount of information a datum contains which is transferable to a learning system based on this concept we propose a systematic sparsification scheme which can drastically reduce the time and space complexity without harming the performance of kernel adaptive filters nonlinear regression short term chaotic time series prediction and long term time series forecasting examples are presented

fast direct methods for gaussian processes a number of problems in probability and statistics can be addressed using the multivariate normal gaussian distribution in the one dimensional case computing the probability for a given mean and variance simply requires the evaluation of the corresponding gaussian density in the n dimensional setting however it requires the inversion of an n times n covariance matrix c as well as the evaluation of its determinant det c in many cases such as regression using gaussian processes the covariance matrix is of the form c sigma 2 i k where k is computed using a specified covariance kernel which depends on the data and additional parameters hyperparameters the matrix c is typically dense causing standard direct methods for inversion and determinant evaluation to require mathcal o n 3 work this cost is prohibitive for large scale modeling here we show that for the most commonly used covariance functions the matrix c can be hierarchically factored into a product of block low rank updates of the identity matrix yielding an mathcal o n log 2 n algorithm for inversion more importantly we show that this factorization enables the evaluation of the determinant det c permitting the direct calculation of probabilities in high dimensions under fairly broad assumptions on the kernel defining k our fast algorithm brings many problems in marginalization and the adaptation of hyperparameters within practical reach using a single cpu core the combination of nearly optimal scaling in terms of problem size with high performance computing resources will permit the modeling of previously intractable problems we illustrate the performance of the scheme on standard covariance kernels

radial basis function network training using a nonsymmetric partition of the input space and particle swarm optimization this paper presents a novel algorithm for training radial basis function rbf networks in order to produce models with increased accuracy and parsimony the proposed methodology is based on a nonsymmetric variant of the fuzzy means fm algorithm which has the ability to determine the number and locations of the hidden node rbf centers whereas the synaptic weights are calculated using linear regression taking advantage of the short computational times required by the fm algorithm we wrap a particle swarm optimization pso based engine around it designed to optimize the fuzzy partition the result is an integrated framework for fully determining all the parameters of an rbf network the proposed approach is evaluated through its application on 12 real world and synthetic benchmark datasets and is also compared with other neural network training techniques the results show that the rbf network models produced by the pso based nonsymmetric fm algorithm outperform the models produced by the other techniques exhibiting higher prediction accuracies in shorter computational times accompanied by simpler network structures

bayesian nonparametric adaptive control using gaussian processes most current model reference adaptive control mrac methods rely on parametric adaptive elements in which the number of parameters of the adaptive element are fixed a priori often through expert judgment an example of such an adaptive element is radial basis function networks rbfns with rbf centers preallocated based on the expected operating domain if the system operates outside of the expected operating domain this adaptive element can become noneffective in capturing and canceling the uncertainty thus rendering the adaptive controller only semiglobal in nature this paper investigates a gaussian process based bayesian mrac architecture gp mrac which leverages the power and flexibility of gp bayesian nonparametric models of uncertainty the gp mrac does not require the centers to be preallocated can inherently handle measurement noise and enables mrac to handle a broader set of uncertainties including those that are defined as distributions over functions we use stochastic stability arguments to show that gp mrac guarantees good closed loop performance with no prior domain knowledge of the uncertainty online implementable gp inference methods are compared in numerical simulations against rbfn mrac with preallocated centers and are shown to provide better tracking and improved long term learning

approximate inference for disease mapping with sparse gaussian processes gaussian process gp models are widely used in disease mapping as they provide a natural framework for modeling spatial correlations their challenges however lie in computational burden and memory requirements in disease mapping models the other difficulty is inference which is analytically intractable due to the non gaussian observation model in this paper we address both these challenges we show how to efficiently build fully and partially independent conditional fic pic sparse approximations for the gp in two dimensional surface and how to conduct approximate inference using expectation propagation ep algorithm and laplace approximation la we also propose to combine fic with a compactly supported covariance function to construct a computationally efficient additive model that can model long and short length scale spatial correlations simultaneously the benefit of these approximations is computational the sparse gps speed up the computations and reduce the memory requirements the posterior inference via ep and laplace approximation is much faster and is practically as accurate as via markov chain monte carlo

rbf based technique for statistical demodulation of pathological tremor this paper presents an innovative technique based on the joint approximation capabilities of radial basis function rbf networks and the estimation capability of the multivariate iterated hilbert transform iht for the statistical demodulation of pathological tremor from electromyography emg signals in patients with parkinson s disease we define a stochastic model of the multichannel high density surface emg by means of the rbf networks applied to the reconstruction of the stochastic process characterizing the disease modeled by the multivariate relationships generated by the karhunen loeve transform in hilbert spaces next we perform a demodulation of the entire random field by means of the estimation capability of the multivariate iht in a statistical setting the proposed method is applied to both simulated signals and data recorded from three parkinsonian patients and the results show that the amplitude modulation components of the tremor oscillation can be estimated with signal to noise ratio close to 30 db with root mean square error for the estimates of the tremor instantaneous frequency additionally the comparisons with a large number of techniques based on all the combinations of the rbf extreme learning machine backpropagation support vector machine used in the first step of the algorithm and iht empirical mode decomposition multiband energy separation algorithm periodic algebraic separation and energy demodulation used in the second step of the algorithm clearly show the effectiveness of our technique these results show that the proposed approach is a potential useful tool for advanced neurorehabilitation technologies that aim at tremor characterization and suppression

fuzzy sets and type 2 under algebraic product and algebraic sum abstract the concept of fuzzy sets of type 2 has been proposed by l a zadeh as an extension of ordinary fuzzy sets a fuzzy set of type 2 can be defined by a fuzzy membership function the grade or fuzzy grade of which is taken to be a fuzzy set in the unit interval 0 1 rather than a point in 0 1 this paper investigates the algebraic properties of fuzzy grades that is fuzzy sets of type 2 under the operations of algebraic product and algebraic sum which can be defined by using the concept of the extension principle and shows that fuzzy grades under these operations do not form such algebraic structures as a lattice and a semiring moreover the properties of fuzzy grades are also discussed in the case where algebraic product and algebraic sum are combined with the well known operations of join and meet for fuzzy grades and it is shown that normal convex fuzzy grades form a lattice ordered semigroup under join meet and algebraic product

laplace approximation for divisive gaussian processes for nonstationary regression the standard gaussian process regression gp is usually formulated under stationary hypotheses the noise power is considered constant throughout the input space and the covariance of the prior distribution is typically modeled as depending only on the difference between input samples these assumptions can be too restrictive and unrealistic for many real world problems although nonstationarity can be achieved using specific covariance functions they require a prior knowledge of the kind of nonstationarity not available for most applications in this paper we propose to use the laplace approximation to make inference in a divisive gp model to perform nonstationary regression including heteroscedastic noise cases the log concavity of the likelihood ensures a unimodal posterior and makes that the laplace approximation converges to a unique maximum the characteristics of the likelihood also allow to obtain accurate posterior approximations when compared to the expectation propagation ep approximations and the asymptotically exact posterior provided by a markov chain monte carlo implementation with elliptical slice sampling ess but at a reduced computational load with respect to both ep and ess

nonstationary fuzzy sets in this paper the notion termed a ldquononstationary fuzzy setrdquo is introduced and the concept of a perturbation function that is used for generating nonstationary fuzzy sets is presented definitions of the basic set operators the union the intersection and the complement for nonstationary fuzzy sets are given together with proofs of selected properties of these operators two case studies were carried out in order to illustrate the use of nonstationary fuzzy sets in a nonstationary fuzzy inference and to provide an initial insight into the relationships between nonstationary and interval type 2 fuzzy sets

a multi resolution approximation for massive spatial datasets abstractautomated sensing instruments on satellites and aircraft have enabled the collection of massive amounts of high resolution observations of spatial fields over large spatial regions if these datasets can be efficiently exploited they can provide new insights on a wide variety of issues however traditional spatial statistical techniques such as kriging are not computationally feasible for big datasets we propose a multi resolution approximation m ra of gaussian processes observed at irregular locations in space the m ra process is specified as a linear combination of basis functions at multiple levels of spatial resolution which can capture spatial structure from very fine to very large scales the basis functions are automatically chosen to approximate a given covariance function which can be nonstationary all computations involving the m ra including parameter inference and prediction are highly scalable for massive datasets crucially the inference algorithms can also be parallelize

quantifying uncertainties in first principles alloy thermodynamics using cluster expansions the cluster expansion is a popular surrogate model for alloy modeling to avoid costly quantum mechanical simulations as its practical implementations require approximations its use trades efficiency for accuracy furthermore the coefficients of the model need to be determined from some known data set training set these two sources of error if not quantified decrease the confidence we can put in the results obtained from the surrogate model this paper presents a framework for the determination of the cluster expansion coefficients using a bayesian approach which allows for the quantification of uncertainties in the predictions in particular a relevance vector machine is used to automatically select the most relevant terms of the model while retaining an analytical expression for the predictive distribution this methodology is applied to two binary alloys sige and mgli including the temperature dependence in their effective cluster interactions the resulting cluster expansions are used to calculate the uncertainty in several thermodynamic quantities ground state line including the uncertainty in which structures are thermodynamically stable at 0 k phase diagrams and phase transitions the uncertainty in the ground state line is found to be of the order of mev atom showing that the cluster expansion is reliable to ab initio level accuracy even with limited data we found that the uncertainty in the predicted phase transition temperature increases when including the temperature dependence of the effective cluster interactions also the use of the bond stiffness versus bond length approximation to calculate temperature dependent properties from a reduced set of alloy configurations showed similar uncertainty to the approach where all training configurations are considered but at a much reduced computational cost

shared kernel information embedding for discriminative inference latent variable models such as the gplvm and related methods help mitigate overfitting when learning from small or moderately sized training sets nevertheless existing methods suffer from several problems 1 complexity 2 the lack of explicit mappings to and from the latent space 3 an inability to cope with multimodality and 4 the lack of a well defined density over the latent space we propose an lvm called the kernel information embedding kie that defines a coherent joint density over the input and a learned latent space learning is quadratic and it works well on small data sets we also introduce a generalization the shared kie skie that allows us to model multiple input spaces e g image features and poses using a single shared latent representation kie and skie permit missing data during inference and partially labeled data during learning we show that with data sets too large to learn a coherent global model one can use the skie to learn local online models we use skie for human pose inference

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions it is well known that standard single hidden layer feedforward networks slfns with at most n hidden neurons including biases can learn n distinct samples x sub i t sub i with zero error and the weights connecting the input neurons and the hidden neurons can be chosen almost arbitrarily however these results have been obtained for the case when the activation function for the hidden neurons is the signum function this paper rigorously proves that standard single hidden layer feedforward networks slfns with at most n hidden neurons and with any bounded nonlinear activation function which has a limit at one infinity can learn n distinct samples x sub i t sub i with zero error the previous method of arbitrarily choosing weights is not feasible for any slfn the proof of our result is constructive and thus gives a method to directly find the weights of the standard slfns with any such bounded nonlinear activation function as opposed to iterative training algorithms in the literature

accuracy analysis for wavelet approximations constructive wavelet networks are investigated as a universal tool for function approximation the parameters of such networks are obtained via some direct monte carlo procedures approximation bounds are given typically it is shown that such networks with one layer of wavelons achieve an l sub 2 error of order o n sup spl rhod where n is the number of nodes d is the problem dimension and spl rho is the number of summable derivatives of the approximated function an algorithm is also proposed to estimate this approximation based on noisy input output data observed from the function under consideration unlike neural network training this estimation procedure does not rely on stochastic gradient type techniques such as the celebrated backpropagation and it completely avoids the problem of poor convergence or undesirable local minima

type 2 fuzzy hidden markov models and their application to speech recognition this paper presents an extension of hidden markov models hmms based on the type 2 t2 fuzzy set fs referred to as type 2 fuzzy hmms t2 fhmms membership functions mfs of t2 fss are three dimensional and this new third dimension offers additional degrees of freedom to evaluate the hmms fuzziness therefore t2 fhmms are able to handle both random and fuzzy uncertainties existing universally in the sequential data we derive the t2 fuzzy forward backward algorithm and viterbi algorithm using t2 fs operations in order to investigate the effectiveness of t2 fhmms we apply them to phoneme classification and recognition on the timit speech database experimental results show that t2 fhmms can effectively handle noise and dialect uncertainties in speech signals besides a better classification performance than the classical hmms

fixed rank kriging for very large spatial data sets summary spatial statistics for very large spatial data sets is challenging the size of the data set n causes problems in computing optimal spatial predictors such as kriging since its computa tional cost is of order a73 in addition a large data set is often defined on a large spatial domain so the spatial process of interest typically exhibits non stationary behaviour over that domain a flexible family of non stationary covariance functions is defined by using a set of basis functions that is fixed in number which leads to a spatial prediction method that we call fixed rank kriging specifically fixed rank kriging is kriging within this class of non stationary covariance functions it relies on computational simplifications when n is very large for obtaining the spatial best linear unbiased predictor and its mean squared prediction error for a hidden spatial process a method based on minimizing a weighted frobenius norm yields best estimators of the covari ance function parameters which are then substituted into the fixed rank kriging equations the new methodology is applied to a very large data set of total column ozone data observed over the entire globe where n is of the order of hundreds of thousands

problems with likelihood estimation of covariance functions of spatial gaussian processes summary maximum likelihood has frequently been suggested as a way to estimate covariance parameters in spatial gaussian processes the results of some simple examples based on simulated data are discussed it is shown by an example with real data that the profile likelihood can be multimodal and that the global maximum may not correspond to a sensible value of the parameters mardia marshall 1984 worked in the following framework we have a gaussian process z x i x e x with mean given by a regression equation e z x f x t for a p dimensional vector the covariance function cov z x z y c x y 0 is also specified by a q dimensional vector 0 this process is observed at n points xi xn and from these observations we wish to estimate 3 and 0 let zn z xi z xn t be the vector of observations and k c xi xj 0 be its covariance matrix finally let f be the matrix with rows f xi t then zn has a multivariate normal distribution with mean f18 dispersion matrix k and the log likelihood is up to an additive constant

diffusion based spatial priors for functional magnetic resonance images we recently outlined a bayesian scheme for analyzing fmri data using diffusion based spatial priors harrison l m penny w ashburner j trujillo barreto n friston k j 2007 diffusion based spatial priors for imaging neuroimage 38 677 695 the current paper continues this theme applying it to a single subject functional magnetic resonance imaging fmri study of the auditory system we show that spatial priors on functional activations based on diffusion can be formulated in terms of the eigenmodes of a graph laplacian this allows one to discard eigenmodes with small eigenvalues to provide a computationally efficient scheme furthermore this formulation shows that diffusion based priors are a generalization of conventional laplacian priors penny w d trujillo barreto n j friston k j 2005 bayesian fmri time series analysis with spatial priors neuroimage 24 350 362 finally we show how diffusion based priors are a special case of gaussian process models that can be inverted using classical covariance component estimation techniques like restricted maximum likelihood patterson h d thompson r 1974 maximum likelihood estimation of components of variance paper presented at 8th international biometrics conference constanta romania the convention in spm is to smooth data with a fixed isotropic gaussian kernel before inverting a mass univariate statistical model this entails the strong assumption that data are generated smoothly throughout the brain however there is no way to determine if this assumption is supported by the data because data are smoothed before statistical modeling in contrast if a spatial prior is used smoothness is estimated given non smoothed data explicit spatial priors enable formal model comparison of different prior assumptions e g that data are generated from a stationary i e fixed throughout the brain or non stationary spatial process indeed for the auditory data we provide strong evidence for a non stationary process which concurs with a qualitative comparison of predicted activations at the boundary of functionally selective regions

gaussian processes for underdetermined source separation gaussian process gp models are very popular for machine learning and regression and they are widely used to account for spatial or temporal relationships between multivariate random variables in this paper we propose a general formulation of underdetermined source separation as a problem involving gp regression the advantage of the proposed unified view is first to describe the different underdetermined source separation problems as particular cases of a more general framework second it provides a flexible means to include a variety of prior information concerning the sources such as smoothness local stationarity or periodicity through the use of adequate covariance functions third given the model it provides an optimal solution in the minimum mean squared error mmse sense to the source separation problem in order to make the gp models tractable for very large signals we introduce framing as a gp approximation and we show that computations for regularly sampled and locally stationary gps can be done very efficiently in the frequency domain these findings establish a deep connection between gp and nonnegative tensor factorizations ntf with the itakura saito distance and lead to effective methods to learn gp hyperparameters for very large and regularly sampled signals

efficient spatio temporal gaussian regression via kalman filtering abstract we study the non parametric reconstruction of spatio temporal dynamical processes via gaussian processes gps regression from sparse and noisy data gps have been mainly applied to spatial regression where they represent one of the most powerful estimation approaches also thanks to their universal representing properties their extension to dynamical processes has been instead elusive so far since classical implementations lead to unscalable algorithms or require some sort of approximation we propose a novel procedure to address this problem by coupling gps regression and kalman filtering in particular assuming space time separability of the covariance kernel of the process and rational time spectrum we build a finite dimensional discrete time state space process representation amenable to kalman filtering with sampling over a finite set of fixed spatial locations our major finding is that the current kalman filter state represents a sufficient statistic to compute the minimum variance estimate of the process at any future time over the entire spatial domain in machine learning a representer theorem states that an important class of infinite dimensional variational problems admits a computable and finite dimensional exact solution in view of this our result can be interpreted as a novel dynamic representer theorem for gps we then extend the study to situations where the spatial input locations set varies over time the proposed algorithms are tested on both synthetic and real field data providing comparisons with standard gp and truncated gp regression techniques

multiscale approximation with hierarchical radial basis functions networks an approximating neural model called hierarchical radial basis function hrbf network is presented here this is a self organizing by growing multiscale version of a radial basis function rbf network it is constituted of hierarchical layers each containing a gaussian grid at a decreasing scale the grids are not completely filled but units are inserted only where the local error is over threshold this guarantees a uniform residual error and the allocation of more units with smaller scales where the data contain higher frequencies only local operations which do not require any iteration on the data are required this allows to construct the network in quasi real time through harmonic analysis it is demonstrated that although a hrbf cannot be reduced to a traditional wavelet based multiresolution analysis mra it does employ riesz bases and enjoys asymptotic approximation properties for a very large class of functions hrbf networks have been extensively applied to the reconstruction of three dimensional 3 13 models from noisy range data the results illustrate their power in denoising the original data obtaining an effective multiscale reconstruction of better quality than that obtained by mra

standing wave decomposition gaussian process we propose a standing wave decomposition swd approximation to gaussian process gp regression gp involves a costly matrix inversion operation which limits applicability to large data analysis for an input space that can be approximated by a grid and when correlations among data are short ranged the kernel matrix inversion can be replaced by analytic diagonalization using the swd we show that this approach applies to uni and multidimensional input data extends to include longer range correlations and the grid can be in a latent space and used as inducing points through simulations we show that our approximate method applied to the squared exponential kernel outperforms existing methods in predictive accuracy per unit time in the regime where data are plentiful our swd gp is recommended for regression analyses where there is a relatively large amount of data and or there are constraints on computation time

reduced hyperbf networks regularization by explicit complexity reduction and scaled rprop based training hyper basis function hyperbf networks are generalized radial basis function neural networks where the activation function is a radial function of a weighted distance such generalization provides hyperbf networks with high capacity to learn complex functions which in turn make them susceptible to overfitting and poor generalization moreover training a hyperbf network demands the weights centers and local scaling factors to be optimized simultaneously in the case of a relatively large dataset with a large network structure such optimization becomes computationally challenging in this paper a new regularization method that performs soft local dimension reduction in addition to weight decay is proposed the regularized hyperbf network is shown to provide classification accuracy competitive to a support vector machine while requiring a significantly smaller network structure furthermore a practical training to construct hyperbf networks is presented hierarchal clustering is used to initialize neurons followed by a gradient optimization using a scaled version of the rprop algorithm with a localized partial backtracking step experimental results on seven datasets show that the proposed training provides faster and smoother convergence than the regular rprop algorithm

a bayesian committee machine the bayesian committee machine bcm is a novel approach to combining estimators that were trained on different data sets although the bcm can be applied to the combination of any kind of estimators the main foci are gaussian process regression and related systems such as regularization networks and smoothing splines for which the degrees of freedom increase with the number of training data somewhat surprisingly we find that the performance of the bcm improves if several test points are queried at the same time and is optimal if the number of test points is at least as large as the degrees of freedom of the estimator the bcm also provides a new solution for on line learning with potential applications to data mining we apply the bcm to systems with fixed basis functions and discuss its relationship to gaussian process regression finally we show how the ideas behind the bcm can be applied in a non bayesian setting to extend the input dependent combination of estimators

scaling multidimensional inference for structured gaussian processes exact gaussian process gp regression has cal o n 3 runtime for data size n making it intractable for large n many algorithms for improving gp scaling approximate the covariance with lower rank matrices other work has exploited structure inherent in particular covariance functions including gps with implied markov structure and inputs on a lattice both enable cal o n or cal o n log n runtime however these gp advances have not been well extended to the multidimensional input setting despite the preponderance of multidimensional applications this paper introduces and tests three novel extensions of structured gps to multidimensional inputs for models with additive and multiplicative kernels first we present a new method for inference in additive gps showing a novel connection between the classic backfitting method and the bayesian framework we extend this model using two advances a variant of projection pursuit regression and a laplace approximation for non gaussian observations lastly for multiplicative kernel structure we present a novel method for gps with inputs on a multidimensional grid we illustrate the power of these three advances on several data sets achieving performance equal to or very close to the naive gp at orders of magnitude less cost

fast bayesian inference of the multivariate ornstein uhlenbeck process the multivariate ornstein uhlenbeck process is used in many branches of science and engineering to describe the regression of a system to its stationary mean here we present an o n bayesian method to estimate the drift and diffusion matrices of the process from n discrete observations of a sample path we use exact likelihoods expressed in terms of four sufficient statistic matrices to derive explicit maximum a posteriori parameter estimates and their standard errors we apply the method to the brownian harmonic oscillator a bivariate ornstein uhlenbeck process to jointly estimate its mass damping and stiffness and to provide bayesian estimates of the correlation functions and power spectral densities we present a bayesian model comparison procedure embodying ockham s razor to guide a data driven choice between the kramers and smoluchowski limits of the oscillator these provide novel methods of analyzing the inertial motion of colloidal particles in optical traps

universal approximation using incremental constructive feedforward networks with random hidden nodes according to conventional neural network theories single hidden layer feedforward networks slfns with additive or radial basis function rbf hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable however as observed in most neural network implementations tuning all the parameters of the networks may cause learning complicated and inefficient and it may be difficult to train networks with nondifferential activation functions such as threshold networks unlike conventional neural network theories this paper proves in an incremental constructive method that in order to let slfns work as universal approximators one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer in such slfns implementations the activation functions for additive nodes can be any bounded nonconstant piecewise continuous functions g r r and the activation functions for rbf nodes can be any integrable piecewise continuous functions g r r and rg x dx 0 the proposed incremental method is efficient not only for sflns with continuous including nondifferentiable activation functions but also for slfns with piecewise continuous such as threshold activation functions compared to other popular methods such a new network is fully automatic and users need not intervene the learning process by manually tuning control parameters

structural optimization with frequency constraints by genetic algorithm using wavelet radial basis function neural network abstract in this study a combination of genetic algorithm ga and neural networks nn is proposed to find the optimal weight of structures subject to multiple natural frequency constraints the optimization is carried out by an evolutionary algorithm using discrete design variables the evolutionary algorithm employed in this investigation is virtual sub population vsp method to reduce the computational time of optimization process the natural frequencies of structures are evaluated using properly trained radial basis function rbf and wavelet radial basis function wrbf neural networks in the wrbf neural network the activation function of hidden layer neurons is substituted with a type of wavelet functions in this new network the position and dilation of the wavelet are fixed and only the weights are optimized the numerical results demonstrate the robustness and high performance of the suggested methods for structural optimization with frequency constraints it is found that the best results are obtained by vsp method using wrbf network

low rank decomposition meets kernel learning a generalized nystrom method abstract low rank matrix decomposition and kernel learning are two useful techniques in building advanced learning systems low rank decomposition can greatly reduce the computational cost of manipulating large kernel matrices however existing approaches are mostly unsupervised and do not incorporate side information such as class labels making the decomposition less effective for a specific learning task on the other hand kernel learning techniques aim at constructing kernel matrices whose structure is well aligned with the learning target which improves the generalization performance of kernel methods however most kernel learning approaches are computationally very expensive to obtain the advantages of both techniques and address their limitations in this paper we propose a novel kernel low rank decomposition formulation called the generalized nystrom method our approach inherits the linear time and space complexity via matrix decomposition while at the same time fully exploits partial label information in computing task dependent decomposition in addition the resultant low rank factors can generalize to arbitrary new samples rendering great flexibility in inductive learning scenarios we further extend the algorithm to a multiple kernel learning setup the experimental results on semi supervised classification demonstrate the usefulness of the proposed method

efficient gaussian process regression for large datasets gaussian processes are widely used in nonparametric regression classification and spatiotemporal modelling facilitated in part by a rich literature on their theoretical properties however one of their practical limitations is expensive computation typically on the order of n3 where n is the number of data points in performing the necessary matrix inversions for large datasets storage and processing also lead to computational bottlenecks and numerical stability of the estimates and predicted values degrades with increasing n various methods have been proposed to address these problems including predictive processes in spatial data analysis and the subset of regressors technique in machine learning the idea underlying these approaches is to use a subset of the data but this raises questions concerning sensitivity to the choice of subset and limitations in estimating fine scale structure in regions that are not well covered by the subset motivated by the literature on compressive sensing we propose an alternative approach that involves linear projection of all the data points onto a lower dimensional subspace we demonstrate the superiority of this approach from a theoretical perspective and through simulated and real data examples

multioutput convolution spectral mixture for gaussian processes multioutput gaussian processes mogps are an extension of gaussian processes gps for predicting multiple output variables also called channels tasks simultaneously in this article we use the convolution theorem to design a new kernel for mogps by modeling cross channel dependencies through cross convolution of time and phase delayed components in the spectral domain the resulting kernel is called multioutput convolution spectral mixture mocsm kernel the results of extensive experiments on synthetic and real life data sets demonstrate the advantages of the proposed kernel and its state of the art performance mocsm enjoys the desirable property to reduce to the well known spectral mixture sm kernel when a single channel is considered a comparison with the recently introduced multioutput sm kernel reveals that this is not the case for the latter kernel which contains quadratic terms that generate undesirable scale effects when the spectral densities of different channels are either very close or very far from each other in the frequency domain

relevance units latent variable model and nonlinear dimensionality reduction a new dimensionality reduction method called relevance units latent variable model rulvm is proposed in this paper rulvm has a close link with the framework of gaussian process latent variable model gplvm and it originates from a recently developed sparse kernel model called relevance units machine rum rum follows the idea of relevance vector machine rvm under the bayesian framework but releases the constraint that relevance vectors rvs have to be selected from the input vectors rum treats relevance units rus as part of the parameters to be learned from the data as a result a rum maintains all the advantages of rvm and offers superior sparsity rulvm inherits the advantages of sparseness offered by the rum and the experimental result shows that rulvm algorithm possesses considerable computational advantages over gplvm algorithm

stable and efficient gaussian process calculations the use of gaussian processes can be an effective approach to prediction in a supervised learning environment for large data sets the standard gaussian process approach requires solving very large systems of linear equations and approximations are required for the calculations to be practical we will focus on the subset of regressors approximation technique we will demonstrate that there can be numerical instabilities in a well known implementation of the technique we discuss alternate implementations that have better numerical stability properties and can lead to better predictions our results will be illustrated by looking at an application involving prediction of galaxy redshift from broadband spectrum data

locally adaptive bayes nonparametric regression via nested gaussian processes we propose a nested gaussian process ngp as a locally adaptive prior for bayesian nonparametric regression specified through a set of stochastic differential equations sdes the ngp imposes a gaussian process prior for the function s mth order derivative the nesting comes in through including a local instantaneous mean function which is drawn from another gaussian process inducing adaptivity to locally varying smoothness we discuss the support of the ngp prior in terms of the closure of a reproducing kernel hilbert space and consider theoretical properties of the posterior the posterior mean under the ngp prior is shown to be equivalent to the minimizer of a nested penalized sum of squares involving penalties for both the global and local roughness of the function using highly efficient markov chain monte carlo for posterior inference the proposed method performs well in simulation studies compared to several alternatives and is scalable to massive data illustrated through a proteomics appli

neural networks for localized approximation we prove that feedforward artificial neural networks with a single hidden layer and an ideal sigmoidal response function cannot provide localized approximation in a euclidean space of dimension higher than one we also show that networks with two hidden layers can be designed to provide localized approximation since wavelet bases are most effective for local approximation we give a discussion of the implementation of spline wavelets using multilayered networks where the response function is a sigmoidal function of order at least two

dual diversified dynamical gaussian process latent variable model for video repairing in this paper we propose a dual diversified dynamical gaussian process latent variable model text d 3 gplvm to tackle the video repairing issue for preservation purposes videos have to be conserved on media however storing on media such as films and hard disks can suffer from unexpected data loss for instance physical damage so repairing of missing or damaged pixels is essential for better video maintenance most methods seek to fill in missing holes by synthesizing similar textures from local patches the neighboring pixels consecutive frames or the whole video however these can introduce incorrect contexts especially when the missing hole or number of damaged frames is large furthermore simple texture synthesis can introduce artifacts in undamaged and recovered areas to address aforementioned problems we introduce two diversity encouraging priors to both of inducing points and latent variables for considering the variety in existing videos in text d 3 gplvm the inducing points constitute a smaller subset of observed data while latent variables are a low dimensional representation of observed data since they have a strong correlation with the observed data it is essential that both of them can capture distinct aspects of and fully represent the observed data the dual diversity encouraging priors ensure that the trained inducing points and latent variables are more diverse and resistant for context aware and artifacts free based video repairing the defined objective function in our proposed model is initially not analytically tractable and must be solved by variational inference finally experimental testing results illustrate the robustness and effectiveness of our method for damaged video repairing

kernel additive models for source separation source separation consists of separating a signal into additive components it is a topic of considerable interest with many applications that has gathered much attention recently here we introduce a new framework for source separation called kernel additive modelling which is based on local regression and permits efficient separation of multidimensional and or nonnegative and or non regularly sampled signals the main idea of the method is to assume that a source at some location can be estimated using its values at other locations nearby where nearness is defined through a source specific proximity kernel such a kernel provides an efficient way to account for features like periodicity continuity smoothness stability over time or frequency self similarity etc in many cases such local dynamics are indeed much more natural to assess than any global model such as a tensor factorization this framework permits one to use different proximity kernels for different sources and to separate them using the iterative kernel backfitting algorithm we describe as we show kernel additive modelling generalizes many recent and efficient techniques for source separation and opens the path to creating and combining source models in a principled way experimental results on the separation of synthetic and audio signals demonstrate the effectiveness of the approach

bayesian generalized kernel mixed models we propose a fully bayesian methodology for generalized kernel mixed models gkmms which are extensions of generalized linear mixed models in the feature space induced by a reproducing kernel we place a mixture of a point mass distribution and silverman s g prior on the regression vector of a generalized kernel model gkm this mixture prior allows a fraction of the components of the regression vector to be zero thus it serves for sparse modeling and is useful for bayesian computation in particular we exploit data augmentation methodology to develop a markov chain monte carlo mcmc algorithm in which the reversible jump method is used for model selection and a bayesian model averaging method is used for posterior prediction when the feature basis expansion in the reproducing kernel hilbert space is treated as a stochastic process this approach can be related to the karhunen loeve expansion of a gaussian process gp thus our sparse modeling framework leads to a flexible approximation method for gps

model robust response surface designs scaling two level factorials summary response surface methods use simple graduating functions to study the relationship between an experimental response variable and a set of continuous explanatory variables in designing a response surface study an experimenter must decide how far apart to set the levels of each factor i e how to scale the design a good choice should be sensitive to the fact that the graduating function is only an approximation to the true response function a bayesian model is proposed that makes explicit assumptions about the inadequacy of an assumed model and a design criterion based on the model leads to reasonable choices of scale for two level factorial designs the choice of scale is found to be insensitive to the prior distributions in the model

the signal extraction approach to nonlinear regression and spline smoothing abstract this article shows how to fit a smooth curve polynomial spline to pairs of data values yi xi prior specification of a parametric functional form for the curve is not required the resulting curve can be used to describe the pattern of the data and to predict unknown values of y given x both point and interval estimates are produced the method is easy to use and the computational requirements are modest even for large sample sizes our method is based on maximum likelihood estimation of a signal in noise model of the data we use the kalman filter to evaluate the likelihood function and achieve significant computational advantages over previous approaches to this problem

gp demo differential evolution for multiobjective optimization based on gaussian process models abstract this paper proposes a novel surrogate model based multiobjective evolutionary algorithm called differential evolution for multiobjective optimization based on gaussian process models gp demo the algorithm is based on the newly defined relations for comparing solutions under uncertainty these relations minimize the possibility of wrongly performed comparisons of solutions due to inaccurate surrogate model approximations the gp demo algorithm was tested on several benchmark problems and two computationally expensive real world problems to be able to assess the results we compared them with another surrogate model based algorithm called generational evolution control gec and with the differential evolution for multiobjective optimization demo the quality of the results obtained with gp demo was similar to the results obtained with demo but with significantly fewer exactly evaluated solutions during the optimization process the quality of the results obtained with gec was lower compared to the quality gained with gp demo and demo mainly due to wrongly performed comparisons of the inaccurately approximated solutions

reversible jump markov chain monte carlo computation and bayesian model determination markov chain monte carlo methods for bayesian computation have until recently been restricted to problems where the joint distribution of all variables has a density with respect to some fixed standard underlying measure they have therefore not been available for application to bayesian model determination where the dimensionality of the parameter vector is typically not fixed this paper proposes a new framework for the construction of reversible markov chain samplers that jump between parameter subspaces of differing dimensionality which is flexible and entirely constructive it should therefore have wide applicability in model determination problems the methodology is illustrated with applications to multiple change point analysis in one and two dimensions and to a bayesian comparison of binomial experiments

variational inference for latent variables and uncertain inputs in gaussian processes the gaussian process latent variable model gp lvm provides a flexible approach for non linear dimensionality reduction that has been widely applied however the current approach for training gp lvms is based on maximum likelihood where the latent projection variables are maximised over rather than integrated out in this paper we present a bayesian method for training gp lvms by introducing a non standard variational inference framework that allows to approximately integrate out the latent variables and subsequently train a gp lvm by maximising an analytic lower bound on the exact marginal likelihood we apply this method for learning a gp lvm from i i d observations and for learning non linear dynamical systems where the observations are temporally correlated we show that a benefit of the variational bayesian procedure is its robustness to overfitting and its ability to automatically select the dimensionality of the non linear latent space the resulting framework is generic flexible and easy to extend for other purposes such as gaussian process regression with uncertain or partially missing inputs we demonstrate our method on synthetic data and standard machine learning benchmarks as well as challenging real world datasets including high resolution video data

gaussian process regressors for multiuser detection in ds cdma systems in this paper we present gaussian processes for regression gpr as a novel detector for cdma digital communications particularly we propose gpr for constructing analytical nonlinear multiuser detectors in cdma communication systems gpr can easily compute the parameters that describe its nonlinearities by maximum likelihood thereby no cross validation is needed as it is typically used in nonlinear estimation procedures the gpr solution is analytical given its parameters and it does not need to solve an optimization problem for building the nonlinear estimator these properties provide fast and accurate learning two major issues in digital communications the gpr with a linear decision function can be understood as a regularized mmse detector in which the regularization parameter is optimally set we also show the gpr receiver to be a straightforward nonlinear extension of the linear minimum mean square error mmse criterion widely used in the design of these receivers we argue the benefits of this new approach in short codes cdma systems where little information on the users codes users amplitudes or the channel is available the paper includes some experiments to show that gpr outperforms linear mmse and nonlinear svm state ofthe art solutions

computationally efficient convolved multiple output gaussian processes recently there has been an increasing interest in regression methods that deal with multiple outputs this has been motivated partly by frameworks like multitask learning multisensor networks or structured output data from a gaussian processes perspective the problem reduces to specifying an appropriate covariance function that whilst being positive semi definite captures the dependencies between all the data points and across all the outputs one approach to account for non trivial correlations between outputs employs convolution processes under a latent function interpretation of the convolution transform we establish dependencies between output variables the main drawbacks of this approach are the associated computational and storage demands in this paper we address these issues we present different efficient approximations for dependent output gaussian processes constructed through the convolution formalism we exploit the conditional independencies present naturally in the model this leads to a form of the covariance similar in spirit to the so called pitc and fitc approximations for a single output we show experimental results with synthetic and real data in particular we show results in school exams score prediction pollution prediction and gene expression data

sparse on line gaussian processes we develop an approach for sparse representations of gaussian process gp models which are bayesian types of kernel machines in order to overcome their limitations for large data sets the method is based on a combination of a bayesian on line algorithm together with a sequential construction of a relevant subsample of the data that fully specifies the prediction of the gp model by using an appealing parameterization and projection techniques in a reproducing kernel hilbert space recursions for the effective parameters and a sparse gaussian approximation of the posterior process are obtained this allows for both a propagation of predictions and bayesian error measures the significance and robustness of our approach are demonstrated on a variety of experiments

maximum likelihood estimation of models for residual covariance in spatial regression we describe the maximum likelihood method for fitting the linear model when residuals are correlated and when the covariance among the residuals is determined by a parametric model containing unknown parameters observations are assumed to be gaussian we give conditions which ensure consistency and asymptotic normality of the estimators our main concern is with the analysis of spatial data and in this context we describe some simulation experiments to assess the small sample behaviour of estimators we also discuss an application of the spectral approximation to the likelihood for processes on a lattice

diffusion based spatial priors for imaging we describe a bayesian scheme to analyze images which uses spatial priors encoded by a diffusion kernel based on a weighted graph laplacian this provides a general framework to formulate a spatial model whose parameters can be optimized the application we have in mind is a spatiotemporal model for imaging data we illustrate the method on a random effects analysis of fmri contrast images from multiple subjects this simplifies exposition of the model and enables a clear description of its salient features typically imaging data are smoothed using a fixed gaussian kernel as a pre processing step before applying a mass univariate statistical model e g a general linear model to provide images of parameter estimates an alternative is to include smoothness in a multivariate statistical model penny w d trujillo barreto n j friston k j 2005 bayesian fmri time series analysis with spatial priors neuroimage 24 350 362 the advantage of the latter is that each parameter field is smoothed automatically according to a measure of uncertainty given the data in this work we investigate the use of diffusion kernels to encode spatial correlations among parameter estimates nonlinear diffusion has a long history in image processing in particular flows that depend on local image geometry romeny b m t 1994 geometry driven diffusion in computer vision kluwer academic publishers can be used as adaptive filters this can furnish a non stationary smoothing process that preserves features which would otherwise be lost with a fixed gaussian kernel we describe a bayesian framework that incorporates non stationary adaptive smoothing into a generative model to extract spatial features in parameter estimates critically this means adaptive smoothing becomes an integral part of estimation and inference we illustrate the method using synthetic and real fmri data

large scale gaussian process inference with generalized histogram intersection kernels for visual recognition tasks we present new methods for fast gaussian process gp inference in large scale scenarios including exact multi class classification with label regression hyperparameter optimization and uncertainty prediction in contrast to previous approaches we use a full gaussian process model without sparse approximation techniques our methods are based on exploiting generalized histogram intersection kernels and their fast kernel multiplications we empirically validate the suitability of our techniques in a wide range of scenarios with tens of thousands of examples whereas plain gp models are intractable due to both memory consumption and computation time in these settings our results show that exact inference can indeed be done efficiently in consequence we enable every important piece of the gaussian process framework learning inference hyperparameter optimization variance estimation and online learning to be used in realistic scenarios with more than a handful of data

bayesian leave one out cross validation approximations for gaussian latent variable models the future predictive performance of a bayesian model can be estimated using bayesian cross validation in this article we consider gaussian latent variable models where the integration over the latent values is approximated using the laplace method or expectation propagation ep we study the properties of several bayesian leave one out loo cross validation approximations that in most cases can be computed with a small additional cost after forming the posterior approximation given the full data our main objective is to assess the accuracy of the approximative loo cross validation estimators that is for each method laplace and ep we compare the approximate fast computation with the exact brute force loo computation secondarily we evaluate the accuracy of the laplace and ep approximations themselves against a ground truth established through extensive markov chain monte carlo simulation our empirical results show that the approach based upon a gaussian approximation to the loo marginal distribution the so called cavity distribution gives the most accurate and reliable results among the fast methods

nonlinear spectral unmixing of hyperspectral images using gaussian processes this paper presents an unsupervised algorithm for nonlinear unmixing of hyperspectral images the proposed model assumes that the pixel reflectances result from a nonlinear function of the abundance vectors associated with the pure spectral components we assume that the spectral signatures of the pure components and the nonlinear function are unknown the first step of the proposed method estimates the abundance vectors for all the image pixels using a bayesian approach an a gaussian process latent variable model for the nonlinear function relating the abundance vectors to the observations the endmembers are subsequently estimated using gaussian process regression the performance of the unmixing strategy is first evaluated on synthetic data the proposed method provides accurate abundance and endmember estimations when compared to other linear and nonlinear unmixing strategies an interesting property is its robustness to the absence of pure pixels in the image the analysis of a real hyperspectral image shows results that are in good agreement with state of the art unmixing strategies and with a recent classification method

orthogonal column latin hypercubes and their application in computer experiments abstract latin hypercubes have been frequently used in conducting computer experiments in this paper a class of orthogonal latin hypercubes that preserves orthogonality among columns is proposed applying an orthogonal latin hypercube design to a computer experiment benefits the data analysis in two ways first it retains the orthogonality of traditional experimental designs the estimates of linear effects of all factors are uncorrelated not only with each other but also with the estimates of all quadratic effects and bilinear interactions second it can facilitate nonparametric fitting procedures because one can select good space filling designs within the class of orthogonal latin hypercubes according to selection criteria

a unifying view of sparse approximate gaussian process regression we provide a new unifying view including all existing proper probabilistic sparse approximations for gaussian process regression our approach relies on expressing the effective prior which the methods are using this allows new insights to be gained and highlights the relationship between existing methods it also allows for a clear theoretically justified ranking of the closeness of the known approximations to the corresponding full gps finally we point directly to designs of new better sparse approximations combining the best of the existing strategies within attractive computational constraints

a framework for evaluating approximation methods for gaussian process regression gaussian process gp predictors are an important component of many bayesian approaches to machine learning however even a straightforward implementation of gaussian process regression gpr requires o n2 space and o n3 time for a data set of n examples several approximation methods have been proposed but there is a lack of understanding of the relative merits of the different approximations and in what situations they are most useful we recommend assessing the quality of the predictions obtained as a function of the compute time taken and comparing to standard baselines e g subset of data and fitc we empirically investigate four different approximation algorithms on four different prediction problems and make our code available to encourage future comparisons

geometric type 1 and type 2 fuzzy logic systems this paper presents a novel approach to the representation of type 1 and type 2 fuzzy sets utilising computational geometry to achieve this our approach borrows ideas from the field of computational geometry and applies these techniques in the novel setting of fuzzy logic we provide new algorithms for various operations on type 1 and type 2 fuzzy sets and for defuzzification results of experiments indicate that this approach reduces the execution speed of these operations