decentralized data reduction with quantization constraints a guiding principle for data reduction in statistical inference is the sufficiency principle this paper extends the classical sufficiency principle to decentralized inference i e data reduction needs to be achieved in a decentralized manner we examine the notions of local and global sufficient statistics and the relationship between the two for decentralized inference under different observation models we then consider the impact of quantization on decentralized data reduction which is often needed when communications among sensors are subject to finite capacity constraints the central question we intend to ask is if each node in a decentralized inference system has to summarize its data using a finite number of bits is it still optimal to implement data reduction using global sufficient statistics prior to quantization we show that the answer is negative using a simple example and proceed to identify conditions under which sufficiency based data reduction followed by quantization is indeed optimal they include the well known case when the data at decentralized nodes are conditionally independent as well as a class of problems with conditionally dependent observations that admit conditional independence structure through the introduction of an appropriately chosen hidden variable

ideal spatial adaptation by wavelet shrinkage summary with ideal spatial adaptation an oracle furnishes information about how best to adapt a spatially variable estimator whether piecewise constant piecewise polynomial variable knot spline or variable bandwidth kernel to the unknown function estimation with the aid of an oracle offers dramatic advantages over traditional linear estimation by nonadaptive kernels however it is a priori unclear whether such performance can be obtained by a procedure relying on the data alone we describe a new principle for spatially adaptive estimation selective wavelet reconstruction we show that variable knot spline fits and piecewise polynomial fits when equipped with an oracle to select the knots are not dramatically more powerful than selective wavelet reconstruction with an oracle we develop a practical spatially adaptive method riskshrink which works by shrinkage of empirical wavelet coefficients riskshrink mimics the performance of an oracle for selective wavelet reconstruction as well as it is possible to do so a new inequality in multivariate normal decision theory which we call the oracle inequality shows that attained performance differs from ideal performance by at most a factor of approximately 2 log n where n is the sample size moreover no estimator can give a better guarantee than this within the class of spatially adaptive procedures riskshrink is essentially optimal relying only on the data it comes within a factor log 2 n of the performance of piecewise polynomial and variableknot spline methods equipped with an oracle in contrast it is unknown how or if piecewise polynomial methods could be made to function this well when denied access to an oracle and forced to rely on data alone

the subspace information criterion for infinite dimensional hypothesis spaces a central problem in learning is selection of an appropriate model this is typically done by estimating the unknown generalization errors of a set of models to be selected from and then choosing the model with minimal generalization error estimate in this article we discuss the problem of model selection and generalization error estimation in the context of kernel regression models e g kernel ridge regression kernel subset regression or gaussian process regression previously a non asymptotic generalization error estimator called the subspace information criterion sic was proposed that could be successfully applied to finite dimensional subspace models sic is an unbiased estimator of the generalization error for the finite sample case under the conditions that the learning target function belongs to a specified reproducing kernel hilbert space rkhs h and the reproducing kernels centered on training sample points span the whole space h these conditions hold only if dim h l where l is the number of training examples therefore sic could be applied only to finite dimensional rkhss in this paper we extend the range of applicability of sic and show that even if the reproducing kernels centered on training sample points do not span the whole space h sic is an unbiased estimator of an essential part of the generalization error our extension allows the use of any rkhss including infinite dimensional ones i e richer function classes commonly used in gaussian processes support vector machines or boosting we further show that when the kernel matrix is invertible sic can be expressed in a much simpler form making its computation highly efficient in computer simulations on ridge parameter selection with real and artificial data sets sic is compared favorably with other standard model selection techniques for instance leave one out cross validation or an empirical bayesian method

some aspects of fusion in estimation theory the problem of fusing or combining various estimates to obtain a single good estimate is investigated the problem of fusion in estimation theory is addressed and several examples using common distributions in which virtually any method of fusion would be useless in approximating the random variable of interest are presented a theorem which for a very general situation shows that fusion resulting in an almost surely exact approximation is always possible is presented in particular this result addresses the situation in which the data consists of the random variable of interest corrupted by additive gaussian noise and the random variable of interest could be any second order random variable finally an example which illustrates the utility of this result is presented

fisher information and stochastic complexity by taking into account the fisher information and removing an inherent redundancy in earlier two part codes a sharper code length as the stochastic complexity and the associated universal process are derived for a class of parametric processes the main condition required is that the maximum likelihood estimates satisfy the central limit theorem the same code length is also obtained from the so called maximum likelihood code

bayesian regularization and pruning using a laplace prior standard techniques for improved generalization from neural networks include weight decay and pruning weight decay has a bayesian interpretation with the decay function corresponding to a prior over weights the method of transformation groups and maximum entropy suggests a laplace rather than a gaussian prior after training the weights then arrange themselves into two classes 1 those with a common sensitivity to the data error and 2 those failing to achieve this sensitivity and that therefore vanish since the critical value is determined adaptively during training pruning in the sense of setting weights to exact zeros becomes an automatic consequence of regularization alone the count of free parameters is also reduced automatically as weights are pruned a comparison is made with results of mackay using the evidence framework and a gaussian regularizer

active learning applied to rcs computations with nonuniform sampling using different objective functions an active learning framework is introduced to reduce the number of frequencies and angles one must consider for wideband monostatic scattering computations or measurements this method is used to optimally select those frequencies and angles that would be most informative resulting in nonuniform sampling and often a reduced number of points vis agrave vis uniform sampling in this paper we focus on jointly two dimensional optimal sampling in frequency and incident angle thetas for monostatic scattering the method consists of two basic steps one step involves estimation of model parameters using a least square ls algorithm the next step is to optimally choose the next point frequency and thetas for analysis by the computational model or experiment this new point is selected with the goal of reducing uncertainty in the parametric model quantified via the fisher information matrix iterating these two steps a sequence of numerical computations or measurements are performed each at the most informative point for learning the parameters of the associated simpler parametric model this idea is demonstrated here in the context of reducing the number of points frequencies and orientations at which a computational model must be employed and in order to avoid repeatedly gathering samplings at the edge of the input space an alternative objective function is applied which makes the actively selected points closer to a region of interest

inferring solutions of differential equations using noisy multi fidelity data for more than two centuries solutions of differential equations have been obtained either analytically or numerically based on typically well behaved forcing and boundary conditions for well posed problems we are changing this paradigm in a fundamental way by establishing an interface between probabilistic machine learning and differential equations we develop data driven algorithms for general linear equations using gaussian process priors tailored to the corresponding integro differential operators the only observables are scarce noisy multi fidelity data for the forcing and solution that are not required to reside on the domain boundary the resulting predictive posterior distributions quantify uncertainty and naturally lead to adaptive solution refinement via active learning this general framework circumvents the tyranny of numerical discretization as well as the consistency and stability issues of time integration and is scalable to high dimensions

learning unrealizable tasks from minimum entropy queries in supervised learning learning from queries rather than from random examples can improve generalization performance significantly we study the performance of query learning for unrealizable tasks where the student cannot learn from the perfectly as a simple model scenario of this kind we consider a linear perceptron student learning a general nonlinear perceptron teacher two kinds of queries for maximum information gain i e minimum entropy are investigated minimum student space entropy msse queries which are appropriate if the teacher space is unknown and minimum teacher space entropy mtse queries which can be used if the teacher space is assumed to be known but a student of a simpler form has deliberately been chosen we find that for msse queries the structure of the student space determines the efficacy of query learning mtse queries on the other hand which we investigate for the extreme case of a binary perceptron teacher lead to a higher generalization error than random examples due to a lack of feedback about the progress of the student in the way queries are selected

incremental active learning for optimal generalization the problem of designing input signals for optimal generalization is called active learning in this article we give a two stage sampling scheme for reducing both the bias and variance and based on this scheme we propose two active learning methods one is the multipoint search method applicable to arbitrary models the effectiveness of this method is shown through computer simulations the other is the optimal sampling method in trigonometric polynomial models this method precisely specifies the optimal sampling locations

dynamic trees for learning and design dynamic regression trees are an attractive option for automatic regression and classification with complicated response surfaces in online application settings we create a sequential tree model whose state changes in time with the accumulation of new data and provide particle learning algorithms that allow for the efficient online posterior filtering of tree states a major advantage of tree regression is that it allows for the use of very simple models within each partition the model also facilitates a natural division of labor in our sequential particle based inference tree dynamics are defined through a few potential changes that are local to each newly arrived observation while global uncertainty is captured by the ensemble of particles we consider both constant and linear mean functions at the tree leaves along with multinomial leaves for classification problems and propose default prior specifications that allow for prediction to be integrated over all model parameters conditional on a given

generalized spike and slab priors for bayesian group feature selection using expectation propagation we describe a bayesian method for group feature selection in linear regression problems the method is based on a generalized version of the standard spike and slab prior distribution which is often used for individual feature selection exact bayesian inference under the prior considered is infeasible for typical regression problems however approximate inference can be carried out efficiently using expectation propagation ep a detailed analysis of the generalized spike and slab prior shows that it is well suited for regression problems that are sparse at the group level furthermore this prior can be used to introduce prior knowledge about specific groups of features that are a priori believed to be more relevant an experimental evaluation compares the performance of the proposed method with those of group lasso bayesian group lasso automatic relevance determination and additional variants used for group feature selection the results of these experiments show that a model based on the generalized spike and slab prior and the ep algorithm has state of the art prediction performance in the problems analyzed furthermore this model is also very useful to carry out sequential experimental design also known as active learning where the data instances that are most informative are iteratively included in the training set reducing the number of instances needed to obtain a particular level of prediction accuracy

near optimal sensor placement for linear inverse problems a classic problem is the estimation of a set of parameters from measurements collected by only a few sensors the number of sensors is often limited by physical or economical constraints and their placement is of fundamental importance to obtain accurate estimates unfortunately the selection of the optimal sensor locations is intrinsically combinatorial and the available approximation algorithms are not guaranteed to generate good solutions in all cases of interest we propose framesense a greedy algorithm for the selection of optimal sensor locations the core cost function of the algorithm is the frame potential a scalar property of matrices that measures the orthogonality of its rows notably framesense is the first algorithm that is near optimal in terms of mean square error meaning that its solution is always guaranteed to be close to the optimal one moreover we show with an extensive set of numerical experiments that framesense achieves state of the art performance while having the lowest computational cost when compared to other greedy methods

information based objective functions for active data selection learning can be made more efficient if we can actively select particularly salient data points within a bayesian learning framework objective functions are discussed that measure the expected informativeness of candidate measurements three alternative specifications of what we want to gain information about lead to three different criteria for data selection all these criteria depend on the assumption that the hypothesis space is correct which may prove to be their main weakness

adaptive design optimization a mutual information based approach to model discrimination in cognitive science discriminating among competing statistical models is a pressing issue for many experimentalists in the field of cognitive science resolving this issue begins with designing maximally informative experiments to this end the problem to be solved in adaptive design optimization is identifying experimental designs under which one can infer the underlying model in the fewest possible steps when the models under consideration are nonlinear as is often the case in cognitive science this problem can be impossible to solve analytically without simplifying assumptions however as we show in this letter a full solution can be found numerically with the help of a bayesian computational trick derived from the statistics literature which recasts the problem as a probability density simulation in which the optimal design is the mode of the density we use a utility function based on mutual information and give three intuitive interpretations of the utility function in terms of bayesian posterior estimates as a proof of concept we offer a simple example application to an experiment on memory retention

an information theoretic approach of designing sparse kernel adaptive filters this paper discusses an information theoretic approach of designing sparse kernel adaptive filters to determine useful data to be learned and remove redundant ones a subjective information measure called surprise is introduced surprise captures the amount of information a datum contains which is transferable to a learning system based on this concept we propose a systematic sparsification scheme which can drastically reduce the time and space complexity without harming the performance of kernel adaptive filters nonlinear regression short term chaotic time series prediction and long term time series forecasting examples are presented

on the rationality of the moduli space of luroth quartics we prove that the moduli space mathfrak m l of luroth quartics in mathbb p 2 i e the space of quartics which can be circumscribed around a complete pentagon of lines modulo the action of mathrm pgl 3 mathbb c is rational as is the related moduli space of bateman seven tuples of points in mathbb p 2

four types of learning curves if machines are learning to make decisions given a number of examples the generalization error e t is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples the generalization error decreases as t increases and the curve e t is called a learning curve the present paper uses the bayesian approach to show that given the annealed approximation learning curves can be classified into four asymptotic types if the machine is deterministic with noiseless teacher signals then 1 e at 1 when the correct machine parameter is unique and 2 e at 2 when the set of the correct parameters has a finite measure if the teacher signals are noisy then 3 e at 1 2 for a deterministic machine and 4 e c at 1 for a stochastic machine

how to modify a neural network gradually without changing its input output functionality it is generally unknown when distinct neural networks having different synaptic weights and thresholds implement identical input output transformations determining the exact conditions for structurally distinct yet functionally equivalent networks may shed light on the theoretical constraints on how diverse neural circuits might develop and be maintained to serve identical functions such consideration also imposes practical limits on our ability to uniquely infer the structure of underlying neural circuits from stimulus response measurements we introduce a biologically inspired mathematical method for determining when the structure of a neural network can be perturbed gradually while preserving functionality we show that for common three layer networks with convergent and nondegenerate connection weights this is possible only when the hidden unit gains are power functions exponentials or logarithmic functions which are known to approximate the gains seen in some biological neurons for practical applications our numerical simulations with finite and noisy data show that continuous confounding of parameters due to network functional equivalence tends to occur approximately even when the gain function is not one of the aforementioned three types suggesting that our analytical results are applicable to more general situations and may help identify a common source of parameter variability in neural network modeling

active learning in approximately linear regression based on conditional expectation of generalization error the goal of active learning is to determine the locations of training input points so that the generalization error is minimized we discuss the problem of active learning in linear regression scenarios traditional active learning methods using least squares learning often assume that the model used for learning is correctly specified in many practical situations however this assumption may not be fulfilled recently active learning methods using importance weighted least squares learning have been proposed which are shown to be robust against misspecification of models in this paper we propose a new active learning method also using the weighted least squares learning which we call alice active learning using the importance weighted least squares learning based on conditional expectation of the generalization error an important difference from existing methods is that we predict the conditional expectation of the generalization error given training input points while existing methods predict the full expectation of the generalization error due to this difference the training input design can be fine tuned depending on the realization of training input points theoretically we prove that the proposed active learning criterion is a more accurate predictor of the single trial generalization error than the existing criterion numerical studies with toy and benchmark data sets show that the proposed method compares favorably to existing methods

a stochastic self organizing map for proximity data we derive an efficient algorithm for topographic mapping of proximity data tmp which can be seen as an extension of kohonen s self organizing map to arbitrary distance measures the tmp cost function is derived in a baysian framework of folded markov chains for the description of autoencoders it incorporates the data by a dissimilarity matrix and the topographic neighborhood by a matrix of transition probabilities from the principle of maximum entropy a nonfactorizing gibbs distribution is obtained which is approximated in a mean field fashion this allows for maximum likelihood estimation using an expectation maximization algorithm in analogy to the transition from topographic vector quantization to the self organizing map we suggest an approximation to tmp that is computationally more efficient in order to prevent convergence to local minima an annealing scheme in the temperature parameter is introduced for which the critical temperature of the first phase transition is calculated in terms o

a new framework for distributed detection with conditionally dependent observations distributed detection with conditionally dependent observations is known to be a challenging problem in decentralized inference this paper attempts to make progress on this problem by proposing a new framework for distributed detection that builds on a hierarchical conditional independence model through the introduction of a hidden variable that induces conditional independence among the sensor observations the proposed model unifies distributed detection with dependent or independent observations this new framework allows us to identify several classes of distributed detection problems with dependent observations whose optimal decision rules resemble the ones for the independent case the new framework induces a decoupling effect on the forms of the optimal local decision rules for these problems much in the same way as the conditionally independent case this is in sharp contrast to the general dependent case where the coupling of the forms of local sensor decision rules often renders the problem intractable such decoupling enables the use of for example the person by person optimization approach to find optimal local decision rules two classical examples in distributed detection with dependent observations are reexamined under this new framework detection of a deterministic signal in dependent noises and detection of a random signal in independent noises

selecting optimal experiments for multiple output multilayer perceptrons where should a researcher conduct experiments to provide training data for a multilayer perceptron this question is investigated and a statistical method for selecting optimal experimental design points for multiple output multilayer perceptrons is introduced multiple class discrimination problems are examined using a framework in which the multilayer perceptron is viewed as a multivariate nonlinear regression model following a bayesian formulation for the case where the variance covariance matrix of the responses is unknown a selection criterion is developed this criterion is based on the volume of the joint confidence ellipsoid for the weights in a multilayer perceptron an example is used to demonstrate the superiority of optimally selected design points over randomly chosen points as well as points chosen in a grid pattern simplification of the basic criterion is offered through the use of hadamard matrices to produce uncorrelated outputs

multi output separable gaussian process towards an efficient fully bayesian paradigm for uncertainty quantification computer codes simulating physical systems usually have responses that consist of a set of distinct outputs e g velocity and pressure that evolve also in space and time and depend on many unknown input parameters e g physical constants initial boundary conditions etc furthermore essential engineering procedures such as uncertainty quantification inverse problems or design are notoriously difficult to carry out mostly due to the limited simulations available the aim of this work is to introduce a fully bayesian approach for treating these problems which accounts for the uncertainty induced by the finite number of observations our model is built on a multi dimensional gaussian process that explicitly treats correlations between distinct output variables as well as space and or time the proper use of a separable covariance function enables us to describe the huge covariance matrix as a kronecker product of smaller matrices leading to efficient algorithms for carrying out inference and predictions the novelty of this work is the recognition that the gaussian process model defines a posterior probability measure on the function space of possible surrogates for the computer code and the derivation of an algorithmic procedure that allows us to sample it efficiently we demonstrate how the scheme can be used in uncertainty quantification tasks in order to obtain error bars for the statistics of interest that account for the finite number of observations

characterizing responses of translation invariant neurons to natural stimuli maximally informative invariant dimensions the human visual system is capable of recognizing complex objects even when their appearances change drastically under various viewing conditions especially in the higher cortical areas the sensory neurons reflect such functional capacity in their selectivity for complex visual features and invariance to certain object transformations such as image translation due to the strong nonlinearities necessary to achieve both the selectivity and invariance characterizing and predicting the response patterns of these neurons represents a formidable computational challenge a related problem is that such neurons are poorly driven by randomized inputs such as white noise and respond strongly only to stimuli with complex high order correlations such as natural stimuli here we describe a novel two step optimization technique that can characterize both the shape selectivity and the range and coarseness of position invariance from neural responses to natural stimuli one step in the optimization is finding the template as the maximally informative dimension given the estimated spatial location where the response could have been triggered within each image the estimates of the locations that triggered the response are updated in the next step under the assumption of a monotonic relationship between the firing rate and stimulus projections on the template at a given position the most likely location is the one that has the largest projection on the estimate of the template the algorithm shows quick convergence during optimization and the estimation results are reliable even in the regime of small signal to noise ratios when we apply the algorithm to responses of complex cells in the primary visual cortex v1 to natural movies we find that responses of the majority of cells were significantly better described by translation invariant models based on one template compared with position specific models with several relevant features

an equivalence between sparse approximation and support vector machines in the first part of this paper we show a similarity between the principle of structural risk minimization principle srm vapnik 1982 and the idea of sparse approximation as defined in chen donoho and saunders 1995 and olshausen and field 1996 then we focus on two specific approximate implementations of srm and sparse approximation which have been used to solve the problem of function approximation for srm we consider the support vector machine technique proposed by v vapnik and his team at at t bell labs and for sparse approximation we consider a modification of the basis pursuit de noising algorithm proposed by chen donoho and saunders 1995 we show that under certain conditions these two techniques are equivalent they give the same solution and they require the solution of the same quadratic programming problem

a practical bayesian framework for backpropagation networks a quantitative and practical bayesian framework is described for learning of mappings in feedforward networks the framework makes possible 1 objective comparisons between solutions using alternative network architectures 2 objective stopping rules for network pruning or growing procedures 3 objective choice of magnitude and type of weight decay terms or additive regularizers for penalizing large weights etc 4 a measure of the effective number of well determined parameters in a model 5 quantified estimates of the error bars on network parameters and on network output and 6 objective comparisons with alternative learning and interpolation models such as splines and radial basis functions the bayesian evidence automatically embodies occam s razor penalizing overflexible and overcomplex models the bayesian approach helps detect poor underlying assumptions in learning models for learning models well matched to a problem a good correlation between generalization ability and the bayesian evidence is obtained

sensor placement by maximal projection on minimum eigenspace for linear inverse problems this paper presents two new greedy sensor placement algorithms named minimum nonzero eigenvalue pursuit mnep and maximal projection on minimum eigenspace mpme for linear inverse problems with greater emphasis on the mpme algorithm for performance comparison with existing approaches in both mnep and mpme we select the sensing locations one by one in this way the least number of required sensor nodes can be determined by checking whether the estimation accuracy is satisfied after each sensing location is determined for the mpme algorithm the minimum eigenspace is defined as the eigenspace associated with the minimum eigenvalue of the dual observation matrix for each sensing location the projection of its observation vector onto the minimum eigenspace is shown to be monotonically decreasing w r t the worst case error variance wcev of the estimated parameters we select the sensing location whose observation vector has the maximum projection onto the minimum eigenspace of the current dual observation matrix the proposed mpme is shown to be one of the most computationally efficient algorithms our monte carlo simulations showed that mpme outperforms the convex relaxation method the sparsense method and the framesense method in terms of wcev and the mean square error mse of the estimated parameters especially when the number of available sensor nodes is very limited

robust submodular observation selection in many applications one has to actively select among a set of expensive observations before making an informed decision for example in environmental monitoring we want to select locations to measure in order to most effectively predict spatial phenomena often we want to select observations which are robust against a number of possible objective functions examples include minimizing the maximum posterior variance in gaussian process regression robust experimental design and sensor placement for outbreak detection in this paper we present the submodular saturation algorithm a simple and efficient algorithm with strong theoretical approximation guarantees for cases where the possible objective functions exhibit submodularity an intuitive diminishing returns property moreover we prove that better approximation algorithms do not exist unless np complete problems admit efficient algorithms we show how our algorithm can be extended to handle complex cost functions incorporating non unit observation cost or communication and path costs we also show how the algorithm can be used to near optimally trade off expected case e g the mean square prediction error in gaussian process regression and worst case e g maximum predictive variance performance we show that many important machine learning problems fit our robust submodular observation selection formalism and provide extensive empirical evaluation on several real world problems for gaussian process regression our algorithm compares favorably with state of the art heuristics described in the geostatistics literature while being simpler faster and providing theoretical guarantees for robust experimental design our algorithm performs favorably compared to sdp based algorithms c 2008 andreas krause h brendan mcmahan carlos guestrin and anupam gupta krause mcmahan guestrin and gupta

sensor centric data reduction for estimation with wsns via censoring and quantization consider a wireless sensor network wsn with a fusion center fc deployed to estimate signal parameters from noisy sensor measurements if the wsn has a large number of low cost battery operated sensor nodes with limited transmission bandwidth then conservation of transmission resources power and bandwidth is paramount to this end the present paper develops a novel data reduction method which requires no inter sensor collaboration and results in only a subset of the sensor measurements transmitted to the fc using interval censoring as a data reduction method each sensor decides separately whether to censor its acquired measurements based on a rule that promotes censoring of measurements with least impact on the estimator mean square error mse leveraging the statistical distribution of sensor data the censoring mechanism and the received uncensored data fc based estimators are derived for both deterministic via maximum likelihood estimation and random parameters via maximum a posteriori probability estimation for a linear gaussian model quantization of the uncensored measurements at the sensor nodes offers an additional degree of freedom in the resource conservation versus estimator mse reduction tradeoff cramer rao bound analysis for the different censor estimators and censor quantizer estimators is also provided to benchmark and facilitate mse based performance comparisons numerical simulations corroborate the analytical findings and demonstrate that the proposed censoring estimation approach performs competitively with alternative methods under different sensing conditions while having lower computational complexity

an information based learning approach to dual control dual control aims to concurrently learn and control an unknown system however actively learning the system conflicts directly with any given control objective for it will disturb the system during exploration this paper presents a receding horizon approach to dual control where a multiobjective optimization problem is solved repeatedly and subject to constraints representing system dynamics balancing a standard finite horizon control objective a knowledge gain objective is defined to explicitly quantify the information acquired when learning the system dynamics measures from information theory such as entropy based uncertainty fisher information and relative entropy are studied and used to quantify the knowledge gained as a result of the control actions the resulting iterative framework is applied to markov decision processes and discrete time nonlinear systems thus the broad applicability and usefulness of the presented approach is demonstrated in diverse problem settings the framework is illustrated with multiple numerical examples

asymptotic theory of information theoretic experimental design we discuss an idea for collecting data in a relatively efficient manner our point of view is bayesian and information theoretic on any given trial we want to adaptively choose the input in such a way that the mutual information between the unknown state of the system and the stochastic output is maximal given any prior information including data collected on any previous trials we prove a theorem that quantifies the effectiveness of this strategy and give a few illustrative examples comparing the performance of this adaptive technique to that of the more usual nonadaptive experimental design in particular we calculate the asymptotic efficiency of the information maximization strategy and demonstrate that this method is in a well defined sense never less efficient and is generically more efficient than the nonadaptive strategy for example we are able to explicitly calculate the asymptotic relative efficiency of the staircase method widely employed in psychophysics research and to demonstrate the dependence of this efficiency on the form of the psychometric function underlying the output responses

information based complexity feedback and dynamics in convex programming we study the intrinsic limitations of sequential convex optimization through the lens of feedback information theory in the oracle model of optimization an algorithm queries an oracle for noisy information about the unknown objective function and the goal is to approximately minimize every function in a given class using as few queries as possible we show that in order for a function to be optimized the algorithm must be able to accumulate enough information about the objective this in turn puts limits on the speed of optimization under specific assumptions on the oracle and the type of feedback our techniques are akin to the ones used in statistical literature to obtain minimax lower bounds on the risks of estimation procedures the notable difference is that unlike in the case of i i d data a sequential optimization algorithm can gather observations in a controlled manner so that the amount of information at each step is allowed to change in time in particular we show that optimization algorithms often obey the law of diminishing returns the signal to noise ratio drops as the optimization algorithm approaches the optimum to underscore the generality of the tools we use our approach to derive fundamental lower bounds for a certain active learning problem overall the present work connects the intuitive notions of information in optimization experimental design estimation and active learning to the quantitative notion of shannon information

active state estimation for nonlinear systems a neural approximation approach in this paper we consider the problem of actively providing an estimate of the state of a stochastic dynamic system over a possibly long finite time horizon the active estimation problem aep is formulated as a stochastic optimal control one in which the minimization of a suitable uncertainty measure is carried out toward this end the use of the renyi entropy as an information measure is proposed and motivated a neural control scheme based on the application of the extended ritz method erim and on the use of a gaussian sum filter gsf is then presented simulation results show the effectiveness of the proposed approach

bounds on error expectation for support vector machines we introduce the concept of span of support vectors sv and show that the generalization ability of support vector machines svm depends on this new geometrical concept we prove that the value of the span is always smaller and can be much smaller than the diameter of the smallest sphere containing the support vectors used in previous bounds vapnik 1998 we also demonstate experimentally that the prediction of the test error given by the span is very accurate and has direct application in model selection choice of the optimal parameters of the svm

minimax bounds for active learning this paper analyzes the potential advantages and theoretical challenges of active learning algorithms active learning involves sequential sampling procedures that use information gleaned from previous samples in order to focus the sampling and accelerate the learning process relative to passive learning algorithms which are based on nonadaptive usually random samples there are a number of empirical and theoretical results suggesting that in certain situations active learning can be significantly more effective than passive learning however the fact that active learning algorithms are feedback systems makes their theoretical analysis very challenging this paper aims to shed light on achievable limits in active learning using minimax analysis techniques we study the achievable rates of classification error convergence for broad classes of distributions characterized by decision boundary regularity and noise conditions the results clearly indicate the conditions under which one can expect significant gains through active learning furthermore we show that the learning rates derived are tight for boundary fragment classes in d dimensional feature spaces when the feature marginal density is bounded from above and below

algebraic analysis for nonidentifiable learning machines this article clarifies the relation between the learning curve and the algebraic geometrical structure of a nonidentifiable learning machine such as a multilayer neural network whose true parameter set is an analytic set with singular points by using a concept in algebraic analysis we rigorously prove that the bayesian stochastic complexity or the free energy is asymptotically equal to 1 log n m1 1 log log n constant where n is the number of training samples and 1 and m1 are the rational number and the natural number which are determined as the birational invariant values of the singularities in the parameter space also we show an algorithm to calculate 1 and m1 based on the resolution of singularities in algebraic geometry in regular statistical models 2 1 is equal to the number of parameters and m1 1 whereas in nonregular models such as multilayer networks 2 1 is not larger than the number of parameters and m1 1 since the increase of the stochastic complexity is equal to the learning curve or the generalization error the nonidentifiable learning machines are better models than the regular ones if bayesian ensemble learning is applied

active learning and basis selection for kernel based linear models a bayesian perspective we develop an active learning algorithm for kernel based linear regression and classification the proposed greedy algorithm employs a minimum entropy criterion derived using a bayesian interpretation of ridge regression we assume access to a matrix bbrn n for which the i j th element is defined by the kernel function k i j bbr with the observed data i bbrd we seek a model m i yi where yi is a real valued response or integer valued label which we do not have access to a priori to achieve this goal a submatrix il ib bbrn m is sought that corresponds to the intersection of n rows and m columns of indexed by the sets il and ib respectively typically m n and n n we have two objectives i determine the m columns of indexed by the set ib that are the most informative for building a linear model m 1 i ib t yi without any knowledge of yi i 1n and ii using active learning sequentially determine which subset of n elements of yi i 1n should be acquired both stopping values ib m and il n are also to be inferred from the data these steps are taken with the goal of minimizing the uncertainty of the model parameters x as measured by the differential entropy of its posterior distribution the parameter vector x bbrm as well as the model bias bbr is then learned from the resulting problem yil il ibx 1 the remaining n n responses labels not included in yil can be inferred by applying x to the remaining n n rows of ib we show experimental results for several regression and classification problems and compare to other active learning methods

a review of bayesian neural networks with an application to near infrared spectroscopy mackay s 1992 bayesian framework for backpropagation is a practical and powerful means to improve the generalization ability of neural networks it is based on a gaussian approximation to the posterior weight distribution the framework is extended reviewed and demonstrated in a pedagogical way the notation is simplified using the ordinary weight decay parameter and a detailed and explicit procedure for adjusting several weight decay parameters is given bayesian backprop is applied in the prediction of fat content in minced meat from near infrared spectra it outperforms early stopping as well as quadratic regression the evidence of a committee of differently trained networks is computed and the corresponding improved generalization is verified the error bars on the predictions of the fat content are computed there are three contributors the random noise the uncertainty in the weights and the deviation among the committee members the bayesian framework is compared to moody s gpe 1992 finally mackay and neal s automatic relevance determination in which the weight decay parameters depend on the input number is applied to the data with improved results

variational bayes for continuous hidden markov models and its application to active learning in this paper we present a variational bayes vb framework for learning continuous hidden markov models chmms and we examine the vb framework within active learning unlike a maximum likelihood or maximum a posteriori training procedure which yield a point estimate of the chmm parameters vb based training yields an estimate of the full posterior of the model parameters this is particularly important for small training sets since it gives a measure of confidence in the accuracy of the learned model this is utilized within the context of active learning for which we acquire labels for those feature vectors for which knowledge of the associated label would be most informative for reducing model parameter uncertainty three active learning algorithms are considered in this paper 1 query by committee qbc with the goal of selecting data for labeling that minimize the classification variance 2 a maximum expected information gain method that seeks to label data with the goal of reducing the entropy of the model parameters and 3 an error reduction based procedure that attempts to minimize classification error over the test data the experimental results are presented for synthetic and measured data we demonstrate that all of these active learning methods can significantly reduce the amount of required labeling compared to random selection of samples for labeling

an interactive approach to solving correspondence problems finding correspondences among objects in different images is a critical problem in computer vision even good correspondence procedures can fail however when faced with deformations occlusions and differences in lighting and zoom levels across images we present a methodology for augmenting correspondence matching algorithms with a means for triaging the focus of attention and effort in assisting the automated matching for guiding the mix of human and automated initiatives we introduce a measure of the expected value of resolving correspondence uncertainties we explore the value of the approach with experiments on benchmark data

machine learning of linear differential equations using gaussian processes this work leverages recent advances in probabilistic machine learning to discover governing equations expressed by parametric linear operators such equations involve but are not limited to ordinary and partial differential integro differential and fractional order operators here gaussian process priors are modified according to the particular form of such operators and are employed to infer parameters of the linear equations from scarce and possibly noisy observations such observations may come from experiments or black box computer simulations as demonstrated in several synthetic examples and a realistic application in functional genomics employ probabilistic machine learning to discover governing equations expressed by parametric linear operators proper placement of gaussian process priors allows one to efficiently infer model parameters via maximum likelihood estimation a general treatment of inverse problems governed by linear operators leading to model discovery from just a handful of noisy measurements

sequential optimal design of neurophysiology experiments adaptively optimizing experiments has the potential to significantly reduce the number of trials needed to build parametric statistical models of neural systems however application of adaptive methods to neurophysiology has been limited by severe computational challenges since most neurons are high dimensional systems optimizing neurophysiology experiments requires computing high dimensional integrations and optimizations in real time here we present a fast algorithm for choosing the most informative stimulus by maximizing the mutual information between the data and the unknown parameters of a generalized linear model glm that we want to fit to the neuron s activity we rely on important log concavity and asymptotic normality properties of the posterior to facilitate the required computations our algorithm requires only low rank matrix manipulations and a two dimensional search to choose the optimal stimulus the average running time of these operations scales quadratically with the dimensionality of the glm making real time adaptive experimental design feasible even for high dimensional stimulus and parameter spaces for example we require roughly 10 milliseconds on a desktop computer to optimize a 100 dimensional stimulus despite using some approximations to make the algorithm efficient our algorithm asymptotically decreases the uncertainty about the model parameters at a rate equal to the maximum rate predicted by an asymptotic analysis simulation results show that picking stimuli by maximizing the mutual information can speed up convergence to the optimal values of the parameters by an order of magnitude compared to using random nonadaptive stimuli finally applying our design procedure to real neurophysiology experiments requires addressing the nonstationarities that we would expect to see in neural responses our algorithm can efficiently handle both fast adaptation due to spike history effects and slow nonsystematic drifts in a neuron s activity

some comments on cp i study the typical configuration of a cp plot when the number of variables in the regression problem is large and there are many weak effects i show that a particular configuration that is very commonly seen can arise in a simple way i give a formula by means of which the risk incurred by the minimum cp rule can be estimated

generalised information criteria in model selection summary the problem of evaluating the goodness of statistical models is investigated from an information theoretic point of view information criteria are proposed for evaluating models constructed by various estimation procedures when the specified family of probability distributions does not contain the distribution generating the data the proposed criteria are applied to the evaluation of models estimated by maximum likelihood robust penalised likelihood bayes procedures etc we also discuss the use of the bootstrap in model evaluation problems and present a variance reduction technique in the bootstrap simulation

bias variance decompositions for likelihood based estimators the bias variance decomposition of mean squared error is well understood and relatively straightforward in this note a similar simple decomposition is derived valid for any kind of error measure that when using the appropriate probability model can be derived from a kullback leibler divergence or log likelihood

near optimal sensor placements in gaussian processes theory efficient algorithms and empirical studies when monitoring spatial phenomena which can often be modeled as gaussian processes gps choosing sensor locations is a fundamental task there are several common strategies to address this task for example geometry or disk models placing sensors at the points of highest entropy variance in the gp model and a d or e optimal design in this paper we tackle the combinatorial optimization problem of maximizing the mutual information between the chosen locations and the locations which are not selected we prove that the problem of finding the configuration that maximizes mutual information is np complete to address this issue we describe a polynomial time approximation that is within 1 1 e of the optimum by exploiting the submodularity of mutual information we also show how submodularity can be used to obtain online bounds and design branch and bound search procedures we then extend our algorithm to exploit lazy evaluations and local structure in the gp yielding significant speedups we also extend our approach to find placements which are robust against node failures and uncertainties in the model these extensions are again associated with rigorous theoretical approximation guarantees exploiting the submodularity of the objective function we demonstrate the advantages of our approach towards optimizing mutual information in a very extensive empirical study on two real world data sets

gaussian processes for object categorization discriminative methods for visual object category recognition are typically non probabilistic predicting class labels but not directly providing an estimate of uncertainty gaussian processes gps provide a framework for deriving regression techniques with explicit uncertainty models we show here how gaussian processes with covariance functions defined based on a pyramid match kernel pmk can be used for probabilistic object category recognition our probabilistic formulation provides a principled way to learn hyperparameters which we utilize to learn an optimal combination of multiple covariance functions it also offers confidence estimates at test points and naturally allows for an active learning paradigm in which points are optimally selected for interactive labeling we show that with an appropriate combination of kernels a significant boost in classification performance is possible further our experiments indicate the utility of active learning with probabilistic predictive models especially when the amount of training data labels that may be sought for a category is ultimately very small

a decision theoretic extension of stochastic complexity and its applications to learning rissanen 1978 has introduced stochastic complexity to define the amount of information in a given data sequence relative to a given hypothesis class of probability densities where the information is measured in terms of the logarithmic loss associated with universal data compression this paper introduces the notion of extended stochastic complexity esc and demonstrates its effectiveness in design and analysis of learning algorithms in on line prediction and batch learning scenarios esc can be thought of as an extension of rissanen s stochastic complexity to the decision theoretic setting where a general real valued function is used as a hypothesis and a general loss function is used as a distortion measure as an application of esc to on line prediction this paper shows that a sequential realization of esc produces an on line prediction algorithm called vovk s aggregating strategy which can be thought of as an extension of the bayes algorithm we derive upper bounds on the cumulative loss for the aggregating strategy both of an expected form and a worst case form in the case where the hypothesis class is continuous as an application of esc to batch learning this paper shows that a batch approximation of esc induces a batch learning algorithm called the minimum l complexity algorithm mlc which is an extension of the minimum description length mdl principle we derive upper bounds on the statistical risk for the mlc which are the least to date through the esc we give a unifying view of the most effective learning algorithms that have been explored in computational learning theory

subspace information criterion for model selection the problem of model selection is considerably important for acquiring higher levels of generalization capability in supervised learning in this article we propose a new criterion for model selection the subspace information criterion sic which is a generalization of mallows s cl it is assumed that the learning target function belongs to a specified functional hilbert space and the generalization error is defined as the hilbert space squared norm of the difference between the learning result function and target function sic gives an unbiased estimate of the generalization error so defined sic assumes the availability of an unbiased estimate of the target function and the noise covariance matrix which are generally unknown a practical calculation method of sic for least mean squares learning is provided under the assumption that the dimension of the hilbert space is less than the number of training examples finally computer simulations in two examples show that sic works well even when the number of training examples is small

design of quantizers for decentralized estimation systems the authors consider parameter estimation in decentralized systems with distributed processors they restrict the local processors to be quantizers and consider the optimal design of the systems to minimize the estimation error they present necessary conditions for the optimal system based on the bayes distortion functions and fisher s information the numerical results compare the resulting quantizers obtained by different distortion criteria

sequential design of computer experiments for the assessment of fetus exposure to electromagnetic fields in this article we describe four sequential sampling strategies for estimating the quantile of y f x where x has a known distribution in rd and f is a deterministic unknown expensive to evaluate real valued function these approaches all consist in modeling f as a sample of a well chosen gaussian process and aim at estimating the quantile by using as few evaluations of f as possible the different methodologies are first compared through various numerical experiments then in the framework of the anr jst fetus project we apply our strategies to a real example corresponding to the exposure of a japanese pregnant woman model and her 26 week old fetus to a plane wave finally we compare our methodologies on a simplified geometric model designed for modeling the fetus exposure to plane waves supplementary materials for this article are available online

a note on distributed estimation and sufficiency in relation to distributed parameter estimation the notion of local and global sufficient statistics is introduced it is shown that when a sufficiency condition is satisfied by the probability distribution of a random sample a global sufficient statistic is obtainable as a function of local sufficient statistics several standard distributions satisfy the said sufficiency condition

bayesian computation using design of experiments based interpolation technique in this article a new deterministic approximation method for bayesian computation known as design of experiments based interpolation technique doit is proposed the method works by sampling points from the parameter space using an experimental design and then fitting a kriging model to interpolate the unnormalized posterior the approximated posterior density is a weighted average of normal densities and therefore most of the posterior quantities can be easily computed doit is a general computing technique that is easy to implement and can be applied to many complex bayesian problems moreover it does not suffer from the curse of dimensionality as much as some quadrature methods it can work using fewer posterior evaluations which is a great advantage over the monte carlo and markov chain monte carlo methods especially when dealing with computationally expensive posteriors this article has supplementary material that is available online

covariate shift adaptation by importance weighted cross validation a common assumption in supervised learning is that the input points in the training set follow the same probability distribution as the input points that will be given in the future test phase however this assumption is not satisfied for example when the outside of the training region is extrapolated the situation where the training input points and test input points follow different distributions while the conditional distribution of output values given input points is unchanged is called the covariate shift under the covariate shift standard model selection techniques such as cross validation do not work as desired since its unbiasedness is no longer maintained in this paper we propose a new method called importance weighted cross validation iwcv for which we prove its unbiasedness even under the covariate shift the iwcv procedure is the only one that can be applied for unbiased classification under covariate shift whereas alternatives to iwcv exist for regression the usefulness of our proposed method is illustrated by simulations and furthermore demonstrated in the brain computer interface where strong non stationarity effects can be seen between training and test sessions

the evidence framework applied to classification networks three bayesian ideas are presented for supervised adaptive classifiers first it is argued that the output of a classifier should be obtained by marginalizing over the posterior distribution of the parameters a simple approximation to this integral is proposed and demonstrated this involves a moderation of the most probable classifier s outputs and yields improved performance second it is demonstrated that the bayesian framework for model comparison described for regression models in mackay 1992a b can also be applied to classification problems this framework successfully chooses the magnitude of weight decay terms and ranks solutions found using different numbers of hidden units third an information based data selection criterion is derived and demonstrated within this framework

multi output local gaussian process regression applications to uncertainty quantification we develop an efficient bayesian uncertainty quantification framework using a novel treed gaussian process model the tree is adaptively constructed using information conveyed by the observed data about the length scales of the underlying process on each leaf of the tree we utilize bayesian experimental design techniques in order to learn a multi output gaussian process the constructed surrogate can provide analytical point estimates as well as error bars for the statistics of interest we numerically demonstrate the effectiveness of the suggested framework in identifying discontinuities local features and unimportant dimensions in the solution of stochastic differential equations

adaptive hessian based nonstationary gaussian process response surface method for probability density approximation with application to bayesian solution of large scale inverse problems we develop an adaptive hessian based non stationary gaussian process gp response surface method for approximating a probability density function pdf that exploits its structure particularly the hessian of its negative logarithm of particular interest to us are pdfs that arise from the bayesian solution of large scale inverse problems which imply very expensive to evaluate pdfs the method can be considered as a piecewise adaptive gaussian approximation in which a gaussian tailored to the local hessian of the negative log probability density is constructed for each subregion in high dimensional parameter space the task of efficiently partitioning the parameter space into subregions is done implicitly through hessian informed membership probability functions the gp machinery is then employed to glue all local gaussian approximations into a global analytical response surface that is far cheaper to evaluate than the original expensive probability density the resulting response surface is also equipp

distributed estimation and quantization an algorithm is developed for the design of a nonlinear n sensor distributed estimation system subject to communication and computation constraints the algorithm uses only bivariate probability distributions and yields locally optimal estimators that satisfy the required system constraints it is shown that the algorithm is a generalization of the classical lloyd max results

an introduction to kernel based learning algorithms this paper provides an introduction to support vector machines kernel fisher discriminant analysis and kernel principal component analysis as examples for successful kernel based learning methods we first give a short background about vapnik chervonenkis theory and kernel feature spaces and then proceed to kernel based learning in supervised and unsupervised scenarios including practical and algorithmic considerations we illustrate the usefulness of kernel algorithms by discussing applications such as optical character recognition and dna analysis

bayesian active learning of neural firing rate maps with transformed gaussian process priors a firing rate map also known as a tuning curve describes the nonlinear relationship between a neuron s spike rate and a low dimensional stimulus e g orientation head direction contrast color here we investigate bayesian active learning methods for estimating firing rate maps in closed loop neurophysiology experiments these methods can accelerate the characterization of such maps through the intelligent adaptive selection of stimuli specifically we explore the manner in which the prior and utility function used in bayesian active learning affect stimulus selection and performance our approach relies on a flexible model that involves a nonlinearly transformed gaussian process gp prior over maps and conditionally poisson spiking we show that infomax learning which selects stimuli to maximize the information gain about the firing rate map exhibits strong dependence on the seemingly innocuous choice of nonlinear transformation function we derive an alternate utility function that selects stimuli to minimize the average posterior variance of the firing rate map and analyze the surprising relationship between prior parameterization stimulus selection and active learning performance in gp poisson models we apply these methods to color tuning measurements of neurons in macaque primary visual cortex

the good bad and ugly distributed detection of a known signal in dependent gaussian noise most results about quantized detection rely strongly on an assumption of independence among random variables with this assumption removed little is known thus in this paper bayes optimal binary quantization for the detection of a shift in mean in a pair of dependent gaussian random variables is studied this is arguably the simplest meaningful problem one could consider if results and rules are to be found they ought to make themselves plain in this problem for certain problem parametrizations meaning the signals and correlation coefficient optimal quantization is achievable via a single threshold applied to each observation the same as under independence in other cases one observation is best ignored or is quantized with two thresholds neither behavior is seen under independence further and again in distinction from the case of independence it is seen that in certain situations an xor fusion rule is optimal and in these cases the implied decision rule is bizarre the analysis is extended to the multivariate gaussian problem

extremal properties of likelihood ratio quantizers m hypotheses and a random variable y with a different probability distribution under each hypothesis are considered a quantizer is applied to form a quantized random variable gamma y the extreme points of the set of possible probability distributions of gamma y as gamma ranges over all quantizers is characterized optimality properties of likelihood ratio quantizers are established for a very broad class of quantization problems including problems involving the maximization of an ali silvey 1966 distance measure and the neyman pearson variant of the decentralized detection problem

bootstrap and cross validation estimates of the prediction error for linear regression models different estimates of the mean squared error of prediction for linear regression models are derived by the bootstrap and cross validation approaches a comparison is made under normal error distributions especially by the biases and the mean square errors the results indicate that the bias corrected bootstrap estimator is best unbiased and should be the first choice while its simulated variant has approximately the same behaviour on the other hand if only a comparison between uncorrected estimators is made with implications for nonlinear regression models in mind then other variants of bootstrap estimates are preferable for a large or a small dimension of the model parameter for a small dimension the cross validation estimate and sometimes grouped variants of it seem also to be acceptable if the model error is known to be small

a clebsch gordan formula for sl3 c and applications to rationality abstract if r s t are irreducible sl 3 c representations we give an easy and explicit description of a basis of the space of equivariant maps r s t theorem 3 1 we apply this method to the rationality problem for invariant function fields in particular we prove the rationality of the moduli space of plane curves of degree 34 this uses a criterion which ensures the stable rationality of some quotients of grassmannians by an sl action proposition 5 4

an optimal selection of regression variables summary an asymptotically optimal selection of regression variables is proposed the key assumption is that the number of control variables is infinite or increases with the sample size it is also shown that mallows s qp akaike s fpe and aic methods are all asymptotically equivalent to this method

asymptotic properties of criteria for selection of variables in multiple regression in normal linear regression analysis many model selection rules proposed from various viewpoints are available for the information criteria aic fpe c p pss and bic the asymptotic distribution of the selected model and the asymptotic quadratic risk based on each criterion are explicitly obtained

nonnegative matrix factorization for rapid recovery of constituent spectra in magnetic resonance chemical shift imaging of the brain we present an algorithm for blindly recovering constituent source spectra from magnetic resonance mr chemical shift imaging csi of the human brain the algorithm which we call constrained nonnegative matrix factorization cnmf does not enforce independence or sparsity instead only requiring the source and mixing matrices to be nonnegative it is based on the nonnegative matrix factorization nmf algorithm extending it to include a constraint on the positivity of the amplitudes of the recovered spectra this constraint enables recovery of physically meaningful spectra even in the presence of noise that causes a significant number of the observation amplitudes to be negative we demonstrate and characterize the algorithm s performance using sup 31 p volumetric brain data comparing the results with two different blind source separation methods bayesian spectral decomposition bsd and nonnegative sparse coding nnsc we then incorporate the cnmf algorithm into a hierarchical decomposition framework showing that it can be used to recover tissue specific spectra given a processing hierarchy that proceeds coarse to fine we demonstrate the hierarchical procedure on sup 1 h brain data and conclude that the computational efficiency of the algorithm makes it well suited for use in diagnostic work up

dethroning the fano factor a flexible model based approach to partitioning neural variability neurons in many brain areas exhibit high trial to trial variability with spike counts that are overdispersed relative to a poisson distribution recent work goris movshon simoncelli 2014 ha

variational inference in nonconjugate models mean field variational methods are widely used for approximate posterior inference in many probabilistic models in a typical application mean field methods approximately compute the posterior with a coordinate ascent optimization algorithm when the model is conditionally conjugate the coordinate updates are easily derived and in closed form however many models of interest like the correlated topic model and bayesian logistic regression are nonconjugate in these models mean field methods cannot be directly applied and practitioners have had to develop variational algorithms on a case by case basis in this paper we develop two generic methods for nonconjugate models laplace variational inference and delta method variational inference our methods have several advantages they allow for easily derived variational algorithms with a wide class of nonconjugate models they extend and unify some of the existing algorithms that have been derived for specific models and they work well on real world data sets we studied our methods on the correlated topic model bayesian logistic regression and hierarchical bayesian logistic regression

soft nearest prototype classification we propose a new method for the construction of nearest prototype classifiers which is based on a gaussian mixture ansatz and which can be interpreted as an annealed version of learning vector quantization lvq the algorithm performs a gradient descent on a cost function minimizing the classification error on the training set we investigate the properties of the algorithm and assess its performance for several toy data sets and for an optical letter classification task results show 1 that annealing in the dispersion parameter of the gaussian kernels improves classification accuracy 2 that classification results are better than those obtained with standard learning vector quantization lvq 2 1 lvq 3 for equal numbers of prototypes and 3 that annealing of the width parameter improved the classification capability additionally the principled approach provides an explanation of a number of features of the heuristic lvq methods