large scale multiple kernel learning while classical kernel based learning algorithms are based on a single kernel in practice it is often desirable to use multiple kernels lanckriet et al 2004 considered conic combinations of kernel matrices for classification leading to a convex quadratically constrained quadratic program we show that it can be rewritten as a semi infinite linear program that can be efficiently solved by recycling the standard svm implementations moreover we generalize the formulation and our method to a larger class of problems including regression and one class classification experimental results show that the proposed algorithm works for hundred thousands of examples or hundreds of kernels to be combined and helps for automatic model selection improving the interpretability of the learning result in a second part we discuss general speed up mechanism for svms especially when used with sparse feature maps as appear for string kernels allowing us to train a string kernel svm on a 10 million real world splice data set from computational biology we integrated multiple kernel learning in our machine learning toolbox shogun for which the source code is publicly available at http www fml tuebingen mpg de raetsch projects shogun

bayesian approaches to gaussian mixture modeling a bayesian based methodology is presented which automatically penalizes overcomplex models being fitted to unknown data we show that with a gaussian mixture model the approach is able to select an optimal number of components in the model and so partition data sets the performance of the bayesian method is compared to other methods of optimal model selection and found to give good results the methods are tested on synthetic and real data sets

load updating for finite element models a method for reducing the differences between experimental testing or real world events and their finite element counterparts is presented in this method system identification or finite element model updating are not performed instead the loads on the finite element model that would create equivaient displacements or strains are identified by various means thus saving a large amount of computational effort this method of finding an applied load is also extremely useful in the analysis of testing data for vehicles and other structures using methods based on classical least squares methods we present the basis for finite element load updating and sample application through the use of a computer program for a beam under static loads we examine the conditioning number of the resulting system of equations and provide a least squares solution that is more robust through the use of singular value decomposition this allows an easier analysis of structural models for example of a prototype vehicle by determining the loads that the prototype is subjected to in testing these loads can then be used on other models of that vehicle during the design process to decrease the time spent in testing while increasing design s reliability by reducing uncertainties in the applied load

adaptive voting rules for k nearest neighbors classifiers a simple form of cooperation between the k nearest neighbors nn approach to classification and the neural like property of adaptation is explored a tunable high level k nearest neighbors decision rule is defined that comprehends most previous generalizations of the common majority rule a learning procedure is developed that applies to this rule and exploits those statistical features that can be induced from the training set the overall approach is tested on a problem of handwritten character recognition experiments show that adaptivity in the decision rule may improve the recognition and rejection capability of standard k nn classifiers

bm e discriminative density propagation for visual tracking we introduce bm3e a conditional bayesian mixture of experts markov model that achieves consistent probabilistic estimates for discriminative visual tracking the model applies to problems of temporal and uncertain inference and represents the unexplored bottom up counterpart of pervasive generative models estimated with kalman filtering or particle filtering instead of inverting a nonlinear generative observation model at runtime we learn to cooperatively predict complex state distributions directly from descriptors that encode image observations typically bag of feature global image histograms or descriptors computed over regular spatial grids these are integrated in a conditional graphical model in order to enforce temporal smoothness constraints and allow a principled management of uncertainty the algorithms combine sparsity mixture modeling and nonlinear dimensionality reduction for efficient computation in high dimensional continuous state spaces the combined system automatically self initializes and recovers from failure the research has three contributions 1 we establish the density propagation rules for discriminative inference in continuous temporal chain models 2 we propose flexible supervised and unsupervised algorithms to learn feed forward multivalued contextual mappings multimodal state distributions based on compact conditional bayesian mixture of experts models and 3 we validate the framework empirically for the reconstruction of 3d human motion in monocular video sequences our tests on both real and motion capture based sequences show significant performance gains with respect to competing nearest neighbor regression and structured prediction methods

kernel methods in machine learning we review machine learning methods employing positive definite kernels these methods formulate learning and estimation problems in a reproducing kernel hilbert space rkhs of functions defined on the data domain expanded in terms of a kernel working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions the latter include nonlinear functions as well as functions defined on nonvectorial data we cover a wide range of methods ranging from binary classifiers to sophisticated methods for estimation with structured data

information based objective functions for active data selection learning can be made more efficient if we can actively select particularly salient data points within a bayesian learning framework objective functions are discussed that measure the expected informativeness of candidate measurements three alternative specifications of what we want to gain information about lead to three different criteria for data selection all these criteria depend on the assumption that the hypothesis space is correct which may prove to be their main weakness

mining data with random forests a survey and results of new tests random forests rf has become a popular technique for classification prediction studying variable importance variable selection and outlier detection there are numerous application examples of rf in a variety of fields several large scale comparisons including rf have been performed there are numerous articles where variable importance evaluations based on the variable importance measures available from rf are used for data exploration and understanding apart from the literature survey in rf area this paper also presents results of new tests regarding variable rankings based on rf variable importance measures we studied experimentally the consistency and generality of such rankings results of the studies indicate that there is no evidence supporting the belief in generality of such rankings a high variance of variable importance evaluations was observed in the case of small number of trees and small data sets

fast svm training using approximate extreme points applications of non linear kernel support vector machines svms to large data sets is seriously hampered by its excessive training time we propose a modification called the approximate extreme points support vector machine aesvm that is aimed at overcoming this burden our approach relies on conducting the svm optimization over a carefully selected subset called the representative set of the training data set we present analytical results that indicate the similarity of aesvm and svm solutions a linear time algorithm based on convex hulls and extreme points is used to compute the representative set in kernel space extensive computational experiments on nine data sets compared aesvm to libsvm chang and lin 2011 cvm tsang et al 2005 bvm tsang et al 2007 lasvm bordes et al 2005 svmperf joachims and yu 2009 and the random features method rahimi and recht 2007 our aesvm implementation was found to train much faster than the other methods while its classification accuracy was similar to that of libsvm in all cases in particular for a seizure detection data set aesvm training was almost 500 times faster than libsvm and lasvm and 20 times faster than cvm and bvm additionally aesvm also gave competitively fast classification times

information criteria for support vector machines this paper presents kernel regularization information criterion kric which is a new criterion for tuning regularization parameters in kernel logistic regression klr and support vector machines svms the main idea of the kric is based on the regularization information criterion ric we derive an eigenvalue equation to calculate the kric and solve the problem the computational cost for parameter tuning by the kric is reduced drastically by using the nystroumlm approximation the test error rate of svms or klr with the regularization parameter tuned by the kric is comparable with the one by the cross validation or evaluation of the evidence the computational cost of the kric is significantly lower than the one of the other criteria

probabilistic classification vector machines in this paper a sparse learning algorithm probabilistic classification vector machines pcvms is proposed we analyze relevance vector machines rvms for classification problems and observe that adopting the same prior for different classes may lead to unstable solutions in order to tackle this problem a signed and truncated gaussian prior is adopted over every weight in pcvms where the sign of prior is determined by the class label i e 1 or 1 the truncated gaussian prior not only restricts the sign of weights but also leads to a sparse estimation of weight vectors and thus controls the complexity of the model in pcvms the kernel parameters can be optimized simultaneously within the training algorithm the performance of pcvms is extensively evaluated on four synthetic data sets and 13 benchmark data sets using three performance metrics error rate err area under the curve of receiver operating characteristic auc and root mean squared error rmse we compare pcvms with soft margin support vector machines svmsoft hard margin support vector machines svmhard svm with the kernel parameters optimized by pcvms svmpcvm relevance vector machines rvms and some other baseline classifiers through five replications of twofold cross validation f test i e 5 times 2 cross validation f test over single data sets and friedman test with the corresponding post hoc test to compare these algorithms over multiple data sets we notice that pcvms outperform other algorithms including svmsoft svmhard rvm and svmpcvm on most of the data sets under the three metrics especially under auc our results also reveal that the performance of svmpcvm is slightly better than svmsoft implying that the parameter optimization algorithm in pcvms is better than cross validation in terms of performance and computational complexity in this paper we also discuss the superiority of pcvms formulation using maximum a posteriori map analysis and margin analysis which explain the empirical success of pcvms

modified cholesky factorizations in interior point algorithms for linear programming we investigate a modified cholesky algorithm typical of those used in most interior point codes for linear programming cholesky based interior point codes are popular for three reasons their implementation requires only minimal changes to standard sparse cholesky algorithms allowing us to take full advantage of software written by specialists in that area they tend to be more efficient than competing approaches that use alternative factorizations and they perform robustly on most practical problems yielding good interior point steps even when the coefficient matrix of the main linear system to be solved for the step components is ill conditioned we investigate this surprisingly robust performance by using analytical tools from matrix perturbation theory and error analysis illustrating our results with computational experiments finally we point out the potential limitations of this approach

computing average shaped tissue probability templates this note presents a framework for generating tissue probability maps that represent the average shape of a number of subjects brain images the procedure is formulated as finding maximum a posteriori estimates within a probabilistic generative model estimating the parameters involves alternating between estimating the deformations that match tissue class images of individual subjects to template and updating the template according to the latest estimates of the deformations a multinomial matching criterion is used such that multiple tissue class images e g grey and white matter are registered simultaneously with the current template estimate in order to generalise the resulting template to a broader range of subjects a template blurriness prior is included within the model

effects of structural modes on vibratory force determination by the pseudoinverse technique the accuracy and effectiveness of the pseudoinverse technique as a means of determining the operating vibratory loads on a structural system from acceleration mobilities obtained by modal superposition can be severely undermined by lack of proper consideration of the participation of the structural modes at the frequency of interest methods of linear algebra and modal analysis are used to establish the limitations of this technique with regard to the number of independent forces determinable at a given frequency in relation to the number and significance of structural modes participating in the response at that frequency results of numerical calculations based on two uniform beams one cantilevered and the other free free show that for an assumed level of random error in the acceleration mobility matrix the accuracy of the determined forces correlates well with a parameter that is related to the number of modes contributing to the response of the structure at a given frequency

comparison of approximate methods for handling hyperparameters i examine two approximate methods for computational implementation of bayesian hierarchical models that is models that include unknown hyperparameters such as regularization constants and noise levels in the evidence framework the model parameters are integrated over and the resulting evidence is maximized over the hyperparameters the optimized hyperparameters are used to define a gaussian approximation to the posterior distribution in the alternative map method the true posterior probability is found by integrating over the hyperparameters the true posterior is then maximized over the model parameters and a gaussian approximation is made the similarities of the two approaches and their relative merits are discussed and comparisons are made with the ideal hierarchical bayesian solution in moderately ill posed problems integration over hyperparameters yields a probability distribution with a skew peak which causes signifi cant biases to arise in the map method in contrast the evidence fram

a practical bayesian framework for backpropagation networks a quantitative and practical bayesian framework is described for learning of mappings in feedforward networks the framework makes possible 1 objective comparisons between solutions using alternative network architectures 2 objective stopping rules for network pruning or growing procedures 3 objective choice of magnitude and type of weight decay terms or additive regularizers for penalizing large weights etc 4 a measure of the effective number of well determined parameters in a model 5 quantified estimates of the error bars on network parameters and on network output and 6 objective comparisons with alternative learning and interpolation models such as splines and radial basis functions the bayesian evidence automatically embodies occam s razor penalizing overflexible and overcomplex models the bayesian approach helps detect poor underlying assumptions in learning models for learning models well matched to a problem a good correlation between generalization ability and the bayesian evidence is obtained

feature selection using probabilistic prediction of support vector regression this paper presents a new wrapper based feature selection method for support vector regression svr using its probabilistic predictions the method computes the importance of a feature by aggregating the difference over the feature space of the conditional density functions of the svr prediction with and without the feature as the exact computation of this importance measure is expensive two approximations are proposed the effectiveness of the measure using these approximations in comparison to several other existing feature selection methods for svr is evaluated on both artificial and real world problems the result of the experiments show that the proposed method generally performs better than or at least as well as the existing methods with notable advantage when the dataset is sparse

bayesian framework for least squares support vector machine classifiers gaussian processes and kernel fisher discriminant analysis the bayesian evidence framework has been successfully applied to the design of multilayer perceptrons mlps in the work of mackay nevertheless the training of mlps suffers from drawbacks like the nonconvex optimization problem and the choice of the number of hidden units in support vector machines svms for classification as introduced by vapnik a nonlinear decision boundary is obtained by mapping the input vector first in a nonlinear way to a high dimensional kernel induced feature space in which a linear large margin classifier is constructed practical expressions are formulated in the dual space in terms of the related kernel function and the solution follows from a convex quadratic programming qp problem in least squares svms ls svms the svm problem formulation is modified by introducing a least squares cost function and equality instead of inequality constraints and the solution follows from a linear system in the dual space implicitly the least squares formulation corresponds to a regression formulation and is also related to kernel fisher discriminant analysis the least squares regression formulation has advantages for deriving analytic expressions in a bayesian evidence framework in contrast to the classification formulations used for example in gaussian processes gps the ls svm formulation has clear primal dual interpretations and without the bias term one explicitly constructs a model that yields the same expressions as have been obtained with gps for regression in this article the bayesian evidence framework is combined with the ls svm classifier formulation starting from the feature space formulation analytic expressions are obtained in the dual space on the different levels of bayesian inference while posterior class probabilities are obtained by marginalizing over the model parameters empirical results obtained on 10 public domain data sets show that the ls svm classifier designed within the bayesian evidence framework consistently yields good generalization performances

hash kernels for structured data we propose hashing to facilitate efficient kernels this generalizes previous work using sampling and we show a principled way to compute the kernel matrix for data streams and sparse feature spaces moreover we give deviation bounds from the exact kernel matrix this has applications to estimation on strings and graphs

generalization performance of regularization networks and support vector machines via entropy numbers of compact operators we derive new bounds for the generalization error of kernel machines such as support vector machines and related regularization networks by obtaining new bounds on their covering numbers the proofs make use of a viewpoint that is apparently novel in the field of statistical learning theory the hypothesis class is described in terms of a linear operator mapping from a possibly infinite dimensional unit ball in feature space into a finite dimensional space the covering numbers of the class are then determined via the entropy numbers of the operator these numbers which characterize the degree of compactness of the operator can be bounded in terms of the eigenvalues of an integral operator induced by the kernel function used by the machine as a consequence we are able to theoretically explain the effect of the choice of kernel function on the generalization performance of support vector machines

sparse bayesian learning for the laplace transform inversion in dynamic light scattering a new method is described using the sparse bayesian learning sbl algorithm of tipping to obtain an optimal and reliable solution to the laplace transform inversion in dynamic light scattering dls the linear inverse problem in dls has numerical solutions that depend on their domains and dimensions for a given domain and dimension a sparse solution in an sbl framework is the most probable solution and can be used for classifying a system of objects by a few relevant values recently we have shown that the sbl algorithm of tipping is suitable for studying cataract in ocular lenses by describing the opacity of a lens with a few dominant sizes of crystallin proteins in the lens however since the sparseness of sbl solutions cannot reflect a true system we need to develop a method by using the sbl algorithm to give a true description and at the same time a useful classification of the opacity of lenses we generate a set of sparse solutions of different domains but of the same dimension and then superimpose them to give a general solution with its dimension treated as a regularization parameter an optimal solution which provides a reliable description of a particle system is determined by the l curve criterion for selecting the suitable value of the regularization parameter the performance of our method is evaluated by analyzing simulated data generated from unimodal and bimodal distributions from the reconstructed distributions we see that our method gives high resolution comparable to the sophisticated bryan s maximum entropy algorithm which gives better resolution than contin our method is then applied to experimental dls data of the ocular lenses of a fetal calf and a rhesus monkey to obtain optimal particle size distributions of crystallins and the crystallin aggregates in the lenses we conclude by discussing possible improvements on our method for analyzing dls data and for solving any linear inverse problem by an sbl algorithm

tpmsvm a novel twin parametric margin support vector machine for pattern recognition a novel twin parametric margin support vector machine tpmsvm for classification is proposed in this paper this tpmsvm in the spirit of the twin support vector machine twsvm determines indirectly the separating hyperplane through a pair of nonparallel parametric margin hyperplanes solved by two smaller sized support vector machine svm type problems similar to the parametric margin email protected support vector machine email protected svm this tpmsvm is suitable for many cases especially when the data has heteroscedastic error structure that is the noise strongly depends on the input value but there is an advantage in the learning speed compared with the email protected svm the experimental results on several artificial and benchmark datasets indicate that the tpmsvm not only obtains fast learning speed but also shows good generalization

visualizing the function computed by a feedforward neural network a method for visualizing the function computed by a feedforward neural network is presented it is most suitable for models with continuous inputs and a small number of outputs where the output function is reasonably smooth as in regression and probabilistic classification tasks the visualization makes readily apparent the effects of each input and the way in which the functions deviate from a linear function the visualization can also assist in identifying interactions in the fitted model the method uses only the input output relationship and thus can be applied to any predictive statistical model including bagged and committee models which are otherwise difficult to interpret the visualization method is demonstrated on a neural network model of how the risk of lung cancer is affected by smoking and drinking

load updating for nonlinear finite element models a measurement based load updating method for finite element models subjected to static loads was studied using a four noded curved beam element for large displacements rotations and a gradient based variable metric optimizer finite element analysis was used to numerically calculate the geometrically linear and nonlinear responses and their sensitivities under a given load the optimizer was used to recursively update the load so that it minimized the square of the difference between the calculated and prefiltered noise free measured displacement or strain data for the basic studies the extracted load was represented within an element by a linear combination of integrated legendre polynomials the coefficients of which were taken as design variables of the least squares problem through a model order analysis the benefits for solving load updating problems using the relative deformation measurement the polynomials of lower orders the elements of larger sizes because of using the high precision four noded beam element and the denser measured points were studied for linear responses under different types of applied loads it was confirmed that using the reduced number of unknown variables to obtain an overdetermined inverse problem helps get unique and stable extracted loads though this conclusion was verified mainly through illustrative examples for a cantilever beam it was generally applicable to other load updating or inverse problems further examples were given for a three dimensional portal frame load updating extracting highly oscillating loads and a strain based cantilever beam load updating the final examples were load updating for geometrically nonlinear finite element models under self weight snow and or pressure loads

relevance vector machine for survival analysis an accelerated failure time aft model has been widely used for the analysis of censored survival or failure time data however the aft imposes the restrictive log linear relation between the survival time and the explanatory variables in this paper we introduce a relevance vector machine survival rvms model based on weibull aft model that enables the use of kernel framework to automatically learn the possible nonlinear effects of the input explanatory variables on target survival times we take advantage of the bayesian inference technique in order to estimate the model parameters we also introduce two approaches to accelerate the rvms training in the first approach an efficient smooth prior is employed that improves the degree of sparsity in the second approach a fast marginal likelihood maximization procedure is used for obtaining a sparse solution of survival analysis task by sequential addition and deletion of candidate basis functions these two approaches denoted by smooth rvms and fast rvms typically use fewer basis functions than rvms and improve the rvms training time however they cause a slight degradation in the rvms performance we compare the rvms and the two accelerated approaches with the previous sparse kernel survival analysis method on a synthetic data set as well as six real world data sets the proposed kernel survival analysis models have been discovered to be more accurate in prediction although they benefit from extra sparsity the main advantages of our proposed models are 1 extra sparsity that leads to a better generalization and avoids overfitting 2 automatic relevance sample determination based on data that provide more accuracy in particular for highly censored survival data and 3 flexibility to utilize arbitrary number and types of kernel functions e g non mercer kernels and multikernel learning

global convergence of delayed dynamical systems we discuss some delayed dynamical systems investigating their stability and convergence in a critical case to ensure the stability the coefficients of the dynamical system must satisfy some inequalities in most existing literatures the restrictions on the coefficients are strict inequalities the tough question is what will happen in the case critical case the strict inequalities are replaced by nonstrict inequalities i e is replaced by spl les the purpose of the paper is to discuss this critical case and give an affirmative answer in the case that the activation functions are hyperbolic tangent

construction of prediction intervals for carbon residual of crude oil based on deep stochastic configuration networks abstract complex components lead to large fluctuations in the physicochemical properties of crude oil which make accurate prediction more difficult to quantify the potential uncertainty associated with prediction this paper proposes a novel approach to construct prediction intervals pis for the carbon residual content of crude oil based on the lower upper bound estimation lube method and deep stochastic configuration networks dscns according to the principle of stochastic configuration networks the input weights and biases of dscn are randomly assigned with a supervisory mechanism and only the output weights need to be evaluated which can greatly reduce the number of parameters to be optimized then combining the coverage width based criterion and mean accumulated width deviation a new cost function of dscn for constructing pis based on the lube method is proposed to center the mean values of the pis as near as possible to the targets hence the average of lower and upper bounds of the pi is calculated as the deterministic output which can solve the problem that the pis based on the original lube method cannot provide the deterministic prediction moreover a modified backtracking search optimization algorithm improving the population diversity and increasing the search capability while maintaining the convergence speed is presented to obtain the optimal pis finally experiments using real world data are carried out and the results demonstrate that the proposed approach can construct pis with high quality

active learning and basis selection for kernel based linear models a bayesian perspective we develop an active learning algorithm for kernel based linear regression and classification the proposed greedy algorithm employs a minimum entropy criterion derived using a bayesian interpretation of ridge regression we assume access to a matrix bbrn n for which the i j th element is defined by the kernel function k i j bbr with the observed data i bbrd we seek a model m i yi where yi is a real valued response or integer valued label which we do not have access to a priori to achieve this goal a submatrix il ib bbrn m is sought that corresponds to the intersection of n rows and m columns of indexed by the sets il and ib respectively typically m n and n n we have two objectives i determine the m columns of indexed by the set ib that are the most informative for building a linear model m 1 i ib t yi without any knowledge of yi i 1n and ii using active learning sequentially determine which subset of n elements of yi i 1n should be acquired both stopping values ib m and il n are also to be inferred from the data these steps are taken with the goal of minimizing the uncertainty of the model parameters x as measured by the differential entropy of its posterior distribution the parameter vector x bbrm as well as the model bias bbr is then learned from the resulting problem yil il ibx 1 the remaining n n responses labels not included in yil can be inferred by applying x to the remaining n n rows of ib we show experimental results for several regression and classification problems and compare to other active learning methods

a review of bayesian neural networks with an application to near infrared spectroscopy mackay s 1992 bayesian framework for backpropagation is a practical and powerful means to improve the generalization ability of neural networks it is based on a gaussian approximation to the posterior weight distribution the framework is extended reviewed and demonstrated in a pedagogical way the notation is simplified using the ordinary weight decay parameter and a detailed and explicit procedure for adjusting several weight decay parameters is given bayesian backprop is applied in the prediction of fat content in minced meat from near infrared spectra it outperforms early stopping as well as quadratic regression the evidence of a committee of differently trained networks is computed and the corresponding improved generalization is verified the error bars on the predictions of the fat content are computed there are three contributors the random noise the uncertainty in the weights and the deviation among the committee members the bayesian framework is compared to moody s gpe 1992 finally mackay and neal s automatic relevance determination in which the weight decay parameters depend on the input number is applied to the data with improved results

statistical regularization of inverse problems in experimental sciences we often need to solve inverse problems that is we want to obtain information about the internal structure of a physical system from indirect noisy observations often the problem is not whether a solution exists on the contrary there are too many solutions that fit the data to a chosen tolerance level the goal is to use prior information to determine a physically meaningful solution here we present some of the basic questions that arise we describe methods that can be used to find inversion estimates as well as ways to assess their performance

lower upper bound estimation method for construction of neural network based prediction intervals prediction intervals pis have been proposed in the literature to provide more information by quantifying the level of uncertainty associated to the point forecasts traditional methods for construction of neural network nn based pis suffer from restrictive assumptions about data distribution and massive computational loads in this paper we propose a new fast yet reliable method for the construction of pis for nn predictions the proposed lower upper bound estimation lube method constructs an nn with two outputs for estimating the prediction interval bounds nn training is achieved through the minimization of a proposed pi based objective function which covers both interval width and coverage probability the method does not require any information about the upper and lower bounds of pis for training the nn the simulated annealing method is applied for minimization of the cost function and adjustment of nn parameters the demonstrated results for 10 benchmark regression case studies clearly show the lube method to be capable of generating high quality pis in a short time also the quantitative comparison with three traditional techniques for prediction interval construction reveals that the lube method is simpler faster and more reliable

multi task learning for classification with dirichlet process priors consider the problem of learning logistic regression models for multiple classification tasks where the training data set for each task is not drawn from the same statistical distribution in such a multi task learning mtl scenario it is necessary to identify groups of similar tasks that should be learned jointly relying on a dirichlet process dp based statistical model to learn the extent of similarity between classification tasks we develop computationally efficient algorithms for two different forms of the mtl problem first we consider a symmetric multi task learning smtl situation in which classifiers for multiple tasks are learned jointly using a variational bayesian vb algorithm second we consider an asymmetric multi task learning amtl formulation in which the posterior density function from the smtl model parameters from previous tasks is used as a prior for a new task this approach has the significant advantage of not requiring storage and use of all previous data from prior tasks the amtl formulation is solved with a simple markov chain monte carlo mcmc construction experimental results on two real life mtl problems indicate that the proposed algorithms a automatically identify subgroups of related tasks whose training data appear to be drawn from similar distributions and b are more accurate than simpler approaches such as single task learning pooling of data across all tasks and simplified approximations to dp

uncertainty representation for mobile robots perception modeling and navigation in unknown environments we present a fuzzy sets based approach to the problem of mobile robot navigation in unknown environments fuzzy sets are used to represent the uncertainty that is inherent to the perception of the environment through the robot sensors this uncertainty is then propagated in the process of map building so that not only a plausible spatial layout of the environment but also the confidence on this layout is obtained the initial map built by the robot is then used for self localization as it continues navigating in the same environment the new information collected by the sensors is matched to the initial map and the transformation that brings them together is used to correct and bound the dead reckoning errors uncertainty representation is a key aspect of this process since it allows to immediately detect pairs of coincident boundaries thus permitting real time self localization the approach is illustrated by experiments in office environments

a product form cholesky factorization method for handling dense columns in interior point methods for linear programming cholesky factorization has become the method of choice for solving the symmetric system of linear equations arising in interior point methods ipms for linear programming lp its main advantages being numerical stability and efficiency for sparse systems however in the presence of dense columns there is dramatic loss of efficiency a typical approach to remedy this problem is to apply the sherman morrison woodbury smw update formula to the dense part this approach while being very efficient is not numerically stable here we propose using product form cholesky factorization to handle dense columns the proposed approach is essentially as stable as the original cholesky factorization and nearly as efficient as the smw approach we demonstrate these properties both theoretically and computationally a key part of our theoretical analysis is the proof that the elements of the cholesky factors of the matrices that arise in ipms for lp are uniformly bounded as the duality gap converges to zero

preventing over fitting during model selection via bayesian regularisation of the hyper parameters while the model parameters of a kernel machine are typically given by the solution of a convex optimisation problem with a single global optimum the selection of good values for the regularisation and kernel parameters is much less straightforward fortunately the leave one out cross validation procedure can be performed or a least approximated very efficiently in closed form for a wide variety of kernel learning methods providing a convenient means for model selection leave one out cross validation based estimates of performance however generally exhibit a relatively high variance and are therefore prone to over fitting in this paper we investigate the novel use of bayesian regularisation at the second level of inference adding a regularisation term to the model selection criterion corresponding to a prior over the hyper parameter values where the additional regularisation parameters are integrated out analytically results obtained on a suite of thirteen real world and synthetic benchmark data sets clearly demonstrate the benefit of this approach

automatic model selection for the optimization of svm kernels this approach aims to optimize the kernel parameters and to efficiently reduce the number of support vectors so that the generalization error can be reduced drastically the proposed methodology suggests the use of a new model selection criterion based on the estimation of the probability of error of the svm classifier for comparison we considered two more model selection criteria gacv generalized approximate cross validation and vc vapnik chernovenkis dimension these criteria are algebraic estimates of upper bounds of the expected error for the former we also propose a new minimization scheme the experiments conducted on a bi class problem show that we can adequately choose the svm hyper parameters using the empirical error criterion moreover it turns out that the criterion produces a less complex model with fewer support vectors for multi class data the optimization strategy is adapted to the one against one data partitioning the approach is then evaluated on images of handwritten digits from the usps database

validation based sparse gaussian process classifier design gaussian processes gps are promising bayesian methods for classification and regression problems design of a gp classifier and making predictions using it is however computationally demanding especially when the training set size is large sparse gp classifiers are known to overcome this limitation in this letter we propose and study a validation based method for sparse gp classifier design the proposed method uses a negative log predictive nlp loss measure which is easy to compute for gp models we use this measure for both basis vector selection and hyperparameter adaptation the experimental results on several real world benchmark data sets show better or comparable generalization performance over existing methods

sparse spectral clustering method based on the incomplete cholesky decomposition a novel sparse spectral clustering method using linear algebra techniques is proposed spectral clustering methods solve an eigenvalue problem containing a graph laplacian the proposed method exploits the structure of the laplacian to construct an approximation not in terms of a low rank approximation but in terms of capturing the structure of the matrix with this approximation the size of the eigenvalue problem can be reduced to obtain the indicator vectors from the eigenvectors the method proposed by zha et al 2002 26 which computes a pivoted lq factorization of the eigenvector matrix is adapted this formulation also gives the possibility to extend the method to out of sample points

representation and fusion of heterogeneous fuzzy information in the 3d space for model based structural recognition application to 3d brain imaging we present a novel approach to model based pattern recognition where structural information and spatial relationships have a most important role it is illustrated in the domain of 3d brain structure recognition using an anatomical atlas our approach performs segmentation and recognition of the scene simultaneously the solution of the recognition task is progressive processing successively different objects and using different pieces of knowledge about the object and about relationships between objects therefore the core of the approach is the knowledge representation part and constitutes the main contribution of this paper we make use of a spatial representation of each piece of information as a spatial fuzzy set representing a constraint to be satisfied by the searched object thanks in particular to fuzzy mathematical morphology operations fusion of these constraints allows us to select segment and recognize the desired object

the evidence framework applied to classification networks three bayesian ideas are presented for supervised adaptive classifiers first it is argued that the output of a classifier should be obtained by marginalizing over the posterior distribution of the parameters a simple approximation to this integral is proposed and demonstrated this involves a moderation of the most probable classifier s outputs and yields improved performance second it is demonstrated that the bayesian framework for model comparison described for regression models in mackay 1992a b can also be applied to classification problems this framework successfully chooses the magnitude of weight decay terms and ranks solutions found using different numbers of hidden units third an information based data selection criterion is derived and demonstrated within this framework

generalized core vector machines kernel methods such as the support vector machine svm are often formulated as quadratic programming qp problems however given m training patterns a naive implementation of the qp solver takes o m 3 training time and at least o m2 space hence scaling up these qps is a major stumbling block in applying kernel methods on very large data sets and a replacement of the naive method for finding the qp solutions is highly desirable recently by using approximation algorithms for the minimum enclosing ball meb problem we proposed the core vector machine cvm algorithm that is much faster and can handle much larger data sets than existing svm implementations however the cvm can only be used with certain kernel functions and kernel methods for example the very popular support vector regression svr cannot be used with the cvm in this paper we introduce the center constrained meb problem and subsequently extend the cvm algorithm the generalized cvm algorithm can now be used with any linear nonlinear kernel and can also be applied to kernel methods such as svr and the ranking svm moreover like the original cvm its asymptotic time complexity is again linear in m and its space complexity is independent of m experiments show that the generalized cvm has comparable performance with state of the art svm and svr implementations but is faster and produces fewer support vectors on very large data sets

efficient svm training using low rank kernel representations svm training is a convex optimization problem which scales with the training set size rather than the feature space dimension while this is usually considered to be a desired quality in large scale problems it may cause training to be impractical the common techniques to handle this difficulty basically build a solution by solving a sequence of small scale subproblems our current effort is concentrated on the rank of the kernel matrix as a source for further enhancement of the training procedure we first show that for a low rank kernel matrix it is possible to design a better interior point method ipm in terms of storage requirements as well as computational complexity we then suggest an efficient use of a known factorization technique to approximate a given kernel matrix by a low rank matrix which in turn will be used to feed the optimizer finally we derive an upper bound on the change in the objective function value based on the approximation error and the number of active constraints support vectors this bound is general in the sense that it holds regardless of the approximation method

joint nonlinear channel equalization and soft ldpc decoding with gaussian processes in this paper we introduce a new approach for nonlinear equalization based on gaussian processes for classification gpc we propose to measure the performance of this equalizer after a low density parity check channel decoder has detected the received sequence typically most channel equalizers concentrate on reducing the bit error rate instead of providing accurate posterior probability estimates we show that the accuracy of these estimates is essential for optimal performance of the channel decoder and that the error rate output by the equalizer might be irrelevant to understand the performance of the overall communication receiver in this sense gpc is a bayesian nonlinear classification tool that provides accurate posterior probability estimates with short training sequences in the experimental section we compare the proposed gpc based equalizer with state of the art solutions to illustrate its improved performance

local results for the gauss newton method on constrained rank deficient nonlinear least squares a nonlinear least squares problem with nonlinear constraints may be ill posed or even rank deficient in two ways considering the problem formulated as min x 1 2vert f 2 x vert 2 2 subject to the constraints f 1 x 0 the jacobian j 1 partial f 1 partial x and or the jacobian j partial f partial x f f 1 f 2 may be ill conditioned at the solution we analyze the important special case when j 1 and or j do not have full rank at the solution in order to solve such a problem we formulate a nonlinear least norm problem next we describe a truncated gauss newton method we show that the local convergence rate is determined by the maximum of three independent rayleigh quotients related to three different spaces in mathbb r n another way of solving an ill posed nonlinear least squares problem is to regularize the problem with some parameter that is reduced as the iterates converge to the minimum our approach is a tikhonov based local linear regularization that converges to a minimum norm problem this approach may be used both for almost and rank deficient jacobians finally we present computational tests on constructed problems verifying the local analysis

gaussian kernel width optimization for sparse bayesian learning sparse kernel methods have been widely used in regression and classification applications the performance and the sparsity of these methods are dependent on the appropriate choice of the corresponding kernel functions and their parameters typically the kernel parameters are selected using a cross validation approach in this paper a learning method that is an extension of the relevance vector machine rvm is presented the proposed method can find the optimal values of the kernel parameters during the training procedure this algorithm uses an expectation maximization approach for updating kernel parameters as well as other model parameters therefore the speed of convergence and computational complexity of the proposed method are the same as the standard rvm to control the convergence of this fully parameterized model the optimization with respect to the kernel parameters is performed using a constraint on these parameters the proposed method is compared with the typical rvm and other competing methods to analyze the performance the experimental results on the commonly used synthetic data as well as benchmark data sets demonstrate the effectiveness of the proposed method in reducing the performance dependency on the initial choice of the kernel parameters

efficient probabilistic classification vector machine with incremental basis function selection probabilistic classification vector machine pcvm is a sparse learning approach aiming to address the stability problems of relevance vector machine for classification problems because pcvm is based on the expectation maximization algorithm it suffers from sensitivity to initialization convergence to local minima and the limitation of bayesian estimation making only point estimates another disadvantage is that pcvm was not efficient for large data sets to address these problems this paper proposes an efficient pcvm epcvm by sequentially adding or deleting basis functions according to the marginal likelihood maximization for efficient training because of the truncated prior used in epcvm two approximation techniques i e laplace approximation and expectation propagation ep have been used to implement epcvm to obtain full bayesian solutions we have verified laplace approximation and ep with a hybrid monte carlo approach the generalization performance and computational effectiveness of epcvm are extensively evaluated theoretical discussions using rademacher complexity reveal the relationship between the sparsity and the generalization bound of epcvm

decomposition methods for linear support vector machines in this letter we show that decomposition methods with alpha seeding are extremely useful for solving a sequence of linear support vector machines svms with more data than attributes this strategy is motivated by keerthi and lin 2003 who proved that for an svm with data not linearly separable after c is large enough the dual solutions have the same free and bounded components we explain why a direct use of decomposition methods for linear svms is sometimes very slow and then analyze why alpha seeding is much more effective for linear than nonlinear svms we also conduct comparisons with other methods that are efficient for linear svms and demonstrate the effectiveness of alpha seeding techniques in model selection

using imprecise environment information for guiding scene interpretation abstract we are concerned with translating knowledge about a real world environment into useful information regarding the content of images taken from this environment in particular knowledge about location and or size of objects expressed at different levels of imprecision is translated into information on region allocation for these objects in an image when the position and camera angle are known imprecisely as well the results objects which may have projection areas of the image where they will be projected and information on their projection can be used to guide an object recognition and scene interpretation system our approach is illustrated by real world real image examples

variational gaussian process classifiers gaussian processes are a promising nonlinear regression tool but it is not straightforward to solve classification problems with them in the paper the variational methods of jaakkola and jordan 2000 are applied to gaussian processes to produce an efficient bayesian binary classifier

on classification with incomplete data we address the incomplete data problem in which feature vectors to be classified are missing data features a supervised logistic regression algorithm for the classification of incomplete data is developed single or multiple imputation for the missing data is avoided by performing analytic integration with an estimated conditional density function conditioned on the observed data conditional density functions are estimated using a gaussian mixture model gmm with parameter estimation performed using both expectation maximization em and variational bayesian em vb em the proposed supervised algorithm is then extended to the semisupervised case by incorporating graph based regularization the semisupervised algorithm utilizes all available data both incomplete and complete as well as labeled and unlabeled experimental results of the proposed classification algorithms are shown

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

combining feature spaces for classification in this paper we offer a variational bayes approximation to the multinomial probit model for basis expansion and kernel combination our model is well founded within a hierarchical bayesian framework and is able to instructively combine available sources of information for multinomial classification the proposed framework enables informative integration of possibly heterogeneous sources in a multitude of ways from the simple summation of feature expansions to weighted product of kernels and it is shown to match and in certain cases outperform the well known ensemble learning approaches of combining individual classifiers at the same time the approximation reduces considerably the cpu time and resources required with respect to both the ensemble learning methods and the full markov chain monte carlo metropolis hastings within gibbs solution of our model we present our proposed framework together with extensive experimental studies on synthetic and benchmark datasets and also for the first time report a comparison between summation and product of individual kernels as possible different methods for constructing the composite kernel matrix

moving force identification optimal state estimation approach existing techniques to identify moving forces exhibit the common weakness of having large fluctuations in the identified results a new method of moving force identification is developed in this paper making use of the dynamic programming technique to overcome this weakness the forces in the state space formulation of the dynamic system are identified in the time domain using a recursive formula based on several distributed sensor measurements and responses of the structure are reconstructed using the identified forces for comparison like all inverse problems the computation is ill conditioned however the dynamic programming technique inherently provides bounds to the ill conditioned forces and results from the simulation studies and laboratory work show great improvements over existing methods in the accuracy of identification