large scale multiple kernel learning while classical kernel based learning algorithms are based on a single kernel in practice it is often desirable to use multiple kernels lanckriet et al 2004 considered conic combinations of kernel matrices for classification leading to a convex quadratically constrained quadratic program we show that it can be rewritten as a semi infinite linear program that can be efficiently solved by recycling the standard svm implementations moreover we generalize the formulation and our method to a larger class of problems including regression and one class classification experimental results show that the proposed algorithm works for hundred thousands of examples or hundreds of kernels to be combined and helps for automatic model selection improving the interpretability of the learning result in a second part we discuss general speed up mechanism for svms especially when used with sparse feature maps as appear for string kernels allowing us to train a string kernel svm on a 10 million real world splice data set from computational biology we integrated multiple kernel learning in our machine learning toolbox shogun for which the source code is publicly available at http www fml tuebingen mpg de raetsch projects shogun

a generalized kernel approach to dissimilarity based classification usually objects to be classified are represented by features in this paper we discuss an alternative object representation based on dissimilarity values if such distances separate the classes well the nearest neighbor method offers a good solution however dissimilarities used in practice are usually far from ideal and the performance of the nearest neighbor rule suffers from its sensitivity to noisy examples we show that other more global classification techniques are preferable to the nearest neighbor rule in such cases for classification purposes two different ways of using generalized dissimilarity kernels are considered in the first one distances are isometrically embedded in a pseudo euclidean space and the classification task is performed there in the second approach classifiers are built directly on distance kernels both approaches are described theoretically and then compared using experiments with different dissimilarity measures and datasets including degraded data simulating the problem of missing values

learning the kernel with hyperkernels this paper addresses the problem of choosing a kernel suitable for estimation with a support vector machine hence further automating machine learning this goal is achieved by defining a reproducing kernel hilbert space on the space of kernels itself such a formulation leads to a statistical estimation problem similar to the problem of minimizing a regularized risk functional we state the equivalent representer theorem for the choice of kernels and present a semidefinite programming formulation of the resulting optimization problem several recipes for constructing hyperkernels are provided as well as the details of common machine learning problems experimental results for classification regression and novelty detection on uci data show the feasibility of our approach

the method of simulated scores for the estimation of ldv models the method of simulated scores mss is presented for estimating limited dependent variables models ldv with flexible correlation structure in the unobservables we propose simulators that are continuous in the unknown parameter vectors and hence standard optimization methods can be used to compute the mss estimators that employ these simulators the first continuous method relies on a recursive conditioning of the multivariate normal density through a cholesky triangularization of its variance covariance matrix the second method combines results about the conditionals of the multivariate normal distribution with gibbs resampling techniques we establish consistency and asymptotic normality of the mss estimators and derive suitable rates at which the number of simulations must rise if biased simulators are used

the evaluation of general non centred orthant probabilities summary the evaluation of the cumulative distribution function of a multivariate normal distribution is considered the multivariate normal distribution can have any positive definite correlation matrix and any mean vector the approach taken has two stages in the first stage it is shown how non centred orthoscheme probabilities can be evaluated by using a recursive integration method in the second stage some ideas of schlafli and abrahamson are extended to show that any non centred orthant probability can be expressed as differences between at most m 1 non centred orthoscheme probabilities this approach allows an accurate evaluation of many multivariate normal probabilities which have important applications in statistical practice

multik mhks a novel multiple kernel learning algorithm in this paper we develop a new effective multiple kernel learning algorithm first we map the input data into m different feature spaces by m empirical kernels where each generated feature space is taken as one view of the input space then through borrowing the motivating argument from canonical correlation analysis cca that can maximally correlate the m views in the transformed coordinates we introduce a special term called inter function similarity loss rifsi into the existing regularization framework so as to guarantee the agreement of multiview outputs in implementation we select the modification of ho kashyap algorithm with squared approximation of the misclassification errors mhks as the incorporated paradigm and the experimental results on benchmark data sets demonstrate the feasibility and effectiveness of the proposed algorithm named multik mhks

sequential quasi monte carlo this is a written discussion of the paper sequential quasi monte carlo sampling by m gerber and n chopin following the presentation given before the royal statistical society in london on december 10th 2014

conjugate bayes for probit regression via unified skew normal distributions regression models for dichotomous data are ubiquitous in statistics besides being useful for inference on binary responses these methods serve also as building blocks in more complex formulations such as density regression nonparametric classification and graphical models within the bayesian framework inference proceeds by updating the priors for the coefficients typically set to be gaussians with the likelihood induced by probit or logit regressions for the responses in this updating the apparent absence of a tractable posterior has motivated a variety of computational methods including markov chain monte carlo routines and algorithms which approximate the posterior despite being routinely implemented markov chain monte carlo strategies face mixing or time inefficiency issues in large p and small n studies whereas approximate routines fail to capture the skewness typically observed in the posterior this article proves that the posterior distribution for the probit coefficients has a unified skew normal kernel under gaussian priors such a novel result allows efficient bayesian inference for a wide class of applications especially in large p and small to moderate n studies where state of the art computational methods face notable issues these advances are outlined in a genetic study and further motivate the development of a wider class of conjugate priors for probit models along with methods to obtain independent and identically distributed samples from the unified skew normal posterior

optimizing the kernel in the empirical feature space in this paper we present a method of kernel optimization by maximizing a measure of class separability in the empirical feature space an euclidean space in which the training data are embedded in such a way that the geometrical structure of the data in the feature space is preserved employing a data dependent kernel we derive an effective kernel optimization algorithm that maximizes the class separability of the data in the empirical feature space it is shown that there exists a close relationship between the class separability measure introduced here and the alignment measure defined recently by cristianini extensive simulations are carried out which show that the optimized kernel is more adaptive to the input data and leads to a substantial sometimes significant improvement in the performance of various data classification algorithms

the normal law under linear restrictions simulation and estimation via minimax tilting summary simulation from the truncated multivariate normal distribution in high dimensions is a recurrent problem in statistical computing and is typically only feasible by using approximate markov chain monte carlo sampling we propose a minimax tilting method for exact independently and identically distributed data simulation from the truncated multivariate normal distribution the new methodology provides both a method for simulation and an efficient estimator to hitherto intractable gaussian integrals we prove that the estimator has a rare vanishing relative error asymptotic property numerical experiments suggest that the scheme proposed is accurate in a wide range of set ups for which competing estimation schemes fail we give an application to exact independently and identically distributed data simulation from the bayesian posterior of the probit regression model

mrm lasso a sparse multiview feature selection method via low rank analysis learning about multiview data involves many applications such as video understanding image classification and social media however when the data dimension increases dramatically it is important but very challenging to remove redundant features in multiview feature selection in this paper we propose a novel feature selection algorithm multiview rank minimization based lasso mrm lasso which jointly utilizes lasso for sparse feature selection and rank minimization for learning relevant patterns across views instead of simply integrating multiple lasso from view level we focus on the performance of sample level sample significance and introduce pattern specific weights into mrm lasso the weights are utilized to measure the contribution of each sample to the labels in the current view in addition the latent correlation across different views is successfully captured by learning a low rank matrix consisting of pattern specific weights the alternating direction method of multipliers is applied to optimize the proposed mrm lasso experiments on four real life data sets show that features selected by mrm lasso have better multiview classification performance than the baselines moreover pattern specific weights are demonstrated to be significant for learning about multiview data compared with view specific weights

alternative sampling methods for estimating multivariate normal probabilities we study the performance of alternative sampling methods for estimating multivariate normal probabilities through the ghk simulator the sampling methods are randomized versions of some quasi monte carlo samples halton niederreiter niederreiter xing sequences and lattice points and some samples based on orthogonal arrays latin hypercube orthogonal array and orthogonal array based latin hypercube samples in general these samples turn out to have a better performance than monte carlo and antithetic monte carlo samples improvements over these are large for low dimensional 4 and 10 cases and still significant for dimensions as large as 50

bayesian analysis of binary and polychotomous response data abstract a vast literature in statistics biometrics and econometrics is concerned with the analysis of binary and polychotomous response data the classical approach fits a categorical response regression model using maximum likelihood and inferences about the model are based on the associated asymptotic theory the accuracy of classical confidence statements is questionable for small sample sizes in this article exact bayesian methods for modeling categorical response data are developed using the idea of data augmentation the general approach can be summarized as follows the probit regression model for binary outcomes is seen to have an underlying normal regression structure on latent continuous data values of the latent data can be simulated from suitable truncated normal distributions if the latent data are known then the posterior distribution of the parameters can be computed using standard results for normal linear models draws from this posterior are used to sample new latent data and t

composite likelihood estimation for the brown resnick process summary genton et al 2011 investigated the gain in efficiency when triplewise rather than pairwise likelihood is used to fit the popular smith max stable model for spatial extremes we generalize their results to the brown resnick model and show that the efficiency gain is substantial only for very smooth processes which are generally unrealistic in applications

on different facets of regularization theory this review provides a comprehensive understanding of regularization theory from different perspectives emphasizing smoothness and simplicity principles using the tools of operator theory and fourier analysis it is shown that the solution of the classical tikhonov regularization problem can be derived from the regularized functional defined by a linear differential integral operator in the spatial fourier domain state of the art research relevant to the regularization theory is reviewed covering occam s razor minimum length description bayesian theory pruning algorithms informational entropy theory statistical learning theory and equivalent regularization the universal principle of regularization in terms of kolmogorov complexity is discussed finally some prospective studies on regularization theory and beyond are suggested

the isometry group of lp μ x is sot contractible abstract we will show that if is an atomless positive measure space x is a banach space and 1 p then the group of isometric automorphisms on the bochner space l p x is contractible in the strong operator topology we do not require or x above to be separable

structural multiple empirical kernel learning multiple kernel learning mkl can boost classification performance through using multiple kernels rather than a single fixed one unlike the traditional mkl with the implicit kernels multiple empirical kernel learning mekl explicitly maps input data into multiple feature spaces this paper focuses on mekl and proposes an effective threefold structural mekl tsmekl the first fold structure is the space structural information between different mapped feature spaces the second one is the class discriminant information within each mapped feature space the third one is the cluster structural information of samples in each mapped feature space the classical mekl mainly pays attention to the first two structures but neglects the last one the proposed tsmekl introduces the cluster structural information into mekl doing so can simultaneously utilize the space the class and the cluster information in the way from globality to locality therefore tsmekl utilizes threefold structural information to result in the improvement of classification performance to the best of our knowledge it is the first time to introduce the cluster information into the mekl framework the main advantage of the developed tsmekl is considering different folds of data information to improve classification performance the experimental results validate the feasibility and effectiveness of tsmekl moreover we discuss the theoretical and experimental generalization risk bound of the proposed algorithm in terms of the rademacher complexity

approximations to multivariate normal rectangle probabilities based on conditional expectations abstract two new approximations for multivariate normal probabilities for rectangular regions based on conditional expectations and regression with binary variables are proposed one is a second order approximation that is much more accurate but also more numerically time consuming than the first order approximation a third approximation based on the moment generating function of a truncated multivariate normal distribution is proposed for orthant probabilities only its accuracy is between the first and second order approximations when the dimension is less than seven and the correlations are not large all of the approximations get worse as correlations get larger these new approximations offer substantial improvements on previous approximations they also compare favorably with the methods of genz for numerical evaluation of the multivariate normal integral the approximation methods should be especially useful within a quasi newton routine for parameter estimation in discrete models that involv

mreklm a fast multiple empirical kernel learning machine abstract multiple empirical kernel learning mekl explicitly maps samples into different empirical feature spaces in which the kernel features of the mapped samples can be directly provided thus mekl is much easier than the conventional multiple kernel learning mkl in terms of processing and analyzing the structure of mapped feature spaces however the computational complexity of mekl with m empirical feature spaces is o mn 3 where n is the number of training samples the dimensions of the generated empirical feature spaces are approximate to n when dealing with large scale problems mekl cannot handle them properly due to the severe computation and memory burden moreover most existing mekl utilizes the gradient decent optimization to learn classifiers but it is time consuming for training therefore this paper proposes a multiple random empirical kernel learning machine mreklm to overcome these problems the proposed mreklm adopts the random projection idea to map samples into multiple low dimensional empirical feature spaces with lower computational complexity o mp 3 where p n is the number of the randomly selected samples after that mreklm adopts an analytical optimization approach to directly deal with multi class problems the computational complexity of mreklm is o m 3 p 3 experimental results also validate both efficiency and effectiveness of the proposed mreklm the contributions of this work are 1 proposing a fast mekl algorithm named mreklm 2 introducing an efficient random empirical kernel mapping approach and 3 extending the capability of mekl to handle large scale problems

model calibration with censored data abstractthe purpose of model calibration is to make the model predictions closer to reality the classical kennedy o hagan approach is widely used for model calibration which can account for the inadequacy of the computer model while simultaneously estimating the unknown calibration parameters in many applications the phenomenon of censoring occurs when the exact outcome of the physical experiment is not observed but is only known to fall within a certain region in such cases the kennedy o hagan approach cannot be used directly and we propose a method to incorporate the censoring information when performing model calibration the method is applied to study the compression phenomenon of liquid inside a bottle the results show significant improvement over the traditional calibration methods especially when the number of censored observations is large

an introduction to kernel based learning algorithms this paper provides an introduction to support vector machines kernel fisher discriminant analysis and kernel principal component analysis as examples for successful kernel based learning methods we first give a short background about vapnik chervonenkis theory and kernel feature spaces and then proceed to kernel based learning in supervised and unsupervised scenarios including practical and algorithmic considerations we illustrate the usefulness of kernel algorithms by discussing applications such as optical character recognition and dna analysis

convex transitivity and function spaces abstract it is shown that if x is a convex transitive banach space and 1 p then l p 0 1 x and l s 0 1 x are convex transitive here l s 0 1 x is the closed linear span of the simple functions in the bochner space l 0 1 x if h is an infinite dimensional hilbert space and c 0 l is convex transitive then c 0 l h is convex transitive some new fairly concrete examples of convex transitive spaces are provided

graph regularized multiset canonical correlations with applications to joint feature extraction abstract multiset canonical correlation analysis mcca is a powerful technique for analyzing linear correlations among multiple representation data however it usually fails to discover the intrinsic geometrical and discriminating structure of multiple data spaces in real world applications in this paper we thus propose a novel algorithm called graph regularized multiset canonical correlations grmccs which explicitly considers both discriminative and intrinsic geometrical structure in multiple representation data grmcc not only maximizes between set cumulative correlations but also minimizes local intraclass scatter and simultaneously maximizes local interclass separability by using the nearest neighbor graphs on within set data thus it can leverage the power of both mcca and discriminative graph laplacian regularization extensive experimental results on the ar cmu pie yale b at t and eth 80 datasets show that grmcc has more discriminating power and can provide encouraging recognition results in contrast with the state of the art algorithms

efficient inference for spatial extreme value processes associated to log gaussian random functions max stable processes arise as the only possible nontrivial limits for maxima of affinely normalized identically distributed stochastic processes and thus form an important class of models for the extreme values of spatial processes until recently inference for max stable processes has been restricted to the use of pairwise composite likelihoods due to intractability of higher dimensional distributions in this work we consider random fields that are in the domain of attraction of a widely used class of max stable processes namely those constructed via manipulation of log gaussian random functions for this class we exploit limiting d dimensional multivariate poisson process intensities of the underlying process for inference on all d vectors exceeding a high marginal threshold in at least one component employing a censoring scheme to incorporate information below the marginal threshold we also consider the d dimensional distributions for the equivalent max stable process and perform full likelihood inference by exploiting the methods of stephenson tawn 2005 where information on the occurrence times of extreme events is shown to dramatically simplify the likelihood the stephenson tawn likelihood is in fact simply a special case of the censored poisson process likelihood we assess the improvements in inference from both methods over pairwise likelihood methodology by simulation

monte carlo evaluation of multivariate normal probabilities abstract this paper extends research on the simulation of multivariate normal probabilities of high order dimension by developing a new family of simulators such simulators are useful for models with limited dependent variables including multinomial probit in panel studies spatial analysis and time series analysis the simulators are derived from a cholesky decomposition of the covariance matrix combined with a suitable choice of an importance sampling distribution the paper studies among others the impact of antithetical sampling the insights gained in this paper are of use in bayesian analysis as well in the evaluation of posterior densities

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

input space versus feature space in kernel based methods this paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector sv kernel functions we first discuss the geometry of feature space in particular we review what is known about the shape of the image of input space under the feature space map and how this influences the capacity of sv methods following this we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel using the example of the class of inhomogeneous polynomial kernels which are often used in sv pattern recognition we then discuss the connection between feature space and input space by dealing with the question of how one can given some vector in feature space find a preimage exact or approximate in input space we describe algorithms to tackle this issue and show their utility in two applications of kernel methods first we use it to reduce the computational complexity of sv decision functions second we combine it with the kernel pca algorithm thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real world data

excursion and contour uncertainty regions for latent gaussian models in several areas of application ranging from brain imaging to astrophysics and geostatistics an important statistical problem is to find regions where the process studied exceeds a certain level estimating such regions so that the probability for exceeding the level in the entire set is equal to some predefined value is a difficult problem connected to the problem of multiple significance testing in this work a method for solving this problem as well as the related problem of finding credible regions for contour curves for latent gaussian models is proposed the method is based on using a parametric family for the excursion sets in combination with a sequential importance sampling method for estimating joint probabilities the accuracy of the method is investigated by using simulated data and an environmental application is presented

on the likelihood function of gaussian max stable processes summary wederiveaclosedformexpressionforthelikelihoodfunctionofagaussianmax stableprocessindexed by r d at p d 1 sites d 1 we demonstrate the gain in efficiency in the maximum composite likelihood estimators of the covariance matrix from p 2t op 3 sites in r 2 by means of a monte carlo