estimation of gradients and coordinate covariation in classification we introduce an algorithm that simultaneously estimates a classification function as well as its gradient in the supervised learning framework the motivation for the algorithm is to find salient variables and estimate how they covary an efficient implementation with respect to both memory and time is given the utility of the algorithm is illustrated on simulated data as well as a gene expression data set an error analysis is given for the convergence of the estimate of the classification function and its gradient to the true classification function and true gradient

component selection and smoothing in multivariate nonparametric regression we propose a new method for model selection and model fitting in multivariate nonparametric regression models in the framework of smoothing spline anova the cosso is a method of regularization with the penalty functional being the sum of component norms instead of the squared norm employed in the traditional smoothing spline method the cosso provides a unified framework for several recent proposals for model selection in linear models and smoothing spline anova models theoretical properties such as the existence and the rate of convergence of the cosso estimator are studied in the special case of a tensor product design with periodic functions a detailed analysis reveals that the cosso does model selection by applying a novel soft thresholding type operation to the function components we give an equivalent formulation of the cosso estimator which leads naturally to an iterative algorithm we compare the cosso with mars a popular method that builds functional anova models in simulations and real examples the cosso method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets the cosso gives very competitive performance in these studies

on convergence conditions of an extended projection neural network the output trajectory convergence of an extended projection neural network was developed under the positive definiteness condition of the jacobian matrix of nonlinear mapping this note offers several new convergence results the state trajectory convergence and the output trajectory convergence of the extended projection neural network are obtained under the positive semidefiniteness condition of the jacobian matrix comparison and illustrative examples demonstrate applied significance of these new results

sparse additive models summary we present a new class of methods for high dimensional non parametric regression and classification called sparse additive models our methods combine ideas from sparse linear modelling and additive non parametric regression we derive an algorithm for fitting the models that is practical and effective even when the number of covariates is larger than the sample size sparse additive models are essentially a functional version of the grouped lasso of yuan and lin they are also closely related to the cosso model of lin and zhang but decouple smoothing and sparsity enabling the use of arbitrary non parametric smoothers we give an analysis of the theoretical properties of sparse additive models and present empirical results on synthetic and real data showing that they can be effective in fitting sparse non parametric models in high dimensional data

stabilising model predictive control for discrete time fractional order systems in this paper a model predictive control scheme is proposed for constrained fractional order discrete time systems we prove that constraints are satisfied and we prescribe conditions for the origin to be an asymptotically stable equilibrium point of the controlled system a finite dimensional approximation of the original infinite dimensional dynamics is employed for which the approximation error can become arbitrarily small the approximate dynamics is used to design a tube based model predictive controller which steers the system state to a neighbourhood of the origin of controlled size stability conditions are finally derived for the mpc controlled system which are computationally tractable and account for the infinite dimensional nature of the fractional order system and the state and input constraints the proposed control methodology guarantees asymptotic stability of the discrete time fractional order system satisfaction of the prescribed constraints and recursive feasibility

learning coordinate covariances via gradients we introduce an algorithm that learns gradients from samples in the supervised learning framework an error analysis is given for the convergence of the gradient estimated by the algorithm to the true gradient the utility of the algorithm for the problem of variable selection as well as determining variable covariance is illustrated on simulated data as well as two gene expression data sets for square loss we provide a very efficient implementation with respect to both memory and time

an adaptive estimation of dimension reduction space summary searching for an effective dimension reduction space is an important problem in regression especially for high dimensional data we propose an adaptive approach based on semiparametric models which we call the conditional minimum average variance estimation mave method within quite a general setting the mave method has the following advantages most existing methods must undersmooth the nonparametric link function estimator to achieve a faster rate of consistency for the estimator of the parameters than for that of the nonparametric function in contrast a faster consistency rate can be achieved by the mave method even without undersmoothing the nonparametric link function estimator the mave method is applicable to a wide range of models with fewer restrictions on the distribution of the covariates to the extent that even time series can be included because of the faster rate of consistency for the parameter estimators it is possible for us to estimate the dimension of the space consistently the relationship of the mave method with other methods is also investigated in particular a simple outer product gradient estimator is proposed as an initial estimator in addition to theoretical results we demonstrate the efficacy of the mave method for high dimensional data sets through simulation two real data sets are analysed by using the mave approach

a constrained em algorithm for independent component analysis we introduce a novel way of performing independent component analysis using a constrained version of the expectation maximization em algorithm the source distributions are modeled as d one dimensional mixtures of gaussians the observed data are modeled as linear mixtures of the sources with additive isotropic noise this generative model is fit to the data using constrained em the simpler soft switching approach is introduced which uses only one parameter to decide on the sub or supergaussian nature of the sources we explain how our approach relates to independent factor analysis

better subset regression using the nonnegative garrote a new method called the nonnegative nn garrote is proposed for doing subset regression it both shrinks and zeroes coefficients in tests on real and simulated data it produces lower prediction error than ordinary subset selection it is also compared to ridge regression if the regression equations generated by a procedure do not change drastically with small changes in the data the procedure is called stable subset selection is unstable ridge is very stable and the nn garrote is intermediate simulation results illustrate the effects of instability on prediction error

a constrained em algorithm for principal component analysis we propose a constrained em algorithm for principal component analysis pca using a coupled probability model derived from single standard factor analysis models with isotropic noise structure the single probabilistic pca especially for the case where there is no noise can find only a vector set that is a linear superposition of principal components and requires postprocessing such as diagonalization of symmetric matrices by contrast the proposed algorithm finds the actual principal components which are sorted in descending order of eigenvalue size and require no additional calculation or postprocessing the method is easily applied to kernel pca it is also shown that the new em algorithm is derived from a generalized least squares formulation

u processes and preference learning preference learning has caused great attention in machining learning in this letter we propose a learning framework for pairwise loss based on empirical risk minimization of u processes via rademacher complexity we first establish a uniform version of bernstein inequality of u processes of degree 2 via the entropy methods then we estimate the bound of the excess risk by using the bernstein inequality and peeling skills finally we apply the excess risk bound to the pairwise preference and derive the convergence rates of pairwise preference learning algorithms with squared loss and indicator loss by using the empirical risk minimization with respect to u processes

model free variable selection computing power has encouraged the modelling of data sets of ever increasing size data mining applications in finance marketing and bioinformatics are obvious examples a limitation of nearly all existing variable selection methods is the need to specify the correct model before selection when the number of predictors is large model formulation and validation can be difficult or even infeasible on the basis of the theory of sufficient dimension reduction we propose a new class of model free variable selection approaches the methods proposed assume no model of any form require no nonparametric smoothing and allow for general predictor effects the efficacy of the methods proposed is demonstrated via simulation and an empirical example is given

hessian eigenmaps locally linear embedding techniques for high dimensional data we describe a method for recovering the underlying parametrization of scattered data m i lying on a manifold m embedded in high dimensional euclidean space the method hessian based locally linear embedding derives from a conceptual framework of local isometry in which the manifold m viewed as a riemannian submanifold of the ambient euclidean space r n is locally isometric to an open connected subset of euclidean space r d because does not have to be convex this framework is able to handle a significantly wider class of situations than the original isomap algorithm the theoretical framework revolves around a quadratic form h f m h f m f dm defined on functions f m r here hf denotes the hessian of f and h f averages the frobenius norm of the hessian over m to define the hessian we use orthogonal coordinates on the tangent planes of m the key observation is that if m truly is locally isometric to an open connected subset of r d then h f has a d 1 dimensional null space consisting of the constant functions and a d dimensional space of functions spanned by the original isometric coordinates hence the isometric coordinates can be recovered up to a linear isometry our method may be viewed as a modification of locally linear embedding and our theoretical framework as a modification of the laplacian eigenmaps framework where we substitute a quadratic form based on the hessian in place of one based on the laplacian

learning theory approach to minimum error entropy criterion we consider the minimum error entropy mee criterion and an empirical risk minimization learning algorithm when an approximation of renyi s entropy of order 2 by parzen windowing is minimized this learning algorithm involves a parzen windowing scaling parameter we present a learning theory approach for this mee algorithm in a regression setting when the scaling parameter is large consistency and explicit convergence rates are provided in terms of the approximation ability and capacity of the involved hypothesis space novel analysis is carried out for the generalization error associated with renyi s entropy and a parzen windowing function to overcome technical difficulties arising from the essential differences between the classical least squares problems and the mee setting an involved symmetrized least squares error is introduced and analyzed which is related to some ranking algorithms

refined generalization bounds of gradient learning over reproducing kernel hilbert spaces gradient learning gl initially proposed by mukherjee and zhou 2006 has been proved to be a powerful tool for conducting variable selection and dimensional reduction simultaneously this approach presents a nonparametric version of a gradient estimator with positive definite kernels without estimating the true function itself so that the proposed version has wide applicability and allows for complex effects between predictors in terms of theory however existing generalization bounds for gl depend on capacity independent techniques and the capacity of kernel classes cannot be characterized completely thus this letter considers gl estimators that minimize the empirical convex risk we prove generalization bounds for such estimators with rates that are faster than previous results moreover we provide a novel upper bound for rademacher chaos complexity of order two which also plays an important role in general pairwise type estimations including ranking and score problems

limit theorems for u processes a u process is a collection of u statistics concretely a u process over a family h of kernels h of m variables based on a probability measure p on s s is the collection u n m h p h in mathcal h of u statistics this chapter is devoted to the asymptotic theory of u processes we are interested in finding conditions on h and p ensuring that the law of large numbers or the central limit theorem or the law of the iterated logarithm hold for u n m h p uniformly in h h the theory of empirical processes deals with the same questions for the case m 1 and u process theory is patterned after it this is a relatively new subject at least in the generality presented here h being an arbitrary collection of kernels which are defined on a general measurable space there is therefore a need to indicate its usefulness to this effect a section on applications is added at the end of the chapter section 5 5 there we illustrate the use of each of the main theorems in this chapter by deriving properties of certain multidimensional generalizations of the median and in general of m estimators and by studying estimators of the cumulative hazard and distribution functions of a random variable based on truncated data

gradient projection network analog solver for linearly constrained nonlinear programming an analog approach is presented for solving nonlinear programming problems with linear constraint conditions the present method is based on transformation of variables with exponential functions which enables every trajectory to pass through an interior of feasible regions along a gradient direction projected onto the feasible space convergence of its trajectory to the solution of optimization problems is guaranteed and it is shown that the present scheme is an extension of the affine scaling method for linear programming to nonlinear programs under a slight modification of riemannian metric an analog electronic circuit is also presented that implements the proposed scheme in real time

on ranking and generalization bounds the problem of ranking is to predict or to guess the ordering between objects on the basis of their observed features in this paper we consider ranking estimators that minimize the empirical convex risk we prove generalization bounds for the excess risk of such estimators with rates that are faster than 1 n we apply our results to commonly used ranking algorithms for instance boosting or support vector machines moreover we study the performance of considered estimators on real data sets

learning gradients predictive models that infer geometry and statistical dependence the problems of dimension reduction and inference of statistical dependence are addressed by the modeling framework of learning gradients the models we propose hold for euclidean spaces as well as the manifold setting the central quantity in this approach is an estimate of the gradient of the regression or classification function two quadratic forms are constructed from gradient estimates the gradient outer product and gradient based diffusion maps the first quantity can be used for supervised dimension reduction on manifolds as well as inference of a graphical model encoding dependencies that are predictive of a response variable the second quantity can be used for nonlinear projections that incorporate both the geometric structure of the manifold as well as variation of the response variable on the manifold we relate the gradient outer product to standard statistical quantities such as covariances and provide a simple and precise comparison of a variety of supervised dimensionality reduction methods we provide rates of convergence for both inference of informative directions as well as inference of a graphical model of variable dependencies

refined rademacher chaos complexity bounds with applications to the multikernel learning problem estimating the rademacher chaos complexity of order two is important for understanding the performance of multikernel learning mkl machines in this letter we develop a novel entropy integral for rademacher chaos complexities as compared to the previous bounds our result is much improved in that it introduces an adjustable parameter e to prohibit the divergence of the involved integral with the use of the iteration technique in steinwart and scovel 2007 we also apply our rademacher chaos complexity bound to the mkl problems and improve existing learning rates

the linear programming approach to approximate dynamic programming the curse of dimensionality gives rise to prohibitive computational requirements that render infeasible the exact solution of large scale stochastic control problems we study an efficient method based on linear programming for approximating solutions to such problems the approach fits a linear combination of pre selected basis functions to the dynamic programming cost to go function we develop error bounds that offer performance guarantees and also guide the selection of both basis functions and state relevance weights that influence quality of the approximation experimental results in the domain of queueing network control provide empirical support for the methodology

local rademacher complexities we propose new bounds on the error of learning algorithms in terms of a data dependent notion of complexity the estimates we establish give optimal rates and are based on a local and empirical version of rademacher averages in the sense that the rademacher averages are computed from the data on a subset of functions with small empirical error we present some applications to classification and prediction with convex function classes and with kernel classes in particular

ranking and empirical minimization of u statistics the problem of ranking ordering instances instead of simply classifying them has recently gained much attention in machine learning in this paper we formulate the ranking problem in a rigorous statistical framework the goal is to learn a ranking rule for deciding among two instances which one is better with minimum ranking risk since the natural estimates of the risk are of the form of a u statistic results of the theory of u processes are required for investigating the consistency of empirical risk minimizers we establish in particular a tail inequality for degenerate u processes and apply it for showing that fast rates of convergence may be achieved under specific noise assumptions just like in classification convex risk minimization methods are also studied

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing