boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

graph theoretic methods for matrix completion problems abstract a pattern is a list of positions in an n n real matrix a matrix completion problem for the class of matrices asks whether every partial matrix whose specified entries are exactly the positions of the pattern can be completed to a matrix we survey the current state of research on matrix completion problems for many subclasses of p 0 matrices including positive definite matrices m matrices inverse m matrices p matrices and matrices defined by various sign symmetry and positivity conditions on p 0 and p matrices graph theoretic techniques used to study completion problems are discussed several new results are also presented including the solution to the m 0 matrix completion problem and the sign symmetric p 0 matrix completion problem

improving spanning trees by upgrading nodes we study budget constrained optimal network upgrading problems we are given an edge weighted graph g v e where node v can be upgraded at a cost of c this upgrade reduces the delay of each link emanating from the goal is to find a minimum cost set of nodes to be upgraded so that the resulting network has a good performance we consider two performance measures namely the weight of a minimum spanning tree and the bottleneck weight of a minimum bottleneck spanning tree and present approximation algorithms

on the generalization ability of on line learning algorithms in this paper it is shown how to extract a hypothesis with small risk from the ensemble of hypotheses generated by an arbitrary on line learning algorithm run on an independent and identically distributed i i d sample of data using a simple large deviation argument we prove tight data dependent bounds for the risk of this hypothesis in terms of an easily computable statistic m sub n associated with the on line performance of the ensemble via sharp pointwise bounds on m sub n we then obtain risk tail bounds for kernel perceptron algorithms in terms of the spectrum of the empirical kernel matrix these bounds reveal that the linear hypotheses found via our approach achieve optimal tradeoffs between hinge loss and margin size over the class of all linear functions an issue that was left open by previous results a distinctive feature of our approach is that the key tools for our analysis come from the model of prediction of individual sequences i e a model making no probabilistic assumptions on the source generating the data in fact these tools turn out to be so powerful that we only need very elementary statistical facts to obtain our final risk bounds

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

online regularized classification algorithms this paper considers online classification learning algorithms based on regularization schemes in reproducing kernel hilbert spaces associated with general convex loss functions a novel capacity independent approach is presented it verifies the strong convergence of the algorithm under a very weak assumption of the step sizes and yields satisfactory convergence rates for polynomially decaying step sizes explicit learning rates with respect to the misclassification error are given in terms of the choice of step sizes and the regularization parameter depending on the sample size error bounds associated with the hinge loss the least square loss and the support vector machine q norm loss are presented to illustrate our method

risk bounds of learning processes for levy processes leevy processes refer to a class of stochastic processes for example poisson processes and brownian motions and play an important role in stochastic processes and machine learning therefore it is essential to study risk bounds of the learning process for time dependent samples drawn from a leevy process or briefly called learning process for leevy process it is noteworthy that samples in this learning process are not independently and identically distributed i i d therefore results in traditional statistical learning theory are not applicable or at least cannot be applied directly because they are obtained under the sample i i d assumption in this paper we study risk bounds of the learning process for time dependent samples drawn from a leevy process and then analyze the asymptotical behavior of the learning process in particular we first develop the deviation inequalities and the symmetrization inequality for the learning process by using the resultant inequalities we then obtain the risk bounds based on the covering number finally based on the resulting risk bounds we study the asymptotic convergence and the rate of convergence of the learning process for leevy process meanwhile we also give a comparison to the related results under the sample i i d assumption

positive definite completions of partial hermitian matrices abstract the question of which partial hermitian matrices some entries specified some free may be completed to positive definite matrices is addressed it is shown that if the diagonal entries are specified and principal minors composed of specified entries are positive then if the undirected graph of the specified entries is chordal a positive definite completion necessarily exists furthermore if this graph is not chordal then examples exist without positive definite completions in case a positive definite completion exists there is a unique matrix in the class of all positive definite completions whose determinant is maximal and this matrix is the unique one whose inverse has zeros in those positions corresponding to unspecified entries in the original partial hermitian matrix additional observations regarding positive definite completions are made

upgrading min max spanning tree problem under various cost functions abstract this paper addresses upgrading min max spanning tree problem mmst given a graph g v e the aim of this problem is to modify edge weights under certain limits and given budget so that the mmst with respect to perturbed graph improves as much as possible we present a complexity result for general non decreasing cost functions in special case it is shown that the problem under linear and sum type hamming cost function can be solved in o e 2 and o e log e log v time respectively

rademacher penalties and structural risk minimization we suggest a penalty function to be used in various problems of structural risk minimization this penalty is data dependent and is based on the sup norm of the so called rademacher process indexed by the underlying class of functions sets the standard complexity penalties used in learning problems and based on the vc dimensions of the classes are conservative upper bounds in a probabilistic sense uniformly over the set of all underlying distributions for the penalty we suggest thus for a particular distribution of training examples one can expect better performance of learning algorithms with the data driven rademacher penalties we obtain oracle inequalities for the theoretical risk of estimators obtained by structural minimization of the empirical risk with rademacher penalties the inequalities imply some form of optimality of the empirical risk minimizers we also suggest an iterative approach to structural risk minimization with rademacher penalties in which the hierarchy of classes is not given in advance but is determined in the data driven iterative process of risk minimization we prove probabilistic oracle inequalities for the theoretical risk of the estimators based on this approach as well

inverse m matrix completions of patterns omitting some diagonal positions abstract let n 1 n which subsets q of n n have the property that whenever a ij for i j q form a partial inverse m matrix i e a ij 0 and if l l q then the inverse of a ij i j l is an m matrix a ij can be defined for i j q so that the inverse of a a ij is an m matrix in terms of a digraph g with vertex set n and arcs i j q with i j our results say 1 it is necessary that the induced subdigraph of any alternate path to an arc be a clique 2 a cycle need not be contained in a clique if i i q for at least one vertex i of 3 when n 4 it is necessary and sufficient that condition 1 is true and every simple cycle has a vertex i with i i q and 4 the general question can be reduced to the case in which g is strongly connected much as it has been when q contains i i for every i n

improved risk tail bounds for on line algorithms tight bounds are derived on the risk of models in the ensemble generated by incremental training of an arbitrary learning algorithm the result is based on proof techniques that are remarkably different from the standard risk analysis based on uniform convergence arguments and improves on previous bounds published by the same authors

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

data dependent margin based generalization bounds for classification we derive new margin based inequalities for the probability of error of classifiers the main feature of these bounds is that they can be calculated using the training data and therefore may be effectively used for model selection purposes in particular the bounds involve empirical complexities measured on the training data such as the empirical fat shattering dimension as opposed to their worst case counterparts traditionally used in such analyses also our bounds appear to be sharper and more general than recent results involving empirical complexity measures in addition we develop an alternative data based bound for the generalization error of classes of convex combinations of classifiers involving an empirical complexity measure that is easier to compute than the empirical covering number or fat shattering dimension we also show examples of efficient computation of the new bounds

nonparametric decentralized detection using kernel methods we consider the problem of decentralized detection under constraints on the number of bits that can be transmitted by each sensor in contrast to most previous work in which the joint distribution of sensor observations is assumed to be known we address the problem when only a set of empirical samples is available we propose a novel algorithm using the framework of empirical risk minimization and marginalized kernels and analyze its computational and statistical properties both theoretically and empirically we provide an efficient implementation of the algorithm and demonstrate its performance on both simulated and real data sets

error bounds for piecewise smooth and switching regression this paper deals with regression problems in which the nonsmooth target is assumed to switch between different operating modes specifically piecewise smooth pws regression considers target functions switching deterministically via a partition of the input space while switching regression considers arbitrary switching laws this paper derives generalization error bounds in these two settings by following the approach based on rademacher complexities for pws regression our derivation involves a chaining argument and a decomposition of the covering numbers of pws classes in terms of the ones of their component function classes and the capacity of the classifier partitioning the input space this yields error bounds with a radical dependence on the number of modes for switching regression the decomposition can be performed directly at the level of the rademacher complexities which yields bounds with a linear dependence on the number of modes by using once more chaining and decomposition at the level of covering numbers we show how to recover a radical dependence examples of applications are given in particular for pws and swichting regression with linear and kernel based component functions

up and downgrading the 1 center in a network abstract we study budget constrained network improvement and degrading problems based on the vertex 1 center problem on graphs given a graph with vertex weights and edge lengths the task is to decrease and increase the vertex weights within certain limits such that the optimal 1 center objective value with respect to the new weights is minimized and maximized respectively the upgrading improvement problem is shown to be solvable in o n 2 time provided that the distance matrix is given the downgrading 1 center problem is shown to be strongly np hard on general graphs but can be solved in o n 2 time on trees as byproduct we suggest an algorithm that solves the problem of minimizing over the upper envelope of n piecewise linear functions in o k n time where k is the total number of breakpoints

on the doubt about margin explanation of boosting margin theory provides one of the most popular explanations to the success of adaboost where the central point lies in the recognition that margin is the key for characterizing the performance of adaboost this theory has been very influential e g it has been used to argue that adaboost usually does not overfit since it tends to enlarge the margin even after the training error reaches zero previously the minimum margin bound was established for adaboost however breiman 1999 9 pointed out that maximizing the minimum margin does not necessarily lead to a better generalization later reyzin and schapire 2006 37 emphasized that the margin distribution rather than minimum margin is crucial to the performance of adaboost in this paper we first present the kth margin bound and further study on its relationship to previous work such as the minimum margin bound and emargin bound then we improve the previous empirical bernstein bounds audibert et al 2009 maurer and pontil 2009 2 30 and based on such findings we defend the margin based explanation against breiman s doubts by proving a new generalization error bound that considers exactly the same factors as schapire et al 1998 39 but is sharper than breiman s 1999 9 minimum margin bound by incorporating factors such as average margin and variance we present a generalization error bound that is heavily related to the whole margin distribution we also provide margin distribution bounds for generalization error of voting classifiers in finite vc dimension space

new concentration inequalities in product spaces we introduce three new ways to measure the distance from a point to a subset of a product space and we prove corresponding concentration inequalities each of them allows to control the fluctuation of a new class of random variables

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

completions of p matrix patterns abstract a list of positions in an n n real matrix a pattern is said to have m completion if every partial m matrix that specifies exactly these positions can be completed to an m matrix let q be a pattern that includes all diagonal entries and let g be its digraph the following are equivalent 1 the pattern q has m completion 2 the pattern q is permutation similar to a block triangular pattern with all the diagonal blocks completely specified 3 any strongly connected subdigraph of g is complete a pattern with some diagonal entries unspecified has m completion if and only if the principal subpattern defined by the specified diagonal positions has m completion

the constrained bottleneck problem in networks we consider problems on networks that are captured by two performance measures one performance measure is any general cost function of a solution the other is a bottleneck measure that describes the worst maximum cost component of the solution the paper contains algorithms to solve three problems in one problem we minimize the bottleneck subject to a constraint on the generalized cost in the second problem we minimize the generalized cost subject to a constraint on the bottleneck in the third problem we consider the two criteria simultaneously and find all the pareto optimum solutions the major result is that the introduction of the bottleneck measure changes the complexity of the original general cost problem by a factor which is at most linear in the number of links

inverse min max spanning tree problem under the weighted sum type hamming distance the inverse optimization problem is to modify the weight or cost length capacity and so on such that a given feasible solution becomes an optimal solution in this paper we consider the inverse min max spanning tree problem under the weighted sum type hamming distance for the model considered we present its combinatorial algorithm that runs in strongly polynomial times

risk bounds for model selection via penalization performance bounds for criteria for model selection are devel oped using recent theory for sieves the model selection criteria are based on an empirical loss or contrast function with an added penalty term moti vated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of ob servations most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function we show that the quadratic risk of the minimum penal ized empirical contrast estimatoris bounded by an index of the accuracy of the sieve this accuracy index quanties the trade off among the candidate models between the approximation error and parameter dimension relative to sample size if we choose a list of models which exhibit good approximation prop erties with respect to different classes of smoothness the estimator can be simultaneously minimax rate optimal in each of those classes this is what is usually called adaptation the type of classes of smoothness in which one gets adaptation depends heavily on the list of models if too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously it may happen that the estimator is only approx

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

on the additive properties of the fat shattering dimension the properties of the vc dimension under various compositions are well understood but this is much less the case for classes of continuous functions in this brief we show that a commonly used scale sensitive dimension v y is much less well behaved under minkowski summation than its vc cousin while the fat shattering dimension retains some compositional similarity to the vc dimension as an application we analyze the fat shattering dimension of trigonometric functions and series

vote counting measures for ensemble classifiers various measures such as margin and bias variance have been proposed with the aim of gaining a better understanding of why multiple classifier systems mcs perform as well as they do while these measures provide different perspectives for mcs analysis it is not clear how to use them for mcs design in this paper a different measure based on a spectral representation is proposed for two class problems it incorporates terms representing positive and negative correlation of pairs of training patterns with respect to class labels experiments employing mlp base classifiers in which parameters are fixed but systematically varied demonstrate the sensitivity of the proposed measure to base classifier complexity

the symmetric m matrix and symmetric inverse m matrix completion problems abstract the symmetric m matrix and symmetric m 0 matrix completion problems are solved and results of johnson and smith linear algebra appl 290 1999 193 are extended to solve the symmetric inverse m matrix completion problem 1 a pattern i e a list of positions in an n n matrix has symmetric m completion i e every partial symmetric m matrix specifying the pattern can be completed to a symmetric m matrix if and only if the principal subpattern r determined by its diagonal is permutation similar to a pattern that is block diagonal with each diagonal block complete or in graph theoretic terms if and only if each component of the graph of r is a clique 2 a pattern has symmetric m 0 completion if and only if the pattern is permutation similar to a pattern that is block diagonal with each diagonal block either complete or omitting all diagonal positions or in graph theoretic terms if and only if every principal subpattern corresponding to a component of the graph of the pattern either omits all diagonal positions or includes all positions 3 a pattern has symmetric inverse m completion if and only if its graph is block clique and no diagonal position is omitted that corresponds to a vertex in a graph block of order 2 the techniques used are also applied to matrix completion problems for other classes of symmetric matrices

rigorous results for the hopfield model with many patterns we perform a thorough investigation of the main aspects of the hopfield model with many patterns advances are made toward the validity of the replica symmetric solution strong evidence of the validity of this solution is given over the entire domain where this validity is conjectured complete proof is given in a subregion that contains strictly the ergodic region

the symmetric inverse m matrix completion problem necessary and sufficient conditions are given on the data for completability of a partial symmetric inverse m matrix the graph of whose specified entries is a cycle and these conditions coincide with those we identify to be necessary in the general nonsymmetric case graphs for which all partial symmetric inverse m matrices have symmetric inverse m matrix completions are identified and these include those that arise in the general positionally symmetric case however the identification of all such graphs is more subtle than the general case finally we show that our new cycle conditions are sufficient for completability of all partial symmetric inverse m matrices the graph of whose specified entries is a block graph

matrix completion problems for pairs of related classes of matrices abstract for a class x of real matrices a list of positions in an n n matrix a pattern is said to have x completion if every partial x matrix that specifies exactly these positions can be completed to an x matrix if x and x0 are classes that satisfy the conditions any partial x matrix is a partial x0 matrix for any x0 matrix a and e 0 a ei is a x matrix and for any partial x matrix a there exists 0 such that a i is a partial x matrix where is the partial identity matrix specifying the same pattern as a then any pattern that has x0 completion must also have x completion however there are usually patterns that have x completion that fail to have x0 completion this result applies to many pairs of subclasses of p and p0 matrices defined by the same restriction on entries including the classes p p0 matrices weakly sign symmetric p p0 matrices and non negative p p0 matrices it also applies to other related pairs of subclasses of p0 matrices such as the pairs classes of p p0 1 matrices weakly sign symmetric p p0 1 matrices and non negative p p0 1 matrices furthermore any pattern that has weakly sign symmetric sign symmetric non negative p0 completion must also have weakly sign symmetric sign symmetric non negative p0 1 completion although these pairs of classes do not satisfy condition 3 similarly the class of inverse m matrices and its topological closure do not satisfy condition 3 but the conclusion remains true and the matrix completion problem for the topological closure of the class of inverse m matrices is solved for patterns containing the diagonal

a new look at independence the concentration of measure phenomenon in product spaces is a farreaching abstract generalization of the classical exponential inequalities for sums of independent random variables we attempt to explain in the simplest possible terms the basic concepts underlying this phenomenon the basic method to prove concentration inequalities and the meaning of several of the most useful inequalities

the dynamics of adaboost cyclic behavior and convergence of margins in order to study the convergence properties of the adaboost algorithm we reduce adaboost to a nonlinear iterated map and study the evolution of its weight vectors this dynamical systems approach allows us to understand adaboost s convergence properties completely in certain cases for these cases we find stable cycles allowing us to explicitly solve for adaboost s output using this unusual technique we are able to show that adaboost does not always converge to a maximum margin combined classifier answering an open question in addition we show that non optimal adaboost where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration may fail to converge to a maximum margin classifier even if optimal adaboost produces a maximum margin also we show that if adaboost cycles it cycles among support vectors i e examples that achieve the same smallest margin

the completion problem for m matrices and inverse m matrices abstract the completion problems for m matrices and inverse m matrices are discussed the former has a simple solution but is not at all analogous to the positive definite completion problem the latter is more subtle and is analogous to the positive definite problem a graph theoretic completion result as well as a determinant maximizing completion that is characterized by having a zero entry in the inverse in every position in which the partial matrix has an unspecified entry is given for the inverse m matrix problem these mirror to some extent the positive definite case but the methodology is necessarily different

inverse m matrices abstract this is an attempt at a comprehensive expository study of those nonnegative matrices which happen to be inverses of m matrices and is aimed at an audience conversant with basic ideas of matrix theory a theme is the parallels anddifferences between the class of m matrices and the class of inverse m matrices among the primary tools used are diagonal multiplications the neumann expansion and the form of the inverse of a partitioned matrix some of the results seem not to have appeared before and several unsolved problems are mentioned

completion of hankel partial contractions of extremal type we find concrete necessary and sufficient conditions for the existence of contractive completions of hankel partial contractions of size 4 4 in the extremal case along the way we introduce a new approach that allows us to solve algorithmically the contractive completion problem for 4 4 hankel matrices as an application we obtain a concrete example of a partially contractive 4 4 hankel matrix which does not admit a contractive completion

p matrix completions under weak symmetry assumptions an n by n matrix is called a matrix if it is one of weakly sign symmetric positive nonnegative p matrix weakly sign symmetric positive nonnegative p0 1 matrix or fischer or koteljanskii matrix in this paper we are interested in matrix completion problems that is when a partial matrix has a matrix completion here we prove that a combinatorially symmetric partial positive p matrix has a positive p matrix completion if the graph of its specified entries is an n cycle in general a combinatorially symmetric partial matrix has a matrix completion if the graph of its specified entries is a 1 chordal graph this condition is also necessary for weakly sign symmetric p0 or p0 1 matrices

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

boosting method for local learning in statistical pattern recognition we propose a local boosting method in classification problems borrowing from an idea of the local likelihood method our proposal local boosting includes a simple device for localization for computational feasibility we proved the bayes risk consistency of the local boosting in the framework of probably approximately correct learning inspection of the proof provides a useful viewpoint for comparing ordinary boosting and local boosting with respect to the estimation error and the approximation error both boosting methods have the bayes risk consistency if their approximation errors decrease to zero compared to ordinary boosting local boosting may perform better by controlling the trade off between the estimation error and the approximation error ordinary boosting with complicated base classifiers or other strong classification methods including kernel machines may have classification performance comparable to local boosting with simple base classifiers for example decision stumps local boosting however has an advantage with respect to interpretability local boosting with simple base classifiers offers a simple way to specify which features are informative and how their values contribute to a classification rule even though locally several numerical studies on real data sets confirm these advantages of local boosting

completions of inverse m matrix patterns abstract a list of positions in an n n real matrix that includes all diagonal positions a pattern is said to have inverse m completion if every partial inverse m matrix that specifies exactly these positions can be completed to an inverse m matrix johnson and smith c r johnson r l smith linear algebra and its applications 241 243 1996 655 667 characterize the positionally symmetric patterns that have inverse m completion as those patterns whose graphs are block clique in this paper the restriction of positional symmetry is removed a pattern has inverse m completion if and only if the digraph g of the pattern has the properties a the induced subdigraph of a cycle is a clique and b if g contains both arc i j and a path of length 1 between i and j then the induced subdigraph of the path is a clique furthermore any irreducible pattern with inverse m completion is positionally symmetric and has a block clique digraph any pattern is permutation similar to a block lower triangular pattern with irreducible diagonal blocks and such a pattern has inverse m completion if and only if i the pattern digraph of each diagonal block is block clique ii each subdiagonal block contains at most one position and iii the pattern digraph of the block structure has no alternate path to a single arc

on completion problems for various classes of p matrices abstract a p matrix is a real square matrix having every principal minor positive and a fischer matrix is a p matrix that satisfies fischer s inequality for all principal submatrices in this paper all patterns of positions for n n matrices n 4 are classified as to whether or not every partial matrix can be completed to a matrix for any of the classes positive p nonnegative p or fischer matrices also all symmetric patterns for 5 5 matrices are classified as to completion of partial fischer matrices and all but two such patterns are classified as to positive p or nonnegative p completion we also show that any pattern whose digraph contains a minimally chordal symmetric hamiltonian induced subdigraph does not have completion for any of the classes positive p nonnegative p fischer matrices

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

sup norm approximation bounds for networks through probabilistic methods we consider the problem of approximating a smooth target function and its derivatives by networks involving superpositions and translations of a fixed activation function the approximation is with respect to the sup norm and the rate is shown to be of order o n sup 1 2 that is the rate is independent of the dimension d the results apply to neural and wavelet networks and extend the work of barren see proc 7th yale workshop on adaptive and learning systems may 1992 and ibid vol 39 p 930 1993 the approach involves probabilistic methods based on central limit theorems for empirical processes indexed by classes of functions

bounds for linear multi task learning we give dimension free and data dependent bounds for linear multi task learning where a common linear operator is chosen to preprocess data for a vector of task specific linear thresholding classifiers the complexity penalty of multi task learning is bounded by a simple expression involving the margins of the task specific classifiers the hilbert schmidt norm of the selected preprocessor and the hilbert schmidt norm of the covariance operator for the total mixture of all task distributions or alternatively the frobenius norm of the total gramian matrix for the data dependent version the results can be compared to state of the art results on linear single task learning

on quantile regression in reproducing kernel hilbert spaces with the data sparsity constraint for spline regressions it is well known that the choice of knots is crucial for the performance of the estimator as a general learning framework covering the smoothing splines learning in a reproducing kernel hilbert space rkhs has a similar issue however the selection of training data points for kernel functions in the rkhs representation has not been carefully studied in the literature in this paper we study quantile regression as an example of learning in a rkhs in this case the regular squared norm penalty does not perform training data selection we propose a data sparsity constraint that imposes thresholding on the kernel function coefficients to achieve a sparse kernel function representation we demonstrate that the proposed data sparsity method can have competitive prediction performance for certain situations and have comparable performance in other cases compared to that of the traditional squared norm penalty therefore the data sparsity method can serve as a competitive alternative to the squared norm penalty method some theoretical properties of our proposed method using the data sparsity constraint are obtained both simulated and real data sets are used to demonstrate the usefulness of our data sparsity constraint