maximum volume clustering a new discriminative clustering approach the large volume principle proposed by vladimir vapnik which advocates that hypotheses lying in an equivalence class with a larger volume are more preferable is a useful alternative to the large margin principle in this paper we introduce a new discriminative clustering model based on the large volume principle called maximum volume clustering mvc and then propose two approximation schemes to solve this mvc model a soft label mvc method using sequential quadratic programming and a hard label mvc method using semi definite programming respectively the proposed mvc is theoretically advantageous for three reasons the optimization involved in hard label mvc is convex and under mild conditions the optimization involved in soft label mvc is akin to a convex one in terms of the resulting clusters secondly the soft label mvc method possesses a clustering error bound thirdly mvc includes the optimization problems of a spectral clustering two relaxed k means clustering and an information maximization clustering as special limit cases when its regularization parameter goes to infinity experiments on several artificial and benchmark data sets demonstrate that the proposed mvc compares favorably with state of the art clustering methods

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

separating k separated ence graph languages an ence graph grammar is k separated k 1 if the distance between any two nonterminal nodes in any of its sentential forms is at least k let sepk denote the class of graph languages generated by k separated grammars then sep1 sep2 is the class of ence boundary ence graph languages and so sep2 sep1 recently engelfriet 1991 showed that sep3 sep2 and conjectured that in fact sepk 1 sepk for each k 1 we prove this conjecture affirmatively

graph grammars with neighbourhood controlled embedding abstract a central feature that distinguishes graph grammars we consider grammars generating sets of node labelled undirected graphs only from string grammars is that in the former one has to provide a mechanism by which a daughter graph the right hand side of a production can be embedded in the rest of the mother graph while in the latter this embedding is provided automatically by the structure that all strings possess left to right orientation in this paper we consider a possible classification of embedding mechanisms for node rewriting graph grammars this classification originates from the basic ideas of 9 on the one hand it allows one to fit a number of existing notions of a graph grammar into a common framework and on the other hand it points out new natural possibilities for defining the embedding mechanism in a graph grammar the relationship between the graph language generating power of graph grammars using various embedding mechanisms is established

exact location of the multicritical point for finite dimensional spin glasses a conjecture we present a conjecture on the exact location of the multicritical point in the phase diagram of spin glass models in finite dimensions by generalizing our previous work we combine duality and gauge symmetry for replicated random systems to derive formulae which make it possible to understand all the relevant available numerical results in a unified way the method applies to non self dual lattices as well as to self dual cases in the former case of which we derive a relation for a pair of values of multicritical points for mutually dual lattices the examples include the j and gaussian ising spin glasses on the square hexagonal and triangular lattices the potts and zq models with chiral randomness on these lattices and the three dimensional j ising spin glass and the random plaquette gauge model

complexity of normal form grammars abstract various types of grammars can be used to describe context free languages such are context free grammars and their normal form restrictions rewriting of a context free grammar to an equivalent grammar in required normal form can cause a change of parameters of the grammar such as the number of rules the number of nonterminals etc greibach normal form grammars and position restricted grammars will be investigated from the point of view of descriptional complexity of context free languages

generating all permutations by context free grammars in greibach normal form we consider context free grammars g n in greibach normal form and particularly in greibach m form m 1 2 which generates the finite language l n of all n strings that are permutations of n different symbols n 1 these grammars are investigated with respect to their descriptional complexity i e we determine the number of nonterminal symbols and the number of production rules of g n as functions of n as in the case of chomsky normal form these descriptional complexity measures grow faster than any polynomial function

generalization error bounds for bayesian mixture algorithms bayesian approaches to learning and estimation have played a significant role in the statistics literature over many years while they are often provably optimal in a frequentist setting and lead to excellent performance in practical applications there have not been many precise characterizations of their performance for finite sample sizes under general conditions in this paper we consider the class of bayesian mixture algorithms where an estimator is formed by constructing a data dependent mixture over some hypothesis space similarly to what is observed in practice our results demonstrate that mixture approaches are particularly robust and allow for the construction of highly complex estimators while avoiding undesirable overfitting effects our results while being data dependent in nature are insensitive to the underlying model assumptions and apply whether or not these hold at a technical level the approach applies to unbounded functions constrained only by certain moment conditions finally the bounds derived can be directly applied to non bayesian mixture approaches such as boosting and bagging

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

on the generalization ability of on line learning algorithms in this paper it is shown how to extract a hypothesis with small risk from the ensemble of hypotheses generated by an arbitrary on line learning algorithm run on an independent and identically distributed i i d sample of data using a simple large deviation argument we prove tight data dependent bounds for the risk of this hypothesis in terms of an easily computable statistic m sub n associated with the on line performance of the ensemble via sharp pointwise bounds on m sub n we then obtain risk tail bounds for kernel perceptron algorithms in terms of the spectrum of the empirical kernel matrix these bounds reveal that the linear hypotheses found via our approach achieve optimal tradeoffs between hinge loss and margin size over the class of all linear functions an issue that was left open by previous results a distinctive feature of our approach is that the key tools for our analysis come from the model of prediction of individual sequences i e a model making no probabilistic assumptions on the source generating the data in fact these tools turn out to be so powerful that we only need very elementary statistical facts to obtain our final risk bounds

hierarchical mixing linear support vector machines for nonlinear classification support vector machines svms play a very dominant role in data classification because of their good generalization performance however they suffer from the high computational complexity in the classification stage when there are a considerable number of support vectors svs it is desirable to design efficient algorithms in the classification stage to deal with datasets obtained from real time pattern recognition systems to this end we propose a novel classifier called hmlsvms hierarchical mixing linear support vector machines which has a hierarchical structure with a mixing linear svm classifier at each node it predicts the label of a sample using only a few hyperplanes we also give a generalization error bound for the class of locally linear svms llsvms based on the rademacher theory which ensures that overfitting can be effectively avoided experimental evaluations show that the proposed classifier achieves a high efficiency in the classification stage while the classification performance approaches that of kernel svms highlightsh mlsvms does not require a large memory to store kernel values in the training process because h mlsvms is composed of lsvms and there are very efficient algorithms for training lsvms 48 49 the hierarchical structure makes h mlsvms predict the labels of new arrived samples via a few of lsvms and it is much faster than nonlinear svms classifiers we quantify the generalization error bound for the class of llsvms based on the rademacher complexity and the stop criterion based on minimizing this bound ensures that h mlsvms can effectively avoid overfitting and have a good classification performance

the boundary between the spin glass and ferromagnetic states of the bond random ising model in the square lattice at t 0 the bond random j ising model in the square lattice is considered in the square cluster approximation the boundary between the ferromagnetic and spin glass states at t 0 is obtained as the transition point from the asymmetric distribution of the effective fields to the symmetric distribution the concentration of this transition pfg is obtained by solving the integral equation for the distribution function of the effective fields

structured sparsity and generalization we present a data dependent generalization bound for a large class of regularized algorithms which implement structured sparsity constraints the bound can be applied to standard squared norm regularization the lasso the group lasso some versions of the group lasso with overlapping groups multiple kernel learning and other regularization schemes in all these cases competitive results are obtained a novel feature of our bound is that it can be applied in an infinite dimensional setting such as the lasso in a separable hilbert space or multiple kernel learning with a countable number of kernels

a chain rule for the expected suprema of gaussian processes the expected supremum of a gaussian process indexed by the image of an index set under a function class is bounded in terms of separate properties of the index set and the function class the bound is relevant to the estimation of nonlinear transformations or the analysis of learning algorithms whenever hypotheses are chosen from composite classes as is the case for multi layer models

upper bounds to the spin correlation functions of the random bond ising model and n vector model with continuously distributed random interactions obtains upper bounds to spin correlation functions in the thermodynamic limit of the zero external field limit for the ising model of general spins and for the n vector model assuming that the exchange integrals are quenched random variables and that their probability distributions are continuous the upper bounds involve as a factor the corresponding spin correlation function of a uniform ferromagnetic ising model of spin or 1 in which the exchange integral is determined by the distributions of the random exchange integrals of the original system using these upper bounds the authors find a sufficient condition on the probability distributions of the exchange integrals for the disappearance of the ferromagnetic and antiferromagnetic states in the system

pac bayesian generalisation error bounds for gaussian process classification approximate bayesian gaussian process gp classification techniques are powerful non parametric learning methods similar in appearance and performance to support vector machines based on simple probabilistic models they render interpretable results and can be embedded in bayesian frameworks for model selection feature selection etc in this paper by applying the pac bayesian theorem of mcallester 1999a we prove distribution free generalisation error bounds for a wide range of approximate bayesian gp classification techniques we also provide a new and much simplified proof for this powerful theorem making use of the concept of convex duality which is a backbone of many machine learning techniques we instantiate and test our bounds for two particular gpc techniques including a recent sparse method which circumvents the unfavourable scaling of standard gp algorithms as is shown in experiments on a real world task the bounds can be very tight for moderate training sample sizes to the best of our knowledge these results provide the tightest known distribution free error bounds for approximate bayesian gpc methods giving a strong learning theoretical justification for the use of these techniques

regularization techniques for learning with matrices there is growing body of learning problems for which it is natural to organize the parameters into a matrix as a result it becomes easy to impose sophisticated prior knowledge by appropriately regularizing the parameters under some matrix norm this work describes and analyzes a systematic method for constructing such matrix based regularization techniques in particular we focus on how the underlying statistical properties of a given problem can help us decide which regularization function is appropriate our methodology is based on a known duality phenomenon a function is strongly convex with respect to some norm if and only if its conjugate function is strongly smooth with respect to the dual norm this result has already been found to be a key component in deriving and analyzing several learning algorithms we demonstrate the potential of this framework by deriving novel generalization and regret bounds for multi task learning multi class learning and multiple kernel learning

on uniform deviations of general empirical risks with unboundedness dependence and high dimensionality the statistical learning theory of risk minimization depends heavily on probability bounds for uniform deviations of the empirical risks classical probability bounds using hoeffding s inequality cannot accommodate more general situations with unbounded loss and dependent data the current paper introduces an inequality that extends hoeffding s inequality to handle these more general situations we will apply this inequality to provide probability bounds for uniform deviations in a very general framework which can involve discrete decision rules unbounded loss and a dependence structure that can be more general than either martingale or strong mixing we will consider two examples with high dimensional predictors autoregression ar with l1 loss and arx model with variable selection for sign classification which uses both lagged responses and exogenous predictors

boundary nlc graph grammars basic definitions normal forms and complexity node label controlled nlc grammars are graph grammars operating on node labeled undirected graphs which rewrite single nodes only and establish connections between the embedded graph and the neighbors of the rewritten node on the basis of the labels of the involved nodes only they define possibly infinite languages of undirected node labeled graphs or if we just omit the labels languages of unlabeled graphs here we consider a restriction of nlc grammars so called boundary nlc bnlc grammars distinguished by the property that whenever in a graph already generated two nodes may be rewritten then these nodes are not adjacent the graph languages generated by this type of grammars are called bnlc languages although we show that this restriction leads to a smaller class of languages still enough generative power remains to define interesting graph languages for example trees complete bipartite graphs maximal outerplanar graphs k trees graphs of bandwidth k graphs of cyclic bandwidth k graphs of binary tree bandwidth k graphs of cutwidth k always for a fixed positive integer k turn out all to be bnlc languages we prove a number of normal forms for bnlc grammars and then we indicate their usefulness by various applications in particular we show that for connected graphs of bounded degree the membership problem for bnlc languages is solvable in deterministic polynomial time

convergence and consistency of regularized boosting with weakly dependent observations this paper studies the statistical convergence and consistency of regularized boosting methods where the samples need not be independent and identically distributed but can come from stationary weakly dependent sequences consistency is proven for the composite classifiers that result from a regularization achieved by restricting the 1 norm of the base classifiers weights the less restrictive nature of sampling considered here is manifested in the consistency result through a generalized condition on the growth of the regularization parameter the weaker the sample dependence the faster the regularization parameter is allowed to grow with increasing sample size a consistency result is also provided for data dependent choices of the regularization parameter

generating all permutations by context free grammars in chomsky normal form let ln be the finite language of all n strings that are permutations of n different symbols n 1 we consider context free grammars gn in chomsky normal form that generate ln in particular we study a few families gn n 1 satisfying l gn ln for n 1 with respect to their descriptional complexity i e we determine the number of nonterminal symbols and the number of production rules of gn as functions of n

from ɛ entropy to kl entropy analysis of minimum information complexity density estimation we consider an extension of e entropy to a kl divergence based complexity measure for randomized density estimation methods based on this extension we develop a general information theoretical inequality that measures the statistical complexity of some deterministic and randomized density estimators consequences of the new inequality will be presented in particular we show that this technique can lead to improvements of some classical results concerning the convergence of minimum description length and bayesian posterior distributions moreover we are able to derive clean finite sample convergence bounds that are not obtainable using previous approaches

existence of the ferromagnetic phase in competing random bond ising models on the square lattice for several types of probability distribution of exchange interaction the authors generalise the arguments given in a preceding letter unpublished 1982 by the present authors to give a sufficient condition for the existence of the spontaneous magnetisation in the random bond ising model on the square lattice they study in detail the system with a gaussian distribution of exchange interaction the system with the exchange interactions j 0 0 and j and the system with the exchange interactions ja 0 and jb 0 it is shown that in each of these systems the ferromagnetic phase certainly exists in a region in the space of the parameters describing the probability distribution of the exchange interaction at sufficiently low temperatures the general condition for the existence of the ferromagnetic phase is given for the system without specifying the form of the probability distribution of the exchange interaction

position restricted grammar forms and grammars this paper deals with the question of canonical types for grammar forms and grammars a grammar form is called position restricted of type m1 mn 1 if n 2 each m1 is a nonnegative integer and each production is either of the type 0 u u a terminal word or 0 w1 1w2 wn nwn 1 where each is a variable and each wi is a terminal word of length mi thus a grammar form is position restricted if all productions in it but the terminal ones have the same relative position of terminals and variables the main results are as follows each grammar form g1 is equivalent to a grammar form g2 of any desired position restricted type m1 mn 0 with the interpretation grammars of g2 of position restricted type m1 mn 0 being sufficient to define all languages in l g1 the above result is no longer true if the position restricted type is such that mn 1 1 the second main result is that each context free language is defined by a grammar g of any desired position restricted type

rademacher chaos complexities for learning the kernel problem we develop a novel generalization bound for learning the kernel problem first we show that the generalization analysis of the kernel learning problem reduces to investigation of the suprema of the rademacher chaos process of order 2 over candidate kernels which we refer to as rademacher chaos complexity next we show how to estimate the empirical rademacher chaos complexity by well established metric entropy integrals and pseudo dimension of the set of candidate kernels our new methodology mainly depends on the principal theory of u processes and entropy integrals finally we establish satisfactory excess generalization bounds and misclassification error rates for learning gaussian kernels and general radial basis kernels

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

learning co sparse analysis operators with separable structures in the co sparse analysis model a set of filters is applied to a signal out of the signal class of interest yielding sparse filter responses as such it may serve as a prior in inverse problems or for structural analysis of signals that are known to belong to the signal class the more the model is adapted to the class the more reliable it is for these purposes the task of learning such operators for a given class is therefore a crucial problem in many applications it is also required that the filter responses are obtained in a timely manner which can be achieved by filters with a separable structure not only can operators of this sort be efficiently used for computing the filter responses but they also have the advantage that less training samples are required to obtain a reliable estimate of the operator the first contribution of this work is to give theoretical evidence for this claim by providing an upper bound for the sample complexity of the learning process the second is a stochastic gradient descent sgd method designed to learn an analysis operator with separable structures which includes a novel and efficient step size selection rule numerical experiments are provided that link the sample complexity to the convergence speed of the sgd algorithm

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

minimax nonparametric classification i rates of convergence this paper studies minimax aspects of nonparametric classification we first study minimax estimation of the conditional probability of a class label given the feature variable this function say f is assumed to be in a general nonparametric class we show the minimax rate of convergence under square l sub 2 loss is determined by the massiveness of the class as measured by metric entropy the second part of the paper studies minimax classification the loss of interest is the difference between the probability of misclassification of a classifier and that of the bayes decision as is well known an upper bound on risk for estimating f gives an upper bound on the risk for classification but the rate is known to be suboptimal for the class of monotone functions this suggests that one does not have to estimate f well in order to classify well however we show that the two problems are in fact of the same difficulty in terms of rates of convergence under a sufficient condition which is satisfied by many function classes including besov sobolev lipschitz and bounded variation this is somewhat surprising in view of a result of devroye gorfi and lugosi see a probabilistic theory of pattern recognition new york springer verlag 1996

non erasing chomsky schutzenberger theorem with grammar independent alphabet abstract the famous theorem by chomsky and schutzenberger cst says that every context free language l over an alphabet is representable as h d r where d is a dyck language over a set of brackets r is a local language and h is an alphabetic homomorphism that erases unboundedly many symbols berstel found that the number of erasures can be linearly limited if the grammar is in greibach normal form berstel and boasson and later independently okhotin proved a non erasing variant of cst for grammars in double greibach normal form in all these cst statements however the size of the dyck alphabet depends on the grammar size for l in the stanley variant of the cst only depends on and not on the grammar but the homomorphism erases many more symbols than in the other versions of cst also the regular language r is strictly locally testable but not local we prove a new version of cst which combines both features of being non erasing and of using a grammar independent alphabet in our construction is polynomial in namely o 46 and the regular language r is strictly locally testable using a recent generalization of medvedev s homomorphic characterization of regular languages we prove that the degree in the polynomial dependence of on may be reduced to just 2 in the case of linear grammars in double greibach normal form

efficient regression in metric spaces via approximate lipschitz extension we present a framework for performing efficient regression in general metric spaces roughly speaking our regressor predicts the value at a new point by computing an approximate lipschitz extension the smoothest function consistent with the observed data after performing structural risk minimization to avoid overfitting we obtain finite sample risk bounds with minimal structural and noise assumptions and a natural runtime precision tradeoff the offline learning and online prediction stages can be solved by convex programming but this naive approach has runtime complexity o n 3 which is prohibitive for large data sets we design instead a regression algorithm whose speed and generalization performance depend on the intrinsic dimension of the data to which the algorithm adapts while our main innovation is algorithmic the statistical results may also be of independent interest

scaled coupled norms and coupled higher order tensor completion recently a set of tensor norms known as coupled norms has been proposed as a convex solution to coupled tensor completion coupled norms have been designed by combining low rank inducing tensor norms with the matrix trace norm though coupled norms have shown good performances they have two major limitations they do not have a method to control the regularization of coupled modes and uncoupled modes and they are not optimal for couplings among higher order tensors in this letter we propose a method that scales the regularization of coupled components against uncoupled components to properly induce the low rankness on the coupled mode we also propose coupled norms for higher order tensors by combining the square norm to coupled norms using the excess risk bound analysis we demonstrate that our proposed methods lead to lower risk bounds compared to existing coupled norms we demonstrate the robustness of our methods through simulation and real data experiments

risk bounds for model selection via penalization performance bounds for criteria for model selection are devel oped using recent theory for sieves the model selection criteria are based on an empirical loss or contrast function with an added penalty term moti vated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of ob servations most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function we show that the quadratic risk of the minimum penal ized empirical contrast estimatoris bounded by an index of the accuracy of the sieve this accuracy index quanties the trade off among the candidate models between the approximation error and parameter dimension relative to sample size if we choose a list of models which exhibit good approximation prop erties with respect to different classes of smoothness the estimator can be simultaneously minimax rate optimal in each of those classes this is what is usually called adaptation the type of classes of smoothness in which one gets adaptation depends heavily on the list of models if too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously it may happen that the estimator is only approx

the bayesian choice a decision theoretic motivation contents introduction decision theoretic foundations of statistical inference from prior information to prior distributions bayesian point estimation tests and confidence regions admissibility and complete classes invariance haar measures and equivariant estimators hierarchical and empirical bayes extensions bayesian calculations a defense of the bayesian choice

on the structure of linear apex nlc graph grammars it is known that linear apex nlc lin a nlc grammars can generate np complete graph languages the present paper examines the source of this expressive power by analyzing complexity and decidability of the so called k connecting lin a nlc k lin a nlc grammars in which the right hand side of each production contains at most k nodes that can be connected to outside nodes the number of connecting nodes is indeed one source of the expressive power of graph grammars in that k lin a nlc k 1 lin a nlc for every k 0 there exists a 2 lin a nlc language which is np complete but every 1 lin a nlc language is in nlog the equivalence problem is undecidable for 1 lin a nlc languages but is decidable for 0 lin a nlc languages even when one of two input languages is an arbitrary b nlc language

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

stability and generalization we define notions of stability for learning algorithms and show how to use these notions to derive generalization error bounds based on the empirical error and the leave one out error the methods we use can be applied in the regression framework as well as in the classification one when the classifier is obtained by thresholding a real valued function we study the stability properties of large classes of learning algorithms such as regularization based algorithms in particular we focus on hilbert space regularization and kullback leibler regularization we demonstrate how to apply the results to svm for regression and classification

node replacement graph languages squeezed with chains trees and forests we present results on language theoretic properties such as closure membership and other decision properties of node replacement graph languages such as nlc b nlc and lin nlc languages squeezed with chains trees and forests

upper bounds to the zero field susceptibilities and magnetisations for the random bound ising model and the random bond n vector model the self avoiding walk approximation is used to give upper bounds to the zero field spin pair correlation function zero field susceptibilities and the spontaneous magnetisation for the systems of the random bond ising model of general spin s and the random bond n vector model where the exchange integrals are assumed to be random variables using these bounds the authors find a sufficient condition for the disappearance of the long range orders the condition is stronger for some cases than the one obtained previously by the authors the present one applies to some cases for which the preceding one does not

online pairwise learning algorithms with convex loss functions online pairwise learning algorithms with general convex loss functions without regularization in a reproducing kernel hilbert space rkhs are investigated under mild conditions on loss functions and the rkhs upper bounds for the expected excess generalization error are derived in terms of the approximation error when the stepsize sequence decays polynomially in particular for lipschitz loss functions such as the hinge loss the logistic loss and the absolute value loss the bounds can be of order o t13logt after t iterations while for the least squares loss the bounds can be of order o t14logt in comparison with previous works for these algorithms a broader family of convex loss functions is studied here and refined upper bounds are obtained

a comparison of boundary graph grammars and context free hypergraph grammars context free hypergraph grammars and boundary graph grammars of bounded nonterminal degree have the same power both for generating sets of graphs and for generating sets of hypergraphs arbitrary boundary graph grammars have more graph generating power than context free hypergraph grammars but they have the same hypergraph generating power to obtain these results several normal forms for boundary graph grammars are given it is also shown that the class of boundary graph languages is closed under the operation of edge contraction where the label of the edge indicates whether or not the edge should be contracted

rigorous results for the hopfield model with many patterns we perform a thorough investigation of the main aspects of the hopfield model with many patterns advances are made toward the validity of the replica symmetric solution strong evidence of the validity of this solution is given over the entire domain where this validity is conjectured complete proof is given in a subregion that contains strictly the ergodic region

an axiomatic definition of context free rewriting and its application to nlc graph grammars abstract an abstract notion of context free grammar is introduced it deals with abstract objects that can be words trees graphs or other combinatorial objects it is applied to nlc graph grammars introduced by rozenberg and janssens the monadic second order theory of a context free nlc set of graphs is decidable

existence of the ferromagnetic phase in a random bond ising model on the square lattice extend griffiths 1972 arguments to show that there exists the thermodynamic limit in a quenched random bond ising model with competing interactions on the square lattice peierls 1936 arguments are extended to give 0 97394 for an upper bound of the critical concentration of the ferromagnetic phase for the system in which the exchange integrals take on j o and j with respective probabilities p and 1 p

a new look at independence the concentration of measure phenomenon in product spaces is a farreaching abstract generalization of the classical exponential inequalities for sums of independent random variables we attempt to explain in the simplest possible terms the basic concepts underlying this phenomenon the basic method to prove concentration inequalities and the meaning of several of the most useful inequalities

information theoretic upper and lower bounds for statistical estimation in this paper we establish upper and lower bounds for some statistical estimation problems through concise information theoretic arguments our upper bound analysis is based on a simple yet general inequality which we call the information exponential inequality we show that this inequality naturally leads to a general randomized estimation method for which performance upper bounds can be obtained the lower bounds applicable for all statistical estimators are obtained by original applications of some well known information theoretic inequalities and approximately match the obtained upper bounds for various important problems moreover our framework can be regarded as a natural generalization of the standard minimax framework in that we allow the performance of the estimator to vary for different possible underlying distributions according to a predefined prior

upper bounds to the correlation functions of the random bond ising model and n vector model with competing interactions studies a correlation function which is given by the canonical average of a product of one or more spin variables for the random bond ising model in which the exchange integrals are j j o and j with probabilities p and 1 p respectively the authors show that an upper bound to the configurational average of the correlation functions calculated in the thermodynamic limit in the zero external field limit is the product of the same correlation function for the corresponding ferromagnetic ising model at the temperature under consideration and the same quantity at the temperature t1 which is determined by the condition t1 2j mod kb1n p 1 p mod where kb is the boltzmann constant the results are given for the random bond ising model of an arbitrary spin s and also for the diluted random bond ising models of an arbitrary spin s with the pair interaction and with general interaction and for the diluted and undiluted random bond n vector model

concentration inequalities using the entropy method we investigate a new methodology worked out by ledoux and massart to prove concentration of measure inequalities the method is based on certain modified logarithmic sobolev inequalities we provide some very simple and general ready to use inequalities one of these inequalities may be considered as an exponential version of the efron stein inequality the main purpose of this paper is to point out the simplicity and the generality of the approach we show how the new method can recover many of talagrand s revolutionary inequalities and provide new applications in a variety of problems including rademacher averages rademacher chaos the number of certain small subgraphs in a random graph and the minimum of the empirical risk in some statistical estimation problems

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

families of locally testable languages kim mcnaughton and mccloskey have produced a polynomial time algorithm in order to test if a deterministic automaton recognizes a locally testable language we provide a characterization in terms of automata for the strictly locally testable languages and for the strongly locally testable languages two subclasses of locally testable languages these two characterizations lead us to polynomial time algorithms for testing these families of languages

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

matrix completion with the trace norm learning bounding and transducing trace norm regularization is a widely used and successful approach for collaborative filtering and matrix completion however previous learning guarantees require strong assumptions such as a uniform distribution over the matrix entries in this paper we bridge this gap by providing such guarantees under much milder assumptions which correspond to matrix completion as performed in practice in fact we claim that previous difficulties partially stemmed from a mismatch between the standard learning theoretic modeling of matrix completion and its practical application our results also shed some light on the issue of matrix completion with bounded models which enforce predictions to lie within a certain range in particular we provide experimental and theoretical evidence that such models lead to a modest yet significant improvement

efficient probabilistic classification vector machine with incremental basis function selection probabilistic classification vector machine pcvm is a sparse learning approach aiming to address the stability problems of relevance vector machine for classification problems because pcvm is based on the expectation maximization algorithm it suffers from sensitivity to initialization convergence to local minima and the limitation of bayesian estimation making only point estimates another disadvantage is that pcvm was not efficient for large data sets to address these problems this paper proposes an efficient pcvm epcvm by sequentially adding or deleting basis functions according to the marginal likelihood maximization for efficient training because of the truncated prior used in epcvm two approximation techniques i e laplace approximation and expectation propagation ep have been used to implement epcvm to obtain full bayesian solutions we have verified laplace approximation and ep with a hybrid monte carlo approach the generalization performance and computational effectiveness of epcvm are extensively evaluated theoretical discussions using rademacher complexity reveal the relationship between the sparsity and the generalization bound of epcvm

griebach normal form transformation revisited we develop a new method for placing a given context free grammar into greibach normal form with only polynomial increase of its size starting with an arbitrarye free context free grammarg we transformginto an equivalent context free grammarhin extended greibach normal form i e in addition to rules fulfilling the greibach normal form properties the grammar can have chain rules the size ofhwill beo g 3 where g is the size ofg moreover in the case thatgis chain rule free hwill be already in greibach normal form ifhis not chain rule free then we use the standard method for chain rule elimination for the transformation ofhinto greibach normal form the size of the constructed grammar iso g 4

a deep connection between the vapnik chervonenkis entropy and the rademacher complexity in this paper we derive a deep connection between the vapnik chervonenkis vc entropy and the rademacher complexity for this purpose we first refine some previously known relationships between the two notions of complexity and then derive new results which allow computing an admissible range for the rademacher complexity given a value of the vc entropy and vice versa the approach adopted in this paper is new and relies on the careful analysis of the combinatorial nature of the problem the obtained results improve the state of the art on this research topic

linear graph grammars power and complexity abstract a graph grammar is linear if it generates graphs with at most one nonterminal node linear graph grammars can simulate nonterminal bounded graph grammars which generate graphs with a bounded number of nonterminal nodes and derivation bounded graph grammars if a linear graph language contains connected graphs of bounded degree only then it is in nspace log n these results are shown for graph grammars with neighbourhood controlled embedding and with dynamic edge relabeling ence grammars

sup norm approximation bounds for networks through probabilistic methods we consider the problem of approximating a smooth target function and its derivatives by networks involving superpositions and translations of a fixed activation function the approximation is with respect to the sup norm and the rate is shown to be of order o n sup 1 2 that is the rate is independent of the dimension d the results apply to neural and wavelet networks and extend the work of barren see proc 7th yale workshop on adaptive and learning systems may 1992 and ibid vol 39 p 930 1993 the approach involves probabilistic methods based on central limit theorems for empirical processes indexed by classes of functions

nonterminal separation in graph grammars abstract the notion of nonterminal separation in ence graph grammars is introduced such a graph grammar is k separated with k 1 if the distance between any two nonterminal nodes in any of its sentential forms is at least k 2 separated ence grammars are known as boundary ence graph grammars there exists a 2 separated ence graph language that is not 3 separated apex ence graph grammars can be arbitrarily separated every apex ence graph language can be generated by a k separated apex ence grammar for every k

covering number bounds of certain regularized linear function classes recently sample complexity bounds have been derived for problems involving linear functions such as neural networks and support vector machines in many of these theoretical studies the concept of covering numbers played an important role it is thus useful to study covering numbers for linear function classes in this paper we investigate two closely related methods to derive upper bounds on these covering numbers the first method already employed in some earlier studies relies on the so called maurey s lemma the second method uses techniques from the mistake bound framework in online learning we compare results from these two methods as well as their consequences in some learning formulations