a second order perceptron algorithm kernel based linear threshold algorithms such as support vector machines and perceptron like algorithms are among the best available techniques for solving pattern classification problems in this paper we describe an extension of the classical perceptron algorithm called second order perceptron and analyze its performance within the mistake bound model of on line learning the bound achieved by our algorithm depends on the sensitivity to second order data information and is the best known mistake bound for efficient kernel based linear threshold classifiers to date this mistake bound which strictly generalizes the well known perceptron bound is expressed in terms of the eigenvalues of the empirical data correlation matrix and depends on a parameter controlling the sensitivity of the algorithm to the distribution of these eigenvalues since the optimal setting of this parameter is not known a priori we also analyze two variants of the second order perceptron algorithm one that adaptively sets the value of the parameter in terms of the number of mistakes made so far and one that is parameterless based on pseudoinverses

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

dynamics of interacting neural networks the dynamics of interacting perceptrons is solved analytically for a directed flow of information the system runs into a state which has a higher symmetry than the topology of the model a symmetry breaking phase transition is found with increasing learning rate in addition it is shown that a system of interacting perceptrons which is trained on the history of its minority decisions develops a good strategy for the problem of adaptive competition known as the bar problem or minority game

ridge regression biased estimation for nonorthogonal problems in multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory if not incorrect if the prediction vectors are not orthogonal proposed is an estimation procedure based on adding small positive quantities to the diagonal of x x introduced is the ridge trace a method for showing in two dimensions the effects of nonorthogonality it is then shown how to augment x x to obtain biased estimates with smaller mean square error

a simplicial approach to the determination of an integer point of a simplex we propose an algorithm to find an integer point of a simplex the algorithm is based on an integer labeling rule and a triangulation of the space starting at an integer point the algorithm leaves it along one of n rays and generates a sequence of adjacent simplices of varying dimension within a finite number of iterations the algorithm either yields an integer point of the simplex or proves that no such point exists

the 2 n 1 2 ray algorithm a new simplicial algorithm to compute economic equilibria a new variable dimension simplicial restart algorithm is introduced to compute economic equilibria the number of rays along which the algorithm can leave the starting point differs from the thusfar known algorithms more precisely the new algorithm has one ray to each of the 2n 1 2 faces of then dimensional price simplex whereas the existing algorithms haven 1 rays either to each facet or to each vertex of the unit simplex the path of points followed by the algorithm can be interpreted as a globally and universally convergent price adjustment process the process is also economically meaningful and therefore it is a good alternative for the well known walras tatonnement process computational results show that the algorithm is competitive with the most efficient simplicial algorithms developed thusfar

on the generalization ability of on line learning algorithms in this paper it is shown how to extract a hypothesis with small risk from the ensemble of hypotheses generated by an arbitrary on line learning algorithm run on an independent and identically distributed i i d sample of data using a simple large deviation argument we prove tight data dependent bounds for the risk of this hypothesis in terms of an easily computable statistic m sub n associated with the on line performance of the ensemble via sharp pointwise bounds on m sub n we then obtain risk tail bounds for kernel perceptron algorithms in terms of the spectrum of the empirical kernel matrix these bounds reveal that the linear hypotheses found via our approach achieve optimal tradeoffs between hinge loss and margin size over the class of all linear functions an issue that was left open by previous results a distinctive feature of our approach is that the key tools for our analysis come from the model of prediction of individual sequences i e a model making no probabilistic assumptions on the source generating the data in fact these tools turn out to be so powerful that we only need very elementary statistical facts to obtain our final risk bounds

generalization error bounds for bayesian mixture algorithms bayesian approaches to learning and estimation have played a significant role in the statistics literature over many years while they are often provably optimal in a frequentist setting and lead to excellent performance in practical applications there have not been many precise characterizations of their performance for finite sample sizes under general conditions in this paper we consider the class of bayesian mixture algorithms where an estimator is formed by constructing a data dependent mixture over some hypothesis space similarly to what is observed in practice our results demonstrate that mixture approaches are particularly robust and allow for the construction of highly complex estimators while avoiding undesirable overfitting effects our results while being data dependent in nature are insensitive to the underlying model assumptions and apply whether or not these hold at a technical level the approach applies to unbounded functions constrained only by certain moment conditions finally the bounds derived can be directly applied to non bayesian mixture approaches such as boosting and bagging

variable dimension algorithms basic theory interpretations and extensions of some existing methods in this paper we establish a basic theory for variable dimension algorithms which were originally developed for computing fixed points by van der laan and talman we introduce a new concept primal dual pair of subdivided manifolds and by utilizing it we propose a basic model which will serve as a foundation for constructing a wide class of variable dimension algorithms our basic model furnishes interpretations to several existing methods lemke s algorithm for the linear complementarity problem its extension to the nonlinear complementarity problem a variable dimension algorithm on conical subdivisions and merrill s algorithm we shall present a method for solving systems of equations as an application of the second method and an efficient implementation of the fourth method to which our interpretation leads us a method for constructing triangulations with an arbitrary refinement factor of mesh size is also proposed

a variable rate refining triangulation a new variable rate refining subdivisionb4 for the solution of equations with piecewise linear homotopies and restart homotopies is described the essential virtue of the subdivisionb4 is that it offers vast latitude to the user in particular the parameters ofb4 can be set so one restriction ofb4 is thatj3 triangulation and another restriction is the octahedral subdivision more generally full flexibility is available in the placement of local focal points and perhaps most importantly b4 permits the new capability of refining coordinates at variable andindependent rates

on the minority game analytical and numerical studies abstract we investigate further several properties of the minority game we have recently introduced we explain the origin of the phase transition and give an analytical expression of 2 n in the n 2 m region the ability of the players to learn a given payoff is also analyzed and we show that the darwinian evolution process tends to a self organized state in particular the lifetime distribution is a power law with exponent 2 furthermore we study the influence of identical players on their gain and on the systems performance finally we show that large brains always take advantage of small brains

a modified integer labeling for complementarity algorithms nonlinear complementary problems can be solved by algorithms based on the principle of simplicial approximation we present such an algorithm that allows the choice of an arbitrary starting point without using an artificial labeling or an extra dimension furthermore the relation between the grid size of the triangulation and the accuracy of an approximate solution is investigated and it is shown that a combination of the algorithm with newton steps yields better accuracy finally the algorithm is adapted for solving systems of equations

predicting a switching sequence of graph labelings we study the problem of predicting online the labeling of a graph we consider a novel setting for this problem in which in addition to observing vertices and labels on the graph we also observe a sequence of just vertices on a second graph a latent labeling of the second graph selects one of k labelings to be active on the first graph we propose a polynomial time algorithm for online prediction in this setting and derive a mistake bound for the algorithm the bound is controlled by the geometric cut of the observed and latent labelings as well as the resistance diameters of the graphs when specialized to multitask prediction and online switching problems the bound gives new and sharper results under certain conditions

improved risk tail bounds for on line algorithms tight bounds are derived on the risk of models in the ensemble generated by incremental training of an arbitrary learning algorithm the result is based on proof techniques that are remarkably different from the standard risk analysis based on uniform convergence arguments and improves on previous bounds published by the same authors

a simplicial algorithm for stationary point problems on polytopes a simplicial variable dimension restart algorithm for the stationary point problem or variational inequality problem on a polytope is proposed given a polytope c in rn and a continuous function f c rn find a point x in c such that fx x fx x for any point x in c starting from an arbitrary point v in c the algorithm generates a piecewise linear path of points in c this path is followed by alternating linear programming pivot steps to follow a linear piece of the path and replacement steps in a simplicial subdivision of c within a finite number of function evaluations and linear programming pivot steps the algorithm finds an approximate stationary point the algorithm leaves the starting point v along a ray pointing to one of the vertices w of c the vertex w is obtained from the optimum solution of the linear programming problem maximize fv x subject to x c

computing an integer point of a simplex with an arbitrary starting homotopy like simplicial algorithm abstract an arbitrary starting homotopy like simplicial algorithm is developed for computing an integer point of an n dimensional simplex the algorithm is derived from the use of an integer labeling rule and a triangulation of r n 0 1 and consists of two interchanging phases one phase of the algorithm constitutes a homotopy simplicial algorithm which generates n 1 dimensional simplices in r n 0 1 and the other phase of the algorithm constitutes a pivoting procedure which generates n dimensional simplices in either r n 0 or r n 1 the algorithm varies from one phase to the other when the matrix defining the simplex is in the so called canonical form starting at an arbitrary integer point in r n 0 the algorithm within a finite number of iterations either yields an integer point of the simplex or proves that no such point exists

data dependent margin based generalization bounds for classification we derive new margin based inequalities for the probability of error of classifiers the main feature of these bounds is that they can be calculated using the training data and therefore may be effectively used for model selection purposes in particular the bounds involve empirical complexities measured on the training data such as the empirical fat shattering dimension as opposed to their worst case counterparts traditionally used in such analyses also our bounds appear to be sharper and more general than recent results involving empirical complexity measures in addition we develop an alternative data based bound for the generalization error of classes of convex combinations of classifiers involving an empirical complexity measure that is easier to compute than the empirical covering number or fat shattering dimension we also show examples of efficient computation of the new bounds

irrelevance of memory in the minority game by means of extensive numerical simulations we show that all the distinctive features of the minority game introduced by challet and zhang physica a 256 514 1998 are completely independent of the memory of the agents the only crucial requirement is that all the individuals must possess the same information irrespective of whether this information is true or false

implementation of a continuation method for normal maps this paper presents an implementation of a nonsmooth continuation method of which the idea was originally put forward by alexander et al we show how the method can be computationally implemented and present numerical results for variational inequality problems in up to 96 variables

online learning with samples drawn from non identical distributions learning algorithms are based on samples which are often drawn independently from an identical distribution i i d in this paper we consider a different setting with samples drawn according to a non identical sequence of probability distributions each time a sample is drawn from a different distribution in this setting we investigate a fully online learning algorithm associated with a general convex loss function and a reproducing kernel hilbert space rkhs error analysis is conducted under the assumption that the sequence of marginal distributions converges polynomially in the dual of a holder space for regression with least square or insensitive loss learning rates are given in both the rkhs norm and the l2 norm for classification with hinge loss and support vector machine q norm loss rates are explicitly stated with respect to the excess misclassification error

moral foundations in an interacting neural networks society a statistical mechanics analysis abstract the moral foundations theory supports that people across cultures tend to consider a small number of dimensions when classifying issues on a moral basis the data also show that the statistics of weights attributed to each moral dimension is related to self declared political affiliation which in turn has been connected to cognitive learning styles by the recent literature in neuroscience and psychology inspired by these data we propose a simple statistical mechanics model with interacting neural networks classifying vectors and learning from members of their social neighbourhood about their average opinion on a large set of issues the purpose of learning is to reduce dissension among agents when disagreeing we consider a family of learning algorithms parametrized by that represents the importance given to corroborating same sign opinions we define an order parameter that quantifies the diversity of opinions in a group with homogeneous learning style using monte carlo simulations and a mean field approximation we find the relation between the order parameter and the learning parameter at a temperature we associate with the importance of social influence in a given group in concordance with data groups that rely more strongly on corroborating evidence sustain less opinion diversity we discuss predictions of the model and propose possible experimental tests

pegasos primal estimated sub gradient solver for svm we describe and analyze a simple and effective stochastic sub gradient descent algorithm for solving the optimization problem cast by support vector machines svm we prove that the number of iterations required to obtain a solution of accuracy epsilon is tilde o 1 epsilon where each iteration operates on a single training example in contrast previous analyses of stochastic gradient descent methods for svms require omega 1 epsilon 2 iterations as in previously devised svm solvers the number of iterations also scales linearly with 1 where is the regularization parameter of svm for a linear kernel the total run time of our method is tilde o d lambda epsilon where d is a bound on the number of non zero features in each example since the run time does not depend directly on the size of the training set the resulting algorithm is especially suited for learning from large datasets our approach also extends to non linear kernels while working solely on the primal objective function though in this case the runtime does depend linearly on the training set size our algorithm is particularly well suited for large text classification problems where we demonstrate an order of magnitude speedup over previous svm learning methods

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

fast rates in statistical and online learning the speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning a fast rate of convergence means less data is needed for the same level of performance the pursuit of fast rates in online and statistical learning has led to the discovery of many conditions in learning theory under which fast learning is possible we show that most of these conditions are special cases of a single unifying condition that comes in two forms the central condition for proper learning algorithms that always output a hypothesis in the given model and stochastic mixability for online algorithms that may make predictions outside of the model we show that under surprisingly weak assumptions both conditions are in a certain sense equivalent the central condition has a re interpretation in terms of convexity of a set of pseudoprobabilities linking it to density estimation under misspecification for bounded losses we show how the central condition enables a direct proof of fast rates and we prove its equivalence to the bernstein condition itself a generalization of the tsybakov margin condition both of which have played a central role in obtaining fast rates in statistical learning yet while the bernstein condition is two sided the central condition is one sided making it more suitable to deal with unbounded losses in its stochastic mixability form our condition generalizes both a stochastic exp concavity condition identified by juditsky rigollet and tsybakov and vovk s notion of mixability our unifying conditions thus provide a substantial step towards a characterization of fast rates in statistical learning similar to how classical mixability characterizes constant regret in the sequential prediction with expert advice setting

the octahedral algorithm a new simplicial fixed point algorithm a new variable dimension simplicial algorithm for the computation of solutions of systems of nonlinear equations or the computation of fixed points is presented it uses the restrart technique of merrill to improve the accuracy of the solution the algorithm is shown to converge quadratically under certain conditions the algorithm should be efficient and relatively easy to implement

the weighted majority algorithm we study the construction of prediction algorithms in a situation in which a learner faces a sequence of trials with a prediction to be made in each and the goal of the learner is to make few mistakes we are interested in the case where the learner has reason to believe that one of some pool of known algorithms will perform well but the learner does not know which one a simple and effective method based on weighted voting is introduced for constructing a compound algorithm in such a circumstance we call this method the weighted majority algorithm we show that this algorithm is robust in the presence of errors in the data we discuss various versions of the weighted majority algorithm and prove mistake bounds for them that are closely related to the mistake bounds of the best algorithms of the pool for example given a sequence of trials if there is an algorithm in the pool a that makes at most m mistakes then the weighted majority algorithm will make at most c log a m mistakes on that sequence where c is fixed constant

on the computation of fixed points in the product space of unit simplices and an application to noncooperative n person games in this paper an algorithm based on the principle of simplicial approximation is introduced to compute fixed points of upper semicontinuous point to set mappings from the product space s of unit simplices into itself the algorithm is a modification of an algorithm introduced in an earlier paper the main feature is that it starts with an arbitrary chosen point in s and that the triangulation of s depends on the starting point moreover the algorithm can terminate with a non full dimensional subsimplex yielding a good approximation an application is given for non cooperative n person games where s is the strategy space some computational experiences are given

a unified approach to the implementation of several restart fixed point algorithms and a new variable dimension algorithm we present a unified description of a class of restart fixed point algorithms including merrill s method and several variable dimension algorithms for their implementation on computers based on this description we show that some pivot saving techniques originally developed for the homotopy methods can be applied to the class we also propose a new variable dimension algorithm having 3 n 1 rays along which we can move toward a solution some numerical comparisons of the simplicial restart algorithms merrill s method the 2n method the octahedral method and the new one support that the latter two methods are more efficient than the others

the d 2 triangulation for simplicial homotopy algorithms for computing solutions of nonlinear equations a triangulation of arbitrary refinement of grid sizes of 0 1 n is proposed for simplicial homotopy algorithms for computing solutions of nonlinear equations on each level the new triangulation called thed2 triangulation subdivides n into simplices according to thed1 triangulation we prove that thed2 triangulation is superior to thek2 triangulation andj2 triangulation in the number of simplices numerical tests show that the simplicial homotopy algorithm based on thed2 triangulation indeed is much more efficient

stability and generalization we define notions of stability for learning algorithms and show how to use these notions to derive generalization error bounds based on the empirical error and the leave one out error the methods we use can be applied in the regression framework as well as in the classification one when the classifier is obtained by thresholding a real valued function we study the stability properties of large classes of learning algorithms such as regularization based algorithms in particular we focus on hilbert space regularization and kullback leibler regularization we demonstrate how to apply the results to svm for regression and classification

a continuous deformation algorithm for variational inequality problems on polytopes a continuous deformation algorithm is proposed for solving a variational inequality problem on a polytopek the algorithm embeds the polytopek intok 0 and starts by applying a variable dimension algorithm onk 0 until an approximate solution is found onk 0 then by tracing the path of solutions of a system of equations the algorithm virtually follows a path of approximate solution ink 0 when the path ink 0 returns to level 0 i e k 0 the variable dimension algorithm is again used until a new approximate solution is found onk 0 the setk 0 is triangulated so that the approximate solution on the path improves the accuracy as the level increases a contrivance for a practical implementation of the algorithm is proposed and tested for some test problems

efficiency and implementation of simplicial zero point algorithms in this paper we compare the efficiency of several simplicial variable dimension algorithms to do so we first treat the issues of degeneracy and accelerating we present a device for solving degeneracy furthermore we compare several accelerating techniques the technique of iterated quasi newton steps after each major cycle of the simplicial algorithm is implemented in a computer code which is used to compare the efficiency of the n 1 ray 2n ray 2n ray and 3n 1 ray algorithms except for the n 1 ray algorithm the number of function evaluations does not differ very much between the various algorithms it appeared however that the 2n algorithm needs considerably less computation time

sequential prediction of individual sequences under general loss functions we consider adaptive sequential prediction of arbitrary binary sequences when the performance is evaluated using a general loss function the goal is to predict on each individual sequence nearly as well as the best prediction strategy in a given comparison class of possibly adaptive prediction strategies called experts by using a general loss function we generalize previous work on universal prediction forecasting and data compression however here we restrict ourselves to the case when the comparison class is finite for a given sequence we define the regret as the total loss on the entire sequence suffered by the adaptive sequential predictor minus the total loss suffered by the predictor in the comparison class that performs best on that particular sequence we show that for a large class of loss functions the minimax regret is either spl theta log n or spl omega spl radic spl lscr log n depending on the loss function where n is the number of predictors in the comparison class and spl lscr is the length of the sequence to be predicted the former case was shown previously by vovk 1990 we give a simplified analysis with an explicit closed form for the constant in the minimax regret formula and give a probabilistic argument that shows this constant is the best possible some weak regularity conditions are imposed on the loss function in obtaining these results we also extend our analysis to the case of predicting arbitrary sequences that take real values in the interval 0 1

a simplicial algorithm for the nonlinear stationary point problem on an unbounded polyhedron a path following algorithm is proposed for finding a solution to the nonlinear stationary point problem on an unbounded convex and pointed polyhedron the algorithm can start at an arbitrary point of the polyhedron the path to be followed by the algorithm is described as the path of zeros of some piecewise continuously differentiable function defined on an appropriate subdivided manifold this manifold is induced by a generalized primal dual pair of subdivided manifolds the path of zeros can be approximately followed by dividing the polyhedron into simplices and replacing the original function by its piecewise linear approximation with respect to this subdivision the piecewise linear path of this function can be generated by alternating replacement steps and linear programming pivot steps a condition under which the path of zeros converges to a solution is also stated and a description of how the algorithm operates when the problem is linear or when the polyhedron is the cartesian product of a poly

simplicial approximation of solutions to the nonlinear complementarity problem with lower and upper bounds ideas of a simplicial variable dimension restart algorithm to approximate zero points onrn developed by the authors and of a linear complementarity problem pivoting algorithm are combined to an algorithm for solving the nonlinear complementarity problem with lower and upper bounds the algorithm can be considered as a modification of the2n ray zero point finding algorithm onrn it appears that for the new algorithm the number of linear programming pivot steps is typically less than for the2n ray algorithm applied to an equivalent zero point problem this is caused by the fact that the algorithm utilizes the complementarity conditions on the variables

the generalization ability of online algorithms for dependent data we study the generalization performance of online learning algorithms trained on samples coming from a dependent source of data we show that the generalization error of any stable online algorithm concentrates around its regret an easily computable statistic of the online performance of the algorithm when the underlying ergodic process is or mixing we show high probability error bounds assuming the loss function is convex and we also establish sharp convergence rates and deviation bounds for strongly convex losses and several linear prediction problems such as linear and logistic regression least squares svm and boosting on dependent data in addition our results have straightforward applications to stochastic optimization with dependent data and our analysis requires only martingale convergence arguments we need not rely on more powerful statistical tools such as empirical process theory

a fully polynomial time approximation algorithm for computing a stationary point of the general linear complementarity problem we apply a potential reduction algorithm to solve the general linear complementarity problem glcp minimize xty subject to ax by cz q and x y z 0 we show that the algorithm is a fully polynomial time approximation scheme fptas for computing an e approximate stationary point of the glcp note that there are some glcps in which every stationary point is a solution xty 0 these include the lcps with row sufficient matrices we also show that the algorithm is a polynomial time algorithm for a special class of glcps

learning by on line gradient descent we study on line gradient descent learning in multilayer networks analytically and numerically the training is based on randomly drawn inputs and their corresponding outputs as defined by a target rule in the thermodynamic limit we derive deterministic differential equations for the order parameters of the problem which allow an exact calculation of the evolution of the generalization error first we consider a single layer perceptron with sigmoidal activation function learning a target rule defined by a network of the same architecture for this model the generalization error decays exponentially with the number of training examples if the learning rate is sufficiently small however if the learning rate is increased above a critical value perfect learning is no longer possible for architectures with hidden layers and fixed hidden to output weights such as the parity and the committee machine we find additional effects related to the existence of symmetries in these problems

optimal rates for zero order convex optimization the power of two function evaluations we consider derivative free algorithms for stochastic and nonstochastic convex optimization problems that use only function values rather than gradients focusing on nonasymptotic bounds on convergence rates we show that if pairs of function values are available algorithms for d dimensional optimization that use gradient estimates based on random perturbations suffer a factor of at most sqrt d in convergence rate over traditional stochastic gradient methods we establish such results for both smooth and nonsmooth cases sharpening previous analyses that suggested a worse dimension dependence and extend our results to the case of multiple m ge 2 evaluations we complement our algorithmic development with information theoretic lower bounds on the minimax convergence rate of such problems establishing the sharpness of our achievable results up to constant sometimes logarithmic factors

a new variable dimension algorithm for the fixed point problem we propose a new variable dimension algorithm for solving a system ofn equations inn variables the algorithm solves a sequence of subproblems consisting of the first several equations the dimension of subproblems does not however increase monotonically a convergence condition is also presented

the p norm generalization of the lms algorithm for adaptive filtering recently much work has been done analyzing online machine learning algorithms in a worst case setting where no probabilistic assumptions are made about the data this is analogous to the h sup spl infin setting used in adaptive linear filtering bregman divergences have become a standard tool for analyzing online machine learning algorithms using these divergences we motivate a generalization of the least mean squared lms algorithm the loss bounds for these so called p norm algorithms involve other norms than the standard 2 norm the bounds can be significantly better if a large proportion of the input variables are irrelevant i e if the weight vector we are trying to learn is sparse we also prove results for nonstationary targets we only know how to apply kernel methods to the standard lms algorithm i e p 2 however even in the general p norm case we can handle generalized linear models where the output of the system is a linear function combined with a nonlinear transfer function e g the logistic sigmoid

an efficient implementation of merrill s method for sparse or partially separable systems of nonlinear equations when merrill s method without an extra dimension is applied to solving sparse or partially separable systems of nonlinear equations the computational efficiency can be further improved i e a larger piece of linearity can be traversed in one step by using a suitable linear system one of the linear systems is updated and the corresponding technique is shown to update information about the linear system and the large piece in the implementation of the method some numerical results of the method support the claim that it is efficient also some mistakes from a previous paper in which the main technique exploiting sparsity was proposed are corrected

online pairwise learning algorithms pairwise learning usually refers to a learning task that involves a loss function depending on pairs of examples among which the most notable ones are bipartite ranking metric learning and auc maximization in this letter we study an online algorithm for pairwise learning with a least square loss function in an unconstrained setting of a reproducing kernel hilbert space rkhs that we refer to as the online pairwise learning algorithm opera in contrast to existing works kar sriperumbudur jain karnick 2013 wang khardon pechyony jones 2012 which require that the iterates are restricted to a bounded domain or the loss function is strongly convex opera is associated with a non strongly convex objective function and learns the target function in an unconstrained rkhs specifically we establish a general theorem that guarantees the almost sure convergence for the last iterate of opera without any assumptions on the underlying distribution explicit convergence rates are derived under the condition of polynomially decaying step sizes we also establish an interesting property for a family of widely used kernels in the setting of pairwise learning and illustrate the convergence results using such kernels our methodology mainly depends on the characterization of rkhss using its associated integral operators and probability inequalities for random variables with values in a hilbert space

a unifying model based on retraction for fixed point algorithms we present a unifying model based on retraction for several restart fixed point algorithms the model embraces the interpretation of the algorithms in terms of stationary point problem by van der laan and talman and fully explains the 2 ray method

how to use expert advice we analyze algorithms that predict a binary value by combining the predictions of several prediction strategies called experts our analysis is for worst case situations i e we make no assumptions about the way the sequence of bits to be predicted is generated we measure the performance of the algorithm by the difference between the expected number of mistakes it makes on the bit sequence and the expected number of mistakes made by the best expert on this sequence where the expectation is taken with respect to the randomization in the predictins we show that the minimum achievable difference is on the order of the square root of the number of mistakes of the best expert and we give efficient algorithms that achieve this our upper and lower bounds have matching leading constants in most cases we then show how this leads to certain kinds of pattern recognition learning algorithms with performance bounds that improve on the best results currently know in this context we also compare our analysis to the case in which log loss is used instead of the expected number of mistakes

on complexity issues of online learning algorithms in this paper some new probabilistic upper bounds are presented for the online learning algorithm proposed in and more generally for linear stochastic approximations in hilbert spaces with these upper bounds not only does one recover almost sure convergence but also relaxes the square summable condition on the step size appeared in the early work furthermore two probabilistic upper bounds are given for an averaging process both of which achieve the same rate with respect to sample size as in batch learning algorithms and one of which is tight in both sample size and regularization parameter

a class of simplicial restart fixed point algorithms without an extra dimension in an earlier paper we introduced an algorithm for approximating a fixed point of a mapping on the product space of unit simplices ideas of that paper are used to construct a class of triangulations ofrn more precisely for somek 1 k n and positive integersm1 mk with sumn a triangulation ofrn is obtained by triangulating the cells which are formed by taking the product of given triangulations ofrmj j 1 k the triangulation of each cell will be defined in relation to an arbitrarily chosen pointv inrn being the starting point of the algorithm fork n we obtain thek triangulation originally due to todd each element of the class can be used to find a simplex which approximates a fixed point of a mapping onrn by generating a unique path of adjacent simplices of variable dimension starting with the pointv

the generalization ability of online svm classification based on markov sampling in this paper we consider online support vector machine svm classification learning algorithms with uniformly ergodic markov chain u e m c samples we establish the bound on the misclassification error of an online svm classification algorithm with u e m c samples based on reproducing kernel hilbert spaces and obtain a satisfactory convergence rate we also introduce a novel online svm classification algorithm based on markov sampling and present the numerical studies on the learning ability of online svm classification based on markov sampling for benchmark repository the numerical studies show that the learning performance of the online svm classification algorithm based on markov sampling is better than that of classical online svm classification based on random sampling as the size of training samples is larger

universal prediction of individual sequences the problem of predicting the next outcome of an individual binary sequence using finite memory is considered the finite state predictability of an infinite sequence is defined as the minimum fraction of prediction errors that can be made by any finite state fs predictor it is proven that this fs predictability can be achieved by universal sequential prediction schemes an efficient prediction procedure based on the incremental parsing procedure of the lempel ziv data compression algorithm is shown to achieve asymptotically the fs predictability some relations between compressibility and predictability are discussed and the predictability is proposed as an additional measure of the complexity of a sequence