exploitation of pairwise class distances for ordinal classification ordinal classification refers to classification problems in which the classes have a natural order imposed on them because of the nature of the concept studied some ordinal classification approaches perform a projection from the input space to one dimensional latent space that is partitioned into a sequence of intervals one for each class class identity of a novel input pattern is then decided based on the interval its projection falls into this projection is trained only indirectly as part of the overall model fitting as with any other latent model fitting direct construction hints one may have about the desired form of the latent model can prove very useful for obtaining high quality models the key idea of this letter is to construct such a projection model directly using insights about the class distribution obtained from pairwise distance calculations the proposed approach is extensively evaluated with 8 nominal and ordinal classifiers methods 10 real world ordinal classification data sets and 4 different performance measures the new methodology obtained the best results in average ranking when considering three of the performance metrics although significant differences are found for only some of the methods also after observing other methods of internal behavior in the latent space we conclude that the internal projections do not fully reflect the intraclass behavior of the patterns our method is intrinsically simple intuitive and easily understandable yet highly competitive with state of the art approaches to ordinal classification

incremental support vector learning for ordinal regression support vector ordinal regression svor is a popular method to tackle ordinal regression problems however until now there were no effective algorithms proposed to address incremental svor learning due to the complicated formulations of svor recently an interesting accurate on line algorithm was proposed for training nu support vector classification nu svc which can handle a quadratic formulation with a pair of equality constraints in this paper we first present a modified svor formulation based on a sum of margins strategy the formulation has multiple constraints and each constraint includes a mixture of an equality and an inequality then we extend the accurate on line nu svc algorithm to the modified formulation and propose an effective incremental svor algorithm the algorithm can handle a quadratic formulation with multiple constraints where each constraint is constituted of an equality and an inequality more importantly it tackles the conflicts between the equality and inequality constraints we also provide the finite convergence analysis for the algorithm numerical experiments on the several benchmark and real world data sets show that the incremental algorithm can converge to the optimal solution in a finite number of steps and is faster than the existing batch and incremental svor algorithms meanwhile the modified formulation has better accuracy than the existing incremental svor algorithm and is as accurate as the sum of margins based formulation of shashua and levin

noise amplitude dependence of the invariant density for noisy fully chaotic one dimensional maps we present some analytic nonperturbative results for the invariant density ensuremath rho x for noisy one dimensional maps at fully developed chaos under periodic boundary conditions the fourier expansion method is used to show precisely how noise makes ensuremath rho x absolutely continuous and smooths it out simple solvable models are used to illustrate the explicit dependence of ensuremath rho x on the amplitude ensuremath eta of the noise distribution all the way from the case of zero noise stackrel ensuremath rightarrow ensuremath eta 0 to the completely noise dominated limit ensuremath eta 1

the invariant density for a class of discrete time maps involving an arbitrary monotonic function operator and an integer parameter when xt and xt 1 represent two random variables each belonging to a real interval 0 1 and being related by a first order difference equation xt 1 f xt called a discrete time map the probability density distribution connected with xt can be translated into that associated with xt 1 this yields an evolution equation by means of which one can construct an infinite sequence wt x t n x 0 1 starting from an integrable function w0 x normalized to unity on 0 1 the question of the convergence of the sequence toward a so called invariant density function w x as t and the problem of finding this limit were examined by a number of authors mostly studying isolated cases the present paper solves the problem for a class of discrete time maps characterized by xt 1 f sn l sn 1f 1 xt l 2 3 whereby f is a real continuous monotonically increasing function mapping 0 1 onto itself and sn is the usual symbol for the sinelike jacobian elliptic function with modulus k 0 1 including the

ensemble dynamics of intermittency and power law decay a spectral decomposition of the frobenius perron operator is constructed for one dimensional maps with intermittent chaos using the method of coherent states a technique using the spectral density function is applied to the the well known cusp map which generates weak type ii intermittency higher order corrections are obtained to the leading 1 t long time behavior of the x x autocorrelation

worst case analysis of the perceptron and exponentiated update algorithms abstract the absolute loss is the absolute difference between the desired and predicted outcome this paper demonstrates worst case upper bounds on the absolute loss for the perception learning algorithm and the exponentiated update learning algorithm which is related to the weighted majority algorithm the bounds characterize the behavior of the algorithms over any sequence of trials where each trial consists of an example and a desired outcome interval any value in the interval is an acceptable outcome the worst case absolute loss of both algorithms is bounded by the absolute loss of the best linear function in a comparison class plus a constant dependent on the initial weight vector plus a per trial loss the per trial loss can be eliminated if the learning algorithm is allowed a tolerance from the desired outcome for concept learning the worst case bounds lead to mistake bounds that are comparable to past results

exponentiated gradient versus gradient descent for linear predictors we consider two algorithm for on line prediction based on a linear model the algorithms are the well known gradient descent gd algorithm and a new algorithm which we call eg they both maintain a weight vector using simple updates for the gd algorithm the update is based on subtracting the gradient of the squared error made on a prediction the eg algorithm uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively we present worst case loss bounds for eg and compare them to previously known bounds for the gd algorithm the bounds suggest that the losses of the algorithms are in general incomparable but eg has a much smaller loss if only a few components of the input are relevant for the predictions we have performed experiments which show that our worst case upper bounds are quite tight already on simple artificial data

extreme value laws for dynamical systems under observational noise abstract in this paper we prove the existence of extreme value laws for dynamical systems perturbed by the instrument like error also called observational noise an orbit perturbed with observational noise mimics the behavior of an instrumentally recorded time series instrument characteristics defined as precision and accuracy act both by truncating and randomly displacing the real value of a measured observable here we analyze both these effects from a theoretical and a numerical point of view first we show that classical extreme value laws can be found for orbits of dynamical systems perturbed with observational noise then we present numerical experiments to support the theoretical findings and give an indication of the order of magnitude of the instrumental perturbations which cause relevant deviations from the extreme value laws observed in deterministic dynamical systems finally we show that the observational noise preserves the structure of the deterministic attractor this goes against the common assumption that random transformations cause the orbits asymptotically fill the ambient space with a loss of information about the fractal structure of the attractor

regularization techniques for learning with matrices there is growing body of learning problems for which it is natural to organize the parameters into a matrix as a result it becomes easy to impose sophisticated prior knowledge by appropriately regularizing the parameters under some matrix norm this work describes and analyzes a systematic method for constructing such matrix based regularization techniques in particular we focus on how the underlying statistical properties of a given problem can help us decide which regularization function is appropriate our methodology is based on a known duality phenomenon a function is strongly convex with respect to some norm if and only if its conjugate function is strongly smooth with respect to the dual norm this result has already been found to be a key component in deriving and analyzing several learning algorithms we demonstrate the potential of this framework by deriving novel generalization and regret bounds for multi task learning multi class learning and multiple kernel learning

compact differentiable transformation groups on exotic spheres 0 introduction 355 1 statement of the main results 357 2 higher dimensional subgroups of o n 1 358 3 higher dimensional transformation groups on homology spheres geometric weight systems and orbit structures 360 4 higher dimensional transformation groups on homotopy spheres and the proof of theorem b 375 5 degree of symmetry of exotic spheres revisited 381 6 large transformation groups on exotic spheres in bp 1 384

the perceptron algorithm versus winnow linear versus logarithmic mistake bounds when few input variables are relevant abstract we give an adversary strategy that forces the perceptron algorithm to make kn mistakes in learning monotone disjunctions over n variables with at most k literals in contrast littlestone s algorithm winnow makes at most o k log n mistakes for the same problem both algorithms use thresholded linear functions as their hypotheses however winnow does multiplicative updates to its weight vector instead of the additive updates of the perceptron algorithm in general we call an algorithm additive if its weight vector is always a sum of a fixed initial weight vector and some linear combination of already seen instances thus the perceptron algorithm is an example of an additive algorithm we show that an adversary can force any additive algorithm to make n k 1 2 mistakes in learning a monotone disjunction of at most k literals simple experiments show that for k n winnow clearly outperforms the perceptron algorithm also on nonadversarial random data

solving nonstationary classification problems with coupled support vector machines many learning problems may vary slowly over time in particular some critical real world applications when facing this problem it is desirable that the learning method could find the correct input output function and also detect the change in the concept and adapt to it we introduce the time adaptive support vector machine ta svm which is a new method for generating adaptive classifiers capable of learning concepts that change with time the basic idea of ta svm is to use a sequence of classifiers each one appropriate for a small time window but in contrast to other proposals learning all the hyperplanes in a global way we show that the addition of a new term in the cost function of the set of svms that penalizes the diversity between consecutive classifiers produces a coupling of the sequence that allows ta svm to learn as a single adaptive classifier we evaluate different aspects of the method using appropriate drifting problems in particular we analyze the regularizing effect of changing the number of classifiers in the sequence or adapting the strength of the coupling a comparison with other methods in several problems including the well known stagger dataset and the real world electricity pricing domain shows the good performance of ta svm in all tested situations

bundle methods for regularized risk minimization a wide variety of machine learning problems can be described as minimizing a regularized risk functional with different algorithms using different notions of risk and different regularizers examples include linear support vector machines svms gaussian processes logistic regression conditional random fields crfs and lasso amongst others this paper describes the theory and implementation of a scalable and modular convex solver which solves all these estimation problems it can be parallelized on a cluster of workstations allows for data locality and can deal with regularizers such as l1 and l2 penalties in addition to the unified framework we present tight convergence bounds which show that our algorithm converges in o 1 e steps to e precision for general convex problems and in o log 1 e steps for continuously differentiable problems we demonstrate the performance of our general purpose solver on a variety of publicly available data sets

the weighted majority algorithm we study the construction of prediction algorithms in a situation in which a learner faces a sequence of trials with a prediction to be made in each and the goal of the learner is to make few mistakes we are interested in the case where the learner has reason to believe that one of some pool of known algorithms will perform well but the learner does not know which one a simple and effective method based on weighted voting is introduced for constructing a compound algorithm in such a circumstance we call this method the weighted majority algorithm we show that this algorithm is robust in the presence of errors in the data we discuss various versions of the weighted majority algorithm and prove mistake bounds for them that are closely related to the mistake bounds of the best algorithms of the pool for example given a sequence of trials if there is an algorithm in the pool a that makes at most m mistakes then the weighted majority algorithm will make at most c log a m mistakes on that sequence where c is fixed constant

on the construction of codimension two minimal immersions of exotic spheres into euclidean spheres it is nowadays a well known fundamental fact in differential topology that there exist many smooth manifolds which are homeomorphic to but non diffeomorphic to spheres such manifolds were christened exotic spheres by their founder milnor 223 the structures of those exotic spheres which bound some parallelizable manifolds were systematically analyzed by the techniques of surgery in 20 for example it is not difficult to show that such exotic spheres can be embedded as codimension two submanifolds of the unit spheres in differential geometry and geometric variational theory the study of closed minimal submanifolds of the euclidean n sphere s 1 is directly related to that of the local structure of singularities of minimal subvarieties in the general riemannian setting therefore closed minimal submanifolds of s 1 are not only interesting nice geometric objects by themselves but are also of general theoretical importance in the study of geometric measure theory 16 18 24 among various problems on the closed minimal submanifolds of euclidean spheres the following two problems naturally distinguish themselves as especially interesting namely

the concave convex procedure the concave convex procedure cccp is a way to construct discrete time iterative dynamical systems that are guaranteed to decrease global optimization and energy functions monotonically this procedure can be applied to almost any optimization problem and many existing algorithms can be interpreted in terms of it in particular we prove that all expectation maximization algorithms and classes of legendre minimization and variational bounding algorithms can be reexpressed in terms of cccp we show that many existing neural network and mean field theory algorithms are also examples of cccp the generalized iterative scaling algorithm and sinkhorn s algorithm can also be expressed as cccp by changing variables cccp can be used both as a new way to understand and prove the convergence of existing optimization algorithms and as a procedure for generating new algorithms

ordinal neural networks without iterative tuning ordinal regression or is an important branch of supervised learning in between the multiclass classification and regression in this paper the traditional classification scheme of neural network is adapted to learn ordinal ranks the model proposed imposes monotonicity constraints on the weights connecting the hidden layer with the output layer to do so the weights are transcribed using padding variables this reformulation leads to the so called inequality constrained least squares icls problem its numerical solution can be obtained by several iterative methods for example trust region or line search algorithms in this proposal the optimum is determined analytically according to the closed form solution of the icls problem estimated from the karush kuhn tucker conditions furthermore following the guidelines of the extreme learning machine framework the weights connecting the input and the hidden layers are randomly generated so the final model estimates all its parameters without iterative tuning the model proposed achieves competitive performance compared with the state of the art neural networks methods for or

phase transition like phenomenon in a piecewise linear map abstract we introduce a one hump fully developed chaotic piecewise linear map with a control parameter which tunes the shape and therefore the value of the lyapounov exponent of the map the graph of the lyapounov exponent as a function of this parameter shows several discontinuities which are explained both qualitatively and quantitatively

efficient learning of label ranking by soft projections onto polyhedra we discuss the problem of learning to rank labels from a real valued feedback associated with each label we cast the feedback as a preferences graph where the nodes of the graph are the labels and edges express preferences over labels we tackle the learning problem by defining a loss function for comparing a predicted graph with a feedback graph this loss is materialized by decomposing the feedback graph into bipartite sub graphs we then adopt the maximum margin framework which leads to a quadratic optimization problem with linear constraints while the size of the problem grows quadratically with the number of the nodes in the feedback graph we derive a problem of a significantly smaller size and prove that it attains the same minimum we then describe an efficient algorithm called sopopo for solving the reduced problem by employing a soft projection onto the polyhedron defined by a reduced set of constraints we also describe and analyze a wrapper procedure for batch learning when multiple graphs are provided for training we conclude with a set of experiments which show significant improvements in run time over a state of the art interior point algorithm

on the consistency of ordinal regression methods many of the ordinal regression models that have been proposed in the literature can be seen as methods that minimize a convex surrogate of the zero one absolute or squared loss functions a key property that allows to study the statistical implications of such approximations is that of fisher consistency fisher consistency is a desirable property for surrogate loss functions and implies that in the population setting i e if the probability distribution that generates the data were available then optimization of the surrogate would yield the best possible model in this paper we will characterize the fisher consistency of a rich family of surrogate loss functions used in the context of ordinal regression including support vector ordinal regression orboosting and least absolute deviation we will see that for a family of surrogate loss functions that subsumes support vector ordinal regression and orboosting consistency can be fully characterized by the derivative of a real valued function at zero as happens for convex margin based surrogates in binary classification we also derive excess risk bounds for a surrogate of the absolute error that generalize existing risk bounds for binary classification finally our analysis suggests a novel surrogate of the squared error loss we compare this novel surrogate with competing approaches on 9 different datasets our method shows to be highly competitive in practice outperforming the least squares loss on 7 out of 9 datasets

negative correlation ensemble learning for ordinal regression in this paper two neural network threshold ensemble models are proposed for ordinal regression problems for the first ensemble method the thresholds are fixed a priori and are not modified during training the second one considers the thresholds of each member of the ensemble as free parameters allowing their modification during the training process this is achieved through a reformulation of these tunable thresholds which avoids the constraints they must fulfill for the ordinal regression problem during training diversity exists in different projections generated by each member is taken into account for the parameter updating this diversity is promoted in an explicit way using a diversity encouraging error function extending the well known negative correlation learning framework to the area of ordinal regression and inheriting many of its good properties experimental results demonstrate that the proposed algorithms can achieve competitive generalization performance when considering four ordinal regression metrics

online markov decision processes we consider a markov decision process mdp setting in which the reward function is allowed to change after each time step possibly in an adversarial manner yet the dynamics remain fixed similar to the experts setting we address the question of how well an agent can do when compared to the reward achieved under the best stationary policy over time we provide efficient algorithms which have regret bounds with no dependence on the size of state space instead these bounds depend only on a certain horizon time of the process and logarithmically on the number of actions

extreme value distributions in chaotic dynamics a theory of extremes is developed for chaotic dynamical systems and illustrated on representative models of fully developed chaos and intermitent chaos the cumulative distribution and its associated density for the largest value occurring in a data set for monotonically increasing or decreasing sequences and for local maxima are evaluated both analytically and numerically substantial differences from the classical statistical theory of extremes are found arising from the deterministic origin of the underlying dynamics

resonances of the cusp family we study a family of chaotic maps with limit cases the tent map and the cusp map the cusp family we discuss the spectral properties of the corresponding frobenius perron operator in different function spaces including spaces of analytical functions and study numerically the eigenvalues and eigenfunctions

online ranking by projecting we discuss the problem of ranking instances in our framework each instance is associated with a rank or a rating which is an integer in 1 to k our goal is to find a rank prediction rule that assigns each instance a rank that is as close as possible to the instance s true rank we discuss a group of closely related online algorithms analyze their performance in the mistake bound model and prove their correctness we describe two sets of experiments with synthetic data and with the eachmovie data set for collaborative filtering in the experiments we performed our algorithms outperform online algorithms for regression and classification applied to ranking

optimal rates for zero order convex optimization the power of two function evaluations we consider derivative free algorithms for stochastic and nonstochastic convex optimization problems that use only function values rather than gradients focusing on nonasymptotic bounds on convergence rates we show that if pairs of function values are available algorithms for d dimensional optimization that use gradient estimates based on random perturbations suffer a factor of at most sqrt d in convergence rate over traditional stochastic gradient methods we establish such results for both smooth and nonsmooth cases sharpening previous analyses that suggested a worse dimension dependence and extend our results to the case of multiple m ge 2 evaluations we complement our algorithmic development with information theoretic lower bounds on the minimax convergence rate of such problems establishing the sharpness of our achievable results up to constant sometimes logarithmic factors

online learning of complex prediction problems using simultaneous projections we describe and analyze an algorithmic framework for online classification where each online trial consists of multiple prediction tasks that are tied together we tackle the problem of updating the online predictor by defining a projection problem in which each prediction task corresponds to a single linear constraint these constraints are tied together through a single slack parameter we then introduce a general method for approximately solving the problem by projecting simultaneously and independently on each constraint which corresponds to a prediction sub problem and then averaging the individual solutions we show that this approach constitutes a feasible albeit not necessarily optimal solution of the original projection problem we derive concrete simultaneous projection schemes and analyze them in the mistake bound model we demonstrate the power of the proposed algorithm in experiments with synthetic data and with multiclass text categorization tasks

statistical properties of time reversible triangular maps of the square time reversal symmetric triangular maps of the unit square are introduced with the property that the time evolution of one of their two variables is determined by a piecewise expanding map of the unit interval we study their statistical properties and establish the conditions under which their equilibrium measures have a product structure i e factorises in a symmetric form when these conditions are not verified the equilibrium measure does not have a product form and therefore provides additional information on the statistical properties of theses maps this is the case of anti symmetric cusp maps which have an intermittent fixed point and yet have uniform invariant measures on the unit interval we construct the invariant density of the corresponding two dimensional triangular map and prove that it exhibits a singularity at the intermittent fixed point

generalized dimensions large deviations and the distribution of rare events abstract generalized dimensions of multifractal measures are usually seen as static objects related to the scaling properties of suitable partition functions or moments of measures of cells when these measures are invariant for the flow of a chaotic dynamical system generalized dimensions take on a dynamical meaning as they provide the rate function for the large deviations of the first hitting time which is the average time required to connect any two different regions in phase space we prove this result rigorously under a set of stringent assumptions as a consequence the statistics of hitting times provides new algorithms for the computation of the spectrum of generalized dimensions numerical examples presented along with the theory suggest that the validity of this technique reaches far beyond the range covered by the theorem we state our result within the framework of extreme value theory this approach reveals that hitting times are also linked to dynamical indicators such as stability of the motion and local dimensions of the invariant measure this suggests that one can use local dynamical indicators from finite time series to gather information on the multifractal spectrum of generalized dimension we show an application of this technique to experimental data from climate dynamics

the exact invariant density for a cusp shaped return map the square root iteration xt 1 1 2 mod xt mod 1 2 has a linear invariant density and a characteristic liapunov exponent lambda 1 2

reduction from cost sensitive ordinal ranking to weighted binary classification we present a reduction framework from ordinal ranking to binary classification the framework consists of three steps extracting extended examples from the original examples learning a binary classifier on the extended examples with any binary classification algorithm and constructing a ranker from the binary classifier based on the framework we show that a weighted 0 1 loss of the binary classifier upper bounds the mislabeling cost of the ranker both error wise and regret wise our framework allows not only the design of good ordinal ranking algorithms based on well tuned binary classification approaches but also the derivation of new generalization bounds for ordinal ranking from known bounds for binary classification in addition our framework unifies many existing ordinal ranking algorithms such as perceptron ranking and support vector ordinal regression when compared empirically on benchmark data sets some of our newly designed algorithms enjoy advantages in terms of both training speed and generalization performance over existing algorithms in addition the newly designed algorithms lead to better cost sensitive ordinal ranking performance as well as improved listwise ranking performance

the resonance spectrum of the cusp map in the space of analytic functions we prove that the frobenius perron operator u of the cusp map f 1 1 1 1 f x 1 2 x which is an approximation of the poincare section of the lorenz attractor has no analytic eigenfunctions corresponding to eigenvalues different from 0 and 1 we also prove that for any q 0 1 the spectrum of u in the hardy space in the disk z c z q 1 q is the union of the segment 0 1 and some finite or countably infinite set of isolated eigenvalues of finite multiplicity

constrained nonnegative matrix factorization for image representation nonnegative matrix factorization nmf is a popular technique for finding parts based linear representations of nonnegative data it has been successfully applied in a wide range of applications such as pattern recognition information retrieval and computer vision however nmf is essentially an unsupervised method and cannot make use of label information in this paper we propose a novel semi supervised matrix decomposition method called constrained nonnegative matrix factorization cnmf which incorporates the label information as additional constraints specifically we show how explicitly combining label information improves the discriminating power of the resulting matrix decomposition we explore the proposed cnmf method with two cost function formulations and provide the corresponding update solutions for the optimization problems empirical experiments demonstrate the effectiveness of our novel algorithm in comparison to the state of the art approaches through a set of evaluations based on real world applications

local coordinate concept factorization for image representation learning sparse representation of high dimensional data is a state of the art method for modeling data matrix factorization based techniques such as nonnegative matrix factorization and concept factorization cf have shown great advantages in this area especially useful for image representation both of them are linear learning problems and lead to a sparse representation of the images however the sparsity obtained by these methods does not always satisfy locality conditions for example the learned new basis vectors may be relatively far away from the original data thus we may not be able to achieve the optimal performance when using the new representation for other learning tasks such as classification and clustering in this paper we introduce a locality constraint into the traditional cf by requiring the concepts basis vectors to be as close to the original data points as possible each datum can be represented by a linear combination of only a few basis concepts thus our method is able to achieve sparsity and locality simultaneously we analyze the complexity of our novel algorithm and demonstrate the effectiveness in comparison with the state of the art approaches through a set of evaluations based on real world applications

efficient learning with virtual threshold gates abstract we reduce learning simple geometric concept classes to learning disjunctions over exponentially many variables we then apply an online algorithm called winnow whose number of prediction mistakes grows only logarithmically with the number of variables the hypotheses of winnow are linear threshold functions with one weight per variable we find ways to keep the exponentially many weights of winnow implicitly so that the time for the algorithm to compute a prediction and update its virtual weights is polynomial our method can be used to learnd dimensional axis parallel boxes whendis variable and unions ofd dimensional axis parallel boxes whendis constant the worst case number of mistakes of our algorithms for the above classes is optimal to within a constant factor and our algorithms inherit the noise robustness of winnow we think that other online algorithms with multiplicative weight updates whose loss bounds grow logarithmically with the dimension are amenable to our methods