nonadditive conditional entropy and its significance for local realism based on the form invariance of the structures given by khinchin s axiomatic foundations of information theory and the pseudoadditivity of the tsallis entropy indexed by q the concept of conditional entropy is generalized to the case of nonadditive nonextensive composite systems the proposed nonadditive conditional entropy is classically nonnegative but can be negative in the quantum context indicating its utility for characterizing quantum entanglement a criterion deduced from it for separability of density matrices for validity of local realism is examined in detail by employing a bipartite spin 1 2 system it is found that the strongest criterion is obtained in the limit q

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

performance measures for neyman pearson classification in the neyman pearson np classification paradigm the goal is to learn a classifier from labeled training data such that the probability of a false negative is minimized while the probability of a false positive is below a user specified level alpha isin 0 1 this work addresses the question of how to evaluate and compare classifiers in the np setting simply reporting false positives and false negatives leaves some ambiguity about which classifier is best unlike conventional classification however there is no natural performance measure for np classification we cannot reject classifiers whose false positive rate exceeds a since among other reasons the false positive rate must be estimated from data and hence is not known with certainty we propose two families of performance measures for evaluating and comparing classifiers and suggest one criterion in particular for practical use we then present general learning rules that satisfy performance guarantees with respect to these criteria as in conventional classification the notion of uniform convergence plays a central role and leads to finite sample bounds oracle inequalities consistency and rates of convergence the proposed performance measures are also applicable to the problem of anomaly prediction

kernel methods in machine learning we review machine learning methods employing positive definite kernels these methods formulate learning and estimation problems in a reproducing kernel hilbert space rkhs of functions defined on the data domain expanded in terms of a kernel working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions the latter include nonlinear functions as well as functions defined on nonvectorial data we cover a wide range of methods ranging from binary classifiers to sophisticated methods for estimation with structured data

global positioning of robot manipulators via pd control plus a class of nonlinear integral actions deals with the position control of robot manipulators proposed is a simple class of robot regulators consisting of a linear proportional derivative pd feedback plus an integral action of a nonlinear function of position errors by using lyapunov s direct method and lasalle s invariance principle the authors characterize a class of such nonlinear functions and they provide explicit conditions on the regulator gains to ensure global asymptotic stability these regulators offer an attractive alternative to global regulation compared with the well known partially model based pd control with gravity compensation and pd control with desired gravity compensation

universal approximation using radial basis function networks there have been several recent studies concerning feedforward networks and the problem of approximating arbitrary functionals of a finite number of real variables some of these studies deal with cases in which the hidden layer nonlinearity is not a sigmoid this was motivated by successful applications of feedforward networks with nonsigmoidal hidden layer units this paper reports on a related study of radial basis function rbf networks and it is proved that rbf networks having one hidden layer are capable of universal approximation here the emphasis is on the case of typical rbf networks and the results show that a certain class of rbf networks with the same smoothing factor in each kernel node is broad enough for universal approximation

unified entropic measures of quantum correlations induced by local measurements we introduce quantum correlation measures based on the minimal change in unified entropies induced by local rank one projective measurements divided by a factor that depends on the generalized purity of the system in the case of nonadditive entropies in this way we overcome the issue of the artificial increasing of the value of quantum correlation measures based on nonadditive entropies when an uncorrelated ancilla is appended to the system without changing the computability of our entropic correlation measures with respect to the previous ones moreover we recover as limiting cases the quantum correlation measures based on von neumann and renyi entropies i e additive entropies for which the adjustment factor becomes trivial in addition we distinguish between total and semiquantum correlations and obtain some inequalities between them finally we obtain analytical expressions of the entropic correlation measures for typical quantum bipartite systems

nonextensive triangle equality and other properties of tsallis relative entropy minimization kullback leibler relative entropy has unique properties in cases involving distributions resulting from relative entropy minimization tsallis relative entropy is a one parameter generalization of kullback leibier relative entropy in the nonextensive thermostatistics in this paper we present the properties of tsallis relative entropy minimization and present some differences with the classical case in the representation of such a minimum relative entropy distribution we highlight the use of the q product an operator that has been recently introduced to derive the mathematical structure behind the tsallis statistics one of our main results is the generalization of triangle equality of relative entropy minimization to the nonextensive case

on the convergence rate of l p norm multiple kernel learning we derive an upper bound on the local rademacher complexity of lp norm multiple kernel learning which yields a tighter excess risk bound than global approaches previous local approaches analyzed the case p 1 only while our analysis covers all cases 1 p assuming the different feature mappings corresponding to the different kernels to be uncorrelated we also show a lower bound that shows that the bound is tight and derive consequences regarding excess loss namely fast convergence rates of the order o n 1 a where is the minimum eigenvalue decay rate of the individual kernels

generalization performance of radial basis function networks this paper studies the generalization performance of radial basis function rbf networks using local rademacher complexities we propose a general result on controlling local rademacher complexities with the l 1 metric capacity we then apply this result to estimate the rbf networks complexities based on which a novel estimation error bound is obtained an effective approximation error bound is also derived by carefully investigating the holder continuity of the l p loss function s derivative furthermore it is demonstrated that the rbf network minimizing an appropriately constructed structural risk admits a significantly better learning rate when compared with the existing results an empirical study is also performed to justify the application of our structural risk in model selection

quantum work fluctuations versus macrorealism in terms of non extensive entropies abstract fluctuations of the work performed on a driven quantum system can be characterized by the so called fluctuation theorems the jarzynski relation and the crooks theorem are famous examples of exact equalities characterizing non equilibrium dynamics such statistical theorems are typically formulated in a similar manner in both classical and quantum physics leggett garg inequalities are inspired by the two assumptions referred to as the macroscopic realism and the non invasive measurability together these assumptions are known as the macrorealism in the broad sense quantum mechanics is provably incompatible with restrictions of the leggett garg type it turned out that leggett garg inequalities can be used to distinguish quantum and classical work fluctuations we develop this issue with the use of entropic functions of the tsallis type varying the entropic parameter we are often able to reach more robust detection of violations of the corresponding leggett garg inequalities in reality all measurement devices suffer from losses within the entropic formulation detection inefficiencies can naturally be incorporated into the consideration this question also shows advantages that are provided due to the use of generalized entropies

convex calibration dimension for multiclass loss matrices we study consistency properties of surrogate loss functions for general multiclass learning problems defined by a general multiclass loss matrix we extend the notion of classification calibration which has been studied for binary and multiclass 0 1 classification problems and for certain other specific learning problems to the general multiclass setting and derive necessary and sufficient conditions for a surrogate loss to be calibrated with respect to a loss matrix in this setting we then introduce the notion of convex calibration dimension of a multiclass loss matrix which measures the smallest size of a prediction space in which it is possible to design a convex surrogate that is calibrated with respect to the loss matrix we derive both upper and lower bounds on this quantity and use these results to analyze various loss matrices in particular we apply our framework to study various subset ranking losses and use the convex calibration dimension as a tool to show both the existence and non existence of various types of convex calibrated surrogates for these losses our results strengthen recent results of duchi et al 2010 and calauzenes et al 2012 on the non existence of certain types of convex calibrated surrogates in subset ranking we anticipate the convex calibration dimension may prove to be a useful tool in the study and design of surrogate losses for general multiclass learning problems

dynamics modeling and tracking control of robot manipulators in random vibration environment in this technical brief the problem of modeling and tracking control for the manipulator with multi revolute joints in random vibration environment is considered by analyzing the effect of environment to the mass points and introducing an equivalent stochastic noise process a stochastic hamiltonian dynamic model is constructed to describe the motion of the manipulator based on the constructed model a state feedback backstepping controller in vector form is designed such that the unique solution of the closed loop system is bounded in probability and the mean square of the tracking error converges to an arbitrarily small neighborhood of zero

improving the sample complexity using global data we study the sample complexity of proper and improper learning problems with respect to different q loss functions we improve the known estimates for classes which have relatively small covering numbers in empirical l sub 2 spaces e g log covering numbers which are polynomial with exponent p 2 we present several examples of relevant classes which have a small fat shattering dimension and hence fit our setup the most important of which are kernel machines

multicategory ψ learning in binary classification margin based techniques usually deliver high performance as a result a multicategory problem is often treated as a sequence of binary classifications in the absence of a dominating class this treatment may be suboptimal and may yield poor performance such as for support vector machines svms we propose a novel multicategory generalization of learning that treats all classes simultaneously the new generalization eliminates this potential problem while at the same time retaining the desirable properties of its binary counterpart we develop a statistical learning theory for the proposed methodology and obtain fast convergence rates for both linear and nonlinear learning examples we demonstrate the operational characteristics of this method through a simulation our results indicate that the proposed methodology can deliver accurate class prediction and is more robust against extreme observations than its svm counterpart

analysis of online composite mirror descent algorithm we study the convergence of the online composite mirror descent algorithm which involves a mirror map to reflect the geometry of the data and a convex objective function consisting of a loss and a

rademacher penalties and structural risk minimization we suggest a penalty function to be used in various problems of structural risk minimization this penalty is data dependent and is based on the sup norm of the so called rademacher process indexed by the underlying class of functions sets the standard complexity penalties used in learning problems and based on the vc dimensions of the classes are conservative upper bounds in a probabilistic sense uniformly over the set of all underlying distributions for the penalty we suggest thus for a particular distribution of training examples one can expect better performance of learning algorithms with the data driven rademacher penalties we obtain oracle inequalities for the theoretical risk of estimators obtained by structural minimization of the empirical risk with rademacher penalties the inequalities imply some form of optimality of the empirical risk minimizers we also suggest an iterative approach to structural risk minimization with rademacher penalties in which the hierarchy of classes is not given in advance but is determined in the data driven iterative process of risk minimization we prove probabilistic oracle inequalities for the theoretical risk of the estimators based on this approach as well

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

statistical performance of support vector machines the support vector machine svm algorithm is well known to the computer learning community for its very good practical results the goal of the present paper is to study this algorithm from a statistical perspective using tools of concentration theory and empirical processes our main result builds on the observation made by other authors that the svm can be viewed as a statistical regularization procedure from this point of view it can also be interpreted as a model selection principle using a penalized criterion it is then possible to adapt general methods related to model selection in this framework to study two important points 1 what is the minimum penalty and how does it compare to the penalty actually used in the svm algorithm 2 is it possible to obtain oracle inequalities in that setting for the specific loss function used in the svm algorithm we show that the answer to the latter question is positive and provides relevant insight to the former our result shows that it is possible to obtain fast rates of convergence for svms

smooth discrimination analysis ties f and g can be based on the estimation of the set g x fx gx we consider applications where it is appropriate to assume that the region g has a smooth boundary or belongs to another nonparametric class of sets in particular this assumption makes sense if discrimination is used as a data analytic tool decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered their rates of convergence are obtained we show that these rules achieve optimal rates for estimation of g and optimal rates of convergence for bayes risks an interesting conclusion is that the optimal rates for bayes risks can be very fast in particular faster than the parametric root n rate these fast rates cannot be guaranteed for plug in rules

estimating divergence functionals and the likelihood ratio by convex risk minimization we develop and analyze m estimation methods for divergence functionals and the likelihood ratios of two probability distributions our method is based on a nonasymptotic variational characterization of f divergences which allows the problem of estimating divergences to be tackled via convex empirical risk optimization the resulting estimators are simple to implement requiring only the solution of standard convex programs we present an analysis of consistency and convergence for these estimators given conditions only on the ratios of densities we show that our estimators can achieve optimal minimax rates for the likelihood ratio and the divergence functionals in certain regimes we derive an efficient optimization algorithm for computing our estimates and illustrate their convergence behavior and practical viability by simulations

properties of q entropies basic properties of the q entropy sq q 1 1 1 tr q 0 q 1 for states of a quantum system are established concavity quasi convexity continuity and failure of additivity and subadditivity for composite systems

information theoretic bell inequalities we formulate information theoretic bell inequalities which apply to any pair of widely separated physical systems if local realism holds the two systems must carry information consistent with the inequalities two spin s particles in a state of zero total spin violate these information bell inequalities

classification using geometric level sets a variational level set method is developed for the supervised classification problem nonlinear classifier decision boundaries are obtained by minimizing an energy functional that is composed of an empirical risk term with a margin based loss and a geometric regularization term new to machine learning the surface area of the decision boundary this geometric level set classifier is analyzed in terms of consistency and complexity through the calculation of its e entropy for multicategory classification an efficient scheme is developed using a logarithmic number of decision functions in the number of classes rather than the typical linear number of decision functions geometric level set classification yields performance results on benchmark data sets that are competitive with well established methods

on the bayes risk consistency of regularized boosting methods the probability of error of classification methods based on convex combinations of simple base classifiers by boosting algorithms is investigated the main result of the paper is that certain regularized boosting algorithms provide bayes risk consistent classifiers under the sole assumption that the bayes classifier may be approximated by a convex combination of the base classifiers nonasymptotic distribution free bounds are also developed which offer interesting new insight into how boosting works and help explain its success in practical classification problems

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

generalized quantum entropy the generalized quantum entropies are introduced in this paper some important properties such as nonnegativity continuity and concavity are proved but different from the von neumann entropy the subadditivity and additivity fail for the quantum unified r s entropy in general

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

on bregman distances and divergences of probability measures this paper introduces scaled bregman distances of probability distributions which admit nonuniform contributions of observed events they are introduced in a general form covering not only the distances of discrete and continuous stochastic observations but also the distances of random processes and signals it is shown that the scaled bregman distances extend not only the classical ones studied in the previous literature but also the information divergence and the related wider class of convex divergences of probability measures an information processing theorem is established too but only in the sense of invariance w r t statistically sufficient transformations and not in the sense of universal monotonicity pathological situations where coding can increase the classical bregman distance are illustrated by a concrete example in addition to the classical areas of application of the bregman distances and convex divergences such as recognition classification learning and evaluation of proximity of various features and signals the paper mentions a new application in 3 d exploratory data analysis explicit expressions for the scaled bregman distances are obtained in general exponential families with concrete applications in the binomial poisson and rayleigh families and in the families of exponential processes such as the poisson and diffusion processes including the classical examples of the wiener process and geometric brownian motion

convexity classification and risk bounds many of the classification algorithms developed in the machine learning literature including the support vector machine and boosting can be viewed as minimum contrast methods that minimize a convex surrogate of the 0 1 loss function the convexity makes these algorithms computationally efficient the use of a surrogate however has statistical consequences that must be balanced against the computational virtues of convexity to study these issues we provide a general quantitative relationship between the risk as assessed using the 0 1 loss and the risk as assessed using any nonnegative surrogate loss function we show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function that it satisfies a pointwise form of fisher consistency for classification the relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications we also present a refined version of this result in the

generalization performance of regularized ranking with multiscale kernels the regularized kernel method for the ranking problem has attracted increasing attentions in machine learning the previous regularized ranking algorithms are usually based on reproducing kernel hilbert spaces with a single kernel in this paper we go beyond this framework by investigating the generalization performance of the regularized ranking with multiscale kernels a novel ranking algorithm with multiscale kernels is proposed and its representer theorem is proved we establish the upper bound of the generalization error in terms of the complexity of hypothesis spaces it shows that the multiscale ranking algorithm can achieve satisfactory learning rates under mild conditions experiments demonstrate the effectiveness of the proposed method for drug discovery and recommendation tasks

stability of tsallis entropy and instabilities of renyi and normalized tsallis entropies a basis for q exponential distributions the q exponential distributions which are generalizations of the zipf mandelbrot power law distribution are frequently encountered in complex systems at their stationary states from the viewpoint of the principle of maximum entropy they can apparently be derived from three different generalized entropies the renyi entropy the tsallis entropy and the normalized tsallis entropy accordingly mere fittings of observed data by the q exponential distributions do not lead to identification of the correct physical entropy here stabilities of these entropies i e their behaviors under arbitrary small deformation of a distribution are examined it is shown that among the three the tsallis entropy is stable and can provide an entropic basis for the q exponential distributions whereas the others are unstable and cannot represent any experimentally observable quantities

possible generalization of boltzmann gibbs statistics with the use of a quantity normally scaled in multifractals a generalized form is postulated for entropy namelys q k 1 i 1 w p i q q 1 whereq characterizes the generalization andp i are the probabilities associated withw microscopic configurations w the main properties associated with this entropy are established particularly those corresponding to the microcanonical and canonical ensembles the boltzmann gibbs statistics is recovered as theq 1 limit

process consistency for adaboost recent experiments and theoretical studies show that adaboost can overfit in the limit of large time if running the algorithm forever is suboptimal a natural question is how low can the prediction error be during the process of adaboost we show under general regularity conditions that during the process of adaboost a consistent prediction is generated which has the prediction error approximating the optimal bayes error as the sample size increases this result suggests that while running the algorithm forever can be suboptimal it is reasonable to expect that some regularization method via truncation of the process may lead to a near optimal performance for sufficiently large sample size

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

on the performance of kernel classes we present sharp bounds on the localized rademacher averages of the unit ball in a reproducing kernel hilbert space in terms of the eigenvalues of the integral operator associated with the kernel we use this result to estimate the performance of the empirical minimization algorithm when the base class is the unit ball of the reproducing kernel hilbert space

fundamental properties of tsallis relative entropy fundamental properties for the tsallis relative entropy in both classical and quantum systems are studied as one of our main results we give the parametric extension of the trace inequality between the quantum relative entropy and the minus of the trace of the relative operator entropy given by hiai and petz the monotonicity of the quantum tsallis relative entropy for the trace preserving completely positive linear map is also shown without the assumption that the density operators are invertible the generalized tsallis relative entropy is defined and its subadditivity is shown by its joint convexity moreover the generalized peierls bogoliubov inequality is also proven

consistency of support vector machines and other regularized kernel classifiers it is shown that various classifiers that are based on minimization of a regularized risk are universally consistent i e they can asymptotically learn in every classification task the role of the loss functions used in these algorithms is considered in detail as an application of our general framework several types of support vector machines svms as well as regularization networks are treated our methods combine techniques from stochastics approximation theory and functional analysis

of quantiles and expectiles consistent scoring functions choquet representations and forecast rankings summary in the practice of point prediction it is desirable that forecasters receive a directive in the form of a statistical functional for example forecasters might be asked to report the mean or a quantile of their predictive distributions when evaluating and comparing competing forecasts it is then critical that the scoring function used for these purposes be consistent for the functional at hand in the sense that the expected score is minimized when following the directive we show that any scoring function that is consistent for a quantile or an expectile functional can be represented as a mixture of elementary or extremal scoring functions that form a linearly parameterized family scoring functions for the mean value and probability forecasts of binary events constitute important examples the extremal scoring functions admit appealing economic interpretations of quantiles and expectiles in the context of betting and investment problems the choquet type mixture representations give rise to simple checks of whether a forecast dominates another in the sense that it is preferable under any consistent scoring function in empirical settings it suffices to compare the average scores for only a finite number of extremal elements plots of the average scores with respect to the extremal scoring functions which we call murphy diagrams permit detailed comparisons of the relative merits of competing forecasts

distance weighted discrimination high dimension low sample size statistical analysis is becoming increasingly important in a wide range of applied contexts in such situations the popular support vector machine suffers from data piling at the margin which can diminish generalizability this leads naturally to the development of distance weighted discrimination which is based on second order cone programming a modern computationally intensive optimization method

universally consistent vertex classification for latent positions graphs in this work we show that using the eigen decomposition of the adjacency matrix we can consistently estimate feature maps for latent position graphs with positive definite link function provided that the latent positions are i i d from some distribution f we then consider the exploitation task of vertex classification where the link

on the consistency of ordinal regression methods many of the ordinal regression models that have been proposed in the literature can be seen as methods that minimize a convex surrogate of the zero one absolute or squared loss functions a key property that allows to study the statistical implications of such approximations is that of fisher consistency fisher consistency is a desirable property for surrogate loss functions and implies that in the population setting i e if the probability distribution that generates the data were available then optimization of the surrogate would yield the best possible model in this paper we will characterize the fisher consistency of a rich family of surrogate loss functions used in the context of ordinal regression including support vector ordinal regression orboosting and least absolute deviation we will see that for a family of surrogate loss functions that subsumes support vector ordinal regression and orboosting consistency can be fully characterized by the derivative of a real valued function at zero as happens for convex margin based surrogates in binary classification we also derive excess risk bounds for a surrogate of the absolute error that generalize existing risk bounds for binary classification finally our analysis suggests a novel surrogate of the squared error loss we compare this novel surrogate with competing approaches on 9 different datasets our method shows to be highly competitive in practice outperforming the least squares loss on 7 out of 9 datasets

generalization performance of fisher linear discriminant based on markov sampling fisher linear discriminant fld is a well known method for dimensionality reduction and classification that projects high dimensional data onto a low dimensional space where the data achieves maximum class separability the previous works describing the generalization ability of fld have usually been based on the assumption of independent and identically distributed i i d samples in this paper we go far beyond this classical framework by studying the generalization ability of fld based on markov sampling we first establish the bounds on the generalization performance of fld based on uniformly ergodic markov chain u e m c samples and prove that fld based on u e m c samples is consistent by following the enlightening idea from markov chain monto carlo methods we also introduce a markov sampling algorithm for fld to generate u e m c samples from a given data of finite size through simulation studies and numerical studies on benchmark repository using fld we find that fld based on u e m c samples generated by markov sampling can provide smaller misclassification rates compared to i i d samples

classifying 50 years of bell inequalities since john s bell demonstrated the interest of studying linear combinations of probabilities in relation with the epr paradox in 1964 bell inequalities have lead to numerous developments unfortunately the description of bell inequalities is subject to several degeneracies which make any exchange of information about them unnecessarily hard here we analyze these degeneracies and propose a decomposition for bell like inequalities based on a set of reference expressions which is not affected by them these reference expressions set a common ground for comparing bell inequalities we provide algorithms based on finite group theory to compute this decomposition implementing these algorithms allows us to set up a compendium of reference bell like inequalities available online at www faacets com this website constitutes a platform where registered bell like inequalities can be explored new inequalities can be compared to previously known ones and relevant information on bell inequalities can be added in a collaborative manner this article is part of a special issue of journal of physics a mathematical and theoretical devoted to 50 years of bell s theorem

on ranking and generalization bounds the problem of ranking is to predict or to guess the ordering between objects on the basis of their observed features in this paper we consider ranking estimators that minimize the empirical convex risk we prove generalization bounds for the excess risk of such estimators with rates that are faster than 1 n we apply our results to commonly used ranking algorithms for instance boosting or support vector machines moreover we study the performance of considered estimators on real data sets

robust support vector machines for classification with nonconvex and smooth losses this letter addresses the robustness problem when learning a large margin classifier in the presence of label noise in our study we achieve this purpose by proposing robustified large margin support vector machines the robustness of the proposed robust support vector classifiers rsvc which is interpreted from a weighted viewpoint in this work is due to the use of nonconvex classification losses besides the robustness we also show that the proposed rscv is simultaneously smooth which again benefits from using smooth classification losses the idea of proposing rsvc comes from m estimation in statistics since the proposed robust and smooth classification losses can be taken as one sided cost functions in robust statistics its fisher consistency property and generalization ability are also investigated besides the robustness and smoothness another nice property of rsvc lies in the fact that its solution can be obtained by solving weighted squared hinge loss based support vector machine problems iteratively we further show that in each iteration it is a quadratic programming problem in its dual space and can be solved by using state of the art methods we thus propose an iteratively reweighted type algorithm and provide a constructive proof of its convergence to a stationary point effectiveness of the proposed classifiers is verified on both artificial and real data sets

entropic test of quantum contextuality we study the contextuality of a three level quantum system using classical conditional entropy of measurement outcomes first we analytically construct the minimal configuration of measurements required to reveal contextuality next an entropic contextual inequality is formulated analogous to the entropic bell inequalities derived by braunstein and caves phys rev lett 61 662 1988 that must be satisfied by all noncontextual theories we find optimal measurements for violation of this inequality the approach is easily extendable to higher dimensional quantum systems and more measurements our theoretical findings can be verified in the laboratory with current technology

flexible high dimensional classification machines and their asymptotic properties classification is an important topic in statistics and machine learning with great potential in many real applications in this paper we investigate two popular large margin classification methods support vector machine svm and distance weighted discrimination dwd under two contexts the high dimensional low sample size data and the imbalanced data a unified family of classification machines the flexible assortment machine flame is proposed within which dwd and svm are special cases the flame family helps to identify the similarities and differences between svm and dwd it is well known that many classifiers overfit the data in the high dimensional setting and others are sensitive to the imbalanced data that is the class with a larger sample size overly influences the classifier and pushes the decision boundary towards the minority class svm is resistant to the imbalanced data issue but it overfits high dimensional data sets by showing the undesired data piling phenomenon the dwd method was proposed to improve svm in the high dimensional setting but its decision boundary is sensitive to the imbalanced ratio of sample sizes our flame family helps to understand an intrinsic connection between svm and dwd and provides a trade off between sensitivity to the imbalanced data and overfitting the high dimensional data several asymptotic properties of the flame classifiers are studied simulations and real data applications are investigated to illustrate theoretical findings

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

task space adaptive control of robotic manipulators with uncertainties in gravity regressor matrix and kinematics thus far most research in adaptive control of robotic manipulators has assumed that models of regressor matrix and kinematics are known exactly to overcome these drawbacks we propose in this paper a task space adaptive law for setpoint control of robots with uncertainties in gravity regressor matrix and kinematics in addition we investigate the stability problem when an estimated task space velocity is used in the feedback loop sufficient conditions for choosing the feedback gains gravity regressor and jacobian matrix are presented to guarantee the stability

least square regularized regression in sum space this paper proposes a least square regularized regression algorithm in sum space of reproducing kernel hilbert spaces rkhss for nonflat function approximation and obtains the solution of the algorithm by solving a system of linear equations this algorithm can approximate the low and high frequency component of the target function with large and small scale kernels respectively the convergence and learning rate are analyzed we measure the complexity of the sum space by its covering number and demonstrate that the covering number can be bounded by the product of the covering numbers of basic rkhss for sum space of rkhss with gaussian kernels by choosing appropriate parameters we tradeoff the sample error and regularization error and obtain a polynomial learning rate which is better than that in any single rkhs the utility of this method is illustrated with two simulated data sets and five real life databases

quantum q divergence the quantum q divergence is the nonadditive relative entropy associated with the tsallis entropy indexed by q here its basic properties including the q deformation and quantum group structure convexity and monotonicity are summarized then it is applied to measuring the degree of state purification it is shown that the quantum q divergence is always well defined if the entropic index q is in the range 0 1 whereas the ordinary quantum relative entropy is singular when the reference state is a pure state thus the additive limit q 1 0 does not exist in general

radial basis function networks and complexity regularization in function learning we apply the method of complexity regularization to derive estimation bounds for nonlinear function estimation using a single hidden layer radial basis function network our approach differs from previous complexity regularization neural network function learning schemes in that we operate with random covering numbers and l sub 1 metric entropy making it possible to consider much broader families of activation functions namely functions of bounded variation some constraints previously imposed on the network parameters are also eliminated this way the network is trained by means of complexity regularization involving empirical risk minimization bounds on the expected risk in terms of the sample size are obtained for a large class of loss functions rates of convergence to the optimal loss are also derived

population theory for boosting ensembles tree ensembles are looked at in distribution space that is the limit case of infinite sample size it is shown that the simplest kind of trees is complete in d dimensional l 2 p space if the number of terminal nodes t is greater than d for such trees we show that the adaboost algorithm gives an ensemble converging to the bayes risk

on the relationship between generalization error hypothesis complexity and sample complexity for radial basis functions feedforward networks together with their training algorithms are a class of regression techniques that can be used to learn to perform some task from a set of examples the question of generalization of network performance from a finite training set to unseen data is clearly of crucial importance in this article we first show that the generalization error can be decomposed into two terms the approximation error due to the insufficient representational capacity of a finite sized network and the estimation error due to insufficient information about the target function because of the finite number of samples we then consider the problem of learning functions belonging to certain sobolev spaces with gaussian radial basis functions using the above mentioned decomposition we bound the generalization error in terms of the number of basis functions and number of examples while the bound that we derive is specific for radial basis functions a number of observations deriving from it apply to any approximation technique our result also sheds light on ways to choose an appropriate network architecture for a particular problem and the kinds of problems that can be effectively solved with finite resources i e with a finite number of parameters and finite amounts of data

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

approximate jacobian control with task space damping for robot manipulators in this note we propose two new approximate jacobian control laws with task space damping for setpoint control of robot manipulators the proposed controllers do not require exact knowledge of the jacobian matrix and dynamics of the robots we will show that the end effector s position converges to a desired position in a finite task space even when the kinematics and jacobian matrix are uncertain experimental results are presented to illustrate the performance of the proposed controllers

boosting method for local learning in statistical pattern recognition we propose a local boosting method in classification problems borrowing from an idea of the local likelihood method our proposal local boosting includes a simple device for localization for computational feasibility we proved the bayes risk consistency of the local boosting in the framework of probably approximately correct learning inspection of the proof provides a useful viewpoint for comparing ordinary boosting and local boosting with respect to the estimation error and the approximation error both boosting methods have the bayes risk consistency if their approximation errors decrease to zero compared to ordinary boosting local boosting may perform better by controlling the trade off between the estimation error and the approximation error ordinary boosting with complicated base classifiers or other strong classification methods including kernel machines may have classification performance comparable to local boosting with simple base classifiers for example decision stumps local boosting however has an advantage with respect to interpretability local boosting with simple base classifiers offers a simple way to specify which features are informative and how their values contribute to a classification rule even though locally several numerical studies on real data sets confirm these advantages of local boosting

stabilization of uncertain nonholonomic systems via time varying sliding mode control this note addresses the robust stabilization problem for a general class of nonholonomic systems in the presence of drift uncertainties the control approach developed is based on the combined applications of the sliding mode control technique and nonlinear time varying systems theory first some properties of nonlinear time varying systems are introduced for the purpose of designing sliding mode controller an explicit time varying feedback form is provided to guarantee the existence and uniqueness of periodic time varying solution for the corresponding linear periodic partial differential equation second an explicit discontinuous feedback control law is presented to guarantee the existence of sliding mode the first integrals obtained by the previous periodic partial differential equation are then directly used to determine the switching function the uniform asymptotic stability of the closed loop system is proved via the invariance principle of nonlinear time varying systems finally an example is given to illustrate the proposed approach

jarzynski equality for quantum stochastic maps jarzynski equality and related fluctuation theorems can be formulated for various setups such an equality was recently derived for nonunitary quantum evolutions described by unital quantum operations i e for completely positive trace preserving maps which preserve the maximally mixed state we analyze here a more general case of arbitrary quantum operations on finite systems and derive the corresponding form of the jarzynski equality it contains a correction term due to nonunitality of the quantum map bounds for the relative size of this correction term are established and they are applied for exemplary systems subjected to quantum channels acting on a finite dimensional hilbert space

information gain within nonextensive thermostatistics we discuss the information theoretical foundations of the kullback information gain recently generalized within a nonextensive thermostatistical formalism general properties are studied and in particular a consistent test for measuring the degree of correlation between random variables is proposed in addition minimum entropy distributions are discussed and the h theorem is proved within the generalized context

inegalites de concentration pour les processus empiriques de classes de parties we propose new concentration inequalities for maxima of set indexed empirical processes our approach is based either on entropy inequalities or on martingale methods the improvements we get concern the rate function which is exactly the large deviations rate function of a binomial law in most of the cases

a unified design method for adaptive visual tracking control of robots with eye in hand fixed camera configuration a unified design method is proposed in this paper to handle adaptive visual tracking control problem of robots in the proposed scheme the robot dynamic parameters camera intrinsic and extrinsic parameters and position parameters of feature points are assumed to be uncertain a unified kinematics model is presented to simultaneously cope with kinematics modeling problem of robots with the eye in hand or fixed camera configuration based on the unified kinematics a unified design method is proposed to solve the visual tracking control problem of robots with the eye in hand or fixed camera configuration by using the depth independent interaction matrix framework adaptive laws are derived to handle the unknown parameters lyapunov stability analysis is provided to show asymptotical convergence of image position and velocity tracking errors to show the effectiveness of the proposed unified design method experimental results for both camera configurations are also given

dependence of computational models on input dimension tractability of approximation and optimization tasks the role of input dimension d is studied in approximating in various norms target sets of d variable functions using linear combinations of adjustable computational units results from the literature which emphasize the number n of terms in the linear combination are reformulated and in some cases improved with particular attention to dependence on d for worst case error upper bounds are given in the factorized form d n where is nonincreasing typically n n 1 2 target sets of functions are described for which the function is a polynomial some important cases are highlighted where decreases to zero as d for target functions extent e g the size of domains in rd where they are defined scale e g maximum norms of target functions and smoothness e g the order of square integrable partial derivatives may depend on d and the influence of such dimension dependent parameters on model complexity is considered results are applied to approximation and solution of optimization problems by neural networks with perceptron and gaussian radial computational units

on ψ learning the concept of large margins have been recognized as an important principle in analyzing learning methodologies including boosting neural networks and support vector machines svms however this concept alone is not adequate for learning in nonseparable cases we propose a learning methodology called learning that is derived from a direct consideration of generalization errors we provide a theory for learning and show that it essentially attains the optimal rates of convergence in two learning examples finally results from simulation studies and from breast cancer classification confirm the ability of learning to outperform svm in generalization

tunable measures for information leakage and applications to privacy utility tradeoffs we introduce a tunable measure for information leakage called maximal alpha leakage this measure quantifies the maximal gain of an adversary in inferring any potentially random function of a dataset from a release of the data the inferential capability of the adversary is in turn quantified by a class of adversarial loss functions that we introduce as alpha loss alpha in 1 infty cup infty the choice of alpha determines the specific adversarial action and ranges from refining a belief about any function of the data for alpha 1 to guessing the most likely value for alpha infty while refining the alpha text th moment of the belief for alpha in between maximal alpha leakage then quantifies the adversarial gain under alpha loss over all possible functions of the data in particular for the extremal values of alpha 1 and alpha infty maximal alpha leakage simplifies to mutual information and maximal leakage respectively for alpha in 1 infty this measure is shown to be the arimoto channel capacity of order alpha we show that maximal alpha leakage satisfies data processing inequalities and a sub additivity property thereby allowing for a weak composition result building upon these properties we use maximal alpha leakage as the privacy measure and study the problem of data publishing with privacy guarantees wherein the utility of the released data is ensured via a hard distortion constraint unlike average distortion hard distortion provides a deterministic guarantee of fidelity we show that under a hard distortion constraint for alpha 1 the optimal mechanism is independent of alpha and therefore the resulting optimal tradeoff is the same for all values of alpha 1 finally the tunability of maximal alpha leakage as a privacy measure is also illustrated for binary data with average hamming distortion as the utility measure

adaptive model selection using empirical complexities given n independent replicates of a jointly distributed pair x y rd r we wish to select from a fixed sequence of model classes f1 f2 a deterministic prediction rule f rd r whose risk is small we investigate the possibility of empirically assessing the complexity of each model class that is the actual difficulty of the estimation problem within each class the estimated complexities are in turn used to define an adaptive model selection procedure which is based on complexity penalized empirical risk the available data are divided into two parts the first is used to form an empirical cover of each model class and the second is used to select a candidate rule from each cover based on empirical risk the covering radii are determined empirically to optimize a tight upper bound on the estimation error an estimate is chosen from the list of candidates in order to minimize the sum of class complexity and empirical risk a distinguishing feature of the approach is that the complexity of each model class is assessed empirically based on the size of its empirical cover finite sample performance bounds are established for the estimates and these bounds are applied to several non parametric estimation problems the estimates are shown to achieve a favorable tradeoff between approximation and estimation error and to perform as well as if the distribution dependent complexities of the model classes were known beforehand in addition it is shown that the estimate can be consistent and even possess near optimal rates of convergence when each model class has an infinite vc or pseudo dimension for regression estimation with squared loss we modify our estimate to achieve a faster rate of convergence appears in annals of statistics vol 27 1830 1864 ams 1991 subject classifications primary 62g07 62g20 secondary 62h30

prediction games and arcing algorithms the theory behind the success of adaptive reweighting and combining algorithms arcing such as adaboost freund schapire 1996a 1997 and others in reducing generalization error has not been well understood by formulating prediction as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set existing arcing algorithms are shown to be algorithms for finding good game strategies the minimax theorem is an essential ingredient of the convergence proofs an arcing algorithm is described that converges to the optimal strategy a bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date schapire freund bartlett and lee 1997 offered an explanation of why adaboost works in terms of its ability to produce generally high margins the empirical comparison of adaboost to the optimal arcing algorithm shows that their explanation is n

local rademacher complexities we propose new bounds on the error of learning algorithms in terms of a data dependent notion of complexity the estimates we establish give optimal rates and are based on a local and empirical version of rademacher averages in the sense that the rademacher averages are computed from the data on a subset of functions with small empirical error we present some applications to classification and prediction with convex function classes and with kernel classes in particular

some general properties of unified entropies basic properties of the unified entropies are examined the consideration is mainly restricted to the finite dimensional quantum case bounds in terms of ensembles of quantum states are given both the continuity in fannes sense and stability in lesche s sense are shown for wide ranges of parameters in particular uniform estimates are obtained for the quantum renyi entropies stability properties in the thermodynamic limit are discussed as well it is shown that the unified entropies enjoy both the subadditivity and triangle inequality for a certain range of parameters non decreasing of all the unified entropies under projective measurements is proved

complexity regularization via localized random penalties in this article model selection via penalized empirical loss minimization in nonparametric classification problems is studied data dependent penalties are constructed which are based on estimates of the complexity of a small subclass of each model class containing only those functions with small empirical loss the penalties are novel since those considered in the literature are typically based on the entire model class oracle inequalities using these penalties are established and the advantage of the new penalties over those based on the complexity of the whole model class is demonstrated

information theoretical properties of tsallis entropies a chain rule and a subadditivity for the entropy of type which is one of the nonadditive entropies were derived by daroczy in this paper we study the further relations among tsallis type entropies which are typical nonadditive entropies the chain rule is generalized by showing it for tsallis relative entropy and the nonadditive entropy we show some inequalities related to tsallis entropies especially the strong subadditivity for tsallis type entropies and the subadditivity for the nonadditive entropies the subadditivity and the strong subadditivity naturally lead to define tsallis mutual entropy and tsallis conditional mutual entropy respectively and then we show again chain rules for tsallis mutual entropies we give properties of entropic distances in terms of tsallis entropies finally we show parametrically extended results based on information theory

some properties of regularized kernel methods in regularized kernel methods the solution of a learning problem is found by minimizing functionals consisting of the sum of a data and a complexity term in this paper we investigate some properties of a more general form of the above functionals in which the data term corresponds to the expected risk first we prove a quantitative version of the representer theorem holding for both regression and classification for both differentiable and non differentiable loss functions and for arbitrary offset terms second we show that the case in which the offset space is non trivial corresponds to solving a standard problem of regularization in a reproducing kernel hilbert space in which the penalty term is given by a seminorm finally we discuss the issues of existence and uniqueness of the solution from the specialization of our analysis to the discrete setting it is immediate to establish a connection between the solution properties of sparsity and coefficient boundedness and some properties of the loss function for the case of support vector machines for classification we also obtain a complete characterization of the whole method in terms of the khun tucker conditions with no need to introduce the dual formulation