fusion of color local spatial and global frequency information for face recognition this paper presents a novel face recognition method by means of fusing color local spatial and global frequency information specifically the proposed method fuses the multiple features derived from a hybrid color space the gabor image representation the local binary patterns lbp and the discrete cosine transform dct of the input image the novelty of this paper is threefold first a hybrid color space the rc rq color space is constructed by combining the r component image of the rgb color space and the chromatic component images c r and q of the yc bc r and yiq color spaces respectively the rc rq hybrid color space whose component images possess complementary characteristics enhances the discriminating power for face recognition second three effective image encoding methods are proposed for the component images in the rc rq hybrid color space to extract features i a patch based gabor image representation for the r component image ii a multi resolution lbp feature fusion scheme for the c r component image and iii a component based dct multiple face encoding for the q component image finally at the decision level the similarity matrices generated using the three component images in the rc rq hybrid color space are fused using a weighted sum rule experiments on the face recognition grand challenge frgc version 2 experiment 4 show that the proposed method improves face recognition performance significantly in particular the proposed method achieves the face verification rate roc iii curve of 92 43 at the false accept rate of 0 1 compared to the frgc baseline performance of 11 86 face verification rate at the same false accept rate

efficient semi supervised learning on locally informative multiple graphs we address an issue of semi supervised learning on multiple graphs over which informative subgraphs are distributed one application under this setting can be found in molecular biology where different types of gene networks are generated depending upon experiments here an important problem is to annotate unknown genes by using functionally known genes which connect to unknown genes in gene networks in which informative parts vary over networks we present a powerful time efficient approach for this problem by combining soft spectral clustering with label propagation for multiple graphs we demonstrate the effectiveness and efficiency of our approach using both synthetic and real biological datasets

partial auc estimation and regression accurate diagnosis of disease is a critical part of health care new diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased from nondiseased states the partial area under the receiver operating characteristic roc curve is a measure of diagnostic test accuracy we present an interpretation of the partial area under the curve auc which gives rise to a nonparametric estimator this estimator is more robust than existing estimators which make parametric assumptions we show that the robustness is gained with only a moderate loss in efficiency we describe a regression modeling framework for making inference about covariate effects on the partial auc such models can refine knowledge about test accuracy model parameters can be estimated using binary regression methods we use the regression framework to compare two prostate specific antigen biomarkers and to evaluate the dependence of biomarker accuracy on the time prior to clinical diagnosis of prostate cancer

classification and saliency detection by semi supervised low rank representation in the area of pattern recognition low rank representation lrr is an efficient method in recovering the subspace structure of the dataset however lrr is unsupervised without any label information lrr constructs an informative graph which is then combined with the mature graph based semi supervised learning gssl framework to complete the classification task in this paper we propose a new low rank learning method which constructs the low rank representation matrix utilizing label information to obtain a more informative graph this method integrates the low rank graph construction and the label information propagation processes together thus the optimization of the low rank representation and the soft label prediction function are calculated iteratively at the same time we name this method as semi supervised low rank learning sslrl it enhanced the classification performance of traditional lrr graph based ssl by 5 30 and the running time is reduced from hundreds to less than ten seconds based on this method a new outlier detection strategy is presented this strategy succeeds with an auc of at least 93 even if the detection condition of lrr is not satisfied the effectiveness of sslrl is demonstrated in semi supervised classification outlier detection and salient detection tasks these extensive experimental results highlight the outperforming of our method over state of the art methods highlightsa semi supervised version of low rank representation is presented label information based outlier and saliency detection strategies are presented semi supervised low rank learning is efficient and effective in classification semi supervised low rank learning is reliable in outlier detection

iterative boolean combination of classifiers in the roc space an application to anomaly detection with hmms hidden markov models hmms have been shown to provide a high level performance for detecting anomalies in sequences of system calls to the operating system kernel using boolean conjunction and disjunction functions to combine the responses of multiple hmms in the roc space may significantly improve performance over a single best hmm however these techniques assume that the classifiers are conditional independent and their of roc curves are convex these assumptions are violated in most real world applications especially when classifiers are designed using limited and imbalanced training data in this paper the iterative boolean combination ibc technique is proposed for efficient fusion of the responses from multiple classifiers in the roc space it applies all boolean functions to combine the roc curves corresponding to multiple classifiers requires no prior assumptions and its time complexity is linear with the number of classifiers the results of computer simulations conducted on both synthetic and real world host based intrusion detection data indicate that the ibc of responses from multiple hmms can achieve a significantly higher level of performance than the boolean conjunction and disjunction combinations especially when training data are limited and imbalanced the proposed ibc is general in that it can be employed to combine diverse responses of any crisp or soft one or two class classifiers and for wide range of application domains

a multi model selection framework for unknown and or evolutive misclassification cost problems in this paper we tackle the problem of model selection when misclassification costs are unknown and or may evolve unlike traditional approaches based on a scalar optimization we propose a generic multi model selection framework based on a multi objective approach the idea is to automatically train a pool of classifiers instead of one single classifier each classifier in the pool optimizing a particular trade off between the objectives within the context of two class classification problems we introduce the roc front concept as an alternative to the roc curve representation this strategy is applied to the multi model selection of svm classifiers using an evolutionary multi objective optimization algorithm the comparison with a traditional scalar optimization technique based on an auc criterion shows promising results on uci datasets as well as on a real world classification problem

measures of residual risk with connections to regression risk tracking surrogate models and ambiguity measures of residual risk are developed as an extension of measures of risk they view a random variable of interest in concert with an auxiliary random vector that helps to manage predict and mitigate the risk in the original variable residual risk can be exemplified as a quantification of the improved situation faced by a hedging investor compared to that of a single asset investor but the notion reaches further with deep connections emerging with forecasting and generalized regression we establish the fundamental properties in this framework and show that measures of residual risk along with generalized regression can play central roles in the development of risk tuned approximations of random variables in tracking of statistics and in estimation of the risk of conditional random variables the paper ends with dual expressions for measures of residual risk which lead to further insights and a new class of distributionally robust optimization models

analysis of decision boundaries in linearly combined neural classifiers combining or integrating the outputs of several pattern classifiers has led to improved performance in a multitude of applications this paper provides an analytical framework to quantify the improvements in classification results due to combining we show that combining networks linearly in output space reduces the variance of the actual decision region boundaries around the optimum boundary this result is valid under the assumption that the a posteriori probability distributions for each class are locally monotonic around the bayes optimum boundary in the absence of classifier bias the error is shown to be proportional to the boundary variance resulting in a simple expression for error rate improvements in the presence of bias the error reduction expressed in terms of a bias reduction factor is shown to be less than or equal to the reduction obtained in the absence of bias the analysis presented here facilitates the understanding of the relationships among error rates classifier boundary distributions and combining in output space

a new approximate maximal margin classification algorithm a new incremental learning algorithm is described which approximates the maximal margin hyperplane w r t norm p 2 for a set of linearly separable data our algorithm called alma p approximate large margin algorithm w r t norm p takes o p 1 2 2 corrections to separate the data with p norm margin larger than 1 where g is the normalized p norm margin of the data alma p avoids quadratic or higher order programming methods it is very easy to implement and is as fast as on line algorithms such as rosenblatt s perceptron algorithm we performed extensive experiments on both real world and artificial datasets we compared alma 2 i e alma p with p 2 to standard support vector machines svm and to two incremental algorithms the perceptron algorithm and li and long s romma the accuracy levels achieved by alma 2 are superior to those achieved by the perceptron algorithm and romma but slightly inferior to svm s on the other hand alma 2 is quite faster and easier to implement than standard svm training algorithms when learning sparse target vectors alma p with p 2 largely outperforms perceptron like algorithms such as alma 2

regression analysis likelihood error and entropy in a regression with independent and identically distributed normal residuals the log likelihood function yields an empirical form of the mathcal l 2 norm whereas the normal distribution can be obtained as a solution of differential entropy maximization subject to a constraint on the mathcal l 2 norm of a random variable the mathcal l 1 norm and the double exponential laplace distribution are related in a similar way these are examples of an inter regenerative relationship in fact mathcal l 2 norm and mathcal l 1 norm are just particular cases of general error measures introduced by rockafellar et al finance stoch 10 1 51 74 2006 on a space of random variables general error measures are not necessarily symmetric with respect to ups and downs of a random variable which is a desired property in finance applications where gains and losses should be treated differently this work identifies a set of all error measures denoted by mathscr e and a set of all probability density functions pdfs that form inter regenerative relationships through log likelihood and entropy maximization it also shows that m estimators which arise in robust regression but in general are not error measures form inter regenerative relationships with all pdfs in fact the set of m estimators which are error measures coincides with mathscr e on the other hand m estimators are a particular case of l estimators that also arise in robust regression a set of l estimators which are error measures is identified it contains mathscr e and the so called trimmed mathcal l p norms

an online auc formulation for binary classification the area under the roc curve auc provides a good scalar measure of ranking performance without requiring a specific threshold for performance comparison among classifiers auc is useful for imprecise environments since it operates independently with respect to class distributions and misclassification costs a direct optimization of this auc criterion thus becomes a natural choice for binary classifier design however a direct formulation based on the auc criterion would require a high computational cost due to the drastically increasing input pair features in this paper we propose an online learning algorithm to circumvent this computational problem for binary classification different from those conventional recursive formulations the proposed formulation involves a pairwise cost function which pairs up a newly arrived data point with those of opposite class in stored data moreover with incorporation of a sparse learning into the online formulation the computational effort can be significantly reduced our empirical results on three different scales of public databases show promising potential in terms of classification auc accuracy and computational efficiency

the hypervolume under the roc hypersurface of near guessing and near perfect observers in n class classification tasks we express the performance of the n class guessing observer in terms of the n sup 2 n conditional probabilities which make up an n class receiver operating characteristic roc space in a formulation in which sensitivities are eliminated in constructing the roc space equivalent to using false negative fraction and false positive fraction in a two class task we then show that the guessing observer s performance in terms of these conditional probabilities is completely described by a degenerate hypersurface with only n 1 degrees of freedom as opposed to the n sup 2 n 1 required in general to achieve a true hypersurface in such a roc space it readily follows that the hypervolume under such a degenerate hypersurface must be zero when n 2 we then consider a near guessing task that is a task in which the n underlying data probability density functions pdfs are nearly identical controlled by n 1 parameters which may vary continuously to zero at which point the pdfs become identical with this approach we show that the hypervolume under the roc hypersurface of an observer in an n class classification task tends continuously to zero as the underlying data pdfs converge continuously to identity a guessing task the hypervolume under the roc hypersurface of a perfect ideal observer in a task in which the n data pdfs never overlap is also found to be zero in the roc space formulation under consideration this suggests that hypervolume may not be a useful performance metric in n class classification tasks for n 2 despite the utility of the area under the roc curve for two class tasks

a two stage evolutionary algorithm based on sensitivity and accuracy for multi class problems the machine learning community has traditionally used correct classification rates or accuracy c values to measure classifier performance and has generally avoided presenting classification levels of each class in the results especially for problems with more than two classes c values alone are insufficient because they cannot capture the myriad of contributing factors that differentiate the performance of two different classifiers receiver operating characteristic roc analysis is an alternative to solve these difficulties but it can only be used for two class problems for this reason this paper proposes a new approach for analysing classifiers based on two measures c and sensitivity s i e the minimum of accuracies obtained for each class these measures are optimised through a two stage evolutionary process it was conducted by applying two sequential fitness functions in the evolutionary process including entropy e for the first stage and a new fitness function area a for the second stage by using these fitness functions the c level was optimised in the first stage and the s value of the classifier was generally improved without significantly reducing c in the second stage this two stage approach improved s values in the generalisation set whereas an evolutionary algorithm ea based only on the s measure obtains worse s levels and obtained both high c values and good classification levels for each class the methodology was applied to solve 16 benchmark classification problems and two complex real world problems in analytical chemistry and predictive microbiology it obtained promising results when compared to other competitive multi class classification algorithms and a multi objective alternative based on e and s

ideal observers and optimal roc hypersurfaces in n class classification the likelihood ratio or ideal observer decision rule is known to be optimal for two class classification tasks in the sense that it maximizes expected utility or equivalently minimizes the bayes risk furthermore using this decision rule yields a receiver operating characteristic roc curve which is never above the roc curve produced using any other decision rule provided the observer s misclassification rate with respect to one of the two classes is chosen as the dependent variable for the curve i e an inversion of the more common formulation in which the observer s true positive fraction is plotted against its false positive fraction it is also known that for a decision task requiring classification of observations into n classes optimal performance in the expected utility sense is obtained using a set of n 1 likelihood ratios as decision variables in the n class extension of roc analysis the ideal observer performance is describable in terms of an n sup 2 n 1 parameter hypersurface in an n sup 2 n dimensional probability space we show that the result for two classes holds in this case as well namely that the roc hypersurface obtained using the ideal observer decision rule is never above the roc hypersurface obtained using any other decision rule where in our formulation performance is given exclusively with respect to between class error rates rather than within class sensitivities

unsupervised learning of finite mixture models this paper proposes an unsupervised algorithm for learning a finite mixture model from multivariate data the adjective unsupervised is justified by two properties of the algorithm 1 it is capable of selecting the number of components and 2 unlike the standard expectation maximization em algorithm it does not require careful initialization the proposed method also avoids another drawback of em for mixture fitting the possibility of convergence toward a singular estimate at the boundary of the parameter space the novelty of our approach is that we do not use a model selection criterion to choose one among a set of preestimated candidate models instead we seamlessly integrate estimation and model selection in a single algorithm our technique can be applied to any type of parametric mixture model for which it is possible to write an em algorithm in this paper we illustrate it with experiments involving gaussian mixtures these experiments testify for the good performance of our approach

online passive aggressive algorithms we present a family of margin based online learning algorithms for various prediction tasks in particular we derive and analyze algorithms for binary and multiclass categorization regression uniclass prediction and sequence prediction the update steps of our different algorithms are all based on analytical solutions to simple constrained optimization problems this unified view allows us to prove worst case loss bounds for the different algorithms and for the various decision problems based on a single lemma our bounds on the cumulative loss of the algorithms are relative to the smallest loss that can be attained by any fixed hypothesis and as such are applicable to both realizable and unrealizable settings we demonstrate some of the merits of the proposed algorithms in a series of experiments with synthetic and real data sets

likelihood ratio based biometric score fusion multibiometric systems fuse information from different sources to compensate for the limitations in performance of individual matchers we propose a framework for the optimal combination of match scores that is based on the likelihood ratio test the distributions of genuine and impostor match scores are modeled as finite gaussian mixture model the proposed fusion approach is general in its ability to handle 1 discrete values in biometric match score distributions 2 arbitrary scales and distributions of match scores 3 correlation between the scores of multiple matchers and 4 sample quality of multiple biometric sources experiments on three multibiometric databases indicate that the proposed fusion framework achieves consistently high performance compared to commonly used score fusion techniques based on score transformation and classification

bayesian and non bayesian approaches to statistical inference and decision making bayesian and non bayesian approaches to statistical inference and decision making are discussed and compared a pragmatic criterion success in practice as well as logical consistency are emphasized in comparing alternative approaches attention is given to learning models probability concepts prior information axiom systems and selected technical aspects of bayesian and non bayesian approaches

confidence intervals around bayes cost in multi state diagnostic settings to estimate optimal performance a critical feature of diagnostic testing is correctly classifying subjects based upon specified thresholds of some measure the commonly employed youden index determines a test s optimal thresholds by maximizing the correct classification rates for a diagnostic scenario an alternative to the youden index is the cost function bayes cost bc bc determines a test s optimal setting by minimizing the sum of all misclassification rates from the test unlike the youden index bc can consider a priori costs of all the diagnostic outcomes including class specific misclassifications regardless of the number of classes delta method approximate confidence intervals around bc are derived under the assumption of normally distributed classes as a means for quantifying a test s performance and comparing classifiers at their optimal settings in a multi state diagnostic framework a simulation study is conducted to demonstrate the performance of the derived confidence intervals that are found to perform well especially for sample sizes of 50 or larger in each diagnostic class finally the proposed methods are applied to a four class breast tissue classification problem where four possible discriminatory features are compared under varying decision cost structures using the confidence intervals around bc the best feature for classification is selected and the optimal thresholds and their 95 confidence intervals are determined

roc graphs for assessing the ability of a diagnostic marker to detect three disease classes with an umbrella ordering receiver operating characteristic roc curves and the area under these curves are commonly used to assess the ability of a continuous diagnostic marker e g dna methylation markers to correctly classify subjects as having a particular disease or not e g cancer these approaches however are not applicable to settings where the gold standard yields more than two disease states or classes roc surfaces and the volume under the surfaces have been proposed for settings with more than two disease classes these approaches however do not allow one to assess the ability of a marker to differentiate two disease classes from a third disease class without requiring a monotone order for the three disease classes under study that is existing approaches do not accommodate an umbrella ordering of disease classes this article proposes the construction of an roc graph that is applicable for an umbrella ordering furthermore this article proposes that a summary measure for this umbrella roc graph can be used to summarize the classification accuracy and corresponding variance estimates can be obtained using u statistics theory or bootstrap methods the proposed methods are illustrated using data from a study assessing the ability of a dna methylation marker to correctly classify lung specimens into three histologic classes squamous cell carcinoma large cell carcinoma and nontumor lung

partial auc maximization in a linear combination of dichotomizers classifier combination is a useful and common methodology to design an effective classification system a large number of combination rules has been proposed hitherto mostly aimed at minimizing the error rate recently some methods have been presented that are devoted to maximize the area under the roc curve auc a more suitable performance measure when dealing with two class problems with imprecise environment and or imbalanced class priors however there are several applications that do not operate in the complete range of the roc curve but only in particular regions of it in these cases it is better to analyze the performance only in a part of the curve and to use the partial auc pauc this paper presents a new method that aims at maximizing the pauc by means of linear combination of classifiers the effectiveness of the proposed method has been proved on two biometric databases

linear combinations of multiple diagnostic markers abstract the receiver operating characteristic roc curve is a simple and meaningful measure to assess the usefulness of diagnostic markers to use the information carried by multiple markers we note that fisher s linear discriminant function provides a linear combination of markers to maximize the sensitivity over the entire specificity range uniformly under the multivariate normal distribution model with proportional covariance matrices with no restriction on covariance matrices we also provide a solution of the best linear combination of markers in the sense that the area under the roc curve of this combination is maximized among all possible linear combinations we illustrate both situations discussed in the article with a cancer clinical trial data

efficient multiclass roc approximation by decomposition via confusion matrix perturbation analysis receiver operator characteristic roc analysis has become a standard tool in the design and evaluation of two class classification problems it allows for an analysis that incorporates all possible priors costs and operating points which is important in many real problems where conditions are often nonideal extending this to the multiclass case is attractive conferring the benefits of roc analysis to a multitude of new problems even though the roc analysis extends theoretically to the multiclass case the exponential computational complexity as a function of the number of classes is restrictive in this paper we show that the multiclass roc can often be simplified considerably because some roc dimensions are independent of each other we present an algorithm that analyzes interactions between various roc dimensions identifying independent classes and groups of interacting classes allowing the roc to be decomposed the resulting decomposed roc hypersurface can be interrogated in a similar fashion to the ideal case allowing for approaches such as cost sensitive and neyman pearson optimization as well as the volume under the roc an extensive bouquet of examples and experiments demonstrates the potential of this methodology

exploiting the relationships among several binary classifiers via data transformation the structural resemblance among several existing classifiers has motivated us to investigate their underlying relationships by exploring into the mapping solutions of these classifiers we found that they can be linked by simple feature data scaling in other words the key to these relationships lies upon how the replica of feature data are being scaled this finding leads us directly to an exploration of novel classifiers beyond existing settings based on an extensive empirical evaluation we show that the proposed formulation facilitates a tuning capability beyond existing settings for classifier generalization highlightsestablishment of relationship among several binary classifiers exploration of a random projection network for such classifier learning proposal of novel classifier which exploits the established relationship observe the performance of novel classifier on several performance measures

maximum entropy principle with general deviation measures an approach to the shannon and renyi entropy maximization problems with constraints on the mean and law invariant deviation measure for a random variable has been developed the approach is based on the representation of law invariant deviation measures through corresponding convex compact sets of nonnegative concave functions a solution to the problem has been shown to have an alpha concave distribution log concave for shannon entropy for which in the case of comonotone deviation measures an explicit formula has been obtained as an illustration the problem has been solved for several deviation measures including mean absolute deviation mad conditional value at risk cvar deviation and mixed cvar deviation also it has been shown that the maximum entropy principle establishes a one to one correspondence between the class of alpha concave distributions and the class of comonotone deviation measures this fact has been used to solve the inverse problem of finding a corresponding comonotone deviation measure for a given alpha concave distribution

a variable step size lms algorithm a least mean square lms adaptive filter with a variable step size is introduced the step size increases or decreases as the mean square error increases or decreases allowing the adaptive filter to track changes in the system as well as produce a small steady state error the convergence and steady state behavior of the algorithm are analyzed the results reduce to well known results when specialized to the constant step size case simulation results are presented to support the analysis and to compare the performance of the algorithm with the usual lms algorithm and another variable step size algorithm they show that its performance compares favorably with these existing algorithms

upscaling levy motions in porous media with long range correlations bacterial motility has been modeled by levy motions which were upscaled in porous media with drift processes such as levy processes and time changed brownian motion via central limit theorems it was possible to prove the limit theorems and upscale the processes because of the independence of their increments therefore it has not been applied to processes with correlated increments such as fractional brownian motions in this paper the upscaling approach is generalized to porous media with long range correlated processes the processes are modeled by a fractional brownian velocity process and p diffusive position processes that were defined and used to classify diffusion processes by o malley and cushman a renormalization group classification of nonstationary and or infinite second moment diffusive processes j stat phys 146 989 1000 2012 10 1007 s10955 012 0448 3 a few examples of p diffusive processes are discussed by computing the values of the parameter p

the weight hierarchy of higher dimensional hermitian codes studies a class of projective systems and linear codes corresponding to hermitian varieties over finite fields the weight hierarchy also known as the set of generalized hamming weights of the code is calculated the higher weight distribution is also found

sensitivity analysis in applications with deviation risk regret and error measures the envelope formula is obtained for optimization problems with positively homogeneous convex functionals defined on a space of random variables those problems include linear regression with general error measures and optimal portfolio selection with the objective function being either a general deviation measure or a coherent risk measure subject to a constraint on the expected rate of return the obtained results are believed to be novel even for markowitz s mean variance portfolio selection but are far more general and include explicit envelope relationships for the rates of return of portfolios that minimize lower semivariance mean absolute deviation deviation measures of cal l p type and semi cal l p type and conditional value at risk in each case the envelope theorem yields explicit estimates for the absolute value of the difference between deviation risk of optimal portfolios with the unperturbed and perturbed asset probability distributions in terms of a norm of the perturbation

stable non gaussian random processes stochastic models with infinite variance stable random variables on the real line multivariate stable distributions stable stochastic integrals dependence structures of multivariate stable distributions non linear regression complex stable stochastic integrals and harmonizable processes self similar processes chentsov random fields introduction to sample path properties boundedness continuity and oscillations measurability integrability and absolute continuity boundedness and continuity via metric entropy integral representation historical notes and extensions

the kernel least mean square algorithm the combination of the famed kernel trick and the least mean square lms algorithm provides an interesting sample by sample update for an adaptive filter in reproducing kernel hilbert spaces rkhs which is named in this paper the klms unlike the accepted view in kernel methods this paper shows that in the finite training data case the klms algorithm is well posed in rkhs without the addition of an extra regularization term to penalize solution norms as was suggested by kivinen kivinen smola and williamson ldquoonline learning with kernels rdquo ieee transactions on signal processing vol 52 no 8 pp 2165 2176 aug 2004 and smale smale and yao ldquoonline learning algorithms rdquo foundations in computational mathematics vol 6 no 2 pp 145 176 2006 this result is the main contribution of the paper and enhances the present understanding of the lms algorithm with a machine learning perspective the effect of the klms step size is also studied from the viewpoint of regularization two experiments are presented to support our conclusion that with finite data the klms algorithm can be readily used in high dimensional spaces and particularly in rkhs to derive nonlinear stable algorithms with comparable performance to batch regularized solutions

on the statistical analysis of roc curves we introduce a new accuracy index for receiver operating characteristic roc curves namely the partial area under the binormal roc graph over any specified region of interest we propose a simple but general procedure based on a conventional analysis of variance for comparing accuracy indices derived from two or more different modalities the proposed method is related to and compared with existing methodology and is illustrated by results from an experiment on optimization of density and contrast yielded by multiform photographic images used for scintigraphy

three class roc analysis a decision theoretic approach under the ideal observer framework receiver operating characteristic roc analysis is well established in the evaluation of systems involving binary classification tasks however medical tests often require distinguishing among more than two diagnostic alternatives the goal of this work was to develop an roc analysis method for three class classification tasks based on decision theory we developed a method for three class roc analysis in this method the objects were classified by making the decision that provided the maximal utility relative to the other two by making assumptions about the magnitudes of the relative utilities of incorrect decisions we found a decision model that maximized the expected utility of the decisions when using log likelihood ratios as decision variables this decision model consists of a two dimensional decision plane with log likelihood ratios as the axes and a decision structure that separates the plane into three regions moving the decision structure over the decision plane which corresponds to moving the decision threshold in two class roc analysis and computing the true class 1 2 and 3 fractions defined a three class roc surface we have shown that the resulting three class roc surface shares many features with the two class roc curve i e using the log likelihood ratios as the decision variables results in maximal expected utility of the decisions and the optimal operating point for a given diagnostic setting set of relative utilities and disease prevalences lies on the surface the volume under the three class surface vus serves as a figure of merit to evaluate different data acquisition systems or image processing and reconstruction methods when the assumed utility constraints are relevant

random variables monotone relations and convex analysis random variables can be described by their cumulative distribution functions a class of nondecreasing functions on the real line those functions can in turn be identified after the possible vertical gaps in their graphs are filled in with maximal monotone relations such relations are known to be the subdifferentials of convex functions analysis of these connections yields new insights the generalized inversion operation between distribution functions and quantile functions corresponds to graphical inversion of monotone relations in subdifferential terms it corresponds to passing to conjugate convex functions under the legendre fenchel transform among other things this shows that convergence in distribution for sequences of random variables is equivalent to graphical convergence of the monotone relations and epigraphical convergence of the associated convex functions measures of risk that employ quantiles var and superquantiles cvar either individually or in mixtures are illuminated in this way formulas for their calculation are seen from a perspective that reveals how they were discovered the approach leads further to developments in which the superquantiles for a given distribution are interpreted as the quantiles for an overlying superdistribution in this way a generalization of koenker basset error is derived which lays a foundation for superquantile regression as a higher order extension of quantile regression

statistical consistency and asymptotic normality for high dimensional robust m estimators we study theoretical properties of regularized robust m estimators applicable when data are drawn from a sparse high dimensional linear model and contaminated by heavy tailed distributions and or outliers in the additive errors and covariates we first establish a form of local statistical consistency for the penalized regression estimators under fairly mild conditions on the error distribution when the derivative of the loss function is bounded and satisfies a local restricted curvature condition all stationary points within a constant radius of the true regression vector converge at the minimax rate enjoyed by the lasso with sub gaussian errors when an appropriate nonconvex regularizer is used in place of an ell 1 penalty we show that such stationary points are in fact unique and equal to the local oracle solution with the correct support hence results on asymptotic normality in the low dimensional case carry over immediately to the high dimensional setting this has important implications for the efficiency of regularized nonconvex m estimators when the errors are heavy tailed our analysis of the local curvature of the loss function also has useful consequences for optimization when the robust regression function and or regularizer is nonconvex and the objective function possesses stationary points outside the local region we show that as long as a composite gradient descent algorithm is initialized within a constant radius of the true regression vector successive iterates will converge at a linear rate to a stationary point within the local region furthermore the global optimum of a convex regularized robust regression function may be used to obtain a suitable initialization the result is a novel two step procedure that uses a convex m estimator to achieve consistency and a nonconvex m estimator to increase efficiency we conclude with simulation results that corroborate our theoretical findings

ordered multiple class roc analysis with continuous measurements receiver operating characteristic roc curves have been useful in two group classification problems in three and multiple class diagnostic problems an roc surface or hyper surface can be constructed the volume under these surfaces can be used for inference using bootstrap techniques or u statistics theory in this article roc surfaces and hyper surfaces are defined and their behaviour and utility in multi group classification problems is investigated the formulation of the problem is equivalent to what has previously been proposed in the general multi category classification problem but the definition of roc surfaces here is less complex and addresses directly the narrower problem of ordered categories in the three class and by extension the multi class problem applied to continuous and ordinal data non parametric manipulation of both continuous and discrete test data and comparison between two diagnostic tests applied to the same subjects are considered a three group classification example in the context of hiv neurological disease is presented and the results are discussed

cvar superquantile norm stochastic case abstract the concept of conditional value at risk cvar is used in various applications in uncertain environment this paper introduces cvar superquantile norm for a random variable which is by definition cvar of absolute value of this random variable it is proved that cvar norm is indeed a norm in the space of random variables cvar norm is defined in two variations scaled and non scaled l 1 and l infinity norms are limiting cases of the cvar norm in continuous case scaled cvar norm is a conditional expectation of the random variable a similar representation of cvar norm is valid for discrete random variables several properties for scaled and non scaled cvar norm as a function of confidence level were proved dual norm for cvar norm is proved to be the maximum of l 1 and scaled l infinity norms cvar norm as a measure of error is related to a regular risk quadrangle trimmed l1 norm which is a non convex extension for cvar norm is introduced analogously to function l p for p

risk tuning with generalized linear regression a framework is set up in which linear regression as a way of approximating a random variable by other random variables can be carried out in a variety of ways which moreover can be tuned to the needs of a particular model in finance or operations research more broadly although the idea of adapting the form of regression to the circumstances at hand has already found advocates in promoting quantile regression as an alternative to classical least squares approaches it is carried here much farther than that axiomatic concepts of error measure deviation measure and risk measure are coordinated with certain statistics that likewise say something about a random variable problems of regression utilizing these concepts are analyzed and the character of their solutions is explored in a range of examples special attention is paid to parametric forms of regression which arise in connection with factor models it is argued that when different aspects of risk enter an optimization problem different forms of regression ought to be invoked for each of those aspects

the least mean fourth lmf adaptive algorithm and its family new steepest descent algorithms for adaptive filtering and have been devised which allow error minimization in the mean fourth and mean sixth etc sense during adaptation the weights undergo exponential relaxation toward their optimal solutions time constants have been derived and surprisingly they turn out to be proportional to the time constants that would have been obtained if the steepest descent least mean square lms algorithm of widrow and hoff had been used the new gradient algorithms are insignificantly more complicated to program and to compute than the lms algorithm their general form is w j 1 w j 2 mu k epsilon j 2k 1 x j where w j is the present weight vector w j 1 is the next weight vector epsilon j is the present error x j is the present input vector mu is a constant controlling stability and rate of convergence and 2k is the exponent of the error being minimized conditions have been derived for weight vector convergence of the mean and of the variance for the new gradient algorithms the behavior of the least mean fourth lmf algorithm is of special interest in comparing this algorithm to the lms algorithm when both are set to have exactly the same time constants for the weight relaxation process the lmf algorithm under some circumstances will have a substantially lower weight noise than the lms algorithm it is possible therefore that a minimum mean fourth error algorithm can do a better job of least squares estimation than a mean square error algorithm this intriguing concept has implications for all forms of adaptive algorithms whether they are based on steepest descent or otherwise

maximizing the area under the roc curve by pairwise feature combination the majority of the available classification systems focus on the minimization of the classification error rate this is not always a suitable metric specially when dealing with two class problems with skewed classes and cost distributions in this case an effective criterion to measure the quality of a decision rule is the area under the receiver operating characteristic curve auc that is also useful to measure the ranking quality of a classifier as required in many real applications in this paper we propose a nonparametric linear classifier based on the maximization of auc the approach lies on the analysis of the wilcoxon mann whitney statistic of each single feature and on an iterative pairwise coupling of the features for the optimization of the ranking of the combined feature by the pairwise feature evaluation the proposed procedure is essentially different from other classifiers using auc as a criterion experiments performed on synthetic and real data sets and comparisons with previous approaches confirm the effectiveness of the proposed method

least median of squares regression abstract classical least squares regression consists of minimizing the sum of the squared residuals many authors have produced more robust versions of this estimator by replacing the square by something else such as the absolute value in this article a different approach is introduced in which the sum is replaced by the median of the squared residuals the resulting estimator can resist the effect of nearly 50 of contamination in the data in the special case of simple regression it corresponds to finding the narrowest strip covering half of the observations generalizations are possible to multivariate location orthogonal regression and hypothesis testing in linear models

large dimensional analysis of robust m estimators of covariance with outliers a large dimensional characterization of robust m estimators of covariance or scatter is provided under the assumption that the dataset comprises independent essentially gaussian legitimate samples as well as arbitrary deterministic samples referred to as outliers building upon recent random matrix advances in the area of robust statistics we specifically show that the so called maronna m estimator of scatter asymptotically behaves similar to well known random matrices when the population and sample sizes grow together to infinity the introduction of outliers leads the robust estimator to behave asymptotically as the weighted sum of the sample outer products with a constant weight for all legitimate samples and different weights for the outliers a fine analysis of this structure reveals importantly that the propensity of the m estimator to attenuate or enhance the impact of outliers is mostly dictated by the alignment of the outliers with the inverse population covariance matrix of the legitimate samples thus robust m estimators can bring substantial benefits over more simplistic estimators such as the per sample normalized version of the sample covariance matrix which is not capable of differentiating the outlying samples the analysis shows that within the class of maronna s estimators of scatter huber estimator is more favorable in a sense to be defined for rejecting outliers than classical alternatives such as tyler s scale invariant estimator often preferred in the literature in fact the analysis reveals that estimators similar to tyler s run the risk of enhancing instead of mitigating some outliers

finite time approach to equilibrium in a fractional brownian velocity field we consider the solution of the equation r t w r t r 0 r0 0 where w is a fractional brownian motion f b m with the hurst exponent 0 1 we show that for almost all realizations of w the trajectory reaches in finite time the nearest equilibrium point i e zero of the f b m either to the right or to the left of r0 depending on whether w r0 is positive or not after reaching the equilibrium the trajectory stays in it forever the problem is motivated by studying the separation between two particles in a gaussian velocity field which satisfies a local self similarity hypothesis in contrast to the case when the forcing term is a brownian motion then an analogous statement is a consequence of the markov property of the process we show our result using as the principal tools the properties of time reversibility of the law of the f b m see lemma 2 4 below and the small ball estimate of molchan commun math phys 205 1999 97 111

roc curves for regression receiver operating characteristic roc analysis is one of the most popular tools for the visual assessment and understanding of classifier performance in this paper we present a new representation of regression models in the so called regression roc rroc space the basic idea is to represent over estimation against under estimation the curves are just drawn by adjusting a shift a constant that is added or subtracted to the predictions and plays a similar role as a threshold in classification from here we develop the notions of optimal operating condition convexity dominance and explore several evaluation metrics that can be shown graphically such as the area over the rroc curve aoc in particular we show a novel and significant result the aoc is equivalent to the error variance we illustrate the application of rroc curves to resource estimation namely the estimation of software project effort

a graphical analysis of cost sensitive regression problems several efforts have been done to bring roc analysis beyond binary classification especially in regression however the mapping and possibilities of these proposals do not correspond to what we expect from the analysis of operating conditions dominance hybrid methods etc in this paper we present a new representation of regression models in the so called regression roc rroc space the basic idea is to represent over estimation on the x axis and under estimation on the y axis the curves are just drawn by adjusting a shift a constant that is added or subtracted to the predictions and plays a similar role as a threshold in classification from here we develop the notions of optimal operating condition convexity dominance and explore several evaluation metrics that can be shown graphically such as the area over the rroc curve aoc in particular we show a novel and significant result the aoc is equal to the error variance multiplied by a factor which does not depend on the model the derivation of rroc curves with non constant shifts and soft regression models and the relation with cost plots is also discussed

a unified view of performance metrics translating threshold choice into expected classification loss many performance metrics have been introduced in the literature for the evaluation of classification performance each of them with different origins and areas of application these metrics include accuracy unweighted accuracy the area under the roc curve or the roc convex hull the mean absolute error and the brier score or mean squared error with its decomposition into refinement and calibration one way of understanding the relations among these metrics is by means of variable operating conditions in the form of misclassification costs and or class distributions thus a metric may correspond to some expected loss over different operating conditions one dimension for the analysis has been the distribution for this range of operating conditions leading to some important connections in the area of proper scoring rules we demonstrate in this paper that there is an equally important dimension which has so far received much less attention in the analysis of performance metrics this dimension is given by the decision rule which is typically implemented as a threshold choice method when using scoring models in this paper we explore many old and new threshold choice methods fixed score uniform score driven rate driven and optimal among others by calculating the expected loss obtained with these threshold choice methods for a uniform range of operating conditions we give clear interpretations of the 0 1 loss the absolute error the brier score the auc and the refinement loss respectively our analysis provides a comprehensive view of performance metrics as well as a systematic approach to loss minimisation which can be summarised as follows given a model apply the threshold choice methods that correspond with the available information about the operating condition and compare their expected losses in order to assist in this procedure we also derive several connections between the aforementioned performance metrics and we highlight the role of calibration in choosing the threshold choice method

the roc manifold for classification systems we define the roc manifold and cc manifold as duals in a given sense their analysis is required to describe the classification system we propose a mathematical definition based on vector space methods to describe both the roc manifolds for n class classification systems fully describe each system in terms of its misclassifications and by conjunction its correct classifications optimal points which minimize misclassifications can be identified even when costs and prior probabilities differ these manifolds can be used to determine the usefulness of a classification system based on a given performance criterion many performance functionals such as summary statistics preferred for cc manifolds can also be evaluated using the roc manifold under certain constraints examples using the roc manifold and performance functionals to compete classification systems are demonstrated with simulated and applied disease detection data

nonparametric analysis of fingerprint data on large data sets by executing different fingerprint image matching algorithms on large data sets it reveals that the match and non match similarity scores have no specific underlying distribution function thus it requires a nonparametric analysis for fingerprint image matching algorithms on large data sets without any assumption about such irregularly discrete distribution functions a precise receiver operating characteristic roc curve based on the true accept rate tar of the match similarity scores and the false accept rate far of the non match similarity scores can be constructed the area under such an roc curve computed using the trapezoidal rule is equivalent to the mann whitney statistic directly formed from the match and non match similarity scores thereafter the z statistic formulated using the areas under roc curves along with their variances and the correlation coefficient is applied to test the significance of the difference between two roc curves four examples from the extensive testing of commercial fingerprint systems at the national institute of standards and technology are provided the nonparametric approach presented in this article can also be employed in the analysis of other large biometric data sets

an interpretation for the roc curve and inference using glm procedures the accuracy of a medical diagnostic test is often summarized in a receiver operating characteristic roc curve this paper puts forth an interpretation for each point on the roc curve as being a conditional probability of a test result from a random diseased subject exceeding that from a random nondiseased subject this interpretation gives rise to new methods for making inference about roc curves it is shown that inference can be achieved with binary regression techniques applied to indicator variables constructed from pairs of test results one component of the pair being from a diseased subject and the other from a nondiseased subject within the generalized linear model glm binary regression framework roc curves can be estimated and we highlight a new semiparametric estimator covariate effects can also be evaluated with the glm models the methodology is applied to a pancreatic cancer dataset where we use the regression framework to compare two different serum biomarkers asymptotic distribution theory is developed to facilitate inference and to provide insight into factors influencing variability of estimated model parameters

locality regularization embedding for face verification graph embedding ge is a unified framework for dimensionality reduction techniques ge attempts to maximally preserve data locality after embedding for face representation and classification however estimation of true data locality could be severely biased due to limited number of training samples which trigger overfitting problem in this paper a graph embedding regularization technique is proposed to remedy this problem the regularization model dubbed as locality regularization embedding lre adopts local laplacian matrix to restore true data locality based on lre model three dimensionality reduction techniques are proposed experimental results on five public benchmark face datasets such as cmu pie feret orl yale and frgc along with nemenyi post hoc statistical of significant test attest the promising performance of the proposed techniques regularized graph embedding approach for face verification a regularization model adopts local laplacian matrix to restore true data locality based on the proposed regularization model three dimensionality reduction techniques are presented

roc representation for the discriminability of multi classification markers in this paper the receiver operating characteristic roc representation and its accuracy measures are well defined and meaningful assessments for the discriminability of multi classification markers are shown given a set of classifiers c a parameterized system can be used to characterize the corresponding optimal roc manifold a connection with the decision set further leads to a better understanding of some geometric features of optimal roc manifolds and preserves the simplicity in computing the hypervolume under the roc manifold hum in addition it motivates us to address the necessary and sufficient conditions for the existence of the hum to sum up this work provides working scientists with an extension of the two class roc analysis to the multi classification roc analysis in a theoretically sound manner highlightsa proper assessment of performance probabilities is shown to be the optimal roc manifold the necessary and sufficient conditions are addressed for the existence of the hum the scientific groundwork is extended to a more general multi class roc analysis

a proposed design and analysis for comparing digital and analog mammography because randomized trials have shown a reduction in breast cancer mortality analog mammography for the early detection of breast cancer has gained widespread use recently several manufacturers have developed digital mammography which promises great advantages in the storage and transmission of images we were asked to design a study to compare the two types of mammography in terms of their performance for the early detection of breast cancer a standard measure of mammography performance is the receiver operating characteristic roc curve which is a plot of false and true positive rates for each ordered classification of the mammography images methods for study design and data analysis based on roc curves have been well developed for diagnostic tests particularly in radiology but for comparing the performance of mammography for the early detection of breast cancer among asymptomatic women special considerations motivate new designs and methodology first digital mammography may cost substantia

online learning with kernels kernel based algorithms such as support vector machines have achieved considerable success in various problems in batch setting where all of the training data is available in advance support vector machines combine the so called kernel trick with the large margin idea there has been little use of these methods in an online setting suitable for real time applications in this paper we consider online learning in a reproducing kernel hilbert space by considering classical stochastic gradient descent within a feature space and the use of some straightforward tricks we develop simple and computationally efficient algorithms for a wide range of problems such as classification regression and novelty detection in addition to allowing the exploitation of the kernel trick in an online setting we examine the value of large margins for classification in the online setting with a drifting target we derive worst case loss bounds and moreover we show the convergence of the hypothesis to the minimizer of the regularized risk functional we present some experimental results that support the theory as well as illustrating the power of the new algorithms for online novelty detection

measuring and estimating diagnostic accuracy when there are three ordinal diagnostic groups this article studies the problem of measuring and estimating the diagnostic accuracy when there are three ordinal diagnostic groups we use a receiver operating characteristic roc surface to describe the probabilities of correct classifications into three diagnostic groups based on various sets of diagnostic thresholds of a test and propose to use the entire and the partial volume under the surface to measure the diagnostic accuracy mathematical properties and probabilistic interpretations of the proposed measure of diagnostic accuracy are discussed under the assumption of normal distributions of the diagnostic test from three diagnostic groups we present the maximum likelihood estimate to the volume under the roc surface and give the asymptotic variance to the estimate we further propose several asymptotic confidence interval estimates to the volume under the roc surface the performance of these confidence interval estimates is evaluated in terms of attaining the nominal coverage probability based on a simulation study in addition we develop a method of sample size determination to achieve an adequate accuracy of the confidence interval estimate finally we demonstrate the proposed methodology by applying it to the clinical diagnosis of early stage alzheimer s disease based on the neuropsychological database of the washington university alzheimer s disease research center

maximum likelihood estimation of receiver operating characteristic roc curves from continuously distributed data we show that truth state runs in rank ordered data constitute a natural categorization of continuously distributed test results for maximum likelihood ml estimation of roc curves on this basis we develop two new algorithms for fitting binormal roc curves to continuously distributed data a true ml algorithm labroc4 and a quasi ml algorithm labroc5 that requires substantially less computation with large data sets simulation studies indicate that both algorithms produce reliable estimates of the binormal roc curve parameters a and b the roc area index az and the standard errors of those estimates

an extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons in a recently published paper in jmlr dem sar 2006 recommends a set of non parametric statistical tests and procedures which can be safely used for comparing the performance of classifiers over multiple data sets after studying the paper we realize that the paper correctly introduces the basic procedures and some of the most advanced ones when comparing a control method however it does not deal with some advanced topics in depth regarding these topics we focus on more powerful proposals of statistical procedures for comparing n n classifiers moreover we illustrate an easy way of obtaining adjusted and comparable p values in multiple comparison procedures

bayesian estimation and prediction using asymmetric loss functions abstract estimators and predictors that are optimal relative to varian s asymmetric linex loss function are derived for a number of well known models their risk functions and bayes risks are derived and compared with those of usual estimators and predictors it is shown that some usual estimators for example a scalar sample mean or a scalar least squares regression coefficient estimator are inadmissible relative to asymmetric linex loss by providing alternative estimators that dominate them uniformly in terms of risk

maximum of a fractional brownian motion probabilities of small values let b t b 0 0 be a fractional brownian motion i e a gaussian process with the structure function e b t b s 2 t s 0 2 we study the logarithmic asymptotics of pt p b t 1 t t as t where is either the interval 0 1 or a bounded region that contains a vicinity of 0 for the case of multidimensional time it is shown that ln pt d ln t 1 o 1 where d is the dimension of zeroes of b t in the former case and the dimension of time in the latter

selecting features in microarray classification using roc curves we present a new method based on the roc receiver operating characteristic curve to efficiently select a feature subset in classifying a high dimensional microarray dataset with a limited number of observations our method has two steps 1 selecting the most relevant features to the target label using the roc curve and 2 iteratively eliminating a redundant feature using the roc curves the roc curve is strongly related with a non parametric hypothesis testing which must be effective for a dataset with small numerical observations experiments with real datasets revealed the significant performance advantage of our method over two competing feature subset selection methods

leaky lms algorithm mse analysis for gaussian data despite the widespread usage of the leaky lms algorithm there has been no detailed study of its performance this paper presents an analytical treatment of the mean square error mse performance for the leaky lms adaptive algorithm for gaussian input data the common independence assumption regarding w n and x n is also used exact expressions that completely characterize the second moment of the coefficient vector and algorithm steady state excess mse are developed rigorous conditions for mse convergence are also established analytical results are compared with simulation and are shown to agree well

extended kernel recursive least squares algorithm this paper presents a kernelized version of the extended recursive least squares ex krls algorithm which implements for the first time a general linear state model in reproducing kernel hilbert spaces rkhs or equivalently a general nonlinear state model in the input space the center piece of this development is a reformulation of the well known extended recursive least squares ex rls algorithm in rkhs which only requires inner product operations between input vectors thus enabling the application of the kernel property commonly known as the kernel trick the first part of the paper presents a set of theorems that shows the generality of the approach the ex krls is preferable to 1 a standard kernel recursive least squares krls in applications that require tracking the state vector of general linear state space models in the kernel space or 2 an ex rls when the application requires a nonlinear observation and state models the second part of the paper compares the ex krls in nonlinear rayleigh multipath channel tracking and in lorenz system modeling problem we show that the proposed algorithm is able to outperform the standard krls and ex rls in both simulations

sgd qn careful quasi newton stochastic gradient descent the sgd qn algorithm is a stochastic gradient descent algorithm that makes careful use of second order information and splits the parameter update into independently scheduled components thanks to this design sgd qn iterates nearly as fast as a first order stochastic gradient descent but requires less iterations to achieve the same accuracy this algorithm won the wild track of the first pascal large scale learning challenge sonnenburg et al 2008

testing forecast optimality under unknown loss empirical tests of forecast optimality have traditionally been conducted under the assumption of mean squared error loss or some other known loss function in this article we establish new testable properties that hold when the forecaster s loss function is unknown but testable restrictions can be imposed on the data generating process trading off conditions on the data generating process against conditions on the loss function we propose flexible estimation of the forecaster s loss function in situations where the loss depends not only on the forecast error but also on other state variables such as the level of the target variable we apply our results to the problem of evaluating the federal reserve s forecasts of output growth forecast optimality is rejected if the fed s loss depends only on the forecast error however the empirical findings are consistent with forecast optimality provided that overpredictions of output growth are costlier to the fed than underpredictions particularly during per