discriminative face alignment this paper proposes a discriminative framework for efficiently aligning images although conventional active appearance models aams based approaches have achieved some success they suffer from the generalization problem i e how to align any image with a generic model we treat the iterative image alignment problem as a process of maximizing the score of a trained two class classifier that is able to distinguish correct alignment positive class from incorrect alignment negative class during the modeling stage given a set of images with ground truth landmarks we train a conventional point distribution model pdm and a boosting based classifier which acts as an appearance model when tested on an image with the initial landmark locations the proposed algorithm iteratively updates the shape parameters of the pdm via the gradient ascent method such that the classification score of the warped image is maximized we use the term boosted appearance models bams to refer to the learned shape and appearance models as well as our specific alignment method the proposed framework is applied to the face alignment problem using extensive experimentation we show that compared to the aam based approach this framework greatly improves the robustness accuracy and efficiency of face alignment by a large margin especially for unseen data

nonparametric quantile estimations for dynamic smooth coefficient models in this article quantile regression methods are suggested for a class of smooth coefficient time series models we use both local polynomial and local constant fitting schemes to estimate the smooth coefficients in a quantile framework we establish the asymptotic properties of both the local polynomial and local constant estimators for mixing time series also a bandwidth selector based on the nonparametric version of the akaike information criterion is suggested together with a consistent estimate of the asymptotic covariance matrix furthermore the asymptotic behaviors of the estimators at boundaries are examined a comparison of the local polynomial quantile estimator with the local constant estimator is presented a simulation study is carried out to illustrate the performance of estimates an empirical application of the model to real data further demonstrates the potential of the proposed modeling procedures

convex regression with interpretable sharp partitions we consider the problem of predicting an outcome variable on the basis of a small number of covariates using an interpretable yet non additive model we propose convex regression with interpretable sharp partitions crisp for this task crisp partitions the covariate space into blocks in a data adaptive way and fits a mean model within each block unlike other partitioning methods crisp is fit using a non greedy approach by solving a convex optimization problem resulting in low variance fits we explore the properties of crisp and evaluate its performance in a simulation study and on a housing price data set

propagation of probabilities means and variances in mixed graphical association models abstract a scheme is presented for modeling and local computation of exact probabilities means and variances for mixed qualitative and quantitative variables the models assume that the conditional distribution of the quantitative variables given the qualitative is multivariate gaussian the computational architecture is set up by forming a tree of belief universes and the calculations are then performed by local message passing between universes the asymmetry between the quantitative and qualitative variables sets some additional limitations for the specification and propagation structure approximate methods when these are not appropriately fulfilled are sketched it has earlier been shown how to exploit the local structure in the specification of a discrete probability model for fast and efficient computation thereby paving the way for exploiting probability based models as parts of realistic systems for planning and decision support the purpose of this article is to extend this computational s

complexity in the case against accuracy estimation some authors have repeatedly pointed out that the use of the accuracy in particular for comparing classifiers is not adequate the main argument concerns some assumptions of seldom validity or correctness underlying the use of this criterion in this paper we study the computational burden of the accuracy s replacement for building and comparing classifiers using the framework of inductive logic programming replacement is investigated in three ways completion of the accuracy with an additional requirement replacement of the accuracy with a bi criterion recently introduced from statistical decision theory the receiver operating characteristic analysis and replacement of the accuracy by a single criterion we prove very hard results for most of the possible replacements a first result shows that allowing the arbitrary multiplication of clauses appears to be totally useless arbitrary is to be taken in its broadest meaning in particular exponential the second point is the sudden appearance of the negative result which is not a function of the criteria s demands the third point is the equivalence in difficulty of all these different criteria in contrast the single accuracy s optimization appears to be tractable in this framework

interval forecasts and parameter uncertainty forecast intervals generalize point forecasts to represent and incorporate uncertainty forecast intervals calculated from dynamic models typically sidestep the issue of parameter estimation this paper shows how to construct asymptotic forecast intervals which incorporate the uncertainty due to parameter estimation our proposed solution is a simple proportional adjustment to the interval endpoints the adjustment factor depending on the asymptotic variance of the interval estimates our analysis is in the context of a forecasting equation with an error independent of the forecasting variables but with unknown distribution the methods are illustrated with a simulation experiment and an application to the us monthly unemployment rate

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

hubs in space popular nearest neighbors in high dimensional data different aspects of the curse of dimensionality are known to present serious challenges to various machine learning methods and tasks this paper explores a new aspect of the dimensionality curse referred to as hubness that affects the distribution of k occurrences the number of times a point appears among the k nearest neighbors of other points in a data set through theoretical and empirical analysis involving synthetic and real data sets we show that under commonly used assumptions this distribution becomes considerably skewed as dimensionality increases causing the emergence of hubs that is points with very high k occurrences which effectively represent popular nearest neighbors we examine the origins of this phenomenon showing that it is an inherent property of data distributions in high dimensional vector space discuss its interaction with dimensionality reduction and explore its influence on a wide range of machine learning tasks directly or indirectly based on measuring distances belonging to supervised semi supervised and unsupervised learning families

a sufficiently fast algorithm for finding close to optimal clique trees abstract we offer an algorithm that finds a clique tree such that the size of the largest clique is at most 2 1 k where k is the size of the largest clique in a clique tree in which this size is minimized and is the approximation ratio of an approximation algorithm for the 3 way vertex cut problem when 4 3 our algorithm s complexity is o 2 4 67k n poly n and it errs by a factor of 3 67 where poly n is the running time of linear programming this algorithm is extended to find clique trees in which the state space of the largest clique is bounded when k o log n our algorithm yields a polynomial inference algorithm for bayesian networks

prediction in dynamic models with time dependent conditional variances abstract this paper considers forecasting the conditional mean and variance from a single equation dynamic model with autocorrelated disturbances following an arma process and innovations with time dependent conditional heteroskedasticity as represented by a linear garch process expressions for the minimum mse predictor and the conditional mse are presented we also derive the formula for all the theoretical moments of the prediction error distribution from a general dynamic model with garch 1 1 innovations these results are then used in the construction of ex ante prediction confidence intervals by means of the cornish fisher asymptotic expansion an empirical example relating to the uncertainty of the expected depreciation of foreign exchange rates illustrates the usefulness of the results

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

robust estimation for the varying coefficient partially nonlinear models abstract in this paper we propose a robust estimation procedure based on the exponential squared loss esl function for the varying coefficient partially nonlinear model under some conditions the asymptotic properties of proposed estimators are established furthermore we develop a new minorization maximization mm algorithm to calculate the estimates for both non parametric and parametric parts and introduce a data driven procedure to select the tuning parameters simulation studies illustrate that the proposed method is more robust and efficient than the classical least squares technique when there are outliers in the dataset finally we apply the proposed methodology to analyze a real dataset the results reveal that the proposed has better the predictive ability

gaussian influence diagrams an influence diagram is a network representation of probabilistic inference and decision analysis models the nodes correspond to variables that can be either constants uncertain quantities decisions or objectives the arcs reveal probabilistic dependence of the uncertain quantities and information available at the time of the decisions the influence diagram focuses attention on relationships among the variables as a result it is increasingly popular for eliciting and communicating the structure of a decision or probabilistic model this paper develops the framework for assessment and analysis of linear quadratic gaussian models within the influence diagram representation the gaussian influence diagram exploits conditional independence in a model to simplify elicitation of parameters for the multivariate normal distribution it is straightforward to assess and maintain a positive semi definite covariance matrix problems of inference and decision making can be analyzed using simple transformations to the assessed model and these procedures have attractive numerical properties algorithms are also provided to translate between the gaussian influence diagram and covariance matrix representations for the normal distribution

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

logistic evolutionary product unit neural networks innovation capacity of poor guatemalan households a new logistic regression algorithm based on evolutionary product unit pu neural networks is used in this paper to determine the assets that influence the decision of poor households with respect to the cultivation of non traditional crops ntc in the guatemalan highlands in order to evaluate high order covariate interactions pus were considered to be independent variables in product unit neural networks punn analysing two different models either including the initial covariates logistic regression by the product unit and initial covariate model or not logistic regression by the product unit model our results were compared with those obtained using a standard logistic regression model and allow us to interpret the most relevant household assets and their complex interactions when adopting ntc in order to aid in the design of rural policies

a boosting approach for supervised mahalanobis distance metric learning determining a proper distance metric is often a crucial step for machine learning in this paper a boosting algorithm is proposed to learn a mahalanobis distance metric similar to most boosting algorithms the proposed algorithm improves a loss function iteratively in particular the loss function is defined in terms of hypothesis margins and a metric matrix base learner specific to the boosting framework is also proposed experimental results show that the proposed approach can yield effective mahalanobis distance metrics for a variety of data sets and demonstrate the feasibility of the proposed approach

time varying combinations of predictive densities using nonlinear filtering abstract we propose a bayesian combination approach for multivariate predictive densities which relies upon a distributional state space representation of the combination weights several specifications of multivariate time varying weights are introduced with a particular focus on weight dynamics driven by the past performance of the predictive densities and the use of learning mechanisms in the proposed approach the model set can be incomplete meaning that all models can be individually misspecified a sequential monte carlo method is proposed to approximate the filtering and predictive densities the combination approach is assessed using statistical and utility based performance measures for evaluating density forecasts of simulated data us macroeconomic time series and surveys of stock market prices simulation results indicate that for a set of linear autoregressive models the combination strategy is successful in selecting with probability close to one the true model when the model set is complete and it is able to detect parameter instability when the model set includes the true model that has generated subsamples of data also substantial uncertainty appears in the weights when predictors are similar residual uncertainty reduces when the model set is complete and learning reduces this uncertainty for the macro series we find that incompleteness of the models is relatively large in the 1970 s the beginning of the 1980 s and during the recent financial crisis and lower during the great moderation the predicted probabilities of recession accurately compare with the nber business cycle dating model weights have substantial uncertainty attached with respect to returns of the s p 500 series we find that an investment strategy using a combination of predictions from professional forecasters and from a white noise model puts more weight on the white noise model in the beginning of the 1990 s and switches to giving more weight to the professional forecasts over time information on the complete predictive distribution and not just on some moments turns out to be very important above all during turbulent times such as the recent financial crisis more generally the proposed distributional state space representation offers great flexibility in combining densities

adaptive nonparametric regression with the k nearest neighbour fused lasso the fused lasso also known as total variation denoising is a locally adaptive function estimator over a regular grid of design points in this article we extend the fused lasso to settings in which the points do not occur on a regular grid leading to a method for nonparametric regression this approach which we call the formula see text nearest neighbours fused lasso involves computing the formula see text nearest neighbours graph of the design points and then performing the fused lasso over this graph we show that this procedure has a number of theoretical advantages over competing methods specifically it inherits local adaptivity from its connection to the fused lasso and it inherits manifold adaptivity from its connection to the formula see text nearest neighbours approach in a simulation study and an application to flu data we show that excellent results are obtained for completeness we also study an estimator that makes use of an formula see text graph rather than a formula see text nearest neighbours graph and contrast it with the formula see text nearest neighbours fused lasso

local propagation in conditional gaussian bayesian networks this paper describes a scheme for local computation in conditional gaussian bayesian networks that combines the approach of lauritzen and jensen 2001 with some elements of shachter and kenley 1989 message passing takes place on an elimination tree structure rather than the more compact and usual junction tree of cliques this yields a local computation scheme in which all calculations involving the continuous variables are performed by manipulating univariate regressions and hence matrix operations are avoided

marginalization and linear opinion pools abstract suppose a decision maker has asked a group of experts to assess subjective probability distributions over some space and he wishes to find a consensus probability distribution as a function of these the assumption that finding the consensus distribution commutes with marginalization of the distributions implies that the consensus must be found using a linear opinion pool weighted average provided the space being considered contains three or more points some of the consequences of this result are discussed

deviance information criterion for latent variable models and misspecified models abstract deviance information criterion dic has been widely used for bayesian model comparison especially after markov chain monte carlo mcmc is used to estimate candidate models this paper first studies the problem of using dic to compare latent variable models when dic is calculated from the conditional likelihood in particular it is shown that the conditional likelihood approach undermines theoretical underpinnings of dic a new version of dic namely dic l is proposed to compare latent variable models the large sample properties of dic l are studied a frequentist justification of dic l is provided like aic dic l provides an asymptotically unbiased estimator to the expected kullback leibler kl divergence between the dgp and a predictive distribution some popular algorithms such as the em kalman and particle filtering algorithms are introduced to compute dic l for latent variable models moreover this paper studies the problem of using dic to compare misspecified models a new version of dic namely dic m is proposed and it can be regarded as a bayesian version of tic a frequentist justification of dic m is provided under misspecification dic l and dic m are illustrated using asset pricing models and stochastic volatility models

beta negative binomial auto regressions for modelling integer valued time series with extreme observations the paper introduces a general class of heavy tailed auto regressions for modelling integer valued time series with outliers the specification proposed is based on a heavy tailed mixture of negative binomial distributions that features an observation driven dynamic equation for the conditional expectation the existence of a stationary and ergodic solution for the class of auto regressive processes is shown under general conditions the estimation of the model can be easily performed by maximum likelihood given the closed form of the likelihood function the strong consistency and the asymptotic normality of the estimator are formally derived two examples of specifications illustrate the flexibility of the approach and the relevance of the theoretical results in particular a linear dynamic equation and a score driven equation for the conditional expectation are studied the score driven specification is shown to be particularly appealing as it delivers a robust filtering method that attenuates the effect of outliers empirical applications to the series of narcotics trafficking reports in sydney and the euro pound sterling exchange rate illustrate the effectiveness of the method in handling extreme observations

simple linear time algorithms to test chordality of graphs test acyclicity of hypergraphs and selectively reduce acyclic hypergraphs chordal graphs arise naturally in the study of gaussian elimination on sparse symmetric matrices acyclic hypergraphs arise in the study of relational data bases rose tarjan and lueker siam j c

self improved gaps almost everywhere for the agnostic approximation of monomials given a learning sample we focus on the hardness of finding monomials having low error inside the interval bounded below by the smallest error achieved by a monomial the best rule and bounded above by the error of the default class the poorest rule it is well known that when its lower bound is zero it is an easy task to find in linear time a monomial with zero error what we prove is that when this bound is not zero regardless of the location of the default class in 0 1 2 it becomes a huge complexity burden to beat significantly the default class in fact under some complexity theoretical assumptions it may already be hard to beat the trivial approximation ratios even when relaxing the time complexity constraint to be quasi polynomial or sub exponential our results also hold with uniform weights over the examples

bayesian nonparametric calibration and combination of predictive distributions abstractwe introduce a bayesian approach to predictive density calibration and combination that accounts for parameter uncertainty and model set incompleteness through the use of random calibration functionals and random combination weights building on the work of ranjan and gneiting we use infinite beta mixtures for the calibration the proposed bayesian nonparametric approach takes advantage of the flexibility of dirichlet process mixtures to achieve any continuous deformation of linearly combined predictive distributions the inference procedure is based on combination gibbs and slice sampling we provide some conditions under which the proposed probabilistic calibration converges in terms of weak posterior consistency to the true underlying density for both cases of iid and markovian observations this calibration property improves upon the earlier calibration approaches we study the methodology in simulation examples with fat tails and multimodal densities and apply it to density forecasts of dail

accelerating score driven time series models abstract we propose a new class of score driven time series models that allows for a more flexible weighting of score innovations for the filtering of time varying parameters the parameter for the score innovation is made time varying by means of an updating equation that accounts for the autocorrelations of past innovations we provide the theoretical foundations for this acceleration method by showing optimality in terms of reducing kullback leibler divergence the empirical relevance of this accelerated score driven updating method is illustrated in two empirical studies first we include acceleration in the generalized autoregressive conditional heteroskedasticity model we adopt the new model to extract volatility from exchange rates and to analyze daily density forecasts of volatilities from all individual stock return series in the standard poor s 500 index second we consider a score driven acceleration for the time varying mean and use this new model in a forecasting study for us inflation

boosted classification trees and class probability quantile estimation the standard by which binary classifiers are usually judged misclassification error assumes equal costs of misclassifying the two classes or equivalently classifying at the 1 2 quantile of the conditional class probability function p y 1 x boosted classification trees are known to perform quite well for such problems in this article we consider the use of standard off the shelf boosting for two more general problems 1 classification with unequal costs or equivalently classification at quantiles other than 1 2 and 2 estimation of the conditional class probability function p y 1 x we first examine whether the latter problem estimation of p y 1 x can be solved with logitboost and with adaboost when combined with a natural link function the answer is negative both approaches are often ineffective because they overfit p y 1 x even though they perform well as classifiers a major negative point of the present article is the disconnect between class probability estimation and classification next we consider the practice of over under sampling of the two classes we present an algorithm that uses adaboost in conjunction with over under sampling and jittering of the data jous boost this algorithm is simple yet successful and it preserves the advantage of relative protection against overfitting but for arbitrary misclassification costs and equivalently arbitrary quantile boundaries we then use collections of classifiers obtained from a grid of quantiles to form estimators of class probabilities the estimates of the class probabilities compare favorably to those obtained by a variety of methods across both simulated and real data sets

evaluating forecasting performance for interval data from the overlapping parts and the non overlapping parts of the actual intervals and the forecast intervals it should be defined a criterion which is more efficient to evaluate forecasting performance for interval data in this paper we present evaluation techniques for interval time series forecasting the forecast results are compared by the mean squared error of the interval mean relative interval error and mean ratio of exclusive or simulation and empirical studies show that our proposed evaluation techniques for interval forecasting can provide a more objective decision space in interval forecasting to policymakers

on the degrees of freedom of the lasso we study the effective degrees of freedom of the lasso in the framework of stein s unbiased risk estimation sure we show that the number of nonzero coefficients is an unbiased estimate forthe degrees of freedom of the lasso a conclusion that requires no special assumption on the predictors in addition the unbiased estimator is shown to be asymptotically consistent with these results on hand various model selection criteria cp aic and bic are available which along with the lars algorithm provide a principled and efficient approach to obtaining the optimal lasso fit with the computational effort of a single ordinary least squares fit

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

generalised density forecast combinations density forecast combinations are becoming increasingly popular as a means of improving forecast accuracy as measured by a scoring rule in this paper we generalise this literature by letting the combination weights follow more general schemes sieve estimation is used to optimise the score of the generalised density combination where the combination weights depend on the variable one is trying to forecast specific attention is paid to the use of piecewise linear weight functions that let the weights vary by region of the density we analyse these schemes theoretically in monte carlo experiments and in an empirical study our results show that the generalised combinations outperform their linear counterparts

optimal prediction pools a prediction model is any statement of a probability distribution for an outcome not yet observed this study considers the properties of weighted linear combinations of n prediction models or linear pools evaluated using the conventional log predictive scoring rule the log score is a concave function of the weights and in general an optimal linear combination will include several models with positive weights despite the fact that exactly one model has limiting posterior probability one the paper derives several interesting formal results for example a prediction model with positive weight in a pool may have zero weight if some other models are deleted from that pool the results are illustrated using s p 500 returns with prediction models from the arch stochastic volatility and markov mixture families in this example models that are clearly inferior by the usual scoring criteria have positive weights in optimal linear pools and these pools substantially outperform their best components

a unified classification model based on robust optimization a wide variety of machine learning algorithms such as the support vector machine svm minimax probability machine mpm and fisher discriminant analysis fda exist for binary classification the purpose of this letter is to provide a unified classification model that includes these models through a robust optimization approach this unified model has several benefits one is that the extensions and improvements intended for svms become applicable to mpm and fda and vice versa for example we can obtain nonconvex variants of mpm and fda by mimicking perez cruz weston hermann and scholkopf s 2003 extension from convex i svm to nonconvex ei svm another benefit is to provide theoretical results concerning these learning methods at once by dealing with the unified model we give a statistical interpretation of the unified classification model and prove that the model is a good approximation for the worst case minimization of an expected loss with respect to the uncertain probability distribution we also propose a nonconvex optimization algorithm that can be applied to nonconvex variants of existing learning methods and show promising numerical results

how biased is the apparent error rate of a prediction rule abstract a regression model is fitted to an observed set of data how accurate is the model for predicting future observations the apparent error rate tends to underestimate the true error rate because the data have been used twice both to fit the model and to check its accuracy we provide simple estimates for the downward bias of the apparent error rate the theory applies to general exponential family linear models and general measures of prediction error special attention is given to the case of logistic regression on binary data with error rates measured by the proportion of misclassified cases several connected ideas are compared mallows s cp cross validation generalized cross validation the bootstrap and akaike s information criterion

of quantiles and expectiles consistent scoring functions choquet representations and forecast rankings summary in the practice of point prediction it is desirable that forecasters receive a directive in the form of a statistical functional for example forecasters might be asked to report the mean or a quantile of their predictive distributions when evaluating and comparing competing forecasts it is then critical that the scoring function used for these purposes be consistent for the functional at hand in the sense that the expected score is minimized when following the directive we show that any scoring function that is consistent for a quantile or an expectile functional can be represented as a mixture of elementary or extremal scoring functions that form a linearly parameterized family scoring functions for the mean value and probability forecasts of binary events constitute important examples the extremal scoring functions admit appealing economic interpretations of quantiles and expectiles in the context of betting and investment problems the choquet type mixture representations give rise to simple checks of whether a forecast dominates another in the sense that it is preferable under any consistent scoring function in empirical settings it suffices to compare the average scores for only a finite number of extremal elements plots of the average scores with respect to the extremal scoring functions which we call murphy diagrams permit detailed comparisons of the relative merits of competing forecasts

a convergent incremental gradient method with a constant step size an incremental aggregated gradient method for minimizing a sum of continuously differentiable functions is presented the method requires a single gradient evaluation per iteration and uses a constant step size for the case that the gradient is bounded and lipschitz continuous we show that the method visits infinitely often regions in which the gradient is small under certain unimodality assumptions global convergence is established in the quadratic case a global linear rate of convergence is shown the method is applied to distributed optimization problems arising in wireless sensor networks and numerical experiments compare the new method with other incremental gradient methods

on measuring and correcting the effects of data mining and model selection abstract in the theory of linear models the concept of degrees of freedom plays an important role this concept is often used for measurement of model complexity for obtaining an unbiased estimate of the error variance and for comparison of different models i have developed a concept of generalized degrees of freedom gdf that is applicable to complex modeling procedures the definition is based on the sum of the sensitivity of each fitted value to perturbation in the corresponding observed value the concept is nonasymptotic in nature and does not require analytic knowledge of the modeling procedures the concept of gdf offers a unified framework under which complex and highly irregular modeling procedures can be analyzed in the same way as classical linear models by using this framework many difficult problems can be solved easily for example one can now measure the number of observations used in a variable selection process different modeling procedures such as a tree based regression and a

estimating the error rate of a prediction rule improvement on cross validation abstract we construct a prediction rule on the basis of some data and then wish to estimate the error rate of this rule in classifying future observations cross validation provides a nearly unbiased estimate using only the original data cross validation turns out to be related closely to the bootstrap estimate of the error rate this article has two purposes to understand better the theoretical basis of the prediction problem and to investigate some related estimators which seem to offer considerably improved estimation in small samples

interval forecasting an analysis based upon arch quantile estimators abstract in this paper we explore techniques for obtaining interval forecasts based on estimated time series models for processes which may exhibit autoregressive conditional heteroskedasticity arch to deal with the available variety of possible interval forecasts we propose a method for combining these forecasts based on quantile regression techniques our approach is practical rather than theoretical with attention focused directly on obtaining interval forecasts for two u s time series a measure of unemployment and a treasury bill rate we evaluate the performance of our procedures using a variety of diagnostics we find interval estimates which perform reasonably well judged by both in sample and out of sample criteria our experience suggests that a certain amount of care is required in order to obtain useful forecasts

forecasting with factor augmented regression a frequentist model averaging approach abstract this paper considers forecast combination with factor augmented regression in this framework a large number of forecasting models are available varying by the choice of factors and the number of lags we investigate forecast combination across models using weights that minimize the mallows and the leave h out cross validation criteria the unobserved factor regressors are estimated by principle components of a large panel with n predictors over t periods with these generated regressors we show that the mallows and leave h out cross validation criteria are asymptotically unbiased estimators of the one step ahead and multi step ahead mean squared forecast errors respectively provided that n t however the paper does not establish any optimality properties for the methods in contrast to well known results in the literature this result suggests that the generated regressor issue can be ignored for forecast combination without restrictions on the relation between n and t simulations show that the mallows model averaging and leave h out cross validation averaging methods yield lower mean squared forecast errors than alternative model selection and averaging methods such as aic bic cross validation and bayesian model averaging we apply the proposed methods to the us macroeconomic data set in stock and watson 2012 and find that they compare favorably to many popular shrinkage type forecasting methods

grafting fast incremental feature selection by gradient descent in function space we present a novel and flexible approach to the problem of feature selection called grafting rather than considering feature selection as separate from learning grafting treats the selection of suitable features as an integral part of learning a predictor in a regularized learning framework to make this regularized learning process sufficiently fast for large scale problems grafting operates in an incremental iterative fashion gradually building up a feature set while training a predictor model using gradient descent at each iteration a fast gradient based heuristic is used to quickly assess which feature is most likely to improve the existing model that feature is then added to the model and the model is incrementally optimized using gradient descent the algorithm scales linearly with the number of data points and at most quadratically with the number of features grafting can be used with a variety of predictor model classes both linear and non linear and can be used for both classification and regression experiments are reported here on a variant of grafting for classification using both linear and non linear models and using a logistic regression inspired loss function results on a variety of synthetic and real world data sets are presented finally the relationship between grafting stagewise additive modelling and boosting is explored

probability estimation for multi class classification using adaboost abstract it is a general viewpoint that adaboost classifier has excellent performance on classification problems but could not produce good probability estimations in this paper we put forward a theoretical analysis of probability estimation model and present some verification experiments which indicate that adaboost can be used for probability estimation with the theory we suggest some useful measures for using adaboost algorithms properly and then we deduce a probability estimation model for multi class classification by pairwise coupling unlike previous approximate methods we provide an analytical solution instead of a special iterative procedure moreover a new problem that how to get a robust prediction with classifier scores is proposed experiments show that the traditional predict framework which chooses one with the highest score from all classes as the prediction is not always good while our model performs well

geometry based ensembles toward a structural characterization of the classification boundary this article introduces a novel binary discriminative learning technique based on the approximation of the non linear decision boundary by a piece wise linear smooth additive model the decision border is geometrically defined by means of the characterizing boundary points points that belong to the optimal boundary under a certain notion of robustness based on these points a set of locally robust linear classifiers is defined and assembled by means of a tikhonov regularized optimization procedure in an additive model to create a final lambda smooth decision rule as a result a very simple and robust classifier with a strong geometrical meaning and non linear behavior is obtained the simplicity of the method allows its extension to cope with some of nowadays machine learning challenges such as online learning large scale learning or parallelization with linear computational complexity we validate our approach on the uci database finally we apply our technique in online and large scale scenarios and in six real life computer vision and pattern recognition problems gender recognition intravascular ultrasound tissue classification speed traffic sign detection chagas disease severity detection clef classification and action recognition using a 3d accelerometer data the results are promising and this paper opens a line of research that deserves further attention

the dynamics of adaboost cyclic behavior and convergence of margins in order to study the convergence properties of the adaboost algorithm we reduce adaboost to a nonlinear iterated map and study the evolution of its weight vectors this dynamical systems approach allows us to understand adaboost s convergence properties completely in certain cases for these cases we find stable cycles allowing us to explicitly solve for adaboost s output using this unusual technique we are able to show that adaboost does not always converge to a maximum margin combined classifier answering an open question in addition we show that non optimal adaboost where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration may fail to converge to a maximum margin classifier even if optimal adaboost produces a maximum margin also we show that if adaboost cycles it cycles among support vectors i e examples that achieve the same smallest margin

a fast genetic method for inducting descriptive fuzzy models under certain inference mechanisms fuzzy rule bases can be regarded as extended additive models this relationship can be applied to extend some statistical techniques to learn fuzzy models from data the interest in this parallelism is twofold theoretical and practical first extended additive models can be estimated by means of the matching pursuit algorithm which has been related to support vector machines boosting and radial basis neural networks learning this connection can be exploited to better understand the learning of fuzzy models in particular the technique we propose here can be regarded as the counterpart to boosting fuzzy classifiers in the field of fuzzy modeling second since matching pursuit is very efficient in time we can expect to obtain faster algorithms to learn fuzzy rules from data we show that the combination of a genetic algorithm and the backfitting process learns faster than ad hoc methods in certain datasets

on robustness properties of convex risk minimization methods for pattern recognition the paper brings together methods from two disciplines machine learning theory and robust statistics we argue that robustness is an important aspect and we show that many existing machine learning methods based on the convex risk minimization principle have besides other good properties also the advantage of being robust robustness properties of machine learning methods based on convex risk minimization are investigated for the problem of pattern recognition assumptions are given for the existence of the influence function of the classifiers and for bounds on the influence function kernel logistic regression support vector machines least squares and the adaboost loss function are treated as special cases some results on the robustness of such methods are also obtained for the sensitivity curve and the maxbias which are two other robustness criteria a sensitivity analysis of the support vector machine is given

robust variable selection with exponential squared loss robust variable selection procedures through penalized regression have been gaining increased attention in the literature they can be used to perform variable selection and are expected to yield robust estimates however to the best of our knowledge the robustness of those penalized regression procedures has not been well characterized in this article we propose a class of penalized robust regression estimators based on exponential squared loss the motivation for this new procedure is that it enables us to characterize its robustness in a way that has not been done for the existing procedures while its performance is near optimal and superior to some recently developed methods specifically under defined regularity conditions our estimators are consistent and possess the oracle property importantly we show that our estimators can achieve the highest asymptotic breakdown point of 1 2 and that their influence functions are bounded with respect to the outliers in either the response or the covari

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

functional gradient ascent for probit regression this paper proposes two gradient based methods to fit a probit regression model by maximizing the sample log likelihood function using the property of the hessian of the objective function the first method performs weighted least square regression in each iteration of the newton raphson framework resulting in probitboost a boosting like algorithm motivated by the gradient boosting algorithm 10 the second proposed approach maximizes the sample log likelihood function by updating the fitted function a small step in the gradient direction performing gradient ascent in functional space resulting in gradient probitboost we also generalize the algorithms to multi class problems by two strategies one of which is to use the gradient ascent to maximize the multi class sample log likelihood function for fitting all the classifiers simultaneously and the second approach uses the one versus all scheme to reduce the multi class problem to a series of binary classification problems the proposed algorithms are tested on typical classification problems including face detection cancer classification and handwritten digit recognition the results show that compared to the alternative methods the proposed algorithms perform similar or better in terms of testing error rates

classification based on hybridization of parametric and nonparametric classifiers parametric methods of classification assume specific parametric models for competing population densities e g gaussian population densities can lead to linear and quadratic discriminant analysis and they work well when these model assumptions are valid violation in one or more of these parametric model assumptions often leads to a poor classifier on the other hand nonparametric classifiers e g nearest neighbor and kernel based classifiers are more flexible and free from parametric model assumptions but the statistical instability of these classifiers may lead to poor performance when we have small numbers of training sample observations nonparametric methods however do not use any parametric structure of population densities therefore even when one has some additional information about population densities that important information is not used to modify the nonparametric classification rule this paper makes an attempt to overcome these limitations of parametric and nonparametric approaches and combines their strengths to develop some hybrid classification methods we use some simulated examples and benchmark data sets to examine the performance of these hybrid discriminant analysis tools asymptotic results on their misclassification rates have been derived under appropriate regularity conditions

gender discriminating models from facial surface normals in this paper we show how to use facial shape information to construct discriminating models for gender classification we represent facial shapes using 2 5d fields of facial surface normals and investigate three different methods to improve the gender discriminating capacity of the model constructed using the standard eigenspace method the three methods are novel variants of principal geodesic analysis pga namely a weighted pga b supervised weighted pga and c supervised pga our starting point is to define a weight map over the facial surface that indicates the importance of different locations in discriminating gender we show how to compute the relevant weights and how to incorporate the weights into the 2 5d model construction we evaluate the performance of the alternative methods using facial surface normals extracted from 3d range images or recovered from brightness images experimental results demonstrate the effectiveness of our methods moreover the classification accuracy which is as high as 97 demonstrates the effectiveness of using facial shape information for gender classification

boosting method for local learning in statistical pattern recognition we propose a local boosting method in classification problems borrowing from an idea of the local likelihood method our proposal local boosting includes a simple device for localization for computational feasibility we proved the bayes risk consistency of the local boosting in the framework of probably approximately correct learning inspection of the proof provides a useful viewpoint for comparing ordinary boosting and local boosting with respect to the estimation error and the approximation error both boosting methods have the bayes risk consistency if their approximation errors decrease to zero compared to ordinary boosting local boosting may perform better by controlling the trade off between the estimation error and the approximation error ordinary boosting with complicated base classifiers or other strong classification methods including kernel machines may have classification performance comparable to local boosting with simple base classifiers for example decision stumps local boosting however has an advantage with respect to interpretability local boosting with simple base classifiers offers a simple way to specify which features are informative and how their values contribute to a classification rule even though locally several numerical studies on real data sets confirm these advantages of local boosting

predictive density and conditional confidence interval accuracy tests abstract this paper outlines testing procedures for assessing the relative out of sample predictive accuracy of multiple conditional distribution models the tests that are discussed are based on either the comparison of entire conditional distributions or the comparison of predictive confidence intervals we also briefly survey existing related methods in the area of predictive density evaluation including methods based on the probability integral transform and the kullback leibler information criterion the procedures proposed in this paper are similar in many ways to andrews 1997 a conditional kolmogorov test econometrica 65 1097 1128 conditional kolmogorov test and to white s 2000 a reality check for data snooping econometrica 68 1097 1126 reality check in particular a predictive density test is outlined that involves comparing square approximation errors associated with models i i 1 n by constructing weighted averages over u of e f i u z t i f 0 u z t 0 2 where f 0 and f i are true and model i distributions u u and u is a possibly unbounded set on the real line a conditional confidence interval version of this test is also outlined and appropriate bootstrap procedures for obtaining critical values when predictions used in the formation of the test statistics are obtained via rolling and recursive estimation schemes are developed an empirical illustration comparing alternative predictive models for u s inflation is given for the predictive confidence interval test

strictly proper scoring rules prediction and estimation scoring rules assess the quality of probabilistic forecasts by assigning a numerical score based on the predictive distribution and on the event or value that materializes a scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distributionf if he or she issues the probabilistic forecast f rather than g f it is strictly proper if the maximum is unique in prediction problems proper scoring rules encourage the forecaster to make careful assessments and to be honest in estimation problems strictly proper scoring rules provide attractive loss and utility functions that can be tailored to the problem at hand this article reviews and develops the theory of proper scoring rules on general probability spaces and proposes and discusses examples thereof proper scoring rules derive from convex functions and relate to information measures entropy functions and bregman divergences in the case of categorical variables we prove a rigorous version of the

gradient tree boosting for training conditional random fields conditional random fields crfs provide a flexible and powerful model for sequence labeling problems however existing learning algorithms are slow particularly in problems with large numbers of potential input features and feature combinations this paper describes a new algorithm for training crfs via gradient tree boosting in tree boosting the crf potential functions are represented as weighted sums of regression trees which provide compact representations of feature interactions so the algorithm does not explicitly consider the potentially large parameter space as a result gradient tree boosting scales linearly in the order of the markov model and in the order of the feature interactions rather than exponentially as in previous algorithms based on iterative scaling and gradient descent gradient tree boosting also makes it possible to use instance weighting as in c4 5 and surrogate splitting as in cart to handle missing values experimental studies of the effectiveness of these two methods as well as standard imputation and indicator feature methods show that instance weighting is the best method in most cases when feature values are missing at random

probabilistic forecasts calibration and sharpness summary probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions we propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only a simple theoretical framework allows us to distinguish between probabilistic calibration exceedance calibration and marginal calibration we propose and study tools for checking calibration and sharpness among them the probability integral transform histogram marginal calibration plots the sharpness diagram and proper scoring rules the diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the stateline wind energy centre in the us pacific northwest in combination with cross validation or in the time series context our proposal provides very general nonparametric alternatives to the use of information criteria for model diagnostics and model selection

complete subset regressions abstract this paper proposes a new method for combining forecasts based on complete subset regressions for a given set of potential predictor variables we combine forecasts from all possible linear regression models that keep the number of predictors fixed we explore how the choice of model complexity as measured by the number of included predictor variables can be used to trade off the bias and variance of the forecast errors generating a setup akin to the efficient frontier known from modern portfolio theory in an application to predictability of stock returns we find that combinations of subset regressions can produce more accurate forecasts than conventional approaches based on equal weighted forecasts which fail to account for the dimensionality of the underlying models combinations of univariate forecasts or forecasts generated by methods such as bagging ridge regression or bayesian model averaging

degrees of freedom in lasso problems we derive the degrees of freedom of the lasso fit placing no assumptions on the predictor matrix x like the well known result of zou et al 2007 which gives the degrees of freedom of the lasso fit when x has full column rank we express our result in terms of the active set of a lasso solution we extend this result to cover the degrees of freedom of the generalized lasso fit for an arbitrary predictor matrix x and an arbitrary penalty matrix d though our focus is degrees of freedom we establish some intermediate results on the lasso and generalized lasso that may be interesting on their own

logistic regression using covariates obtained by product unit neural network models we propose a logistic regression method based on the hybridation of a linear model and product unit neural network models for binary classification in a first step we use an evolutionary algorithm to determine the basic structure of the product unit model and afterwards we apply logistic regression in the new space of the derived features this hybrid model has been applied to seven benchmark data sets and a new microbiological problem the hybrid model outperforms the linear part and the nonlinear part obtaining a good compromise between them and they perform well compared to several other learning classification techniques we obtain a binary classifier with very promising results in terms of classification accuracy and the complexity of the classifier

a midas approach to modeling first and second moment dynamics abstract we propose a new approach to predictive density modeling that allows for midas effects in both the first and second moments of the outcome specifically our modeling approach allows for midas stochastic volatility dynamics generalizing a large literature focusing on midas effects in the conditional mean and allows the models to be estimated by means of standard gibbs sampling methods when applied to monthly time series on growth in industrial production and inflation we find strong evidence that the introduction of midas effects in the volatility equation leads to improved in sample and out of sample density forecasts our results also suggest that model combination schemes assign high weight to midas in volatility models and produce consistent gains in out of sample predictive performance