consistency of random forests random forests are a learning algorithm proposed by breiman mach learn 45 2001 5 32 that combines several randomized decision trees and aggregates their predictions by averaging despite its wide usage and outstanding practical performance little is known about the mathematical properties of the procedure this disparity between theory and practice originates in the difficulty to simultaneously analyze both the randomization process and the highly data dependent tree structure in the present paper we take a step forward in forest exploration by proving a consistency result for breiman s mach learn 45 2001 5 32 original algorithm in the context of additive regression models our analysis also sheds an interesting light on how random forests can nicely adapt to sparsity 1 introduction random forests are an ensemble learning method for classification and regression that constructs a number of randomized decision trees during the training phase and predicts by averaging the results since its publication in the seminal paper of breiman 2001 the procedure has become a major data analysis tool that performs well in practice in comparison with many standard methods what has greatly contributed to the popularity of forests is the fact that they can be applied to a wide range of prediction problems and have few parameters to tune aside from being simple to use the method is generally recognized for its accuracy and its ability to deal with small sample sizes high dimensional feature spaces and complex data structures the random forest methodology has been successfully involved in many practical problems including air quality prediction winning code of the emc data science global hackathon in 2012 see http www kaggle com c dsg hackathon chemoinformatics svetnik et al 2003 ecology prasad iverson and liaw 2006 cutler et al 2007 3d

a stochastic maximum principle for systems with jumps with applications to finance abstract we consider a stochastic control problem with linear dynamics with jumps convex cost criterion and convex state constraint in which the control enters the drift the diffusion and the jump coefficients we allow these coefficients to be random and do not impose any l p bounds on the control we obtain a stochastic maximum principle for this model that provides both necessary and sufficient conditions of optimality this is the first version of the stochastic maximum principle that covers the consumption investment problem in which there are jumps in the price system

recursive partitioning for heterogeneous causal effects in this paper we propose methods for estimating heterogeneity in causal effects in experimental and observational studies and for conducting hypothesis tests about the magnitude of differences in treatment effects across subsets of the population we provide a data driven approach to partition the data into subpopulations that differ in the magnitude of their treatment effects the approach enables the construction of valid confidence intervals for treatment effects even with many covariates relative to the sample size and without sparsity assumptions we propose an honest approach to estimation whereby one sample is used to construct the partition and another to estimate treatment effects for each subpopulation our approach builds on regression tree methods modified to optimize for goodness of fit in treatment effects and to account for honest estimation our model selection criterion anticipates that bias will be eliminated by honest estimation and also accounts for the effect of making additional splits on the variance of treatment effect estimates within each subpopulation we address the challenge that the ground truth for a causal effect is not observed for any individual unit so that standard approaches to cross validation must be modified through a simulation study we show that for our preferred method honest estimation results in nominal coverage for 90 confidence intervals whereas coverage ranges between 74 and 84 for nonhonest approaches honest estimation requires estimating the model with a smaller sample size the cost in terms of mean squared error of treatment effects for our preferred method ranges between 7 22

approximation dans les espaces metriques et theorie de l estimation we investigate the relations between the speed of estimation and the metric structure of the parameter space especially in the case when its metric dimension is infinite given some distance d on generally hellinger distance in the case of n i i d variables we consider the minimax risk for n observations r n q mathop inf sup limits t n theta in theta text ie theta d q theta t n t n being any estimate of we shall look for functions r such that for positive constants c 1 q and c 2 q c 1 r q n r n q c2 r q n r n is the speed of estimation and we shall show under fairly general conditions including i i d variables and regular cases of markov chains and stationnary gaussian processes that r n is determined up to multiplicative constants by the metric structure of we shall also give a construction for some sort of universal estimates the risk of which is bounded by c 2 r q n in all cases where the preceding theory applies

stochastic maximum principle for optimal control with multiple priors abstract the necessary condition is derived for optimal control with multiple priors which are mutually singular the tool we use is the theory of g expectation

lower bounds for the minimax risk using f divergences and applications lower bounds involving f divergences between the underlying probability measures are proved for the minimax risk in estimation problems our proofs just use simple convexity facts special cases and straightforward corollaries of our bounds include well known inequalities for establishing minimax lower bounds such as fano s inequality pinsker s inequality and inequalities based on global entropy conditions two applications are provided a new minimax lower bound for the reconstruction of convex bodies from noisy support function measurements and a different proof of a recent minimax lower bound for the estimation of a covariance matrix

necessary conditions for optimal control of stochastic systems with random jumps a maximum principle is proved for optimal controls of stochastic systems with random jumps the control is allowed to enter into both diffusion and jump terms the form of the maximum principle turns out to be quite different from the one corresponding to the pure diffusion system the word pure here means the absence of the jump term in calculating the first order coefficient for the cost variation only a property for lebesgue integrals of scalar valued functions in the real number space cal r is used this shows that there is no essential difference between deterministic and stochastic systems as far as the derivation of maximum principles is concerned

quasi oracle estimation of heterogeneous treatment effects flexible estimation of heterogeneous treatment effects lies at the heart of many statistical challenges such as personalized medicine and optimal resource allocation in this paper we develop a general class of two step algorithms for heterogeneous treatment effect estimation in observational studies we first estimate marginal effects and treatment propensities in order to form an objective function that isolates the causal component of the signal then we optimize this data adaptive objective function our approach has several advantages over existing methods from a practical perspective our method is flexible and easy to use in both steps we can use any loss minimization method e g penalized regression deep neural networks or boosting moreover these methods can be fine tuned by cross validation meanwhile in the case of penalized kernel regression we show that our method has a quasi oracle property even if the pilot estimates for marginal effects and treatment propensities are not particularly accurate we achieve the same error bounds as an oracle who has a priori knowledge of these two nuisance components we implement variants of our approach based on penalized regression kernel ridge regression and boosting in a variety of simulation setups and find promising performance relative to existing baselines

the central role of the propensity score in observational studies for causal effects abstract the results of observational studies are often disputed because of nonrandom treatment assignment for example patients at greater risk may be overrepresented in some treatment group this paper discusses the central role of propensity scores and balancing scores in the analysis of observational studies the propensity score is the estimated conditional probability of assignment to a particular treatment given a vector of observed covariates both large and small sample theory show that adjustment for the scalar propensity score is sufficient to remove bias due to all observed covariates applications include matched sampling on the univariate propensity score which is equal percent bias reducing under more general conditions than required for discriminant matching multivariate adjustment by subclassification on balancing scores where the same subclasses are used to estimate treatment effects for all outcome variables and in all subpopulations and visual representation of multivariate adjustment by a two dimensional plot author

rates of convergence of minimum distance estimators and kolmogorov s entropy on construit des estimateurs robustes et on montre que le taux de convergence resultant depend naturellement d une fonction entropie

estimation of a regression function by maxima of minima of linear functions in this paper estimation of a regression function from independent and identically distributed random variables is considered estimates are defined by minimization of the empirical l2 risk over a class of functions which are defined as maxima of minima of linear functions results concerning the rate of convergence of the estimates are derived in particular it is shown that for smooth regression functions satisfying the assumption of single index models the estimate is able to achieve up to some logarithmic factor the corresponding optimal one dimensional rate of convergence hence under these assumptions the estimate is able to circumvent the so called curse of dimensionality the small sample behavior of the estimates is illustrated by applying them to simulated data

estimation and inference of heterogeneous treatment effects using random forests many scientific and engineering challenges ranging from personalized medicine to customized marketing recommendations require an understanding of treatment effect heterogeneity in this article we

deformations of dihedral 2 group extensions of fields given a g galois extension of number fields l k we ask whether it is a specialization of a regular g galois cover of pk this is the inverse of the usual use of the hilbert irreducibility theorem in the inverse galois problem we show that for many groups such arithmetic liftings exist by observing that the existence of generic extensions implies the arithmetic lifting property we explicitly construct generic extensions for dihedral 2 groups under certain assumptions on the base field k we also show that dihedral groups of order 8 and 16 have generic extensions over any base field k with characteristic different from 2

probability inequalities for likelihood ratios and convergence rates of sieve mles let y 1 y n be independent identically distributed with density p 0 and let f be a space of densities we show that the supremum of the likelihood ratios i 1 n p y i p 0 y i where the supremum is over p f with p l 2 p 0 1 2 2 e is exponentially small with probability exponentially close to 1 the exponent is proportional to ne 2 the only condition required for this to hold is that e exceeds a value determined by the bracketing hellinger entropy of f a similar inequality also holds if we replace f by f n and p 0 by q n where q n is an approximation to p 0 in a suitable sense these results are applied to establish rates of convergence of sieve mles furthermore weak conditions are given under which the optimal rate e n defined by h e n f ne n 2 where h f is the hellinger entropy of f is nearly achievable by sieve estimators

estimation and accuracy after model selection classical statistical theory ignores model selection in assessing estimation accuracy here we consider bootstrap methods for computing standard errors and confidence intervals that take model selection into account the methodology involves bagging also known as bootstrap smoothing to tame the erratic discontinuities of selection based estimators a useful new formula for the accuracy of bagging then provides standard errors for the smoothed estimators two examples nonparametric and parametric are carried through in detail a regression model where the choice of degree linear quadratic cubic is determined by the cp criterion and a lasso based estimation problem

on maximum likelihood estimation in infinite dimensional parameter spaces an approximate maximum likelihood estimate is known to be consistent under some compactness and integrability conditions in this paper we study its convergence rate and its asymptotic efficiency in estimating smooth functionals of the parameter we provide conditions under which the rate of convergence can be established this rate is essentially governed by the size of the space of score functions as measured by an entropy index we also show that for a large class of smooth functionals the plug in maximum likelihood estimate is asymptotically efficient that is it achieves the minimal fisher information bound the theory is illustrated by several nonparametric or semiparametric examples

ϵ nash mean field games for linear quadratic systems with random jumps and applications this paper studies linear quadratic gaussian lqg games of stochastic large population system with jump diffusion processes the most distinguishing feature compared with the well studied mean fi

the stochastic maximum principle for linear convex optimal control with random coefficients this paper considers a stochastic control problem with linear dynamics convex cost criterion and convex state constraint in which the control enters both the drift and diffusion coefficients these coefficients are allowed to be random and no l p bounds are imposed on the control an explicit solution for the adjoint equation and a global stochastic maximum principle are obtained for this model this is evidently the first version of the stochastic maximum principle that covers the consumption investment problem the mathematical tools are those of stochastic calculus and convex analysis when it is assumed as in other versions of the stochastic maximum principle that the admissible controls are square integrable not only a necessary but also a sufficient condition for optimality is obtained it is then shown that this particular case of the general model may be applied to solve a variety of problems in stochastic control including the linear regulator predicted miss and benes problems

wavelet shrinkage asymptopia much recent effort has sought asymptotically minimax methods for recovering infinite dimensional objects curves densities spectral densities images from noisy data a now rich and complex body of work develops nearly or exactly minimax estimators for an array of interesting problems unfortunately the results have rarely moved into practice for a variety of reasons among them being similarity to known methods computational intractability and lack of spatial adaptivity we discuss a method for curve estimation based on n noisy data translate the empirical wavelet coefficients towards the origin by an amount 2 log n n the proposal differs from those in current use is computationally practical and is spatially adaptive it thus avoids several of the previous objections further the method is nearly minimax both for a wide variety of loss functions pointwise error global error measured in l p norms pointwise and global error in estimation of derivatives and for a wide range of smoothness classes including standard holder and sobolev classes and bounded variation this is a much broader near optimality than anything previously proposed we draw loose parallels with near optimality in robustness and also with the broad near eigenfunction properties of wavelets themselves finally the theory underlying the method is interesting as it exploits a correspondence between statistical questions and questions of optimal recovery and information based complexity

a semiparametric approach to model effect modification one fundamental statistical question for research areas such as precision medicine and health disparity is about discovering effect modification of treatment or exposure by observed covariates we

metalearners for estimating heterogeneous treatment effects using machine learning there is growing interest in estimating and analyzing heterogeneous treatment effects in experimental and observational studies we describe a number of metaalgorithms that can take advantage of any supervised learning or regression method in machine learning and statistics to estimate the conditional average treatment effect cate function metaalgorithms build on base algorithms such as random forests rfs bayesian additive regression trees barts or neural networks to estimate the cate a function that the base algorithms are not designed to estimate directly we introduce a metaalgorithm the x learner that is provably efficient when the number of units in one treatment group is much larger than in the other and can exploit structural properties of the cate function for example if the cate function is linear and the response functions in treatment and control are lipschitz continuous the x learner can still achieve the parametric rate under regularity conditions we then introduce versions of the x learner that use rf and bart as base learners in extensive simulation studies the x learner performs favorably although none of the metalearners is uniformly the best in two persuasion field experiments from political science we demonstrate how our x learner can be used to target treatment regimes and to shed light on underlying mechanisms a software package is provided that implements our methods

the minimum description length principle in coding and modeling we review the principles of minimum description length and stochastic complexity as used in data compression and statistical modeling stochastic complexity is formulated as the solution to optimum universal coding problems extending shannon s basic source coding theorem the normalized maximized likelihood mixture and predictive codings are each shown to achieve the stochastic complexity to within asymptotically vanishing terms we assess the performance of the minimum description length criterion both from the vantage point of quality of data compression and accuracy of statistical inference context tree modeling density estimation and model selection in gaussian linear regression serve as examples

convergence of numerical solutions to stochastic age structured system of three species abstract in general most of stochastic age structured system of three species do not have explicit solutions thus numerical approximation schemes are invaluable tools for exploring their properties the aim of this paper is to investigate the convergence of numerical approximation solution to the true solution for stochastic age structured system of three species

optimality conditions for partial information stochastic control problems driven by levy processes abstract in this paper we consider a partial information stochastic control problem where the system is governed by a nonlinear stochastic differential equation driven by teugels martingales associated with some levy process and an independent brownian motion we prove optimality necessary conditions in the form of a maximum principle these conditions turn out to be sufficient under some convexity assumptions to illustrate the general results an example is solved

covering numbers for convex functions in this paper we study the covering numbers of the space of convex and uniformly bounded functions in multidimension we find optimal upper and lower bounds for the epsilon covering number of cal c a b d b in the l p metric 1 leq p in terms of the relevant constants where d geq 1 a b 0 and cal c a b d b denotes the set of all convex functions on a b d that are uniformly bounded by b we summarize previously known results on covering numbers for convex functions and also provide alternate proofs of some known results our results have direct implications in the study of rates of convergence of empirical minimization procedures as well as optimal convergence rates in the numerous convexity constrained function estimation problems

resampling consistency of substitution estimators on the basis of n i i d random variables with a common unknown distribution p we wish to estimate a functional n p an obvious and very general approach to this problem is to find an estimator p n of p first and then construct a so called substitution estimator n p n of n p in this paper we investigate how to choose the estimator p n so that the substitution estimator n p n will be consistent

minimax estimation in sparse canonical correlation analysis canonical correlation analysis is a widely used multivariate statistical technique for exploring the relation between two sets of variables this paper considers the problem of estimating the leading canonical correlation directions in high dimensional settings recently under the assumption that the leading canonical correlation directions are sparse various procedures have been proposed for many high dimensional applications involving massive data sets however there has been few theoretical justification available in the literature in this paper we establish rate optimal nonasymptotic minimax estimation with respect to an appropriate loss function for a wide range of model spaces two interesting phenomena are observed first the minimax rates are not affected by the presence of nuisance parameters namely the covariance matrices of the two sets of random variables though they need to be estimated in the canonical correlation analysis problem second we allow the presence of the residual canonical correlation directions however they do not influence the minimax rates under a mild condition on eigengap a generalized sin theta theorem and an empirical process bound for gaussian quadratic forms under rank constraint are used to establish the minimax upper bounds which may be of independent interest