Sparse Contextual CDF Regression
Authors: Kamyar Azizzadenesheli
William Lu
Anuran Makur
Qian Zhang
Venue: Accepted by TMLR ()

Abstract: Estimating cumulative distribution functions (CDFs) of context-dependent random variables is a central statistical task underpinning numerous applications in machine learning and economics. In this work, we extend a recent line of theoretical inquiry into this domain by analyzing the problem of \emph{sparse contextual CDF regression}, wherein data points are sampled from a convex combination of $s$ context-dependent CDFs chosen from a set of $d$ basis functions. We show that adaptations of several canonical regression methods serve as tractable estimators in this functional sparse regression setting under standard assumptions on the conditioning of the basis functions. In particular, given $n$ data samples, we prove estimation error upper bounds of $\tilde{O}(\sqrt{s/n})$ for functional versions of the lasso and Dantzig selector estimators, and $\tilde{O}(\sqrt{s}/\sqrt[4]{n})$ for a functional version of the elastic net estimator. Our results match the corresponding error bounds for finite-dimensional regression and improve upon CDF ridge regression which has $\tilde{O}(\sqrt{d/n})$ estimation error. Finally, we obtain a matching information-theoretic lower bound which establishes the minimax optimality of the lasso and Dantzig selector estimators up to logarithmic factors.

Published in Transactions on Machine Learning Research (07 (07/2024)
Sparse Contextual CDF Regression
Kamyar Azizzadenesheli Nvidia Corporation
kannyaru@nvidaaacommmm com
William Lu Department of Computer Science
lu9099purdue... edu
Purdue University
Anuran Makur
amakur@purdue..c edu
Department of Computer Science and School of Electricol and Computer Engineering
Purdue University
Qian Zhang
zhan3761 @purdue. edu
Deportment of Statistics
Purdue University
Reviewed on OpenReview: https: //ppe//evee..eettttt.....e wetpii/wwwwee//eeeeeeeeeeeeeeeee forrm id= AIc48TjuSt
Abstract
Estimating cumulative distribution functions (CDFs) of context-dependent random variables a central statistical task underpinning numerous applications in machine learning and economics. In this work, we extend a recent line of theoretical inquiry into this domain by analyzing the problem of sparse contextual CDF regression, wherein data points are sampled from convex combination of context-dependent CDFs chosen from set of d basis functions. We show that adaplations of several canonical regression methods serve as tractable estimators in this functional sparse regression setting under standard assumptions on the conditioning of the basis functions. In particular, given n data samples, we prove estimation error upper bounds of ((vs/n) for functional versions of the lasso and Dantzig selector estimators, and Ö(VB/Vm vm) for a functional version of the elastic net estimator. Our results match the corresponding error bounds for finite-dimensional regression and improve upon CDF ridge regression which has Ó(vd/v) estimation error. Finally, obtain matching information-theoredd lower bound which establishes the minimax optimality of the lasso and Dantzig selector estimators up to logarithmic factors.
1 Introduction
The estimation of cumulative distribution functions (CDFs) is classical problem in mathematical statistics stemming back to the Giivenko-Cantelly theorem (Cantelli, 1933; Glivenko, 1933; Devroye et al., 2013), which states that empirical CDEs constructed from independent samples of single random variable converge uniformly to the random variable's true CDF, Subsequent classical research in this area has focused on deriving tight non-asymptotic sample complexity results terms of the Kolmogoroy-Sminat distance among others, such as the Dvoretzky-Kiefer-ee Wiierkk---Kifee-Wolowiiia inequality (Dvoretzky et al., 1956) and improved bounds by Massart (1990).
Motivated by applications to modern learning tasks such as contextual bandits and Markov decision processes (Huang et al., 2021; 2022), & recent. line of research (Zhang et al., 2024) introduced the problem of content-dependen CDF estimation, which requires a learner to simultancously estimate (possibly infinite) family of CDFs parameterized by some context variable. As an initial simplification, the authors considered
The author ordering is alphabetical.

---

Published in Transactions on Machine Learning Research (07/2024)
the restricted setting of contertual CDF regression, wherein the true contextual CDF is convex combination of d context-dependent basis functions. The authors generalized the classical ridge regression method (Hoerl & Kennard, 1970; Abbasi-Yadkony et al., 2011) to this functional regression problem and derived a tight ((dd/n) estimation error bound given n samples a variety of data gencration settings. However, when d is large, specifically unstructured settings where a massive set of potential CDFs are considered, ridge regression utilizes all CDF basis functions for the purpose of estimation without regard to their relevance.
In this paper, as further step towards developing general algorithms for contextual CDF estimation, we propose sparse regression and basis selection techniques for the aforementioned CDF regression problem based on functional versions of lasso (Tibshirani, 1996), clastic net (Zon & Hastic, 2005), and Dantzig selector (Candes & Tao, 2007) methods. Crucially, all of our techniques achieve estimation bounds with no polynomial dependence on d, allowing accurate recovery of the true contextual CDF from bases containing exponentially many irrelevant functions. We also establish minimax optimality results for sparse CDF regression.
1.1 Outline
We briefly delineate the structure of our paper. We introduce the sparse contextual CDF regression problem under present investigation Section 1.2 and define relevant notation in Appendix A. In Section 2, we outline some applications of CDFs in machine learning and economics, and provide an overview of related functional regression schemes in the previous scientific literature. In Section 3, state our main contributions and discuss the overarching proof techniques used throughout the paper. Section 4 provides the formal derivation of our upper bound on lasso estimation error in the fixed design setling. We include numerical simulations of our data generation and parameter estimation processes in Section 5. We defer remaining proofs and tochnical details to Appendiccs B to E, provide examples of CDF bascs which satisfy the preconditions for our main results Appendix F, and briefly discuss the computational complexity of our estimators in Appendix G.
1.2 Model and Setup
In this subsection, we formally define the sparse contextual CDF regression problem under investigation in this paper. (Recall that we utilize the notation defined in Appendix A.) Let X' denote a general context space. We refer to a function [(x, () X' IR [0, 1] as a contertual CDF if for any x E X, f(x,,) is valid CDF for real-valued random variable with range contained in some set S C R. Let be any probability measure on S. Let {øi...., Φa} denote fixed basis of d E N contextual CDFs indexed by i E [d]. For convenience, we often conceptualize this basis as a single vector-valued function $ X x R + [0, 1] defined by (#, ()} = (x, t) for all i E [d].
Let F; X + [0, 1] denote the true contextual CDF we aim to recover. Following the precedent established in Zhang al. (2024), we assume that F is a convex combination of the basis functions (øi...., od). Hence, the problem reduces to recovering the true parameter vector 0, E A Ad-1 such that
Vz E X, VE R, F(z,t) 0_ (xx
Let S. =upp((.). denote the support of the true parameter. For notational simplicity, let s s= = = =10011o. Let ()))),")........ (x be a set of n E N observed samples indexed by j E [m], generated according to the sixred design or random design settings described below:
Fixed design. For each j E [n], the context variable x(1) E X is fixed a priori, and the response variable yاو R is independently sampled from the CDF F(x()),.).
Random design. For each j E [n]. the context variable z00) E X is independently sampled from an unknown probability distribution pK) over X. Then, conditional on x(1), the response variable gوو) R is independently sampled from the CDF F(x(6),,)
These design settings extend the data generation processes described in Abbasi-Yadkori et al. (2011) and Hsu et al. (2012). Tet (t) (zx(0),0 denote the CDF basis at the jth context variable, and let zim
2

---

Published in Transactions on Machine Learning Research (07/2024)
(xl1), x(1) denote collection of sampled variables, with analogous definitions for y and (x. "m Let
Un E Ruxd denote the empirical n-sample Gramian matrir given by
Vi, E [d], (nnli
Un
Σ $ n
1
dm.
(1)
In the random design setting, let En E Rdxd denote the expected n-sample Gramion malric given by
Vi,i' [d], Eileg
(1
En
E X
2
2
dm
(2)
n
X
T1
As mentioned previously, the objective of contextual CDF regression is to recover O. given the samples (z. 1mm Each yú) defines a one-sample empirical CDF (t) = 1ft 2 y0) which approximates the true CDF --x = (.) X = زافر in expectation conditioned on X x()), We investigate three estimators for 0- based on the paradigm of empirical risk minimization. Firstly, the lasso estimator 0₂ imposes an e' -e-enalyyy on the parameter vector, weighted by X 0:
0^ arg min DERd
Σ 10:
1 + A 100
(3)
Secondly, the elastic net estimator 4x imposes both el and l2 (e-penalties on the parameter vector, weighted by d and X2 respectively:
A arg min BERd
Σ ==1
1l - 0 -TT0,1(22(s.m) + A |101, + A2 |1ll?
(4)
Thirdly, the Dantzig selector 02 is given by the following variation the constrained optimization formulation of lasso estimation:
0x minimize Begd
||0|
(5)
1
subject to
(100) - 0 ,j SA.
(6)
n
For all three estimators, analyze the eslimation error given by the Fuclidean distance ||0 (0-00.1) 0.11A bet ween the estimated and true parameter vectors, possibly weighted by some matrix A. We remark that Equations (3) to (5) define improper estimators which do not necessarily lie in A and are thus not guaranteed to define valid CDFs However, since 0, Ad-1 and A closed and convex, any upper bound on the estimation error of an improper estimator also holds for its projection onto A by Beck (2014, Equation 9.10).
Lastly, remark that our results hold for any d, although we principally interested in the advantages of our proposed methods over the ridge estimator baseline (Zhang et al., 2024, Section 3.1) for small values of s. We emphasize that our results have no dependence on the cardinality of the context space X.
2 Related Work
Broadly speaking, CDFs underpin the computation of risk functionals which inform docision-making in mathematical finance and actuarial science. For example, generic law invariant risk functions such as conditional value-at-risk (Rockafellar et al., 2000; Artzner et al., 1999) are parameterized by CDFs, and notion of distortion risk measure (Wirch & Hardy, 2001) ariscs from the composition of a CDF with distortion function. Coherent risk measures are instrumental in portfolio management and optimization (Krokhmal, 2007) and can be formulated in terms of CDFs (Shapiro al., 2014). Lastly, spectral risk
3

---

Published in Transactions on Machine Learning Research (07/2024)
measures are computed as weighted averages of outcomes (Acerbi, 2002) and hence depend on the entire trajectory of the underlying random variable's CDF.
Similar considerations exist in further scopes of learning theory, where the aforementioned risk functionals are incorporated into supervised learning tasks (Liu et al.; 2022) and multi-armed bandit problems (Cassel et al., 2023) to model fairness, risk aversion, and distribution shift (Wong et al., 2022). The recently proposed off-policy risk assessment framework (Huang al., 2021) includes CDF estimation block and has been applied to contextual bandit problems and Markov decision processes (Huang et al., 2022). Other work in this vein has employed the mean-variance (Sani et al., 2013; Zimin et al., 2014) and value-at-risk (Vakili & Zhao, 2015) paradigms in the multi-armed bandit setting to analyze risk-reward trade-offs. Furthermore, cumulative prospect theory is intimately tied to risk distortion and the ensuing dependence on CDF estimation (Prashanth et al., 2016), and has found relevance in reinforcement learning and stochastic optimization (Jie et al., 2018).
key building
We touch on the well-established precedent in the scientific community for framing function estimation through the lens of linear regression and basis selection. Romo et al. (2013) generalized lasso variable selection to linear models with scalar regressors and functional responses to develop interpretable analyses of car accident data. A related method exploiting feature sparsity and output function smoothness was proposed in Barber et al. (2017) to perform genome-wide association studies. Linear models with functional regressors and scalar responses have been used genomics, MRI data amalysis, and chemometrics, spurring the development of functional group-lasso (Pannu Billor, 2017) and wavelet-based lasso methods (Zhao et al., 2012; 2015) for such models. Prior studies on genetic regulatory networks (Hong & Lian, 2011) have incorporated function-on-function regression models with E'-regularization to encode sparsity in pairwise interactions between genes. Lastly, recent developments on lasso estimation for function-on-ffucctoo regression (Centofanti et al., 2022; Maranzano et al., 2023) find downstream application in geostatistical models and mortality data analysis.
Compared to the prior literature, the main novelty of our contributions lies in the simultaneous estimation of an entire family of CDFs. In this sense, our work may be interpreted as a generalization of the canonical mixture model with known basis distributions (Murphy, 2012, Section 11.2) the context-dependent setting, focus on CDF cstimation instead of PDFs or quantiles which require restrictions
For maximum generality, for existence and well-definednnss.. We formulate estimation through the lens of functional linear regression and show that our task reduces to integral computation and fiite-dimensional optimization, making our approach computationally efficient using existing numerical methods. Lastly, our derivations utilize intrinsic properties of CDFs instead of performing discretization to avoid introducing approximation error, in contrast to the approach taken in Hong Lian (2011) among others.
3
Main Results
3.1
Lasso
Before presenting our main contributions, we introduce the following condition on symmetric matrices which is used in the statements of our main results:
and S. S [d]. Let Definition 1 (Restricted eigenvalue condition). Fix d E N, K 2 0, 7 2,0,
Cs().) = ( Rd | S> S
A symmetric matrix A E Rdxd satisfies the (1,, 7) -restrirdt eigenvalue condition over S, iff
Ve E (S.),, T. Av 2 1|
Intuitively, the restricted eigenvalue condition specializes the notions of positive definiteness and strong convexity the subset of diroctions whose support is close to S,. The significance of this condition lies in its application to the Gramian matrices U, and Σ,, since bound-b the Gramian eigenvalues sufficient to ensure the well-conditioning of the basis functions comprising the true CDF. Hence, our work instantiates
4

---

Published in Transactions on Machine Learning Research (07/2024)
the standard definition from Wainwright (2019, Definition 7.12) on the general inner product spaces used in the definitions of Un and En, We provide examples of non-trivial CDF bases which satisly the restricted eigenvalue condition in Appendix F.
Under this assumption, we state our first main result, a high- probabbobayylity upper bound on the error of the lasso estimator in the fixed design setting:
Theorem (Lasso fixed design upper bound). Pir ò E (0, and R > 0. Assume the samples (x, y) Am are generated according to the fized design setting. Assume Un sotisfies the (1s, 3e-rttiieeeed eigenvalue condition over S.. Let 02 be the lasso estimator (3) with regularization hyperparameter with probability at least 1- -, 6, the estimation error satisfies the bounds
(2/n)) log (2g(2) Then
062 10a-0,11
-o11l. =6
2s
2d log
|0x 0n and
6
2s
2d 8
log
Kn
K n
We provide the technical details of our proof in Section 4. The crux of our argument lies in the definition of Ox the minimizer of the empirical risk objective in (3). Substituting 0x and 0, into this objective results in an inequality (15) in terms of 0₂ and 0.. Rearranging yields an upper-bound (17) on the estimation error Alu, in terms of two quantities:
The scalar projections of the sampling errors 1 8_ o, onto the d basis functions $j which we upper-bound with high probability (10) using Hoeffding's inequality and the union bound over i e [d].
The difference between the e-regularizers |10.1|: - ||0x||, which we upper-bound by utilizing the sparsity of 0g and the definition of -o-om.m.
Combining these results gives rise to a bound on the relative magnitudes of the components of corresponding the true support S. and its complement (21). Then, upper-bounding the component [[A]].]l: 11 is sufficient (22) to characterize the overall estimation error ||]|,,, Applying various norm equivalences and invoking the restricted cigenvalue condition on U,, completes the proof. We remark that the restricted eigenvalue condition plays comparable role in ensuring the well-conditioning of the Gramian matrix Un in our work, as it does for X X in classical formulations of finite-dimensional regression methods.
Theorem 1 establishes the asymplotic complexity of lasso CDF regression as O(y log(d)/n) = = 8/)),,), in direct analogy to the ((vd/n) result for ridge regression in Zhang et al. (2024). The dependence from our sparsity analysis on ||6.]|, ||λIl],, which reduces the problem to bounding |||A|s_]| |l discussed above. The factor of 1/vnn is consequence of Hoeffding's inequality, and the factor of v /log(d) (d) arises from the union bound over E [d] in our analysis of the sampling errors. Ultimately, the lasso estimator's charactoristic f'-penalty term plays an instrumental role in deriving result with sub-polynomial dependence on d.
v arises
Our second main result is a high brgbbprobability upper bound on the error of the lasso estimator in the random design setting, under similar assumptions on the well-conditioning of the CDF basis:
Theorem (Lasso random design upper bound). Fir on E (0,1), 82 E (0,1-81). and > Assume the somples (I, (I are generated according to the random design setting. Assume that for ony E N, the matrix Σn satisfies the (k,,) -restricid eigenvalue condition over S.. Assume the sample size is at least 2 (32d2/n² /x2) log(d/61). Let 0x be the lasso estimator (3) with regulorization hyperporametee A (2/m)) (og(2)/8)). Then with probability at least 1 -5 -, 82, the estimation error satisfies the bounds
0₂x 1.11. S
2d 82
and 100 0.
12 K
2s
2d
12
log kn
log
n
52
We defer the technical details of our proof to Appendix B. In a nutshell, since Un is a sum of n independent matrices and is an unbiased estimator of Σ,, we can employ a matrix analogue of Hoeffding's incquality (23) to justify approximating Σ, with Un given sufficiently many samples n. This reduces the problem to the fixed design setting, and invoking Theorem completes the proof.
5

---

Published in Transactions on Machine Learning Research (07 /224)
3.2
Elastic Net
Our third main result is a high-probability upper bound on the error of the elastic net estimator in the fixed design setting:
Theorem 3 (Elastic net fixed design upper bound). Fir 5 E 1). Assume the samples (z. y) are generaled according to the fixed design setling. Assume Un salisfies the (1s + -resrri-eeetrreeeed eigenvalue condition over S.. Let 0x be the elastic net estimator (4) with e -regularization hyperparameter X1 = Av (2/n))oog(d//// (2d(6/ Then with probubilily at least 1 - 8, the eslimation error salisfies the bounds
10siNs -0a
On -olles,tatt
2
2d
s
6
log
+ 2A2
V A2
(7)
72
10:1120 - 0,
2
2d
V8
66
log
+ 2A2
K + 22
Furthermore, (3/2) (2/2) log(2d/0)) and X2 (2/n)) log (2d/6)
2n. the bound (7) implies
|xx,20 - 0.11 Littelll
S $Av5
3
2
log
2d
(8)
n
18s2 log
2d
S 4
8
We emphasize that Theorem 3 produces non-trivial estimation bounds even when no assumptions are placed on Un (i.e., k = 0), in which case the bound in (7) has Ö(Vs/ /nn) complexity. However, when Λ > 0 and the e P-regularization hyperparameter sufficiently small with X2 = O(1/Vn), the elastic net estimation bound exhibits ((vs/n) scaling, comparable with the result for the lasso estimator Theorem 1.
At a high level, the elastic net estimator with its characteristic l-penalty term may be perceived performing lasso regression on the "regularized" Gramian matrix U, + >pld, whose smallest eigenvalue is strictly positive. Additional conditions on Un further tighten the estimation bounds by bolstering the minimum cigenvalue of Un + A2Ta, Hence, the clastic net estimator simultancously selects features and regularizes the problem in an integrated fashion by ensuring strong convexity of the objective function (4).
We defer the technical details of our proof to Appendix C. Our arguments mirror the proof structure for Theorem 1, with adjustments to accommodate the clastic net estimator's e--penalty term. At the outsot, substituting 0x,ر> and 0, into the empirical risk objective (4) and rearranging yields an upper-bound on | U. + +A2ly instead of |l., The resulting expression (32) contains additional quantity A which upper-bound in terms of the error component [|[A]s. |I in (34). Combining this contribution with the terms inherited from the lasso objective gives rise to the parenthesized sum in (7), and choosing the optimal value X2 to balance the summands achieves the bound in (8).
By the same technique based on matrix Hoeffding that we introduced in Section 3.1, we also obtain the following brobbbrbbyyylityy upper bound on the error of the elastic net estimator in the random design setting, which we prove in Appendix C:
Theorem 4 (Elastic net random design upper bound). Fix d, E (0. 82 E (0,1-8:), and 0. Assume the samples (x, y) are generated according to the random design setting. Assume that for any n E N, the matria En satisfies the (r,3 + 412eAA))
eigenvalue condition over S.. Assume the sample size is at least n 2 (32d// /x²) log(d/51). Let 0x,22 be the elastic net estimalor (4) with et errregaarrziaionn hyperporamneter X1 (2/2/)) log (2) Then with probability at least - on 62, the estimation error satisfies the bounds
O 0. 1
(6)
2
2d
2s
Unttell <
6
log
82
+ 2A2
V
+ 2
(9)
6

---

Published in Transactions on Machine Learning Research (07/2024)
10 0u1 (61)
2
2d
2/8
log
+ 222
82
+ 2X2
Furthermore, i (// log( 22) and A2 = 3/ log (ogg(2//0)) (2d/52)
Ki the bound (9) implies
1
1 0, o.
S 22/28
6/ n
2
log
2d
82
18s?
2d
S 4
log
82
3.3 Dantzig Selector
Before presenting our fourth main result, we introduce two more conditions on symmetric matrices which underlie standard assumptions in the statistics and compressed sensing literature (Bandeira et al., 2013):
Definition 2 (Restricted isometry property). Fir c 2 0 and p N. symmetric matriz A E Idddd satisfies the (e.t)-restricted isometry property iff
Vu E 1oolo (1-e) ||=1 S Au (1 (1+
The restricted isometry property states that the curvature of the quadratic form represented by A is bounded around along directions residing in sparse axis-aligned subspaces, or alternatively, that the linear operator represented by A is approximately scale-preserving when applied to sparse inputs. A weaker variant of this property bears resemblance to the Cauchy-Schie Schwarz inequality:
Definition (Restricted orthogonality property). Fir Ç 2 0, O.PE E N, and q N. A symmetric matrir A Rdxd satisfies the (S.p. 9) rrestriceed orthogonalily properly iff
Yu,u E Rd. ||u/oo s P ||o||o S supp(p) (ssup(r)) = 0 = |u Au| S [|]]||
Under the restricted orthogonality property, A maps sparse vectors with disjoint support to dissimilar outputs. Our fourth main result is a high-probability upper bound on the error of the Dantzig selector in the fixed design setting, under assumptions on Un similar to the prior literature (Candes & Tao, 2007):
Theorem 5 (Dantzig selector fixed design upper bound). Fix 5 E (0,1), CE (0,1), and E -0,1-c) c). Assume the samples (I, (m are generated according to the fized design setting. Assume Un satisfies the (e, rrsttrict isometry property and the (,, resttricd orthogonality property. Let 0x be the Dantzig selector (5) with regularization hyperparameter X = ((2/n)log(2//)) the estimation error satisfies the bound
(og(2d/6). Then with probability at least 5,
2s 100 -0.11 S I- - <V n
2d 8
log
We include the Dantzig selector our investigation of sparse contextual CDF regression as an example of variable selection formulated as a lincar programming problem (6), as opposed to the usual quadratic programming perspective of lasso estimation. We remark that the (e, rrestrieed isometry properly implies the (c,p, restrectd orthogonality property, by Candes & Tao (2005, Lemma 1.1). Hence, Theorem 5 also holds when the sole assumption placed on U, is the (c, reostttted isometry property.
We defer the technical details of our proof to Appendix D and provide a high-level summary below. Using a result from the prior statistical literature, we upper-bound the estimation error || in terms of the components |[A]s, and [[[]]s; lI, where Sy superset of S, (43). Utilizing the sparsity of 0,, we bound the latter component terms of the former (44). Using another prior result, we opper-bounde [|[A] s, in terms of [[U[],,,)) All (46), which in turn depends on quantities (48):
7

---

Published in Transactions on Machine Learning Research (07/2024)
The scalar projections of the sampling errors I,00 6, 2 onto the d basis functions 9,, which we upper-bound using similar probabilistic arguments as the proof of Theorem 1.
The scalar projections of Loo - 8x D3 onto 4y. which we upper-bound using the constraint in (6).
Combining these bounds and applying various norm equivalences completes the proof. The restricted isometry and restricted orthogonality assumptions on Un are preconditions for the prior results we use, but none of our subsequent arguments depend these assumptions.
As previously done for the lasso and elastic net estimators, we use a matrix version of Hoeffding's inequality to derive the following high-probability upper bound on the error of the Dantzig selector in the random design setting, which we prove in Appendix D:
Theorem (Dantzig selector random design upper bound). Fir 51 E (0,1), 62 E (0,1-&)), (0, 1/2), and E 10,1/2-c c). Assume the samples (x.y) im are generated according to the random design setting. Assume that for any n E N, the malrie E,, solisfies the (e, 2s) rrestricd isomelry properly and the (5,8, (5.;,2s)-re restrrcidd orthogonality property. Assume the sample size is at least Dantzig selector (5) with regularization hyperparameter À = ((2/n) log(2d/b)) 62). Then with probability at least 1 - 61 82, the eslimation error satisfies the bound
(8d²) minfe, ()²) log(2d/81). Let 02 be the
28
2d
|002 0.1 - 2E 2Ç
log
82
3.4 Lower Bound
Our last main result is lower bound the minimar P-risk of sparse contextual CDF regression. To begin this discussion, introduce some relevant notation. For any d e N, let Ba be the universe of all d-dimensional bases of contextual CDFs, i.e.,
B²
X x
10, 11= Vi [d]. Vz E X, [0(2, )), is CDF)
For any x E X, E Ad11 and + E Ba, let pa y| denote the probability distribution corresponding to the CDF 0 (xx,))) Given context variables æli E X", let P denote the family of product distributions of ylm which are convex combinations of contextual CDFs, i.e,
Pd
p Ylml)), : 1 1 A Ba
lm
denote the family of product distributions of ylm which are convex combinations of s contextual CDFs chosen from a d-dimensional basis, i.e.,
For any S d, let pds C pd
al
pd.e zim
p& Y : Ad-1 A 11@1 = s SA@E Ba
j=1
Given a distribution E Pd ginn let ((P) E d-1 denote its parameter and let S, = supp(0(P)) c [d] Consequently, [P)] for any P pd.s randomized) estimators 0 R + Rd.
gin For any n N, let θa, be the universe of all (possibly
Our main result establishes a lower bound on the estimation error of any estimator for sparse contextual CDF regression, and is obtained as a corollary of the lower bound for general contextual CDF regression in Zhang et al. (2024, Theorem 8):
Corollary 1 (Lower bound). Fir d E N, S d, and any sufficiently large 2 8/2. Fix w E X", Then, the minimax 12 risk of sparse conteztual CDF regression satisfies the bound
[(r2m)) e(P) = - (V)
27 ((0(P)))))
E -
inf de@am
sup Pepdee ylu kim

---

Published in Transactions on Machine Learning Research (07/2024)
We defer the technical details of our proof to Appendix E. In nutshell, it suffices to consider only the istimation error component corresponding to the indices in (P), thereby reducing the problem to general s-dimensional CDF regression. Furthermore, we obtain minimax upper bounds from the high-probability upper bounds in Theorems 1 and by the arguments from Zhang et al. (2024, p. 12). Hence, Corollary establishes the minimax optimality of the lasso and Danizig selector estimators up to logarithmic factors.
3.5 Comparison to liiite-Dimessioal Setting
We conchude Section 3 with a summary of the technical differences between the present CDF regression setting and the well-studied fmite-dimeensonnal regression setting. Firstly, known upper bounds on estimation error for finite-dimensional sparse regression methods (cf. Wainwright (2019, Equation 7.26)) typically depend on the scales of the noise and features, and assume the sampling noise is sub-Gaussian random variable. However, the sampling noise Iyo) - 0_ 3 in CDF regression is random function, which presents challenges when attempting to define its distribution. To circumvent this issue, we analyze the effect of sampling error by considering instead the inner product (defined as an integral over the support S) between the sampling noise and the feature CDFs 3-- We incorporate this inner product in our formulation of the Dantzig selector (6), and we refer the reader to Equations (13) and (18) in the proof of Theorem 1, Equations (29) and (33) in the proof of Theorem 3, and Equations (40) and (47) in the proof of Theorem for examples of tochnical steps in our derivations which utilize the inner product between sampling noise and feature CDFs. Ultimately, our proposed estimators admit error bounds independent of the response and noise scales in both the fixed and random design settings (Theorems to 6).
Secondly, the Gramian matrix in inite-dimensions regression is given by the matrix product between the design matrix and its transpose, equivalent to taking the dot product between each pair of training instances. In CDF regression, we instead define the empirical Gramian matrix Un (1) and expected Gramian matrix En (2) in terms of an integral over the support S of the underlying random variable of the contextual CDFs, corresponding to the inner product between functions in the Hilbert space (SS, m) opposed to the dot product between vectors in ordinary d-dimensional Euclidean space. In similar vein, the aforementioned inner product between the sampling noise and feature CDFs has support [-1,1] due to the boundedness of CDFs (11), enabling us to derive a concentration result on the inner product over n samples using Hoeffding's incquality (12) in Lemma 1.
Thirdly, our analysis of random design settings in CDF regression differs from similar work in the fiite-dimennional regime by employing a matrix version of Hoeffding's inequality (Lemma 3) to establish preconditions on the conditioning of Un given similar assumptions on Σn and sufficiently many random samples. To justify this step, we bound the operator norm of the empirical Gramian error by using properties of CDFs (26), whereas additional assumptions on the design matrix are required in the analogous finte-dimensiona case. We refer readers to (28) in the proof of Theorem 2, (38) in the proof of Theorem 4, and (51) in the proof of Theorem 6 for specific details on the usage of this concentration result in our derivations. We note that we impose the restricted eigenvalue, restricted isometry, and restricted orthogonality assumptions directly on the Gramian matrices Un and E,, although such conditions are typically applied to the design matrix in the finitedimeensoonll setting.
Finally, the boundedness of CDFs plays a role in our derivation of the lower bound on minimax --risk. In nutshell, the minimax --iikk scales as the inverse square root of the empirical Gramian matrix's smallest eigenvalue, which in turn is upper-bounded by the Gramian entries and hence the integral of CDF products by (1). For specific details, we refer readers to (54) in the proof of Corollary 1.
Notwithssanding the technical details discussed above, we note that our derivations of the lasso and clastic not error bounds (Theorems and 3) share structural similarities with classical arguments for the corresponding fiite-dimeensonnal settings, in that we utilize the optimality of the estimator 0₂ to derive a bound relating 0x and 8. by rearranging the empirical risk objective. We refer readers to Wainwright (2019, p. 212-213) for example of this general technique applied to fiittedimennionall regression.
9

---

Published in Transactions on Machine Learning Research (07 (07/2024)
4 Proof of Lasso Fixed Design Upper Bound
In this section, we prove Theorem 1. We begin by deriving a concentration bound on the inner products between the sampling errors lyos -0_ 2y and the basis functions 4 as a consequence of Hoeflding's inequality and the union bound:
Lemma 1 (Concentration bound). Fir d E (0, With probability at least 1 6, it holds that
2d
Σ (60 0. 0. Φ1)
log
(10)
8
Proof of Lemma 1. For each E [n] and i E [d], define random variable
Z = (Tru: 0. (o)l))
Let J, = =((1--1) denote the o-algebra generated by the random variables y The mean of Z is
B Z,, 1,
@ B - 0_ o (0)k)) [5]]
@ [B[ - 0_ Φ, (5). (
@ [[, [+,},}]
0,
where (a) holds by the tower rule of expectation, (b) holds by definition of Zz,j (c) holds by Fubini's theorem, and (d) holds because E[[yo] Fj] = 0, 4, Furthermore, the support of Z is [0, 1] because
(4)
/ (1))) 0_ 2 2t))) Φ, (z)), dm
(8)
(t) - 0_ 0t)) 0. (2()) dm
8
(S))
(11)
(4)
where (a) holds by definition of inner product, (b) holds by the triangle inequality, (c) holds because CDFs are bounded between 0 and 1, and (d) holds because is probability measure. Thus, for any 0,
(a)
d
n
P
(rrs) 0T
27
T
Zi
27
1=1
j=1
n7²
2d exp
(12)
2
where (a) holds by definition of Zı, and the union bound, and (b) holds by Hoeffding's inequality. Choosing 22/n log (onnlo/(2/))) (2d/8) and rearranging, we get 8 = 2d exp( -nr2 /2), and thus
2 log
2d x
P
Σ Iym 0 ,
S
2 1 -
n
as desired.
This lemma also applies the random design setting by taking F, = ax11 yl5-1).
10

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Now, are ready to prove Theorem 1.
Proof of Theorem 1. Throughout this proof, we restrict to the subset of the probability space where
2
2d
L 0, B3)
log
(13)
8
4
which holds with probability at least 1 - ô by Lemma 1. For notational simplicity, let A = 02 A..
We follow the argument from Wainwright (2019, p. 212-213). We have
| U.
(a) A
AUU
A
1 Φ dm
T1 j=1
1
A
$
A dm
11
2
(S,
12
0x
2
+ |10_,,112=(0,mm
2
0 2
0 2
(²(Sm
11 j=1
- | $3 (2(S,m)
-100000 10 (3((,
Φ3 0.
(14)
where (a) holds by definition of weighted P-norm induced by Un, (b) holds by definition of Un, (c) holds by the linearity of integration, (d) holds by definition inner product between functions, (e) holds by definition of A, and (f) holds by definition of and the lincarity of inner product. Next, since & minimizes the objective in (3), it follows that
L20 - 00 (erss
+ +
s - n
Σ Lyus - 0. 11 |10.111
(15)
(=(s,m)
Expanding the squared norms and rearranging, follows that (ef. Wainwright (2019, Equation 7.29))
Σ (10T | (=((sm) (0
< (Lyon, A ATΦ,) + > (110.11, - -18211.)
(16)
Combining Equations (14) and (16), we obtain
S 2 (L,0, A (T,,
0, TA 0_ 2,) +1 (10011, - 110
| U.
(17)
==1
==1
Next, we upper-bound ( We have
(a)
(yyn - 0. Φ 2,,
(18)
==1
11

---

Published in Transactions on Machine Learning Research (07 2022)
(h)
2 m
S
||
1
Σ
100 0 pz. Bj)
(e)
_
||
(19)
where (a) holds by the linearity of inner product, (b) follows from Hölder's inequality, and (c) holds by We have
substituting in (13). Next, we upper-bound 2
d
d 00 + A]
[0+1,1
1=1
[[01]]1
Σ
|[e + A1,1
Σ [[A]]]
IES.
iESS
113
[10.1s.
||
|101|s. + [Als. ll,
| 9:
IA
|
(20)
where (a) holds by definition of -norm and A, (b) holds because supp(0.) S., (c) holds by definition of
el norm, and (d) holds by the triangle incquality. Combining Equations (17), (19) and (20), we obtain
3
| l1.)
|
|| S. 11
(21)
2
31
(2) 1A
| 11a
(22)
2
3A
Vs
||A|s,
2
3A
I In
V ||||
2
31
||
2
where (a) holds by the equivalence between l'- and --norms, and (b) follows from the restricted eigenvalue condition on Un since rearranging (21) yields ||[A|slll < 3]][A]s. |I1 Thus, the estimation error satisfies the bounds
31
S (a)
2s
2d
(b) |
6 K
2s log
2d
all,, S
6
log
and
|
Ur
2
Kn
n
where (a) holds by substituting À and (b) follows from the restricted eigenvalue condition on Un.
5 Numerical Simulations
In this section, numerically simulate the data generation processes described in Section 1.2 and empirically evaluate the accuracy of our proposed lasso (3) and clastic net (4) estimators synthetic data. For mathematical conciseness, we consider a basis of contextual Bernoulli CDFs, which yields a closed-form expression for the induced norm (l:))2>(5.m) in the training objective. Formally, let x= [o, 1]a be the space of all d-tuples x = (21...., rd) of Bernoulli parameters, and define the basis functions
[1 1-1: Die if
Vi [d], (x(xt))
it=11,
12

---

Published in Transactions on Machine Learning Research (07/2024)
where (1,,,)) is the CDF of Bernoull(zz.) random variable. It follows that F(x,,) = 0__ (z is the CDF of a Bernoulli(0 x) random variable, because
t(z,t)= = Σ 0o1, [ - Di, 1,
d
ir 1-
(a) 1 0,x, ir0sb<l,
ift=1
if 1,
1=1
where (a) holds because 0. is a PMF. Let m be the uniform measure over the support S = 1]. Then, the log-liken bigliielihood term in Equations (3) and (4) is equivalent to the canonical least squares formulation because
1000 - 0 , @)
E/ (b)n)(t) - -TT (t)))))) dt
j=1
(Lf8 (()))) (1-10002())) 2
Σ
(0 x111 y(0))²
j=1
( A0 bl|
where (a) follows from the support S = 10, 1] and the choice of measure, (b) holds by substituting in the definitions of yyon and $ and the matrix A E Raxd and vector b E IK" in (c) are defined as [A]g] = z() and [b]; = yd)). For specific details regarding our implementation, we refer interested readers to our Python htt.://///ccccaaaaangaaaaa//aara
code at
In our experiments, we compare the e--morm estimation error of our proposed lasso and elastic net estimators against the ridge regression baseline introduced in Zhang et al. (2024, Section 3.1). For the P-regularization hyperparameter of ridge regression and the e'-regularization hyperparameter of lasso and elastic net regression, we use = (o(2/))))g(((/)) as specified
Theorems and 3, with 5 = 0.001. For experimental convenience, we use various fixed values of A2 for the elastic net estimator, which we report in Figures 1 and 2. Different values of \ and 5 produced qualitatively similar results. We investigate how the estimation errors scale with various problem dimensions, and report means and standard deviations over 30 independent random trials in Figures 1 and 2 for each configuration under consideration. We train all models on the same set of generated samples in each random trial.
5.1 Fixed Design
First, we consider the fixed design setting with context variables x01)= = =a(()) .......))
given by
1 (x) if i =i=) j (mod d)
if S d if >
with
2
1 vall
(i)
if i + (mod d)
Ammnn (M
aj
and Mj- = (1- xli)) (1-x01) +
Σ k=
j-1
(1 1 x(t))
(1-2(H))
T
where oj is initialized as add1 = Mmin (Ma)/2 and is doubled as necessary on each j iteration to ensure x00 X. To investigate the effect of the sample size n, we choose 100 logarithmically spaced points for n from 10 to 10", and fix the CDF basis dimension d = 10 and parameter sparsity = Figure 1(a) graphs the estimation errors against n a log-log plot. The lasso trend line has slope - 1/2, matching the theoretical ((1//ń) bound in Theorem and substantially outperforming the ridge bascline. The clastic net estimator achieves results in between the lasso and ridge methods. As a further point of comparison, we repeat. this experiment using handpicked regularization hyperparameters for the ridge estimator and plot the results in Figure 1(b). Values of less than 10-3 or greater than 10-1 produced results comparable to the yellow and purple trend lines, respectively. Observe that the accuracy of the ridge estimator with 10-3 matches the lasso estimator when n < 10S, but quickly plateaus for larger sample sizes. This control experiment confirms
13

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
10
10P
a.l
10-1
10
Hidge
Hope
10
Lasso
Elastic Net (Ag
9)
10
(a) Effect of sample size
(b) Effect of ridge regularization À
1.0
Ridge
Ag 10
0.7
Lasso
Ag 10
Elactic Not (As 10-2)
Ag 10-1
25
0.5
0.1
0.4
0.2
0.2
20
25
10
15
20
25
30
(c) Effect of sparsity
(d) Efffect elastic net regularization A2
0.5
Ridge
Lasso
0.4
Elastic Net 10
0.3
20
40
60
80 100
d
(e) Effect CDF basis dimension d
Figure Means and standard deviations ridge, lasso, and elastic net estimation errors against various hyperparameters in the synthetic Bernoulli experiments for the fixed design setting. Figures 1(a), 1(c) and ((c) contrast the ridge, lasso, and clastic net estimators with common regularization hyperparameter X == X 4 (2/n) log(2d/8), as defined in Theorems and 3. For further comparison, Figure (bb) plots the ridge ostimation error against n for various values of X, and Figure 1(d) plots the elastic net estimation enror against for various values of X2.
14

---

Published in Transactions on Machine Learning Research (07/2024)
that the superior accuracy of our lasso estimator in the large setting cannot be emulated by the ridge baseline regardless of hyperparameter tuning.
Secondly, we investigate how the estimation errors scale with the sparsity of the true parameter 0,. We fix the sample size n = =10 and CDF basis dimension d = 30, and consider s from to 30. Figure 1(c) graphs the estimation errors against 8 on a linear plot. The lasso trend line reflects the theoretical ((v3) bound in Theorem 1, and our lasso estimator notably outperforms the ridge baseline in the sparse regime s < d. To interpret the decreasing ridge trend line, note that ||0-|| 1 for any value of s (because 8. is a PMF), and so denser parameter vectors (with greater s) tend to have smaller ||@+||- Thus, the ridge regularization penalty 2/ at the true parameter vector decreases as incrcases, loading to improved accuracy. For additional comparison, we repeat this experiment for the elastic net estimator using handpicked values of X2 and plot the results in Figure 1(d). As expected, the elastic net estimation error interpolates continuously between the lasso and ridge estimation errors as X2 increases. Values of A2 outside the range visualized in Figure (dd) produced similar results.
Lastly, we investigate the dependence of the estimation errors on the CDF basis dimension d. We fix the sample size = 10a and parameter sparsity == 10, and consider d from 10 to 100. Figure I(e) graphs the estimation errors against d on a linear plot. The ridge and lasso trend lines indicate the respective theoretical bounds of O(vi) (Zhang et al., 2024, Section 3.2) and O d) (Theorem 1).
5.2 Random Design
Next, consider the random design setting with context vectors x01) = "..")...,,.?" sampled i..d. from Uniform(O, 1)d distribution. We investigate the effect of the sample size n, sparsity 8, and CDF basis dimension d on the ridge, lasso, and elastic net estimation errors, as was done for the fixed design setting above, and plot the results in Figures 2(a) to 2(c) respectively. The qualitative trends in the random design experiments are comparable to the corresponding fixed design experiments, although the variance among the 30 trials for cach configuration lower in the random design setting. We conjecture that the variance is reduced by virtue of the context vectors in the random design experiments being more typical instances of the problem, as opposed to the fixed design setting where the context vectors were specifically chosen to be hard instances which may amplify the effect of the randomness in O.. On a different note, we remark that the clastic net error asymptotes for = 10t in Figure 2(a) because X2 is fixed for experimental convenience and does not, decrease with n, although a more refined hyperparameter setting could avoid this effect in principle. Lastly, we compare different values of 22 for the elastic net estimator in Figure 2(d) and observe the expected interpolation between the ridge and lasso trendlines as d ranges from 10 to 100.
6 Conclusion
In this paper, we introduced the task of sparse contextual CDF regression and proposed three basis selection tochniques for this problem stemming from the canonical lasso, clastic net, and Dantzig selector regression methods. We derived upper bounds of Ō ///n) and ((V3/ V7) estimation error, and obtained matching lower bound on minimax risk to establish the optimality of our proposed lasso and Dantzig selector estimators. In particular, our estimation bounds have sub-polynomial dependence on the dimension d of the CDF regression basis, enabling our methods to perform basis selection with exponentially many irrelevant features and furthering progress towards the ultimate goal of general contextual CDF estimation.
We suggest three directions for future work. Firstly, our present analysis holds only when d is finite. A natural continuation of our research may investigate similar basis selection methods for CDF regression with difnite-dimensionn
feature maps. Another promising follow-up direction is to generalize our results to CDF regression with the least absolute deviation (e'-) loss in the vein of previous work which combines robust regression and variable selection (Wang el al., 2007). Lastly, prior literalure has established looser estimation error bounds on Enite-dimensional lasso regression without the restricted isometry property (Zhao & Yu, 2006; Mcinshausen & Yu, 2009). In this spirit, future work may aim to determine unified and weaker necessary conditions for analyzing the functional lasso, clastic net, and Dantzig estimators proposed in this paper.
15

---

Published in Transactions on Machine Learning Research (07 (222)
10a
Ridge
Ridge
Lasso
Lasso
Elastic (As 10
0.35
Elastic Net ()g 10-2)
0.30
0.25
10-1
0.20
0.15
stope 1/2
0.10
0.05
10-2
10
10
10
(a) Effect of sample size n
(b) Effect of sparsity 8
0.200
0.25
0.178
0.20
0.150
B46 0.
Ridge
10
0.125
Lasso
0.15
1.5
Elastic Net (Ag 10 2)
0.100
10
0.06
40
100
80
100
(c) Effect of CDF basis dimension
(d) Effect of elastic net regularization A2
Figure 2: Mcans and standard deviations of ridge, lasso, and clastic net cstimators against various hyperparameters in the synthetic Bernoulli experiments for the random design setting, Figures 2(a) to 2(c) contrast the ridge, lasso, and elastic net estimators with common regularization hyperparameter A = \ 4 (2/n)) log (2d(),,), as defined in Theorems and 3. For further comparison, Figure 2(d) plots the elastic net estimation error against for various values of X2.
Overall, our main contributions and proposed future directions indicate that contextual CDF estimation remains fruitful area of theoretical investigation, accompanied by immediate relevance profusion of downstream scientific applications.
16

---

Published in Transactions on Machine Learning Research (07 2024)
References
Yasin AlbassiYadkenn Dávid Pål, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
Carlo Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance, (6(7):1505-1514 2002.
Philippe Artzner, Freddy Delbaen, Jean-Mare Eber, and David Heath. Coherent measures of risk. Mathematical finance, 9(3):203 228, 1999.
Afonso S Bandeira, Matthew Fickus, Dustin G Mixon, and Percy Wong, The road to deterministic matrices with the restricted isometry property. Journal of Fourier Analysis and Applications, 19(6):1133-1149 2013.
Rina Foygel Barber, Malthew Reimherr, and Thomas Schill. The function-on--salltt LASSO with applications to longitudinal GWAS. Electromic Journal of Statistics, 11(1):1351 1389, 2017. doi: 10.1214/17-FJ512260 URL http::///bd..orr//011111//13311
Amir Beck. Introduction to nonlinear optimization: Theory, algorithms, and applications with MATLAB. SIAM, 2014.
Amir Beck and Mare Teboulle. A fast iterative shrinkagg--hressholdnng algorithm for linear inverse problems. STAM Journal on Imaging Sciences, 2(1):183 202, 2009. doi: 10.1137/080716542. URL httppt:///doioorr// 10.1137/08071642..
E.J. Candes and T. Tao. Decoding by linear programming. IEEE Transactions on Information Theory, 51 (12):4203-4211) 4215, 2005. doi: 10.1109/TII 101109/TTT.005..055 20899858979.
Emmanuel Candes and Terence Tao. The Dantzig selector: Statistical estimation when p much larger than The Annals of Statistics, 35(6):2313 2351, 2007. doi: 10.1214 1.121/000903300000011222 URL
Francesco Paolo Cantelli. Sulla determinazione empirica delle leggi di probabilita. Giorn. Ist. Ital. Attuari, 4(421-424), 1933.
Asaf Cassel, Shie Mannor, and Assaf Zeevi. A general framework for bandit problems beyond cumulative objectives. Mothematics of Operations Research, 2023.
Fabio Centofanti, Matteo Fontana, Antonio Lepore, and Simone Vantini. Smooth lasso estimator for the function-on--function linear regression model. Computational Statistics & Data Amalysis, 176:107556, 2022.
Michael B. Cohen, Yin Tat Lee, and Zhao Song. Solving lincar programs the current matrix multiplication time. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing. STOC 2019, pp. 938 942, New York, NY, USA, 2019. Association for Computing Machinery. TSBN 9781450367059. URL httpp://aa.oogg//00///////113313
doi: 00.1145/3313766.31666
Luc Devroye, László Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition, volume 31. Springer Science Business Media, 2013.
Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. The Annals of Matheratical Statistics. pp. 642 669, 1956.
Valery Glivenko. Sulla determinazione empirica delle leggi di probabilita. Gion. 1st. Ital, Attauri., 4:92 99, 1933.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometries, 12(1):55-66 67, 1970.
17

---

Published in Transactions on Machine Learning Research (07/2024)
Zhaoping Hong and Heng Lian. Inference of genetic networks from time course expression data using functional regression with lasso penalty. Communicaatios in Statislics- Statislics-Theory and Methods, 40(10): 1768 1779, 2011.
Daniel Hsu, Sham M Kakade, and Tong Zhang. Random design analysis of ridge regression. In Conference on learning theory, pp. 9 1. JMLR Workshop and Conference Proceedings, 2012.
Audrey Huang, Leqi Liu, Zachary Lipton, and Kamyar Arizzadenssheli. Off-policy risk assessment contextual bandits. Advances in Neural Information Processing Systems, 34, 2021.
Audrey Huang, Leqi Liu, Zachary C Lipton, and Kamyar Arizzadonesheli. Off-policy risk assessment for markov decision processes. In Artificial Intelligence and Stalistics, 2022.
Cheng Jie, LA Prashanth, Michael Fu, Steve Marcus, and Csaba Szepesvári. Stochastic optimization in a cunulative prospect theory framework. IEEE Transactions on Aulomatic Control, 63(9):2867 2882, 2018.
PAVLO A. Krokhmal. Higher moment coherent risk mcasures. Quantitative Finance, 7(4):373-33) 387, 2007. doi: 10. 00980/166976800015555399
Liang Li, Yongcheng Li, and Qing Ling. Fast algorithms to solve the dantzig selector. In 2013 10th IEEE International Conference on Control and Aulomation (TCCA), pp. 1544-1549. TEFE, 2013.
Leqi Liu, Audrey Huang, Zachary Lipton, and Kamyar Azizzadenesheli. Supervised learning with general risk functionals. In International Conference on Machine Learning, pp. 12570-12592. PMLR, 2022.
Paolo Maranzano, Philipp Otto, and Alessandro Fassò. Adaptive lasso estimation for functional hidden dynamic geostatistical models. Stochastic Enwironmental Research and Risk Assessment, pp. 1-23, 2023.
Pascal Massart. The tight constant in the pp. 1269-1283, 1990.
inequality. The annals of Probability,
Nicolai Meinshausen and Bin Yu. Lasso-type recovery of sparse representations for high-dimensional data. The annals of statistics, 37(1), 2009.
Kevin Murphy, Machine learnóng: a probabilistic perspective. MIT press, 2012.
Jasdeep Panmu and Nedret Billor. Robust group-lasso for functional regression model. Communications in and computation, 46(5):3356 3374, 2017.
LA Prashanth, Cheng Jie, Michael Fu, Steve Marcus, and Csaba Szepesvári. Cumulative prospect theory meets reinforcement learning: Prediction and control. In International Conference on Machine Learning, pp. 1406-1415 PMLR, 2016.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2: 21-42, 2000.
Juan Romo, Rosa E. Lillo, and Nicola Mingolti. Lasso variable selection functional regression. In DES- Working Papers. Statisties and Econometrics, 2013. URL 118891754.
Amir Sani, Alessandro Lazaric, and Rémi Munos. Risk-aversion multi-armed bandits. arXiv preprint arXiv:1301.1936, 2013.
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszezyúski. Lectures on stochastic programming: modeling and theory. SIAM, 2014.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Slatistical Methodology, 58(1):267-28) 288, 1996.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathemalics, 12(4))389 -334, 2012.
18

---

Published in Transactions on Machine Learning Research (07/2024)
Sattar Vakili and Qing Zhao. Mean-variane and value at risk in multi-armed bandit problems. In 2015 53rd Annual Allerton Conference on Communication, Control, and Compuling (Allerton); pp. 1330-1335.
LEEE, 2015.
Jan van den Brand. A deterministic linear program solver in current matrix multiplication time. Tn Proceedings of the Thirty-Firss Annual ACM-SIAM Symposium Discrete Algorithms, SODA '20, pp. 259 278, USA, 2020. Sociely for Industrial and Applied Mathematics.
Martin J Wainwright. High-dimensional statistics: A non-asymptotic vieupoint, volume 48. Cambridge university press, 2019.
Hansheng Wang, Guodong Li, and Guohua Jiang. Robust regression shrinkage and consistent variable selection through the lad-lasso. Journal of Business Economic Statistics, 25(3):347-38) 335, 2007.
Virginia Vassilevska Williams, Yinzhan Xu, Zixuan Xu, and Renfei Zhou. New bounds for matrix multiplication: from alpha to omega. In Proceedings of the 2024 Annual ACM-SIAM Symposium on Discrele Algorithms (SODA), pp. 3792 3835. SIAM, 2024.
Julia L Wirch and Mary R Hardy. Distortion risk measures: Coherence and stochastic dominance. In International congress on insurance: Malhematicc and economics, pp. 15 17, 2001.
William Wong, Audrey Huang, Liu Leqi, Kamyar Azizzadenesheli, and Zachary C Lipton. Riskyzoo: A library for risk-sensitive supervised learning. In ICML 2022 Workshop on Responsible Decision Making in Dynamic Environments, 2022.
Qian Zhang, Anuran Makur, and Kamyar Azizzadenesheli. Functional linear regression of cumulative distribution functions. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
Peng Zhao and Bin Yu. On model selection consistency of lasso. Journal of Machine Learning Research, 7: 2541-2563, 2563, 2006.
Yihong Zhao, R Todd Ogden, and Philip T Reiss. Wavelet-based lasso in functional linear regression. Journal of compulational and graphical statistics, (1(3::000-11 111-600--77,,.0 2012.
Yihong Zhao, Huaihou Chen, and R Todd Ogden. saaelet-based weighted lasso and screening approaches in functional línear regression. Journal of Computational and Graphical Statistics, (4(3):6556675 2015.
Yujie Zhao and Xiaoming Huo. survey of numerical algorithms that can solve the lasso problems.
WIREs Computational Statistics, e5(4)2e1 2023. doi: https://doi.org. http///ooo.or11.11/0/22/// 10.1102//wccs.1000 URL https://wires. onl www.ieeerel witerrr..wleey.ooom /ttp////a///////////////////////
Alexander Zimin, Rasmus Ibsen-Jensen, and Krishnendu Chatterjee. Generalized risk-aversion in stochastic
multi-armed bandits. arXiv preprint orXie:1405 0833, 2014.
Hui Zou and Trevor Hastie. Regularization and variable selection via the clastic net. Journal of the Royal Stalistical Sociely Series B: Slatisticul Methodology, 67(2):301-33) -20, 2005.
19

---

Published in Transactions on Machine Learning Research (07 (07/2024)
A
Notation
Let N denote the natural numbers starting from 1. Lct [n] = n [1. n] denote the set of natural numbers from 1 to n. Let 1(-} denole the Iverson bracket. When discussing empirical CDEs, we use [( = 1( 2 a} to denote translated unit step functions. Let La denote the x d identity matrix. Let A C R denote the (d - --dmmmnnnion probability simplex. Let argı max f() denote the top-k maximizers of function f over a set S. In the context of Landau notation, lct O(-) denote asymptotic upper bounds with hidden logarithmic or sub-logarithmic factors. Throughout this paper, all logarithms have base and all vectors are column vectors. When invoking theorem, use the notation theorem statement.
y pass value to variable in the
Given a vector x E IR,, let [at]s denote its ith entry. For any set 5 C [d], let [a]s E RISI denote the entries of x corresponding to the indices in S. In this context, let S = [d] - denote set complement with respect to the universe of indices. Let supp(æ)) denote the support of x and let ||æ]|o supp(z)) denote the number of non-zero entries in r. Let |allle denote the (--norm of for any p so], let ||>||| denote its Euclidean (E2-) norm, and let || = vx Ar denote its weighted e-norm induced by a positive definite matrix A E Rdxd
Given a matrix E R let [A]ts denote its entry at row i and column j. For any set S S [m]. let [Aks] E | denote the submatrix obtained from A by extracting the rows corresponding the indices in S. For any sets S C [m]] and T S [n], let. [A]s.T E RISSx|TI denote the submatrix obtained from A by extracting the rows corresponding to the indices S and the columns corresponding to the indices in T. Let ||A| denote the induced & Poooprrator norm of A, let Pmin (A) denote its minimum eigenvalue, let (max(() denole its maximum eigenvalue, and let miin (A) denote its minimum singular value.
We denote random variables with uppercase letters and realizations with lowercase letters. We use the notation Ex-øff] to emphasize the random variable and distribution which an expectation is taken over. Given probability distributions Pı,..., Pn over random variables XI...., X, respectively, let 0²= P3 be the product distribution over (X1,.... X,)) Given a probability measure m on a set S, define an inner product between functions f.g S + I as (f.g) = Sg [(t) g(t) dm, and denote the norm induced by this inner product as [l (f, Throughout this paper, integrals and inner products are defined entry-wise for vector-valuee valued and matrix-valued functions.
j=1
B Proof of Lasso Random Design Upper Bound
In this section, we prove Theorem 2. We begin by establishing a restricted eigenvalue condition on the empirical Gramian matrix Un with high probability:
Lemma 2 (Empirical Gramian restricted eigenvalue condition). Fix E (0, 1). Assume that for any n E N, the matriz Σn satisfies the (N, resstrictte eigenvalue condition over S.. Then, for any Z (32d// ( log(d/5), with probobility at least 1 5, the matrir Un satisfies the (m/2, -resttricee eigenvalue condition over S..
Proof of Lemma 2. Fix 2 (32d)) /x²) log(d/d) and CS.).). We have
v U- = v Enu TE + vT (Un En)u
2 1 En + Mmin (Un En, ||0/|2
6 k ||||||| + pmiu (Un Ex) ||u|12
where (a) holds by the restricted eigenvalue condition on Σn, For each j E [n), define a matrix
E 1 TT dm
D, T dm
Observe that U, E,, (1/n) Ej Dy. Our proof uses the following result which we distill from the random matrix theory literature and restate for convenience:
20

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Lemma 3 (Matrix Hoeffding, Tropp 2012, Theorem 1.3). Let AI,..., An Raxd be independent random symmetric matrices, each with mean R [[Ag]] O and operaoor norm ||AB|| s bk- Then, for any T > 0,
7²
Mmax
Σ
A 27
- dexp
(23)
8 b
k=1
Furthermore, two direct consequences of (23) are
1
n
Minxx
Ak
27
=
Umax
AE
2 nT
k=1
(a)
n²272
dexp
(24)
8 83
8
where (a) holds by applying (28), and
n Σ AE
P
Mmin
Σ
Ak
S -
=
-mmin
nn nt
1==1
k=
P
Mmax
As
2 nT
k=1
(0)
n=²²2
dexp
(25)
8
8k 82
where (b) holds by (24) because B[-Ax] -B[As] 0 and ||-Ax| || || < bk-
By inspection, the mean of D3 is E [D,]] 0. Moreover, the norm of D, can be upper-bounded as
|
max
u Dy
||0 =1
T ( 4
dm v v _T
$ T J dm w
max 1|0 =
(u T Φ dm E
(v ))) 2 dm
max
[|]]
1AS
max max ||v==1
(v TT)²²
dm, E
(v) (v)TT,) )² dm
||0|| 1 11? dm }
||||| 110 dm, E
max max [|e] =1
11, dm,
|1, dm
max
d
Φ (2.10)
2 dm, Σ
[ Φi (x()),) (amm
max
m(S)
(26)
d,
(27)
where (a) holds bocause D, symmetric, (b) holds by definition of Dj, (c) holds because |a- S max max (a, B8} for all non-negative numbers and 3, (d) holds by the Cauchy-Schwarz inequality, (e) holds by definition
of f-norm, (f) holds because 6; is contextual CDF for all i E [d], and (g) holds because is probability measure. Thus, with probability at least 1 - 8, it holds that
T UNV 2 x ||lll + Pumin ( n
||/112
21

---

Published in Transactions on Machine Learning Research (07 (07/2024)
log (
2 16 ||ll1
2d
||ill
8
(6)
2 16
||lll
11111
2
( |ull?
where (a) follows from (25) after invoking Lemma with As == D, and T : 2dv(2/n log(d/0), and (b) follows because n 2 (32d2/r² log(d/6). Since (S.) is arbitrary, (c) is the (x/2, reestrict cigenvalue condition over S. desired.
Now, we are ready to prove Theorem 2.
Proof of Theorem 2. Let be the event that Un satisfies the (restt3) eigenvalue condition over S., Invoking Lemma with =1 -j and 7 == 3, we get
P(E) 2 2- 51
(28)
Next, let JF be the event that the estimation error satisfies the desired bounds
00x 0. 8.11s S /
2s
2d
2d
log
12
log
(k/2)n
82
Kn
82
100 - -all $ k/2
2s
[dd
2d
log
log
82
82
Invoking Theorem with = x/2 and 8 = 82, we get P(F 8) 2 82. Thus,
2 ( 8) 1
(1 61) 52) 2 - 51 - 82
desired.
C Proofs of Elastic Net Upper Bounds
In this section, we prove Theorems and 4.
Proof of Theorem 3. Throughout this proof, we restrict to the subset of the probability space where
2d
A1
mo 0. 0_ +,)
log
(29)
which holds with probability at least 1 5 by Lemma 1. For notational simplicity, let A = 0211 0.. We follow the argument from Wainwright (2019, 212-213). We have
A
(a)
(U,, + Aala) A
tAg
T dm
A + da ||
J=1
g 1
A ATD, 2 3 A dm + ||
7
22

---

Published in Transactions on Machine Learning Research (07 2024)
(
2
+ + ||
(3)
72
[c)
+ +110,0yl1e @3](c>(s ) 12 (1ehenee) 1|2 + +110-11
72
"
Σ
OT O1lin2 2y 23)
-2220711200
72
j=l
S
0T A2ida $3 21|2000
- -1167 @ 1)ll(c(smm)
+ 12 (1erecs 12 - -110.1²
71
)=1
A T 0. - 2.,
(30)
where (a) holds by definition of weighted e-norm induced by Un (ef. Zou & Hastie (2005)), (b) holds by definition of Un, (c) holds by the linearity of integration, (d) holds by definition of inner product wet functions, (e) holds by definition of A, and (f) holds by definition of A and the linearity of inner product.
Next, since وx,2 minimizes the objective in (4), it follows that
0 2 ||
| >a |00sniall" 2
+ +1
yon
c3(s,m)
1
gis 0_ 0T 0 01112
Au 110.11 + A2 110.112
n
Expanding the squared norms and rearranging, follows that (cf. Wainwright (2019, Equation 7.29))
1
( - 110TT00;12 (0:::1222(smm) + da (10mn.ee/ "0 -1100113
03
1
c=(8.m)
d1 (10..11- - |02120 l1)
(31)
Combining Equations (30) and (31), we obtain
|||| t (Lyn A 2,)
2 (4
2 n
(A (ATO),, 22 ToATO +11 (10011,
0 10 11)
(32)
Next, we upper-bound ①. We have
2 n
y - 00_ A,
222A
n1
==1
(8)
T
2
(I (Iy() - D. b.)
- -22 00
(33)
(c)
T
2
(100 - 0. 0.
2Aa [A] [0~]s.
1=1
2 Σ
|1I|
(L10) - T. D3)
+ 2A₂ ||A]s, |l, 110.11.0
3=1
23

---

Published in Transactions on Machine Learning Research (07 (07/2024)
(e) A1
- | |||| + 2X2 ||A|| s. |1, |10
22 A1
||هه| 2A2 ||ا| Il
(34)
where (a) and (b) hold by the linearity of inner product, (c) holds because supp(0.) S.,, (d) follows from Hölder's inequality, (e) holds by substituting in (29), and (f) holds because 0, PMF. Next, we
upper-bound ②. We have
0
10.1,
Σ
10. + A11
(
[101,11
Σ 110 + +1111 Σ |[]]]]
FES.
LES.
$E.5
|1
|I
|10-1s. + -A1s, 111
|
s. ll
| ss
(35)
where (a) holds by definition of ll -norm and A, (b) holds because supp(0.) S., (c) holds by definition of
el noorm, and (d) holds by the triangle incquality. Combining Equations (32), (34) and (35), we obtain
3λ1
X |
U
+ 222
||||s. s. 1
(36)
2
2
3d1
IV VA
+ 272
||A|s, 1
2
3λ
+ 22²
V8 ||A|s s. ||
3d1
IA
+ +21²
v
+
2X₂
s
| llu +A2la
2
+ A2
where (a) holds by the equivalence between l- and -noomms, and (b) follows from the restricted eigenvalue
condition on Un since rearranging (36) yields ||A]se]] S (3 (l/A))lllsss satisfies the bounds
lh. Thus, the estimation error
Aallu I
1311 2
+ 222
8
V
+ A2
2d
log
2A2
(37)
8
K + X2
where (a) holds by substituting in Å1, and
(9/12 log
(b) <
-(0/
2d
v
||||
| D
=
+ 2212
V
K
K + A2
where (b) follows from the restricted eigenvalue condition on Un. Finally, choosing d2 (2/n2) (og(2d/8)- 2x results in
(a)
2
2d
| Alr Apla
S
12, log
- 4m V8
3
log
24

---

Published in Transactions on Machine Learning Research (07 (07/2024)
2
2d
4
3
log
8
(6)
18s²
2d
11
log
n
desired, where (a) holds by substituting X2 into (37) and is well-defined because < 3//22n)) log (2d/0). and (b) holds because & 2 0.
Next, prove Theorem 4.
Proof of Theorem 4. Let be the event that Un satisfies the (x/2,3+422/A)) ) eigenvalue condition over S., Invoking Lemma with ò ;= ò1 and r == + 422/A1, we get
P(E) 2 1 - 81.
(38)
Next, let F be the event that the estimation error satisfies the desired bounds
10 - -0.1
o.llentbete
2
2d
6
log
+
+22
82
(n/2) A2
2
2d
28
log
2Xe
(39)
n1
82
K. + 2A2
1 - 0-1l
2d
61
log
+ 2A₂
82
(R/2) A2
2
2d
2/8
6
log
2A₂
82
K. + 2\2
Invoking Theorem with x = x/2 and 8 = 82, we get P(F 2 82. Thus,
P(F) 2 P( n FF) = (P(E) 8) (1 - 61) (1) - 52) 2 1 - 81 - 52
desired. Finally, choosing A2- / (2/ n log (2)
results in
| 0,
(0)
2
2d
2
2d
5
12
log
2k
v28
6
log
82
82
2
2d
=
2s 2s
6
n
log
82
(h)
18s2
2d
4
log
82
as desired, where (a) holds by substituting A2 into (39) and is well-defined because x < (2/2 log (2d/2)) and (b) holds because i 2 O.
D Proofs of Dantzig Selector Upper Bounds
In this section, we prove Theorems 5 and 6.
25

---

Published in Transactions on Machine Learning Research (07/2024)
Proof of Theorem 5. Throughout this proof, we restrict to the subset of the probability space where
7 n Σ (1)) - 0- 0, 3)
2d X
log 5
(40)
which holds with probability at least 1 1-d ò by Lemma 1. Our proof uses the following result which we distill from the statistics literature and restate for convenience:
Lemma 4 (Candes & Tao (2007, Lemma 3.1)). Let A E Rgdxd be symmetric matrix. Assume A satisfies the (c.p))restricted isometry property and the (C, P, -rsttritdd orthogonality property with c c + < 1 and S d/2. Fix u Ie and S C [d] with |5|= =p Let T S U argp max [[]]al. (Refer to Appender for the definition of arg, maxield)-S: -s)) Then,
1- ||4 (1y ul|
S
||
I-c
(1 o) I1
(41)
||l|2 s ||1] + 1
p 11llsol1..
(42)
For notational simplicity, let A = 02 - 0- and St = S, U arg, maxield- s. [[l] Invoking Lemma 4 with A = Un. V ::= A, and S == S., it follows from (42) that
| = VIIAls,/
1|+
+ 1 Assc
(43)
Next, upper-bound ||[A] ss Il..
Observe that
Σ ||
igs.
(a)
|A]++
+ Σ 110. + A1,1-
Σ
[[0.].1
- Σ
iES.
SE
iES.
VES.
d
d
||0, + A Σ
[[0.1.1
Σ AA],
iES.
110, + Al1 + ||| 11, ||0_11 11
16
|OA1| + ||[|] s. 11, 110-11
WV
||. 11,
V8
|A]]s.]]
S
|
(44)
where (a) holds by the reverse triangle incquality, (b) holds bocause supp(0=)= = S., (c) holds by definition
of A, (d) holds because 02 and 0- both satisfy the constraint (6) and 02 minimizes the f--norm among all such vectors by (5), and (e) holds by the equivalence between et and 22-nrrss. Combining Equations (43) and (44), we obtain
|| S v2 Als
(45)
Next, we upper-bound [[[A]s, |]. We have
|
(a)
Unlls
S all + (1- -s-e -
Als:||
--e
26

---

Published in Transactions on Machine Learning Research (07/2024)
@ - [nl(ss 1
S |واا|
where (a) holds by (41) and (b) holds by (44). Rearranging, follows that
مااو |
s i c- ||U A||
1
(46)
Next, we upper-bound (s, All.
We have
(a)
A
S 228
[U A
$
<2s
|U A 1l:
12
V28
f. 83)
20
( 28
Σ
(1/00) 0g. - (Iyin) o; 221))
(47)
71
1
(d)
1 n Σ
n
S
28
(Tyt) 0. p3. b3)
(I (10))
0 33 13)
(48)
12
7 j=1
8
(c)
n
228
A
(Iy0 - 01 03 bT)
n j=1
5 S 28
(49)
where (a) holds by the equivalence between e- and normss, (b) holds by definition of Un, (c) holds by the linearity of inner product, (d) holds by the triangle inequality, (e) holds by substituting in (40), and (f) holds because Ox satisfies the constraint (6). Combining Equations (45), (46) and (49) and substituting in X, we obtain
4XV8 ||||| S 1-c- 1- c- 5\ n
4
288
2d
log
desired.
Next, prove Theorem 6.
Proof of Theorem 6. Let E be the event that
Mmax (Un - En) S min {e, () and Amin (Un - En) 2 min [e.
(50)
Under this event, Un satisfies the (2e, 2s) restriitd isometry property, because for any v Idd with |ello s 2s, 0 T Un v T En + (Un - En)w
S v E,, T prnaxe (Un Enn ||0/|
(2) (1+ ||0|2 + Umax(Un En) ||/||2
(0)
02 (1+ 2c) ||,1|
where (a) holds by the restricted isometry property on E,, and (b) holds by (50), and
T U 2 En + pmnin (U, End 10122
27

---

Published in Transactions on Machine Learning Research (07/2024)
2 (e)
(1 - |1012 + pmin (Ux - En) |101
2 (1 2e) ||0112
where (c) holds by the restricted isometry property on E, and (b) holds by (50). Furthermore, Un also satisfies the (2C, 8, --esstiiied orthogonality properly under event &, because for any u, v E Rd such that |lullo p, I|v|lo $ g, and supp(u) supp((p)
=0,
lu Un
[u En u T (U,, En)ol
W4 11 3
11 E + T (Un En)ul
1 E + ||/|| IlU, En
En max {Mmax (Un En); - Umin(Us Σn)] [|u]]] ||]]
< ||4||| ||i||| + max {mmaxe (Un En), -Mmin (Un En [|u]] [|]]
2
[|]]]] ||ll|
where (a) holds by the triangle inequality, (b) holds by the Cauchy- inequality, (c) holds by the restricted orthogonality property on En, and (d) holds by (50). The probability of event & is
P(E) P(/max (Un - En) 2 min te, (}) P' (Un - En) S -min {c, (})
(6) 1 2dexpp
n² min ()
(51)
8nd2
(c)
2 1 - 81.
where (a) holds by the union bound, (b) holds by Equations (24) and (25) by the argument in (27) after
invoking Lemma 3 with 7 := minfe, {), and (c) holds because n 2 (&d2 (iin(,,)))) log(d//))).
Next, let F be the event that the estimation error satisfics the desired bound
28
2d
|00₂ - 0.1 - 1-2e 2e 2ç V
log
82
Invoking Theorem with c: 2c, Ç = 2G, and 0 = 62, we get P(F [) 2 1 82. Thus,
P(F) 2 P( = P( PP 8 8) 61) - 52) 2 1 - 8, 82
as desired,
E Proof of Lower Bound
In this section, we prove Corollary 1.
Proof. We have
2e 0 ((ddit)) ))
[((y)m)]
inf sup E
- -10(P)ls.(P)
a pepdaa y
S_(P)
[[[])))]]]] 10(P)ss.())
inf sup
sup SCid] repd kin
E y
| S.(P=S
28

---

Published in Transactions on Machine Learning Research (07/2024)
( inf
[((]m)) 0(2)
sup
sup
s |5|=a
QEP 2im Y Q
[ e(0)
inf 6
inf
sup
S d
QEP
inf
sup
E PI
@( (m) (1))
(52)
QEP
where in (a) we partition the distributions in Pais based on the support of their parameter, (b) holds because every distribution P E pd,a has an equivalent distribution Q Patm with ((Q) = [0(P))s.. (P), and (c) holds because every estimator 0 E θa, and set S S [d] with cardinality |S| defines estimator ((.) = [0))ss θ.,n. Our proof uses the following result which we distill from the literature and restate for convenience:
Lemma 5 (Zhang et al. (2024, Theorem 8)). For any d E N, n E N, and x E xe, the minimax C-risk of general contertual CDF regression satisfies the lower bound
R(O(P inf sup BeGa,n Pepa v zimn
-
d
2 Ω min 1,
1 sup PEPA alin
Punin (U, (Un(P))
Invoking Lemma 5 with d == s and combining the result with (52), we obtain
Hi 0 (pd)))
Ω
min 1,
(53)
sup
Pmin (Un (Q))
QEP
Next, we upper-boune pmin (U)) for any Q E pi Observe that
(a)
Punn (U,
<
(r(U,(())
(
(n)l)
dr (1(2))))))
dm
(8) 1
(S)
(54)
72
(55)
where (a) holds because the irace of matrix is the sum of its eigenvalues, (b) holds because the trace of a matrix is the sum of its diagonal entries, (c) holds by definition of Un. (d) holds because (x(x)),, is CDF.
and (c) holds because m is a probability measure. Combining Equations (53) and (55), we obtain
be (8 (p(p....) mdin )) 2 Ω min 1,
1 n
S
> Ω
min
1
2n
29

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
(a) Ω V
as desired, where (a) holds because n 2 8/2.
F
Examples
In this section, we present three examples of contextual CDF bases which non-trivially satisfy the restricted cigenvalue condition (i.e., with K > 0 in Definition 1).
F.1
Categorical CDFs
Our first example considers the expected Gramian matrix En of family of categorical CDFs drawn independently from a fixed set of Dirichlet priors:
Proposition 1 (Categorical CDFS). Pir d E N Dirichlet distributions Dirichlet(ar.1,... ai,K) indezed by E [d]. For each Dirichlet distribution, let 0.o E Ek=1 0i,k denote the sum of its porameters and let ãs,k ai.x/ou, denote its normalized kth parometer. Assume E N i.i.d. context samples generated, where each context sample z(56 = =xx)) (x.")....;"?) the respective Dirichlet priors, i.e.,
(?!) consists of d categorical PMF's sampled independently from
(3) Vi [d], 2! 1i........ F(KK
(1) Diri aiik)
Given contert sample as defined above, let the ith contertual basis function di(2, t) be the CDF of Categorical(,........ .(,K) random variable. Let mn be the uniform measure over the support S = [1, K]. Then the matrix En sotisfies the (ve restsiiiete eigenvalue condition over S. for
K-
K
K
min seld ai,0 +1
K
k=1
t=1
and arbitrory 2 and S, S [d].
Proof. The entries of Σn are
1
Enke
E
Φ: (x(), Φr (xUi, dm
7
(6)
IE
0i t) oe (X, dm
K /
(c)
TE
(x(X, 1) de (x,1) di
K-
K-1
k+1
E
Σ Xi,e
Σ Xv.
E
Σ Xi.
X
K
where (a) holds by definition of E,, (b) holds because the context samples X are i.i.d., (c) follows from the support S = [1, K] and the choice of mcasure, and (d) holds by splitting the domain of integration and substituting in the definition of d.. The diagonal entries of Σn (i.e., 1') are
K-1
k
&
Enkss
Σ
E
K
30

---

Published in Transactions on Machine Learning Research (07 (07/2024)
K
0,,0 + āie
(
0,100
04,0 +1
04,0 +
t=1
0i.0
Σ
+
Cit
(56)
01,0 +
05
where (a) follows from the properties of the Dirichlet distribution. The off-diagonal entries of Σn (i.e., i + 1')
are
K
1
k
X
E
Σ
X
K
k=
1=1
K-1
k
6
Σ
(57)
K
k=
where (a) holds because the categorical distributions are pairwise independent and (b) follows from the properties of the Dirichlet distribution. Thus, for any IR,
d
En
Eoley [o], [u],
2
06i.0
Σ
+
Öi,t
[u]
0,i.0
06,0
1
P=1
2=1
Σ
[o]
Lole
Σ
āi,
Σ
āi,t
[0]
Σ ār.e
K
04,0 +1
K-1
Σ
āid
K
04.0 +
K
}
min EE|d]
Σ
āie
|0/1
K
04,00 +
k=1
f=1
1=k+1
K ||
as desired, where (a) holds by substituting in Equations (56) and (57).
F.2 Step CDFs
Our second example considers the empirical Gramian matrix Un of a family of shifted step-function CDFs:
Proposition 2 (Step CDFs). Fix d E N. Let X be the space of all d-tuples x = (#1...., Ia) such that
0 0 x < ... < Ea < 1. Consider the contextual CDF basis
Vi [d], dx t) = 1 2 n;}
Lel m be the uniform measure over the support 5 = [0, 1]. Then for any n E N and I E X", the matric Un satisfies the (n, resttriced eigenvalue condition over S. for
1 min
Σ
(
(1)
a
4n
31

---

Published in Transactions on Machine Learning Research (07 (07/2024)
and arbilrary y and S. S [d],
(5) where for notalional simplicity we define xd+1 1 for each j E [n].
Proof. For any E Rd, we have
UnU
(4)
Σ
0 J dm
j=1
(6)
n
T (Tb)) dm
71
j=1
ΣΣ
n T $
2 di
Σ
Σ
[0], 1( t 2 {(()
})
)
dt
d
Σ [0] dt
d
h
(
x
Σ |00,
144
20 1 #
13 [o]
(58)
where (a) holds by definition of Un. (b) holds by the linearity of integration, (c) follows from the support 5 = [0,1] and the choice of measure, (d) holds by substituting in the definition of on and (e) holds by splitting the domain of integration. For each j E [n],, define a matrix A₁ E Rdddd given by
Al 2 2t 1ii k).
(59)
Combining Equations (58) and (59), we obtain
1 U
A
n
j=1
1
|| 2
n
1
mmin (A₂)² ||11
n
| 2
(60)
Next, we upper-bound |1.7 Since Ay is lower triangular with identical values from the main diagonal leftwards in each row, its inverse is the bidiagonal matrix AJ D, + Cj given by
DDiks = (1)) (1/41 - =110) - 1fi = i'}, =1},
1Cy,e = (xlf)) -((1it) 2(!") -
lli i' +1}.
32

---

Published in Transactions on Machine Learning Research (07 (2742)
It follows that
(a)
||47||| = |2l ||0/|
(6)
Tit1 - 2z -
(1)
seld-1] 1141 1; (11) -
(5)
-(1))²
max
+ max
KEId]
1++1 - 2z )ー
(1)
-
I 2 max
sE|d
iE(d] ii1 - (1)) 1))
-
= min (1)
(61)
where (a) holds by the triangle inequality and (b) holds by the diagonal structure of D, and Cj. Combining Equations (60) and (61), we obtain
v U 2
Σ
min ( j=1 tl|d
21 (3) ||u11
-itl
4n
K ||u||
F.3
Polynomial CDFs
Our third example considers the empirical Gramian matrix Un of family of shifted and scaled polynomial CDFs of varying degrees:
Proposition 3 (Polynomial CDFs). Fir d E N and E (0.1/d). Let X be the space of all d-tuples x (xi,...,Ia) Id) of pairs d4 = (b,, r;) E [0, 1] x (0, 00) such that bit1 by 2 for all i E [d - 1] and 1 - ba 2 a.
Consider the contextual CDF basis
i < bi if t E [b,, by + a)
[d], (((x,))
[t=by
ift t 2 b; a.
Let in be the uniform measure over the support S = [0, 1]. Then for any n N and al E Ex", the matriz Un satisfies the (F,77) -restticcte eigenvalue over S. for
1 n
(1 min OJA.A max St,j
4 kE|d]
sEld
and arbilrury 7 2 and S. C [d], where for notational simplicity we define
())
+0(1)
is k d,
(rL
(12 +1)
Oj,K
+0 + -
bd
if k = d.
ar!
(7)
Big
r1 +
25 +1
for all i [d]. j E [m), and is [d].
Proof. The entries of U,
UUnla (
(a)
Σ
2i (2)),) or (26)))) dm
n 1=1
33

---

Published in Transactions on Machine Learning Research (07/2024)
1 n2
[ @i (ab)).) 1) or (200), 1) di,
d)
where (a) holds by definition of Un and (b) follows from the support S =00, 1] and the choice of measure. The diagonal entries of Un (i.e., i = 2') are
Unles
Φ (z()) t) dt
20(0)
1(21 +o
(1 - 1101
(a)
dt +
1 dtt +,
((0)
a
+ 1 (1)) a
2r +1 1
ar!
6(00
1
(1) 1
11
ri
(i)
+
(2 (0)
+
Bus
where (a) holds by substituting in the definition of oz. The off- off-diagonal entries of Un (i.e., i < <z) are
(
+0
1
(5)
dt + [
(Un]e
1 de
Joy+a
j=l
a
+ +1 - 8/
(
11
where (a) holds by substituting in the definitions of Φ; and dv. Next, we characterize ① For any j E [m] and i [d] such that i < i', we have
d-1
a
a
(1)
((6)
a
+ - a - (66)
( +1
k+1
-
+
(
km
+1 1
k+1
d
+
(10) - (1)
-1())
K+1
(1)
(
+ a 1661
(1 +1) ('htl +1)
k+1
+
Id +1
(2)
[+l -r(())
+ k-1
1[k 2 1) 1 21 +
+ 1
((6)
(
+ 1)
(n/6) +1)
(1) dd
+1
kk
aj,a
where (a) holds because the sum over k telescopes and (b) holds because S f'. For each j E [n], define
matrices Aj, B5 E Rdxx given by
[Ajle, Vajkk 1 i
[Bilag = Bis lii 1ii= :1=i/] -'}.
Combining the above, we obtain U, = (1/n) ΣØー (AJ A, - -B,). Thus, for any u E Rd,
0. -
1 Σ
(uTA]
A7 A₁e - BB,)
34

---

Published in Transactions on Machine Learning Research (07/2024)
[14 - -||//| || [|
n (omia 1|u)? - ||B [|=l²))
1
Σ
1
Σ
|| [| -1131 ||o12.
(62)
Next, upper-bound || Since A, is lower triangular with identical values from the main diagonal leftwards in each row, its inverse is the bidiagonal matrix A7 = D2 + Cj given by
1 li; =
[Dile
Vogj,
1[i= = k: + 13.
Vag.
It follows that
(0)
|4,4| S
| |1lll
(6)
1
1
max
max
kE di
Vajik
kE[d-1]
s max kE(d]
Vaj.k
= min ai
(63)
kE(d
where (a) holds by the triangle incquality and (b) holds by the diagonal structure of D, and Cj. Furthermore, |,رl = maxxeld Big by the diagonal structure of Bj. Combining Equations (62) and (63), obtain
U 2
Σ5
n
1
min as, max Sc,) |101 kE(d]
n2
4
sE|d]
= K ||01
Lastly, present specific instance of the polynomial CDF basis where x 0 is readily apparent. Let d, n E N be arbitrary and let 1/(d+ 1). For each i [d] and [m], let 6,(7) (i-1)/(d+ 1) and r(()) i.
Then,
1 min 0j,s
1
min (41)(-1I(k+2)
I+I
if << d k
4 kE[d]
kEld
(d+1)= + d+1'
1
4(d + 1)
max xe[4]
(d+ + 1)(i + 1)(2i + 1)
max Bij =[[d]
for each j [n]. and so > 0 desired.
35

---

Published in Transactions on Machine Learning Research (07 (2742)
G Computational Complexity
In this appendix, we discuss the computational complexity of computing lasso, clastic net, and Dantzig selector estimators. First, observe that after pre-computing the matrix A = -ー Ss 4, , dm E Rdddd and the vector b = -ド= Js yyo F,dm Rd, our lasso estimator, elastic net estimator, and Dantzig selector can be directly computed using any method for the standard lasso estimator, clastic net estimator, and Dantzig selector in Rdd Specifically, the standard lasso estimator, elastic net estimator, and Dantzig selector in R with feature matrix X E [Rd and response vector E R" solve
BLass := arg min BERd
y x0| +110111
n
(0T Tx 20 (x v)) + 311111
arg min DE
n
Dlastiic = arg min BERd
lly x01 111111111 + >=110113
n
(0 (X x +21000 20 (XT v)) =110111 }
arg min BERd n
and
@Dantigg == minimize
||0|
EER""
1 | y x x s,,
subject to |
n
respectively. Any algorithm A that solves the above problems only accesses the matrix X X E Raxd and the vector X y TRd from the problems. Therefore, noticing the similarity in the structures of our problems and the above standard problems, by replacing X X with A and X y with b defined previously, we can solve our estimator directly using A.
Now, we analyze the computational complexity for calculating A and b, which consists of the evaluation of O(nd)) onc-dimensional integrals. In practice, users can pick an appropriate numerical integration mothod based on the specific form of the contextual CDFs & if it hard or impossible to analytically calculate the integrals. Here, we consider the standard trapezoidal rule for numerical integration. For m grid points, the computational complexity of integrating using the trapezoidal rule is ((m) (assuming single CDF evaluation is constant time). Thus, the total computational complexity for the pre-processing step is 0(mnd*).
Next, we use the above reasoning to analyze the computational complexity of our estimators using existing algorithms for standard lasso, elastic net, and Dantzig selector problems. Lasso and elastic net estimators can be computed using the fast iterative srrinkagg-thrssooddng algorithm (FISTA) (Beck & Teboulle, 2009) which iteratively optimizes the objective function based on gradient evaluations. In the pre-processing step of FISTA, we evaluate the matrix A and the vector b, which takes (mmdd) time as analyzed above. After the pre-processing step, each iteration of FISTA takes O(d²) computation time. Thus, for k E N iterations of FISTA applied for lasso and elastic net estimators, the computational complexity is O(mn + k)d?), where m is the number of grid points used for numerical integration. Furthermore, the convergence rate of FISTA is O(1/k²) (Zhao& & Huo, 2023).
For the Dantzig selector, we also first calculate A and b in O(mnd?). Then, after introducing slack variables u, U E Rdd, (5) can be recast as the following standard lincar program (Li et al., 2013):
minimize G,u.1
Σ d1
subject to - A0 = V.
U; S 0, S ue Vi [d] -nd vi nd, Vi e (d.
36

---

Published in Transactions on Machine Learning Research (07 (07/2024)
One can invoke any preferred linear programming solver to solve the above problem. Theoretically, the computational complexity for solving the above linear programming problem with accuracy 8 is O(d=+oll) log(d/6)) in expectation for the randomized solver in Cohen et al. (2019) and O(d (d)lo((//6)) for the deterministic solver in van den Brand (2020), where w the exponent of matrix multiplication complexity with the current bound w S 2.371552 (Williams al., 2024). Thus, the overall computational complexity of solving the Dantzig selector problem O(mnd d= log 2 (d) log(d/6) (for the deterministic
solver).
Finally, remark that in the classical finite-dimensionall lasso, elastic net, and Dantzig selector problems, the overall computational complexities are the same as those listed above with m = 1. This is because cach integral can be replaced with a single scalar multiplication in the pre-processing step.
37


Reviews:
[{"review_id": "8cnHbc5dnD", "reviewer": "Reviewer_G4mk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper is exceptionally well-written and easy to follow.\n\n- The problem addressed in this work is relevant and appealing to a wide audience within the machine learning community.\n\n- The mathematical derivations are precise and rigorous, based on my review. I did not identify any significant mathematical errors.\n\n- The paper proposes and analyzes several estimators, providing a comprehensive investigation into the topic.\n\n-------------------------\n\n**Weaknesses**:\n\n- The primary weakness is that the contribution of this work appears somewhat incremental. While the combination of contextual CDF estimation and sparse regression is novel, the individual ideas are not surprising or groundbreaking. The results align with expectations.\n\n- The proofs are relatively straightforward and follow standard procedures from (for example) Wainwright's works combined with prior recent works on CDF regression, which is understandable given the similarity between sparse regression and sparse contextual CDF regression. The same simplicity extends to the lower-bound proof, which basically is a slight varaition of the bound given by Zhang et al. (also outlined by the authors).\n\n- Theorem 2, which presents the error upper-bound for random design, has a high sample complexity. Although the error decreases as $\\mathcal{O}\\left((s/n)^{1/2}\\right)$, the requirement that $n$ be greater than $\\mathcal{O}\\left(d^2\\right)$ contradicts the claimed sample complexity improvement at the paper's outset. This issue primarily affects the random design and not fixed design scenarios.\n\n- Proposing the Restricted Eigenvalue Condition as an analogue of RIP for sparse contextual CDF regression is a natural and promising approach. However, it would enhance the paper if the authors could provide converse results to demonstrate that this condition is not only sufficient but also necessary for the problem at hand.\n\n- The paper lacks a detailed discussion on the computational complexity of the proposed estimators. It would be beneficial for the authors to address this aspect, especially in comparing the computational demands of CDF regression versus ordinary sparse regression as found in existing literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates sparse contextual CDF regression using several newly proposed estimators. A contextual CDF is a parameterized cumulative distribution function (CDF) where an additional input, known as the \"context\", can vary the form of the CDF. To manage the complexity, the authors limit the number of potential contextual CDFs by employing a fixed finite basis, making the problem theoretically approachable.\n\nIn this scenario, the authors posit an unknown true contextual CDF that is a (sparse) convex combination of $d$ known base CDFs. For each observation $j \\in [n]$, a context $x^{(j)} \\in \\mathcal{X}$ is chosen either through fixed design or randomly sampled from an unknown distribution (in random design). Conditioned on the context, a sample is obtained from the convex combination. The objective of the paper is to recover the coefficients of this convex combination based on the $n$ observed samples, with the coefficients assumed to be $s$-sparse.\n\nThe authors introduce three estimators for this purpose: i) the Lasso estimator, ii) Elastic Net estimator, and iii) the Dantzig selector. Assuming a geometric regularity condition on the basis CDFs (termed as the Restricted Eigenvalue condition in Definition 1, similar to RIP), the authors provide upper-bound estimates for the error across all scenarios, complemented by some lower-bound analyses. The upper-bound results, akin to conventional sparse regression, indicate an error reduction rate of $(s/n)^{1/2}$ rather than $(d/n)^{1/2}$, demonstrating how leveraging sparsity as a structural property can diminish the required sample size, as anticipated.\n\nThe paper includes various numerical simulations towards the end, although I have not validated these results myself.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the Weaknesses section.", "broader_impact_concerns": "none.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uhXdMZcsXk", "reviewer": "Reviewer_eGni", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\nThe problem setting sounds interesting, and potentially generalizable to other settings. \n\n**Weakness**\n\nThe main weakness of this work is the novelty. The algorithm and analysis seem to solve more-or-less standard sparse regression problems with RIP style conditions. I wonder why the paper had to be specifically about “CDF” regression. Compared to a more well-studied setting of sparse linear regression, can the authors elaborate what is the technical novelty in this work to solve the “CDF” regression? (also, what is the main technical challenge that was not in sparse linear regression?)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies sparse CDF regression. For Lasso and Dantzig selector formulation, the paper obtains $\\sqrt{s \\log d/n}$  rate, and for elastic net estimator, the paper obtains $(s^2 \\log d / n)^{1/4}$.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In the abstract, it might be better to clarify the following points:\n\n-\tWhat does s-context dependent CDF mean?\n\n-\tWhen we have $1/\\sqrt{n}$ estimators, why should we care $n^{-1/4}$ estimator?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XMkTrpuo9I", "reviewer": "Reviewer_tKKA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nI enjoyed reading this paper, as all the theoretical results and proofs are presented clearly. Furthermore, the sketch of the proof provided under each theorem is quite helpful in understanding the key technical steps. The setting considered in the paper is general enough for the audience of TMLR.\n \n \n \nWeaknesses: \n\nLack of experiment justification: \nOnly synthetic results for LASSO under fixed design are provided. Can the authors enlarge the numerical section by including results for LASSO for random design settings and fixed design settings for either Elastic Net or Dantzig Selector?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of learning a sparse mixture weight of contextual cumulative distribution functions (CDFs). Specifically, data points are generated from a convex combination of $s$ context-dependent CDFs from a set of $d$ known basis functions. This paper adapts three conical regression methods, i.e., LASSO, Elastic Net, and Dantzig Selector, to this new problem and provides the corresponding estimation error upper bounds. It is shown that these $L_1$ based approaches achieve better rates compared to $L_2$ regularized ridge regression. Moreover, a matching information-theoretic lower bound is provided, establishing the minimax optimality of the lasso and Dantzig selector estimators.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I noticed that random design results are only provided for LASSO. Could the authors generalize such an analysis to Elastic Net and Dantzig Selector? Or at least describe the challenges of such an extension. I believe such a discussion could be insightful for the readers.\n \n- I felt quite confused reading the first two pages of the paper, but the considered problem became much more apparent after going through the math model on page 3. I think the basic math model of the problem can be moved to section I to improve readability, as the formulation is quite neat.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies sparse contextual CDF regression, and provides convergence rate analysis for Lasso, elastic net, and Dantzig selector estimators. The results match the corresponding error bounds for finite-dimensional regression. Furthermore, the Lasso and Dantzig selector estimators are shown to be minimax optimal (up to logarithmic factors). The revised version clarifies a few technical questions raised in the reviews, and also elaborates technical differences from the finite-dimensional case. While there are concerns over significance, the paper contains useful contributions, is well-written, and should be accepted to TMLR.
Decision: Accept (poster)