Title: Agnostically Learning Single-Index Models using Omnipredictors
Venue: NeurIPS 2023
Accepted: yes
Keywords: generalized linear models

single-index models

agnostic learning

pac learning

logistic regression

omnipredictors

multiaccuracy

calibration
Abstract: We give the first result for agnostically learning Single-Index Models (SIMs) with arbitrary monotone and Lipschitz activations. All prior work either held only in the realizable setting or required the activation to be known. Moreover, we only require the marginal to have bounded second moments, whereas all prior work required stronger distributional assumptions (such as anticoncentration or boundedness). Our algorithm is based on recent work by Gopalan et al. [2023] on Omniprediction using predictors satisfying calibrated multiaccuracy. Our analysis is simple and relies on the relationship between Bregman divergences (or matching losses) and $\ell_p$ distances. We also provide new guarantees for standard algorithms like GLMtron and logistic regression in the agnostic setting.

Agnostically Learning Single-Index Models using Omnipredictors

Abstract

We give the first result for agnostically learning Single-Index Models (SIMs) with arbitrary monotone and Lipschitz activations. All prior work either held only in the realizable setting or required the activation to be known. Moreover, we only require the marginal to have bounded second moments, whereas all prior work required stronger distributional assumptions (such as anticoncentration or boundedness). Our algorithm is based on recent work by Gopalan et al. [2023] on omniprediction using predictors satisfying calibrated multiaccuracy. Our analysis is simple and relies on the relationship between Bregman divergences (or matching losses) and ℓ p distances. We also provide new guarantees for standard algorithms like GLMtron and logistic regression in the agnostic setting.

1 Introduction

Generalized Linear Models (GLMs) and Single-Index Models (SIMs) constitute fundamental frameworks in statistics and supervised learning [McCullagh, 1984, Agresti, 2015], capturing and generalizing basic models such as linear and logistic regression. In the GLM framework, labeled examples ( x , y ) are assumed to satisfy u ( E [ y | x ]) = w · x (or E [ y | x ] = u -1 ( w · x ) ), where u is a known monotone function (called the link function) and w is an unknown vector. Single-Index Models (SIMs) are a generalization in which the monotone link function u is also unknown.

In the realizable setting where the labels are indeed generated according to a GLM with a known Lipschitz link function, the GLMtron algorithm of Kakade et al. [2011] is a simple and efficient learning algorithm. When the ground truth is only assumed to be a SIM (i.e. the link function is unknown), it can be learned efficiently by the Isotron algorithm [Kalai and Sastry, 2009, Kakade et al., 2011].

In this work, we consider the significantly more challenging agnostic setting, where the labels are arbitrary and not necessarily realizable by any SIM. Importantly, we do not fix a link function in advance; our goal is to output a predictor that has squared error comparable to that of the optimal SIM with an arbitrary monotone and Lipschitz link function. We can equivalently view this as a natural squared-error regression problem in which the final optimality guarantee must hold with respect to all SIMs with bounded weights and monotone, Lipschitz link functions. 1

Formally, consider a distribution D over R d × [0 , 1] and denote the squared error of a function h : R d → R by err 2 ( h ) = E ( x ,y ) ∼ D [( y -h ( x )) 2 ] . Let opt(SIM) denote optimal value of err 2 ( h ) over all SIMs h with bounded weights and arbitrary 1 -Lipschitz monotone activations (we call the inverse u -1 of a link function u the activation function). Given access to samples from D , the goal of an agnostic learning algorithm is to compute a predictor p : R d → [0 , 1] with error err 2 ( p ) that, with high probability over the samples, is comparable to the error of the optimal SIM:

Our main result is the first efficient learning algorithm with a guarantee of this form.

Theorem 1.1 (Informal, see Theorem 3.1) . Let SIM B denote the class of SIMs of the form x ↦→ u -1 ( w · x ) for some 1 -Lipschitz function u -1 and ∥ w ∥ 2 ≤ B . Let D be any distribution over R d × [0 , 1] whose marginal on R d has bounded second moments. There is an efficient algorithm (Algorithm 1) that agnostically learns SIM B over D up to error

This result provides a guarantee comparable to that of the Isotron algorithm [Kalai and Sastry, 2009, Kakade et al., 2011] (which also tackles the SIM setting, where the link function is unknown) but for the challenging agnostic setting rather than the realizable setting (where opt(SIM B , D ) = 0 ). Moreover, Isotron's guarantees require the distribution to be supported on the unit ball, whereas we only require a mild second moment condition.

In our view, this result helps establish Algorithm 1 (and indeed any algorithm to compute omnipredictors, as we introduce and discuss shortly) as an efficient and powerful off-the-shelf supervised learning algorithm, akin to random forests or gradient-boosted trees.

Anatural question is whether our guarantees are near-optimal, e.g., whether we can obtain a guarantee of the form err 2 ( p ) ≤ opt(SIM) + ε . However, there is strong evidence that such results cannot be obtained using efficient algorithms [Goel et al., 2019, Diakonikolas et al., 2020a, Goel et al., 2020, Diakonikolas et al., 2021]. We partially justify the form of our guarantee by showing in Section 5 (adapting a result due to Diakonikolas et al. [2022a]) that one cannot avoid a dependence on the norm bound B . Further closing the gap between upper and lower bounds is an important direction for future work.

Overview of techniques. We now describe the main ingredients and techniques that go into proving Theorem 1.1. Our starting point is the connection between GLMs and so-called matching losses [Auer et al., 1995]. This connection arises from the fact that fitting a GLM with a known link function is equivalent to minimizing (over the class of all linear functions) a certain convex loss known as the matching loss corresponding to that link function (see Definition 1.8). (We refer the reader to [Gopalan et al., 2023, Sec 5.1] for a more detailed discussion.)

Importantly for us, minimizing the matching loss corresponding to a link function u yields a meaningful guarantee even in the agnostic setting, where the Bayes optimal predictor (i.e. x ↦→ E [ y | x ] ) is not necessarily a GLM at all. Specifically, it turns out to be equivalent (via Fenchel-Legendre duality) to finding the closest predictor to the Bayes optimal predictor in the metric of the Bregman divergence associated with the link function u (see e.g. [Gopalan et al., 2023, Lemma 5.4]).

This is a powerful observation, but it still assumes that we have a fixed and known link function u in mind. In the agnostic setting, it is arguably unrealistic to assume that we know the best link function for the distribution at hand. Remarkably, recent work by Gopalan et al. [2022, 2023] has shown that there exist efficient learning algorithms that simultaneously minimize all matching losses corresponding to arbitrary monotone and Lipschitz link functions. Their solution concept is called an omnipredictor , i.e., a single predictor that is able to compete with the best-fitting classifier in a class C as measured by a large class of losses (as opposed to just a single pre-specified loss, as is standard in machine learning). They obtain such predictors through calibrated multiaccuracy [Gopalan et al., 2023] or multicalibration [Gopalan et al., 2022].

From the point of view of ordinary supervised learning or regression, however, an optimality guarantee in terms of such matching losses or Bregman divergences is hard to interpret. A much more standard metric is simply that of squared error. The key final step for our results is to find a way of translating (a) optimality (ranging over all linear functions) in terms of all matching losses simultaneously into (b) optimality (ranging over all SIMs) in terms of squared error. We do so by proving simple analytic distortion inequalities relating matching losses to ℓ p losses, which we believe may be of independent interest.

On a technical level, to prove these distortion inequalities we first prove strong versions of such inequalities for matching losses arising from bi-Lipschitz link functions (see Lemma 2.2). We then obtain our results for general Lipschitz link functions by carefully approximating them using bi-Lipschitz link functions (see Lemma 3.3).

Further applications. As further applications of our approach, if we let opt(GLM u -1 ,B ) denote the optimal value of err 2 ( h ) over all GLMs of the form x ↦→ u -1 ( w · x ) , where ∥ w ∥ 2 ≤ B , we obtain the following result about bi-Lipschitz link functions (including, for example, the Leaky ReLU).

Theorem 1.2 (Informal, see Theorem 2.1) . Let u : R → R be a bi-Lipschitz invertible link function. Then, any predictor p : R d → R that is an ε -approximate minimizer of the population matching loss that corresponds to u , with respect to a distribution D over R d × [0 , 1] satisfies

This guarantee holds under milder distributional assumptions than are required by comparable prior work on agnostically learning GLMs or single neurons [Frei et al., 2020, Diakonikolas et al., 2022b]. Moreover, when we focus on distortion bounds between the logistic loss and the squared loss, we obtain a near-optimal guarantee of ˜ O (opt(GLM u -1 ,B )) for logistic regression, when u is the logit link function (i.e., when GLM u -1 ,B is the class of sigmoid neurons).

Theorem 1.3 (Informal, see Theorem 4.1) . Let u ( t ) = ln( t 1 -t ) . Then, any predictor p : R d → R that is an approximate ε -minimizer of the population logistic loss, with respect to a distribution D over R d × [0 , 1] whose marginal on R d has subgaussian tails in every direction satisfies

While our error guarantee for this problem is weaker than that of Diakonikolas et al. [2022b], we do not make the anti-concentration assumptions their results require.

1.1 Background and Relation to Prior Work

We note that matching losses have been studied in various previous works either implicitly [Kakade et al., 2011] or explicitly [Auer et al., 1995, Diakonikolas et al., 2020b, Gopalan et al., 2023] and capture various fundamental algorithms like logistic and linear regression [McCullagh, 1984, Agresti, 2015]. However, to the best of our knowledge, our generic and direct approach for transforming matching loss guarantees to squared error bounds, has not been explored previously. Furthermore, our results do not depend on the specific implementation of an algorithm, but only on the matching loss bounds achieved by its output. In this sense, we provide new agnostic error guarantees for various existing algorithms of the literature. For example, our results imply new guarantees for the GLMtron algorithm of Kakade et al. [2011] in the agnostic setting, since GLMtron can be viewed as performing gradient descent (with unit step size) on the matching loss corresponding to a specified link function.

Matching losses over linear functions are also linked to the Chow parameters [O'Donnell and Servedio, 2008] through their gradient with respect to w , as observed by Diakonikolas et al. [2020b]. In fact, the norm of the matching loss gradient is also linked to multiaccuracy, a notion that originates to fairness literature [Hebert-Johnson et al., 2018, Kim et al., 2019]. A stationary point w of a matching loss that corresponds to a GLM with link u is associated with a multiaccurate predictor p ( x ) = u -1 ( w · x ) , i.e., a predictor such that for all i ∈ [ d ] , E [ x i ( y -p ( x ))] = 0 . The work of [Gopalan et al., 2022, 2023] on omnipredictors presents a single predictor that is better than any linear model w · x for every matching loss. The results of Gopalan et al. [2022] show that a multicalibrated predictor (with respect to the features x i ) is an omnipredictor for all convex losses, whereas Gopalan et al. [2023] shows that the simpler condition of calibrated multiaccuracy suffices for matching losses that arise from GLMs. In view of the relationship between multiaccuracy and the gradient of the matching loss, our results show that, while multiaccuracy implies bounds on agnostically learning GLMs, the additional guarantee of calibration is sufficient for agnostically learning all SIMs.

The work of Shalev-Shwartz et al. [2011] showed strong agnostic learning guarantees in terms of the absolute error (rather than the squared error) of the form opt + ε for a range of GLMs, but their work incurs an exponential dependence on the weight norm B . In comparison, for the absolute loss, we get a bound of the form B opt log(1 / opt) for logistic regression (see Theorem 4.3). In more recent years, the problem of agnostically learning GLMs has frequently also been phrased as the problem of agnostically learning single neurons (with a known activation). For the ReLU activation, work by Goel et al. [2017] showed an algorithm achieving error opt + ε in time poly( d ) exp(1 /ε ) over marginals on the unit sphere, and Diakonikolas et al. [2020b] showed an algorithm achieving error

O (opt) + ε in fully polynomial time over isotropic log-concave marginals. The work of Frei et al. [2020], Diakonikolas et al. [2022b] both show guarantees for learning general neurons (with known activations) using the natural approach of running SGD directly on the squared loss (or a regularized variant thereof). Frei et al. [2020] achieves error O (opt) for any given strictly increasing activation and O ( √ opt) for the ReLU activation, but they assume that the marginal distribution is bounded. In contrast, we only assume that the marginal distribution has bounded second moments and we do not consider that the activation is known. Diakonikolas et al. [2022b] proved an O (opt) guarantee for a wide range of activations (including the ReLU), in the setting where the activation is known and over a large class of structured marginals, which need, however, to satisfy strong concentration and anti-concentration properties.

In terms of lower bounds and hardness results for this problem, the work of [Goel et al., 2019, Diakonikolas et al., 2020a, Goel et al., 2020, Diakonikolas et al., 2021, 2022a] has established superpolynomial hardness even for the setting of agnostically learning single ReLUs over Gaussian marginals.

Limitations and directions for future work. While we justify a certain dependence on the norm bound B in our main result on agnostically learning SIMs, we do not completely justify the exact form of Theorem 1.1. An important direction for future work is to tightly characterize the optimal bounds achievable for this problem, as well as to show matching algorithms.

1.2 Preliminaries

For the following, ( x , y ) is used to denote a labelled sample from a distribution D over R d × Y , where Y denotes the interval [0 , 1] unless it is specified to be the set { 0 , 1 } . We note that, although we provide results for the setting where the labels lie within [0 , 1] , we may get similar results for any bounded label space. We use P D (resp. E D ) to denote the probability (resp. expectation) over D and, similarly, P S (resp. E S ) to denote the corresponding empirical quantity over a set S of labelled examples. Throughout the paper, we will use the term differentiable function to mean a function that is differentiable except on finitely many points. Our main results will assume the following about the marginal distribution on R d .

Definition 1.4 (Bounded moments) . For λ ≥ 1 and k ∈ N , we say that a distribution D x over R d has λ -bounded 2 k -th moments if for any v ∈ S d -1 we have E x ∼ D x [( v · x ) 2 k ] ≤ λ .

For a concept class C : R d → R , we define opt( C , D ) to be the minimum squared error achievable by a concept c : R d → R in C with respect to the distribution D . We now define our main learning task.

Definition 1.5 (Agnostic learning) . Let C : R d → R be a concept class, let Ψ : [0 , 1] → [0 , 1] be an increasing function, let D be a distribution over R d × [0 , 1] and ε > 0 . We say that an algorithm A agnostically learns the class C up to error Ψ(opt( C , D )) + ε if algorithm A , given a number of i.i.d. samples from D , outputs, with probability at least 2 / 3 over the samples and the randomness of A , a hypothesis h : R d → [0 , 1] with E D [( y -h ( x )) 2 ] ≤ Ψ(opt( C , D )) + ε .

We will also provide results that are specific to the sigmoid activation and work under the assumption that the marginal distribution is sufficiently concentrated.

Definition 1.6 (Concentrated marginals) . For λ > 0 and γ , we say that a distribution D x over R d is ( λ, γ ) -concentrated if for any v ∈ S d -1 and r ≥ 0 we have P x ∼ D x [ | v · x | ≥ r ] ≤ λ · exp( -r γ ) .

Definition 1.7 (Fenchel-Legendre pairs) . We call a pair of functions ( f, g ) a Fenchel-Legendre pair if the following conditions hold.

g ′ : R → R is continuous, non-decreasing, differentiable and 1 -Lipschitz with range ran ( g ′ ) ⊇ (0 , 1) and g ( t ) = ∫ t 0 g ′ ( τ ) dτ , for any t ∈ R .

f : ran ( g ′ ) → R is the convex conjugate (Fenchel-Legendre transform) of g , i.e., we have f ( r ) = sup t ∈ R r · t -g ( t ) for any r ∈ ran ( g ′ ) .

For such pairs of functions (and their derivatives f ′ , g ′ ), the following are true for r ∈ ran ( g ′ ) and t ∈ ran ( f ′ ) (note that ran ( f ′ ) is not necessarily R when g ′ is not invertible).

Note that g ′ will be used as an activation function for single neurons and f ′ corresponds to the unknown link function of a SIM (or the known link function of a GLM). We say that g ′ is bi-Lipschitz if for any t 1 < t 2 ∈ R we have that ( g ′ ( t 2 ) -g ′ ( t 1 )) / ( t 2 -t 1 ) ∈ [ α, β ] . If g ′ is [ α, β ] bi-Lipschitz, then f ′ is [ 1 β , 1 α ] bi-Lipschitz. However, the converse implication is not necessarily true when g ′ is not strictly increasing.

Definition 1.8 (Matching Losses) . For a non-decreasing and Lipschitz activation g ′ : R → R , the matching loss ℓ g : Y × R → R is defined pointwise as follows:

where g ( t ) = ∫ t 0 g ′ . The function ℓ g is convex and smooth with respect to its second argument. The corresponding population matching loss is

In Equation (1.3), c : R d → R is some concept and D is some distribution over R d × [0 , 1] . In the specific case where c is a linear function, i.e., c ( x ) = w · x , for some w ∈ R d , then we may alternatively denote L g ( c ; D ) with L g ( w ; D ) .

Wealso define the Bregman divergence associated with f to be D f ( q, r ) = f ( q ) -f ( r ) -( q -r ) f ′ ( r ) , for any q, r ∈ ran ( g ′ ) . Note that D f ( q, r ) ≥ 0 with equality iff q = r .

Definition 1.9 (SIMs and GLMs as Concept Classes) . For B > 0 , we use SIM B to refer to the class of all SIMs of the form x ↦→ g ′ ( w · x ) where ∥ w ∥ 2 ≤ B and g ′ : R → R is an arbitrary 1 -Lipschitz monotone activation that is differentiable (except possibly at finitely many points). We define GLM g ′ ,B similarly except for the case where g ′ is fixed and known.

We also define the notion of calibrated multiaccuracy that we need to obtain omnipredictors in our context.

Definition 1.10 (Calibrated Multiaccuracy) . A predictor p : R d → [0 , 1] is called ε -multiaccurate if for all i ∈ [ d ] , | E [ x i ( y -p ( x ))] | ≤ ε . It is called ε -calibrated if | E p ( x ) E y | p ( x ) [ y -p ( x )] | ≤ ε .

2 Distortion Bounds for the Matching Loss

In this section, we propose a simple approach for bounding the squared error of a predictor that minimizes a (convex) matching loss, in the agnostic setting. We convert matching loss bounds to squared loss bounds in a generic way, through appropriate pointwise distortion bounds between the two losses. In particular, for a given matching loss L g , we transform guarantees on L g that are competitive with the optimum linear minimizer of L g to guarantees on the squared error that are competitive with the optimum GLM whose activation ( g ′ ) depends on the matching loss at hand.

We now provide the main result we establish in this section.

Theorem 2.1 (Squared Error Minimization through Matching Loss Minimization) . Let D be a distribution over R d × [0 , 1] , let 0 < α ≤ β and let ( f, g ) be a Fenchel-Legendre pair such that g ′ : R → R is [ α, β ] bi-Lipschitz. Suppose that for a predictor p : R d → ran ( g ′ ) we have

Then we also have: err 2 ( p ) ≤ β α · opt(GLM g ′ ,B ) + 2 βε .

The proof of Theorem 2.1 is based on the following pointwise distortion bound between matching losses corresponding to bi-Lipschitz link functions and the squared distance.

Lemma 2.2 (Pointwise Distortion Bound for bi-Lipschitz link functions) . Let 0 < α ≤ β and let ( f, g ) be a Fenchel-Legendre pair such that f ′ : ran ( g ′ ) → R is [ 1 β , 1 α ] bi-Lipschitz. Then for any y, p ∈ ran ( g ′ ) we have

In the case that f ′ is differentiable on (0 , 1) , the proof of Lemma 2.2 follows from an application of Taylor's approximation theorem of degree 2 on the function f , since the Bregman divergence D f ( y, p ) is exactly equal to the error of the second degree Taylor's approximation of f ( y ) around p and f ′′ ( ξ ) ∈ [ 1 β , 1 α ] for any ξ ∈ ran ( g ′ ) . The relationship between ℓ g and D f follows from property (1.2). Note that when g ′ is [ α, β ] bi-Lipschitz, then f ′ is [ 1 β , 1 α ] bi-Lipschitz.

Theorem 2.1 follows by applying Lemma 2.2 appropriately to bound the error of a predictor p by its matching loss L g ( f ′ ◦ p ) and bound the matching loss of the linear function corresponding to w ∗ by the squared error of g ′ ( w ∗ · x ) , where g ′ ( w ∗ · x ) is the element of GLM g ′ ,B with minimum squared error.

Although Theorem 2.1 only applies to bi-Lipschitz activations g ′ , it has the advantage that the assumption it makes on p corresponds to a convex optimization problem and, when the marginal distribution has certain concentration properties (for generalization), can be solved efficiently through gradient descent on the empirical loss function. As a consequence, for bi-Lipschitz activations we can obtain O (opt) efficiently under mild distributional assumptions in the agnostic setting.

3 Agnostically Learning Single-Index Models

In this section, we provide our main result on agnostically learning SIMs. We combine the distortion bounds we established in Section 2 with results from Gopalan et al. [2023] on Omniprediction, which can be used to learn a predictor p that satisfies the guarantee of Theorem 2.1 simultaneously for all bi-Lipschitz activations. By doing so, we obtain a result for all Lipschitz and non-decreasing activations simultaneously.

Theorem 3.1 (Agnostically Learning SIMs) . Let λ, B ≥ 1 , ε > 0 and let D be a distribution over R d × [0 , 1] with second moments bounded by λ . Then, Algorithm 1 agnostically learns the class SIM B over D up to squared error O ( B √ λ √ opt(SIM B , D )) + ε using time and sample complexity poly( d, B, λ, 1 ε ) . Moreover, the same is true for any algorithm with an omniprediction guarantee like the one of Theorem 3.2.

In order to apply Theorem 2.1, we use the following theorem which is a combination of results in Gopalan et al. [2023], where they show that the matching losses corresponding to a wide class of functions can all be minimized simultaneously by an efficiently computable predictor.

Theorem 3.2 (Omnipredictors for Matching Losses, combination of results in Gopalan et al. [2023]) . Let λ, L, R, B ≥ 1 , ε > 0 and let D be a distribution over R d × [0 , 1] whose marginal on R d has λ -bounded second moments. Then, Algorithm 1, given sample access to D , with probability at least 2 / 3 returns a predictor p : R d → (0 , 1) with the following guarantee. For any Fenchel-Legendre pair ( f, g ) such that g ′ : R → R is L -Lipschitz, and f ′ takes values within the interval [ -R,R ] , p satisfies

The algorithm requires time and sample complexity poly( λ, B, L, R, 1 ε ) .

Algorithm 1 is a version of the algorithm of Gopalan et al. [2023] for calibrated multiaccuracy, specific to our setting. For a more quantitative version of Theorem 3.2, see Theorem C.3 in the appendix.

We aim to apply Theorem 3.2 to the class of all Lipschitz activations (which is wider than the class of bi-Lipschitz activations). This is enabled by the following lemma, whose proof is based on Theorem 2.1 and the fact that the error of a predictor is bounded by the sum of the error of another predictor and the squared expected distance between the two predictors.

Lemma 3.3. Let D be a distribution over R d × [0 , 1] . Let g ′ : R → R be some fixed activation, and f ′ its dual. Consider the class GLM g ′ ,B , and let w ∗ be the weights achieving opt(GLM g ′ ,B , D ) . Let ϕ ′ : R → R be an [ α, β ] bi-Lipschitz function (differentiable except possibly at finitely many points) that we wish to approximate g ′ by. Any predictor p : R d → R that satisfies

also satisfies the following ℓ 2 error guarantee:

Algorithm 1: Calibrated Multiaccuracy (modification of Algorithm 2 in Gopalan et al. [2023])

Input: Sufficiently large training set S over R d × [0 , 1] , parameters ε , λ , B , L , R

Output: A predictor p : R d → (0 , 1)

Form a training set S ′ by substituting each element ( x , y ) in S with ( x , y ′ ) where y ′ ∈ { 0 , 1 } is formed as follows. Conditioned on y , let y ′ be an independent Bernoulli random variable with probability of success y .

Let A = B · L , C > 0 a sufficiently large universal constant and let δ = ε 2 CA 2 λR 2 , σ = ε -δR 2 CA 2 λR . Set p ← 0 (the zero function).

repeat

repeat

Let S ′′ be a subset of S ′ with | S ′′ | = C d 2 A 2 λR 2 ε 2 and set S ′ ← S ′ \ S ′′ . Form a set T by substituting each element ( x , y ′ ) of S ′′ with ( x , z ) , where z = y ′ -p ( x ) . 1 ∑

|

T

∥

Let

v

=

Reject if

|

v

∥

2

(

x

,z

)

∈

T

3(

ε

z

-

x

.

A

λδ

)

/

(4

A

)

.

Otherwise , let w = B v / ∥ v ∥ 2 and update p to be the following predictor

until some iteration rejects (which happens when it has found a multiaccurate predictor) ;

Estimate the calibration error | E p δ ( x ) E y | p δ ( x ) [ y -p δ ( x )] | of p δ using the samples in S ′′ according to the following expression

δ

If the estimate has value at most ε/ 2 , then output p and terminate .

Otherwise , update p to be the following (calibrated) predictor until some iteration terminates the execution ;

By combining Theorem 3.2 with Lemma 3.3 (whose proofs can be found in Appendix C), we are now ready to prove our main theorem.

Proof of Theorem 3.1. We will combine Theorem 3.2, which states that there is an efficient algorithm that simultaneously minimizes the matching losses corresponding to bounded, non-decreasing and Lipschitz activations, with Lemma 3.3, which implies that minimizing the matching loss corresponding to the nearest bi-Lipschitz activation is sufficient to obtain small error. Note that we may assume that ε < 1 / 2 , since otherwise the problem is trivial (output the zero function and pick C = 2 ).

As a first step, we show that link functions corresponding to bi-Lipschitz activations are bounded (according to Definition C.1, for γ = 0 ). In particular, let ϕ ′ : R → R be an [ α, β ] bi-Lipschitz activation for some β ≥ α > 0 such that ϕ ′ ( s ) ∈ [ -1 , 2] for some s ∈ R and let ψ ′ be the inverse of ϕ ′ ( ϕ ′ is invertible since it is strictly increasing). We will show that ψ ′ ( r ) ∈ [ -R,R ] for any r ∈ [0 , 1] and R = O ( | s | +1 /α ) .

√

We pick r 0 = 0 , r 1 = 1 and get that | ψ ′ ( ϕ ′ ( s )) -ψ ′ ( r 0 ) | ≤ 1 α | ϕ ′ ( s ) -r 0 | ≤ 2 α . Hence ψ ′ ( r 0 ) ≥ ψ ′ ( ϕ ′ ( s )) -2 α = s -1 α . Similarly, we get ψ ′ ( r 1 ) ≤ s + 2 α . Therefore, ψ ′ ( r ) ∈ [ ψ ′ (0) , ψ ′ (1)] ⊆ [ -| s | -2 α , | s | + 2 α ] , for any r ∈ [0 , 1] , due to monotonicity of ψ ′ .

For a given non-decreasing and 1 -Lipschitz g ′ , we will now show that there is a bounded bi-Lipschitz activation ϕ ′ such that if the assumption of Lemma 3.3 is satisfied for ϕ ′ by a predictor p , then the error of p is bounded by

Suppose, first, that opt(GLM g ′ ,B ) ≤ ε 2 . Then, we pick ϕ ′ ( t ) = g ′ ( t ) + εt . Note that ϕ ′ is [ ε, 1 + ε ] bi-Lipschitz. Moreover, since opt(GLM g ′ ,B ) ≤ ε 2 , we must have some s ∈ R with | s | ≤ 2 λB 2 such that g ′ ( s ) ∈ [ -1 , 2] . Otherwise, we would have opt(GLM g ′ ,B ) = E [( g ′ ( w ∗ · x ) -y ) 2 ] ≥ E [( g ′ ( w ∗ · x ) -y ) 2 | | w ∗ · x ≤ 2 λB 2 | ] · P [ | w ∗ · x | ≤ 2 λB 2 ] ≥ P [ | w ∗ · x | ≤ 2 λB 2 ] = 1 -P [ | w ∗ · x | > 2 λB 2 ] ≥ 1 4 > ε 2 , due to Chebyshev's inequality, the fact that w ∗ ∈ W and the bounded moments assumption. Therefore, ψ ′ is ( R = 2 λB 2 + 2 ε , γ = 0) -bounded and we have

As a consequence, under the assumption of Lemma 3.3 for ϕ ′ , the error of the corresponding predictor p is err 2 ( p ) ≤ 2(1 + ε ) ε +2(1 + ε ) ελB 2 +2(1 + ε ) ε = O ( λB 2 ε ) .

In the case that opt(GLM g ′ ,B ) > ε 2 , we pick ϕ ′ ( t ) = g ′ ( t )+ t √ opt(GLM g ′ ,B ) B √ λ . Wemay also assume that opt(GLM g ′ ,B ) ≤ 1 / 2 , since otherwise any predictor with range [0 , 1] will have error at most 2opt(GLM g ′ ,B ) . Then, ψ is ( O ( λB 2 + B √ λ ε ) , 0) -bounded, ϕ ′ is [ 1 B √ opt(GLM g ′ ,B ) / √ λ, 1 + 1 B ] bi-Lipschitz which gives

As a consequence, under the assumption of Lemma 3.3 for ϕ ′ , the error of the corresponding predictor p is err 2 ( p ) ≤ 4(1 + 1 B ) B √ λ √ opt(GLM g ′ ,B ) + 2(1 + 1 B ) ε . Using a similar approach as for the case opt(GLM g ′ ,B ) ≤ ε , we can show that ψ ′ is polynomially bounded (as per Definition C.1), since opt(GLM g ′ ,B ) ≤ 1 2 .

To conclude the proof of Theorem 3.1, we apply Theorem 3.2 with appropriate (polynomial) choice of parameters ( R = O ( λB 2 + B √ λ ε ) , L = 2 ), to show that there is an efficient algorithm that outputs a predictor p : R d → (0 , 1) for which the assumption of Lemma 3.3 holds simultaneously for all bi-Lipschitz activations ( ϕ ′ ) with sufficiently bounded inverses ( ψ ′ ).

4 Stronger Guarantees for Logistic Regression

In this section, we follow the same recipe we used in Section 2 to get distortion bounds similar to Theorem 2.1 for the sigmoid activation (or, equivalently, the logistic model) under the assumption that the marginal distribution is sufficiently concentrated (see Definition C.1). In particular, Theorem 2.1 does not hold, since the sigmoid is not bi-Lipschitz and our main Theorem 3.1 only provides a guarantee of O ( √ opt) for squared error. We use appropriate pointwise distortion bounds for the matching loss corresponding to the sigmoid activation and provide guarantees of ˜ O (opt) for logistic regression with respect to both squared and absolute error, under appropriate assumptions about the concentration of the marginal distribution. The proofs of this section are provided in Appendix D.

For the logistic model, the link function f ′ is defined as f ′ ( r ) = ln( r 1 -r ) , for r ∈ (0 , 1) and the corresponding activation g ′ is the sigmoid g ′ ( t ) = 1 1+ e -t for t ∈ R . The corresponding matching loss is the logistic loss.

Squared error. We first provide a result for squared loss minimization. In comparison to Theorem 2.1, the qualitative interpretation of the following theorem is that, while the sigmoid activation is not bi-Lipschitz, it is effectively bi-Lipschitz under sufficiently concentrated marginals.

Theorem 4.1 (Squared Loss Minimization through Logistic Loss Minimization) . Let D be a distribution over R d × [0 , 1] whose marginal on R d is (1 , 2) -concentrated. Let g ′ : R → R be the sigmoid activation, i.e., g ′ ( t ) = (1 + e -t ) -1 for t ∈ R . Assume that for some B > 0 , ε > 0 and a predictor p : R d → (0 , 1) we have

If we let opt g = min ∥ w ∥ 2 ≤ B err 2 ( g ′ w ) , then for the predictor p and some universal constant C > 0 we also have

In particular, the squared error of p is upper bounded by ˜ O (opt g ) , since the function t ↦→ exp(log 1 / 2 t ) is asymptotically smaller than any polynomial function t ↦→ t γ with γ > 0 .

Once more, the proof of our result is based on an appropriate pointwise distortion bound which we provide below. It follows by the fact that the Bregman divergence corresponding to the sigmoid is the Kullback-Leibler divergence and by combining Pinsker's inequality (lower bound) with Lemma 4.1 of Götze et al. [2019] (upper bound).

Lemma 4.2 (Pointwise Distortion Bound for Sigmoid) . Let g ′ be the sigmoid activation. Then, for any y, p ∈ (0 , 1) we have that D f ( y, p ) = D KL ( y ∥ p ) = y ln( y/p ) + (1 -y ) ln( 1 -y 1 -p ) . Moreover

We translate Lemma 4.2 to a bound on the squared error of a matching loss minimizer (in this case, the logistic loss) using an approach similar to the one for Theorem 2.1. In order to use the upper bound on the surrogate loss provided by Lemma 4.2, we apply it to p ← g ′ ( w · x ) , where g ′ is the sigmoid function, and observe that the quantity 1 p (1 -p ) is exponential in | w · x | . Hence, when the marginal is ( λ, 2) -concentrated (subgaussian concentration), then 1 p (1 -p ) is effectively bounded.

Absolute error. All of the results we have provided so far have focused on squared error minimization. We now show that our approach yields results even for the absolute error, which can also be viewed as learning in the p-concept model [Kearns and Schapire, 1994]. In particular, for a distribution D over R d × [0 , 1] , we define the absolute error of a predictor p : R d → [0 , 1] as follows.

In the specific case when the labels are binary, i.e., y ∈ { 0 , 1 } , we have

̸

where the distribution D p is over R d ×{ 0 , 1 }×{ 0 , 1 } and is formed by drawing samples ( x , y ) from D and, given x , forming y p by drawing a conditionally independent Bernoulli random variable with parameter p ( x ) . We provide the following result.

Theorem 4.3 (Absolute Loss Minimization through Logistic Loss Minimization) . Let D be a distribution over R d ×{ 0 , 1 } whose marginal on R d is (1 , 1) -concentrated. Let g ′ : R → R be the sigmoid activation, i.e., g ′ ( t ) = (1 + e -t ) -1 for t ∈ R . Assume that for some B > 0 , ε > 0 and a predictor p : R d → (0 , 1) we have

If we let opt g = min ∥ w ∥ 2 ≤ B err 1 ( g ′ w ) , then for the predictor p and some universal constant C > 0 we also have

The corresponding distortion bound in this case is between the absolute and logistic losses and works when the labels are binary.

Lemma 4.4 (Pointwise Distortion between Absolute and Logistic Loss) . Let g ′ be the sigmoid activation. Then, there is a constant c ∈ R such that for any y ∈ { 0 , 1 } and p ∈ (0 , 1) , we have

The bound of Theorem 4.3 implies an algorithm for learning an unknown sigmoid neuron in the p-concept model, by minimizing a convex loss. While there are algorithms achieving stronger guarantees [Diakonikolas et al., 2022b] for agnostically learning sigmoid neurons, such algorithms typically make strong distributional assumptions including concentration, anti-concentration and anti-anti-concentration or boundedness.

Moreover, it is useful to compare the bound we provide in Theorem 4.3 to a lower bound by Diakonikolas et al. [2020c, Theorem 4.1], which concerns the problem of agnostically learning halfspaces by minimizing convex surrogates. In particular, they show that even under log-concave marginals, no convex surrogate loss can achieve a guarantee better than O (opt log(1 / opt)) , where opt is measured with respect to the ℓ 1 error (which is equal to the probability of error). The result is not directly comparable to our upper bound, since we examine the sigmoid activation. Their setting can be viewed as a limit case of ours by letting the norm of the weight vector grow indefinitely (the sigmoid tends to the step function), but the main complication is that our upper bound is of the form O ( B opt log(1 / opt)) , which scales with B . However, their lower bound concerns marginal distributions that are not only concentrated, but are also anti-concentrated and anti-anticoncentrated, while our results only make concentration assumptions.

5 Necessity of Norm Dependence

In this final section, we use a lower bound due to Diakonikolas et al. [2022a] on agnostic learning of GLMs using SQ algorithms and compare it with our main result (Theorem 3.1). For simplicity, we specialize to the case of the standard sigmoid or logistic function. A modification of their proof ensures that the bound holds under isotropic marginals. 2

Theorem 5.1 (SQ Lower Bound for Agnostically Learning GLMs, variant of [Diakonikolas et al., 2022a, Thm C.3]) . Let g ′ : R d → R be the standard logistic function. Any SQ algorithm either requires d ω (1) queries or d -ω (1) tolerance to distinguish between the following two labeled distributions:

(Labels have signal.) D signal on R d × R is such that opt(GLM g ′ ,B , D signal ) ≤ exp( -Ω(log 1 / 4 d )) = o (1) for some B = poly( d ) .

(Labels are random.) D random on R d × R is such that the labels y are drawn i.i.d. from { a, b } for certain universal constants a, b ∈ [0 , 1] . In particular, opt(GLM g ′ ,B , D random ) = Ω(1) for any B .

Both D signal and D random have the same marginal on R d , with 1 -bounded second moments.

Let us consider applying our main theorem (Theorem 3.1) to this setting, with D being either D signal or D random , and with the same B = poly( d ) as is required to achieve small error in the 'labels have signal' case. We would obtain a predictor with ℓ 2 error at most B √ opt(GLM g ′ ,B ) (or indeed with SIM B in place of GLM g ′ ,B ). Since this is ω (1) , this guarantee is insufficient to distinguish the two cases above, which is as it should be since our main algorithm indeed fits into the SQ framework.

Theorem 5.1 does, however, justify a dependence on the norm B in our main result. In particular, it is clear that a guarantee of the form opt(GLM g ′ ,B ) c for any universal constant c > 0 (independent of B ) would be too strong, as it would let us distinguish the two cases above. In fact, this lower bound rules out a large space of potential error guarantees stated as functions of B and opt(GLM g ′ ,B ) . For instance, for sufficiently large d , it rules out any error guarantee of the form exp( O (log 1 / 5 B )) · opt(GLM g ′ ,B ) c ′ for any universal constant c ′ > 0 .

Acknowledgments and Disclosure of Funding

We wish to thank the anonymous reviewers of NeurIPS 2023 for their constructive feedback. Aravind Gollakota was at UT Austin while this work was done, supported by NSF award AF-1909204 and the NSF AI Institute for Foundations of Machine Learning (IFML). Adam R. Klivans was supported by NSF award AF-1909204 and the NSF AI Institute for Foundations of Machine Learning (IFML). Konstantinos Stavropoulos was supported by NSF award AF-1909204, the NSF AI Institute for Foundations of Machine Learning (IFML), and by scholarships from Bodossaki Foundation and Leventis Foundation.

References

Parikshit Gopalan, Lunjia Hu, Michael P. Kim, Omer Reingold, and Udi Wieder. Loss minimization through the lens of outcome indistinguishability. In Yael Tauman Kalai, editor, 14th Innovations in Theoretical Computer Science Conference, ITCS 2023, January 10-13, 2023, MIT, Cambridge, Massachusetts, USA , volume 251 of LIPIcs , pages 60:1-60:20. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2023. (document), 1, 1.1, 3, 3, 3.2, 3, 1, C.1, C.3, C.1, C.1, C.1

Peter McCullagh. Generalized linear models. European Journal of Operational Research , 16(3): 285-292, 1984. ISSN 0377-2217. doi: https://doi.org/10.1016/0377-2217(84)90282-0. URL https://www.sciencedirect.com/science/article/pii/0377221784902820 . 1, 1.1

Alan Agresti. Foundations of linear and generalized linear models . John Wiley & Sons, 2015. 1, 1.1

Sham M Kakade, Varun Kanade, Ohad Shamir, and Adam Kalai. Efficient learning of generalized linear and single index models with isotonic regression. Advances in Neural Information Processing Systems , 24, 2011. 1, 1, 1.1

Adam Tauman Kalai and Ravi Sastry. The isotron algorithm: High-dimensional isotonic regression. In COLT 2009 - The 22nd Conference on Learning Theory, Montreal, Quebec, Canada, June 18-21, 2009 , 2009. 1, 1

Surbhi Goel, Sushrut Karmalkar, and Adam Klivans. Time/accuracy tradeoffs for learning a relu with respect to gaussian marginals. Advances in neural information processing systems , 32, 2019. 1, 1.1

Ilias Diakonikolas, Daniel Kane, and Nikos Zarifis. Near-optimal sq lower bounds for agnostically learning halfspaces and relus under gaussian marginals. Advances in Neural Information Processing Systems , 33:13586-13596, 2020a. 1, 1.1

Surbhi Goel, Aravind Gollakota, and Adam Klivans. Statistical-query lower bounds via functional gradients. Advances in Neural Information Processing Systems , 33:2147-2158, 2020. 1, 1.1

Ilias Diakonikolas, Daniel M. Kane, Thanasis Pittas, and Nikos Zarifis. The optimality of polynomial regression for agnostic learning under gaussian marginals in the sq model. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 1552-1584. PMLR, 15-19 Aug 2021. 1, 1.1

Ilias Diakonikolas, Daniel Kane, Pasin Manurangsi, and Lisheng Ren. Hardness of learning a single neuron with adversarial label noise. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 8199-8213. PMLR, 28-30 Mar 2022a. 1, 1.1, 5, 5.1

Peter Auer, Mark Herbster, and Manfred K. K Warmuth. Exponentially many local minima for single neurons. In D. Touretzky, M.C. Mozer, and M. Hasselmo, editors, Advances in Neural Information Processing Systems , volume 8. MIT Press, 1995. URL https://proceedings.neurips.cc/ paper_files/paper/1995/file/3806734b256c27e41ec2c6bffa26d9e7-Paper.pdf . 1, 1.1

Parikshit Gopalan, Adam Tauman Kalai, Omer Reingold, Vatsal Sharan, and Udi Wieder. Omnipredictors. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022) . Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2022. 1, 1.1

Spencer Frei, Yuan Cao, and Quanquan Gu. Agnostic learning of a single neuron with gradient descent. Advances in Neural Information Processing Systems , 33:5417-5428, 2020. 1, 1.1

Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning a single neuron with adversarial label noise via gradient descent. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4313-4361. PMLR, 02-05 Jul 2022b. URL https://proceedings. mlr.press/v178/diakonikolas22c.html . 1, 1, 1.1, 4

Ilias Diakonikolas, Surbhi Goel, Sushrut Karmalkar, Adam R Klivans, and Mahdi Soltanolkotabi. Approximation schemes for relu regression. In Conference on Learning Theory , pages 1452-1485. PMLR, 2020b. 1.1

Ryan O'Donnell and Rocco A Servedio. The chow parameters problem. In Proceedings of the fortieth annual ACM symposium on Theory of computing , pages 517-526, 2008. 1.1

Ursula Hebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (Computationally-identifiable) masses. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1939-1948. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/hebert-johnson18a.html . 1.1

Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages 247-254, 2019. 1.1

Shai Shalev-Shwartz, Ohad Shamir, and Karthik Sridharan. Learning kernel-based halfspaces with the 0-1 loss. SIAM Journal on Computing , 40(6):1623-1646, 2011. 1.1

Surbhi Goel, Varun Kanade, Adam Klivans, and Justin Thaler. Reliably learning the relu in polynomial time. In Conference on Learning Theory , pages 1004-1042. PMLR, 2017. 1.1

Friedrich Götze, Holger Sambale, and Arthur Sinulis. Higher order concentration for functions of weakly dependent random variables. Electronic Journal of Probability , 24:1-19, 2019. 4

Michael J Kearns and Robert E Schapire. Efficient distribution-free learning of probabilistic concepts. Journal of Computer and System Sciences , 48(3):464-497, 1994. 4

Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Non-convex sgd learns halfspaces with adversarial label noise. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1854018549. Curran Associates, Inc., 2020c. URL https://proceedings.neurips.cc/paper_ files/paper/2020/file/d785bf9067f8af9e078b93cf26de2b54-Paper.pdf . 4

A Technical Lemmas

In this section, we provide some technical Lemmas that we use in our proofs.

Proposition A.1 (Weak Learner for Linear Functions) . Let D be a distribution over R d × [ -1 , 1] whose marginal on R d has λ -bounded second moments and B > 0 . For any ε > 0 and δ ∈ (0 , 1) , there is a universal constant C > 0 and an algorithm that given a set S of i.i.d. samples from D of size at least C · d 2 λB 2 ε 2 log 1 δ , runs in time O ( d | S | ) and satisfies the following specifications with probability at least 1 -δ

If E ( x ,z ) ∼ D [ z ( w · x )] ≥ ε for some w ∈ R d with ∥ w ∥ 2 ≤ B , then the algorithm accepts. Otherwise, it may or may not reject and return a special symbol.

If the algorithm accepts then it returns w ∈ R d with ∥ w ∥ 2 ≤ B such that we have E ( x ,z ) ∼ D [ z ( w · x )] ≥ ε/ 4 .

Proof. We will prove the proposition for δ = 1 / 6 . We may boost the probability of success with repetition.

The algorithm computes the vector v = E S [ z x ] . If ∥ v ∥ 2 ≤ 3 ε 4 B , then the algorithm rejects and outputs a special symbol. Otherwise, it outputs the vector B ∥ v ∥ 2 v .

Suppose, first, that E ( x ,z ) ∼ D [ z ( w · x )] ≥ ε for some w with ∥ w ∥ 2 ≤ B . Then, due to Chebyshev's inequality we have for any i ∈ [ d ]

Hence, with probability at least 5 / 6 , we have ∥ E S [ z x ] -E D [ z x ] ∥ 2 ≤ ε 8 B , due to a union bound. Therefore, ∥ v ∥ 2 ≥ ∥ E D [ z x ] ∥ 2 -ε 8 B ≥ E D [ z ( w · x )] B -ε 8 B ≥ 7 B 8 and the algorithm accepts.

Suppose, now, that the algorithm accepts. Then, we have ∥ v ∥ 2 > 3 ε 4 B and (with probability at least 5 / 6 ) we have

since ∥ E S [ z x ] -E D [ z x ] ∥ 2 ≤ ε 8 B . This concludes the proof.

Proposition A.2. Let D be a distribution over R d × { 0 , 1 } and p : R d → [0 , 1] . Consider the distribution D p over R d × { 0 , 1 } × { 0 , 1 } , which is formed by drawing samples ( x , y ) from D and, given x , forming y p by drawing a conditionally independent Bernoulli random variable with parameter p ( x ) . Then we have

̸

Proof. Since over D p , y and y p are conditionally independendent, we have

̸

̸

B Proofs from Section 2

B.1 Proof of Theorem 2.1

To prove Theorem 2.1, we first prove the following more general theorem. Theorem 2.1 may then be easily recovered from this by setting H = GLM g ′ ,B and observing that f ′ ( g ′ ( w · x )) = w · x , since g ′ is invertible.

Theorem B.1 (Squared Error Minimization through Distorted Matching Loss Minimization) . Let D be a distribution over R d × [0 , 1] , let 0 < α ≤ β and let ( f, g ) be a pair of Fenchel-Legendre dual functions such that g ′ : R → R is continuous, non-decreasing and f ′ : ran ( g ′ ) → R is [ 1 β , 1 α ] bi-Lipschitz. Let ε > 0 and H ⊆ { R d → ran ( g ′ ) } . Assume that for a predictor p : R d → ran ( g ′ ) we have

Then, for the predictor p , we also have: err 2 ( p ) ≤ β α · min h ∈H err 2 ( h ) + 2 βε .

Proof. We apply Lemma 2.2 with y ← y and p ← p ( x ) and take expectations over D on both sides. We have that

Therefore, we can bound the squared error of p as follows.

where Q ∗ = E ℓ g ( y, f ′ ( y )) .

We now apply Lemma 2.2 again with y ← y and p ← h ( x ) and we similarly have

Therefore, for any h ∈ H , we have, in total: err 2 ( p ) ≤ β α err 2 ( h ) + 2 βε

We first prove Lemma 2.2, which we restate here for convenience.

Lemma B.2. Assume f ′ is [1 /β, 1 /α ] bi-Lipschitz and differentiable on all except from a finite number of points on any bounded interval. Then for any y, p ∈ ran ( g ′ ) we have

Proof. We first show that ℓ g ( y, f ′ ( p )) -ℓ g ( y, f ′ ( y )) = D f ( y, p ) . In particular, we have

since f ′ ( p ) ∈ ran ( f ′ ) and we know that g ( t ) = tg ′ ( t ) -f ( g ′ ( t )) for any t ∈ ran ( f ′ ) as well as g ′ ( f ′ ( p )) = p for any p ∈ ran ( g ′ ) . Therefore, we have

Let ψ : ran ( g ′ ) → R be such that ψ ′ ( p ) = f ′ ( p ) and ψ ′ is differentiable on the open interval between y and p , with ψ ′′ ( ξ ) ∈ [1 /β, 1 /α ] for any ξ between y and p . Let γ y := f ( y ) -ψ ( y ) , γ p := f ( p ) -ψ ( p ) and γ ψ := 2 max {| γ y | , | γ p |} . Then we have that

for any ψ as defined above (say ψ ∈ Ψ ). In particular, we have

Since, we have only a finite number of points where the derivative is not well defined, a simple smoothening technique may give us Ψ such that inf ψ ∈ Ψ γ ψ = 0 .

(by assumption, for any h ∈ H )

.

C Proofs from Section 3

C.1 Proof of Theorem 3.2

We first define a boundedness property which we use in order to apply the results from Gopalan et al. [2023]. The property states that the activation function (the partial inverse of the link function) must either have a range that covers all possible labels, or has a range whose closure covers all possible labels and the rate with which the labels are covered as we tend to the limits of the domain is at least polynomial. For example, the sigmoid activation tends to 1 (resp. 0 ) exponentially fast as its argument increases (resp. decreases).

Definition C.1 (Bounded Functions) . Let u : (0 , 1) → R be a non-decreasing function defined on the interval (0 , 1) . For R,γ ≥ 0 , we say that u is ( R,γ ) -bounded on [0 , 1] if for any ε > 0 , there are r 0 ≤ r 1 ∈ [0 , 1] such that if we let u ( r i ) = lim r → r i u ( r ) , i = 0 , 1 then

Proposition C.2. Let u : (0 , 1) → ( -R,R ) be non-decreasing and continuous, then u is ( R, 0) -bounded.

We restate a more quantitative version of Theorem 3.2 here for convenience.

Theorem C.3 (Omnipredictors for Matching Losses, combination of results in Gopalan et al. [2023]) . Let D be a distribution over R d × [0 , 1] whose marginal on R d has λ -bounded second moments. There is an algorithm that, given sample access to D , with high probability returns a predictor p : R → (0 , 1) with the following guarantee. For any pair of Fenchel-Legendre dual functions ( f, g ) such that g ′ : R → R is continuous, non-decreasing and L -Lipschitz, and f ′ is ( R,γ ) -bounded (see Definition C.1), p satisfies

The algorithm requires time

and sample complexity

Proof of Theorem 3.2. The idea of the proof (given by Gopalan et al. [2023]) is that in each repetition of both the inner and the outer loop of Algorithm 1, there is a potential function which reduces by some amount that is bounded away above zero. The potential function is in fact the function E D [( p ∗ ( x ) -p ( x )) 2 ] , where p ∗ ( x ) = E D [ y ′ | x ] (for us, y ′ is formed by drawing a Bernoulli random variable with probability of success y , given an example ( x , y ) ∈ R d × [0 , 1] drawn from D ). Since E D [( p ∗ ( x ) -p ( x )) 2 ] ∈ [0 , 1] , the number of iterations of each of the loops has to be bounded. Moreover, after the completion of each of the inner loops, the current value of p corresponds to an approximately multiaccurate predictor and note that the algorithm terminates if and only if a discretized version p δ of (the multiaccurate) p is approximately calibrated. The output is then p δ which is close to p (and hence also approximately multiaccurate), but also approximately calibrated. We will now present a small number of technical modifications we need to make in the proof of Gopalan et al. [2023] in order to specialize it to our setting. The main difference here is that we do not consider the distribution of x to have bounded norm with probability 1 , but we only assume it to have bounded second moments. For the following, we assume that g is 1 -Lipschitz, since we can set B ← B · L and push the Lipschitz constant in the domain of w .

Suppose first that the marginal of D on R d is supported on the unit ball B d and that the labels are binary. Then, the result would follow from Theorems 7.7 and A.4 of Gopalan et al. [2023]. In particular,

Theorem 7.7 states that given access to a weak learner with the specifications of Proposition A.1, there is an efficient algorithm that computes an ε 1 -calibrated and ( C , ε 1 ) -multiaccurate predictor p , where the notions of calibration and multiaccuracy originate to the literature of fairness and are defined, e.g., in Definitions 3.1 and 3.2 of Gopalan et al. [2023] and C = { x → w · x | ∥ w ∥ 2 ≤ B } ∪ { x → 1 } (the class C is bounded as long as ∥ x ∥ 2 ≤ 1 almost surely). Theorem A.4 states that for ε 2 > 0 , any ε 1 -calibrated and ( C , ε 1 ) -multiaccurate predictor p minimizes simultaneously the matching loss corresponding to any pair ( f, g ) ∈ F (where f is ( R,γ ) -bounded) up to error

The expression above is formed by proving that any pair ( f, g ) ∈ F has the property that f ′ is ( ε 2 , R (1 /ε 2 ) γ ) -approximately optimal (as per the Definition A.1 of Gopalan et al. [2023]), for any ε 2 > 0 . In particular, we would like to show that for any ε 2 > 0 there exists ̂ f ′ such that the following is true for any r ∈ [0 , 1]

We may pick ̂ f ′ as follows (for r 0 ≤ r 1 as given by Definition C.1 for ε ← ε 2 , since f ′ is ( R,γ ) -bounded).

The desired result follows from using the expression for ℓ g , the convexity of g (since g ′ is non decreasing) and the guarantees of Definition C.1. In particular, we have that | ̂ f ′ | ≤ max {| f ′ ( r 0 ) | , | f ′ ( r 1 ) |} since f ′ is increasing. Moreover, for r ∈ [ r 0 , r 1 ] we have ℓ g ( r, ̂ f ′ ( r )) = ℓ g ( r, f ′ ( r )) and for r < r 0 we have

Similarly, for r > r 1 , we have

In order to acquire R (1 /ε 2 ) γ ε 1 + ε 1 + ε 2 ≤ ε , we set ε 2 = ε/ 3 and ε 1 = ( ε/ 3) 1+ γ /R .

However, we only assume that the marginal distribution has λ -bounded second moments and we, therefore, need to make certain modifications to the proof of their Theorem 7.7. In particular, the boundedness assumption is used in the proofs of Lemma 7.2, Lemma 7.6 and Theorem 7.7 in Gopalan et al. [2023].

During the execution of Algorithm 1, two types of updates are made. The first type of update is done within the inner loop and corresponds to beginning from a predictor p old and acquiring p new which is the function x ↦→ ( p old ( x ) + σ ( w · x )) [0 , 1] , where w is the output of the weak learner of our Proposition A.1, run with ε ← ε 3 ( ε 3 will be specified later). To lower bound the decrease in the potential function during this type of update, we use a version of Lemma 7.6 in Gopalan et al. [2023]. In this case, one needs to pick a step size σ that is polynomially smaller than the guarantee that the

weak learner provides. In particular, if the weak learner of our Proposition A.1 is run with ε ← ε 3 , then one acquires (following the proof of Lemma 7.6 in Gopalan et al. [2023])

If σ is picked to be σ = ε 3 4 B 2 λ , then the quantity of interest E [( p ∗ ( x ) -p old ( x )) 2 ] -E [( p ∗ ( x ) -p new ( x )) 2 ] is lower bounded by ε 2 3 16 B 2 λ . Hence the number of iterations of the inner loop is upper bounded by O ( B 2 λ ε 2 3 ) . Note that in their original algorithm, σ was picked equal to ε 3 and this is why ε 3 (or another corresponding parameter) does not appear in their proofs. The updated choice of σ generates a polynomial overhead in time and sample complexity. We pick ε 3 = 1 2 ( ε 1 -B √ λδ ) so that each time we exit the inner loop, we have that, with high probability, the current value of p corresponds to an ( ε 1 -B √ λδ ) -multiaccurate predictor.

The second type of update (the outer loop update) is a calibration step where p new is set to be the following function with the notation of Algorithm 1

In this case, Corollary 7.5 of Gopalan et al. [2023] can be used as is to acquire that E [( p ∗ ( x ) -p old ( x )) 2 ] -E [( p ∗ ( x ) -p new ( x )) 2 ] ≥ ε 2 1 / 8 if we set δ ≤ ε 2 1 /C for some large enough universal constant C > 0 . Hence, the number of iterations of the outer loop is upper bounded by O (1 /ε 2 1 ) .

It remains to show that once the algorithm terminates, the output is approximately calibrated and multiaccurate. Regarding multiaccuracy, whenever we exit the inner loop, the current value of p corresponds (with high probability) to an ( ε 1 -B √ λδ ) -multiaccurate predictor. The predictor p δ is close to p in absolute distance and therefore can be shown to be approximately multiaccurate using some version of Lemma 7.2 in Gopalan et al. [2023]. In particular, following the proof of Lemma 7.2, the multiaccuracy guarantee for p δ changes to ( C , α + B √ λδ ) -multiaccuracy, where α is the multiaccuracy guarantee for p , by using a Cauchy-Schwarz inequality and bounding E [( w · x ) 2 ] by B 2 · λ . Since α = ε 1 -B √ λδ , the multiaccuracy guarantee is ε 1 . The calibration guarantee follows by the fact that the termination criterion corresponds to empirically checking whether p δ is calibrated and by using Lemma 7.4 in Gopalan et al. [2023], the empirical estimate should be close to the true calibration error. Overall, once we terminate, the output is ε 1 calibrated and multiaccurate.

Since the time and sample complexity of each of the calls of the weak learner depends on ε 3 , we pick δ ← ε 2 1 C (1+ B 2 λ ) so that ε 3 = Θ( ε 1 ) and δ ≤ ε 2 1 C as required.

Sample Complexity. For each of the inner loop iterations, we require a fresh sample of size O ( d 2 B 2 λ ε 2 1 ) , as specified by Proposition A.1 (by setting the probability of success to a sufficiently small constant). For the outer loop, we require O ( 1+ B 8 λ 4 ε 8 1 log( 1+ Bλ ε 1 )) samples per iteration, so that the calibration error estimate is accurate enough, according to Lemma 7.4 in Gopalan et al. [2023].

Time Complexity. Each of the inner loop iterations requires O ( d 3 B 2 λ ε 2 1 ) time and each of the outer loop iterations requires an additional O ( d 1+ B 2 λ ε 2 1 + 1+ B 10 λ 5 ε 10 1 log( 1+ Bλ ε 1 )) time.

The final technical complication we need to address is that their algorithm works only given binary labels. We can, however, form binary labels as follows. Let ( x , y ) be drawn from D . We have that y ∈ [0 , 1] . Given y , we draw an independent Bernoulli random variable y ′ with probability of success y , forming the distribution D ′ over R d ×{ 0 , 1 } . We run the algorithm of Gopalan et al. [2023] on D ′ and receive a predictor p such that

We have that

This concludes the proof of Theorem 3.2.

C.2 Proof of Lemma 3.3

We first apply Theorem 2.1 with g ′ ← ϕ ′ to get that for ϕ ′ w ( x ) = ϕ ′ ( w · x ) , we have

Moreover, we have

This concludes the proof of lemma 3.3.

D Proofs from Section 4

D.1 Proof of Theorem 4.1

In the case we consider here, g ′ is the sigmoid activation, i.e., g ′ ( t ) = (1 + e -t ) -1 for any t ∈ R . In particular, the pointwise surrogate loss we consider satisfies

for any y ∈ [0 , 1] and p ∈ (0 , 1) . We may extend Lemma 4.2 to also capture y ∈ { 0 , 1 } , by defining ℓ g (0 , f ′ (0)) = ℓ g (1 , f ′ (1)) = -ln 2 (the inequality would hold under this definition). Hence, following a similar procedure as the one used for proving Theorem B.1, we obtain the following by applying Lemma 4.2

Therefore, in order to prove Theorem 4.1, it is sufficient to provide a strong enough upper bound for the quantity of the right hand side of Equation (D.2) in terms of opt g . We observe that

It remains to bound the quantity E [ e | w ∗ · x | ( y -g ′ w ( x )) 2 ] . Set Q = e | w ∗ · x | ( y -g ′ w ( x )) 2 ( Q is a random variable). Then for any r ≥ 0 we have

✶ Since E [ e | w ∗ · x | {| w ∗ · x | > r } ] = ∫ ∞ s =0 F ( s ) ds = ∫ ∞ s =0 Pr[ | w ∗ · x | ≥ max { ln s, r } ] ds , we obtain

✶ To bound the quantity E [ e | w ∗ · x | ✶ {| w ∗ · x | > r } ] , consider F ( s ) := Pr[ e | w ∗ · x | ✶ {| w ∗ · x | > r } ≥ s ] . We have that F ( s ) = { s = 0 } Pr[ | w ∗ · x | ≤ r ] + Pr[ | w ∗ · x | ≥ max { ln s, r } ] .

Therefore, in total, we have that err 2 ( p ) ≤ 8 e r opt g +8 Ce B 2 e r e -( r/B ) 2 +2 ε and we may obtain Theorem 4.1 by picking r = B (ln 1 opt ) 1 / 2 .

D.2 Proof of Theorem 4.3

We first prove Lemma 4.4. We have that for any y ∈ { 0 , 1 } and p ∈ (0 , 1)

where CE( y, p ) is the cross entropy function. It is sufficient to show that for y ∈ { 0 , 1 } and p ∈ [0 , 1] ,

where the series on the right converges for all p < 1 . We can also write

with the series converging for p > 0 .

For the lower bound, we observe the following inequalities hold for all p ∈ [0 , 1]

Observe that

For the upper bound, we first prove the claim for y = 0 , where it states that

When p ≤ 1 / 2 we can use Eq. (D.5) to bound

The bounds holds by observing that since p (1 -p ) ≥ 1 / 4 ,

When p ≥ 1 / 2 , we note that

and 2 p ≥ 1 . Hence the bound holds in this case too.

In the case where y = 1 , the bound states that

This is implied by our bound for y = 0 by taking q = 1 -p . This concludes the proof of Lemma 4.4 and we are ready to prove Theorem 4.3. The following are true

Similarly to the proof of Theorem 4.1, we observe that

It remains to bound the quantity E [ | w ∗ · x | · | y -g ′ w ( x ) | ] . Set Q = | w ∗ · x | · | y -g ′ w ( x ) | ( Q is a random variable). Then for any r ≥ 0 we have

✶ To bound the quantity E [ | w ∗ · x | · ✶ {| w ∗ · x | > r } ] , consider F ( s ) := Pr[ | w ∗ · x | · ✶ {| w ∗ · x | > r } ≥ s ] . We have that F ( s ) = { s = 0 } Pr[ | w ∗ · x | ≤ r ] + Pr[ | w ∗ · x | ≥ max { s, r } ] .

Therefore, in total, we have that err 1 ( p ) ≤ 2 ln 4 · opt g +2( r + B ) e -r/B +2 r · opt g + ε and we may obtain Theorem 4.3 by picking r = B · ln 1 opt .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of learning SIMs agnostically. The authors proposed an algorithm achieving $B\\sqrt{opt}$ $\\ell_2$-error under mild distributional assumptions. Their main contributions are twofold: 1) they linked the $\\ell_2$ loss of a bi-lipschitz activation with its matching loss. 2) they proposed an algorithm that minimizes the matching loss for all Lipschitz activations. They also showed stronger guarantees for logistic regression and argued that the dependence of norm $B$ is inevitable in the final $\\ell_2$ loss bound achievable for any efficient algorithm.\n\nSTRENGTHS:\nThe authors bring in the technique of fenchel duality to study the distortion bound between the $\\ell_2$ loss and the matching loss for GLMs. Even though their idea is similar to theorem 5.5 in [GHK+23], the result is original. In addition, the authors proposed a method using omnipredictors to find a predictor that minimizes the matching loss with respect to all 1-Lipschitz activations. This idea is new to the reviewer.\n\nWEAKNESSES:\nThe result about the $\\ell_2$ error of learning sigmoid activation (thm 4.1) seems weak, as there are already similar results in the literature, see for example, thm 3.3 [FCG20], though in [FCG20] they used lightly stronger assumptions (bounded $D_x$), their $\\ell_2$ error does not have the logarithm factor. A detailed comparison with related works would be helpful to prove the significance of this paper.\n\nThe presentation of this paper lacks clarity and is confusing. There is no clear description of the algorithm proposed by the authors, and the sample complexity and runtime are not clearly stated throughout the paper. There are also a lot of symbols used in the paper that are not defined, to name a few: the term $opt_g$ appeared in the proof of thm 3.1, the $\\varepsilon$ appeared in the statement of thm C.2, and the function $l_2$ used in the proof of thm 3.2.\n\nThe proofs lack readability. For example, in line 261, it is hard to understand why $opt_g\\geq Pr[|w^*\\cdot x|\\leq 2\\lambda B^2]$ without further clarification. It would also be helpful to have more details in the proof of thm 3.2, like the relation between $\\epsilon_1$ and $\\epsilon_2$, and the statement from line 533 to 537.\n\nQUESTIONS:\nCan the authors provide some intuition about multi-accuracy and multi-caliberation? How do they relate to the omnipredictors for SIMs and matching loss? In particular, what exactly is the relation between multi-accuracy and the gradient of the matching loss, as mentioned in line 110?\n\nLIMITATIONS:\nI agree with the authors that the potential limitation of this work is the exact dependence of norm $B$ is not justified. It would be interesting future work to find the tightest $\\ell_2$ error bound of the SIMs.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives an efficient algorithm for learning Single Index Models with arbitrary monotone and Lipschitz function under the condition the marginal distribution of x has bounded variance in all directions.\n\nThe error guarantee of the algorithm is $O(B \\sqrt{\\lambda} \\sqrt{\\mathrm{opt}})+\\epsilon$ ($B=\\|w\\|_2$ and $\\lambda$ is the bound on variance) which is weaker than some previous result but this paper also has less assumption on the marginal distribution.\n\nThe authors give an SQ lower bound justifying the dependence of their error on B. However, that lower bound does not imply the error need to have a polynomial dependence on B (i.e. a $\\exp (\\log^{1/2} B$) dependence should be suffice to not contradict the lower bound).\n\nThe high level idea of the algorithm is this:\nThe authors considers matching loss as the surrogate loss function.\nIn the agnostic setting, it no longer holds that the matching loss and \nSquared loss has the same minimizer.\nInstead the authors observes that there is a “distortion bound” between matching loss (for bi-Lipschitz function $u$) and squared loss (lemma 2.2).\nNamely, this means for any prediction $p$ and true value $y$, the matching loss is at most off by some offset (which is the matching loss between $y$ and itself) and then a multiplicative factor (which depends on the bi-Lipschitz parameter of $u$). \nThis implies small matching loss implies small squared loss.\n(Up to some $O(\\mathrm{opt})$ error.)\n\nThen the argument follows that in order to learn a SIM with an unknown activation function $u$. \nWe can use existing result to learn a “omnipredictor” that has small matching loss for any bi-Lipschitz function.\nLet $u’$ be nearest bi-Lipschitz function of $u$.\nDue to the boundedness, it suffices to consider $u’$ instead of $u$.\nThis gives their algorithm for SIM.\n\nSome other result they show (using similar idea) are (ignoring the dependence on $B$ in the error):\n1. An distribution independent efficient algorithm for leaning GLM up to error $O(\\mathrm{opt})$ when the activation function is bi-Lipschitz\n2. An efficient algorithm for logistic regression up to $O(\\mathrm{opt})$ under some concentration assumption of the distribution\n\nSTRENGTHS:\nThe algorithm holds under very mild assumption on the marginal distribution.\nThe distortion bound the authors prove might also have other interesting applications.\n\nWEAKNESSES:\nThe main weakness is the authors do not have a lower bound that can match the upper bound result they give.\nIt would be interesting to see if one can get a better algorithm or prove a tighter lower bound.\n\nQUESTIONS:\nIt would be better if the authors can have a paragraph summarizing the high level ideas of the algorithms.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.\nThere is potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the learning of Single-Index Models (SIMs) with arbitrary monotone and Lipschitz activations. In SIM model, labeled examples $(x, y)$ are assumed to satisfy $E[y|x] = u^{-1}(w.x)$, where $w$ is an unknown vector, and  $u$ is an unknown monotone function (a.k.a. link function). Given IID-drawn samples from an unknown distribution $D$ over $\\mathbb{R}^d\\times [0,1]$, the learner's objective is to minimize a predefined loss (e.g., square loss) among the class of SIM models. This paper presents a learning algorithm with a square loss $O(\\sqrt{opt})+\\epsilon$ if $u^{-1}$ is 1-Lipschitz, where $opt$ is the minimum square error among all SIM models with bounded-norm  $w$.  Moreover, the authors provide related results for standard algorithms like GLMtron and logistic regression.\n\nSTRENGTHS:\nThe paper is solid, and the proof techniques are interesting. This paper considers a harder problem of the non-realizable case ($opt \\neq 0$) and unknown activation function. Moreover, the analysis is based on more relaxed conditions compared to prior works as it requires the marginals to have bounded second moments.\n\nWEAKNESSES:\nThe paper claims (in the title and abstract) to prove agostic learnability. This is misleading! First, agnostic in the PAC learning context implies no distributional assumptions on the labeled samples, but a bounded second moment is assumed in this work. \nSecond, the agnostic learner has an error $opt+\\epsilon$ which is clearly not the case here! The bound in Theorem 3.1 is of the form $O(B\\sqrt{opt})+\\epsilon$, where $B$ is the bound on $||w||$. \nGiven that the paper shows \"weak learnability\" as best.   \n\nThe paper is well-written in general but there are several typos that need to be corrected: \n- The notation for sphere, $\\mathbb{S}^{d-1}$, is not defined. \n- Line 158: is $f'$ the derivative of $f$?\n- Line 169: $d$ is missing:  $c:\\mathbb{R}\\to \\mathbb{R}$ need to be  $c:\\mathbb{R}^d\\to \\mathbb{R}$\n- Line 180: $d$ is also missing. \n\n\nI think it would be better if Definition 1.4 and 1.5 were written as Assumptions.\n\nQUESTIONS:\nQ1. When do we get the $opt+\\epsilon$ error? \n\nQ2. What about other loss functions beyond the square error and logistic loss? \n\nQ3. The bound for logistic loss scales with B as $e^{B^2}$, isn't that problematic?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
This work studies the problem of agnostically learning single index models with arbitrary Monotone and Lipschitz activations. Compared prior work, this work establishes the existence of an learning algorithm under more relaxed assumptions. This work is based on recent work by Gopalan et al. [2023] on Omniprediction using predictors satisfying calibrated multiaccuracy.

STRENGTHS:
This paper studies a difficult but realistic setting, and the topic of agonistically learning single index model is very interesting and important in itself.

WEAKNESSES:
I find this paper a bit difficult to read for general researchers in the machine learning community, especially those who are not familiar with the related literature. 

I also have a (perhaps naive) question about the claims in the abstract and line 33. It seems to claim that this paper devises an "algorithm" that can efficiently learn SIM in an agnostic setting, I wonder what is the algorithm actually? It seems all theoretical results in this paper are all pointing to the existence of such an algorithm, but not an actual algorithm that can be used to estimate it.

QUESTIONS:
My questions are raised in the previous section.

LIMITATIONS:
This paper may benefit from polishing up the write-up and make it more friendly for general ML researchers or practitioners.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09