Title: An Efficient Tester-Learner for Halfspaces
Venue: NeurIPS 2023
Accepted: no
Keywords: testable learning

pac learning

agnostic learning

Massart label noise

adversarial label noise

distribution testing
Abstract: We give the first efficient algorithm for learning halfspaces in the testable learning model recently defined by Rubinfeld and Vasilyan [2022].  In this model, a learner certifies that the accuracy of its output hypothesis is near optimal whenever the training set passes an associated test, and training sets drawn from some target distribution must pass the test.  This model is more challenging than distribution-specific agnostic or Massart noise models where the learner is allowed to fail arbitrarily if the distributional assumption does not hold. We consider the setting where the target distribution is the standard Gaussian in $d$ dimensions and the label noise is either Massart or adversarial (agnostic).  For Massart noise, our tester-learner runs in polynomial time and outputs a hypothesis with (information-theoretically optimal) error $\mathrm{opt} + \epsilon$ (and extends to any fixed strongly log-concave target distribution). For adversarial noise, our tester-learner obtains error $O(\mathrm{opt}) + \epsilon$ in polynomial time. Prior work on testable learning ignores the labels in the training set and checks that the empirical moments of the covariates are close to the moments of the base distribution.  Here we develop new tests of independent interest that make critical use of the labels and combine them with the moment-matching approach of Gollakota et al. [2022].  This enables us to implement a testable variant of the algorithm of Diakonikolas et al. [2020a, 2020b] for learning noisy halfspaces using nonconvex SGD.

Abstract

We give the first efficient algorithm for learning halfspaces in the testable learning model recently defined by Rubinfeld and Vasilyan [RV23]. In this model, a learner certifies that the accuracy of its output hypothesis is near optimal whenever the training set passes an associated test, and training sets drawn from some target distribution must pass the test. This model is more challenging than distributionspecific agnostic or Massart noise models where the learner is allowed to fail arbitrarily if the distributional assumption does not hold. We consider the setting where the target distribution is the standard Gaussian in d dimensions and the label noise is either Massart or adversarial (agnostic). For Massart noise, our tester-learner runs in polynomial time and outputs a hypothesis with (informationtheoretically optimal) error opt + ϵ (and extends to any fixed strongly log-concave target distribution). For adversarial noise, our tester-learner obtains error O ( opt )+ ϵ in polynomial time. Prior work on testable learning ignores the labels in the training set and checks that the empirical moments of the covariates are close to the moments of the base distribution. Here we develop new tests of independent interest that make critical use of the labels and combine them with the momentmatching approach of [GKK23]. This enables us to implement a testable variant of the algorithm of [DKTZ20a, DKTZ20b] for learning noisy halfspaces using nonconvex SGD.

1 Introduction

Learning halfspaces in the presence of noise is one of the most basic and well-studied problems in computational learning theory. A large body of work has obtained results for this problem under a variety of different noise models and distributional assumptions (see e.g. [BH21] for a survey). A major issue with common distributional assumptions such as Gaussianity, however, is that they can be hard or impossible to verify in the absence of any prior information.

The recently defined model of testable learning [RV23] addresses this issue by replacing such assumptions with efficiently testable ones. In this model, the learner is required to work with an arbitrary input distribution D XY and verify any assumptions it needs to succeed. It may choose to reject a given training set, but if it accepts, it is required to output a hypothesis with error close to opt ( C , D XY ) , the optimal error achievable over D XY by any function in a concept class C . Further, whenever the training set is drawn from a distribution D XY whose marginal is truly a well-behaved target distribution D ∗ (such as the standard Gaussian), the algorithm is required to accept with high probability. Such an algorithm, or tester-learner, is then said to testably learn C with respect to target marginal D ∗ . (See Definition 2.1.) Note that unlike ordinary distribution-specific agnostic learners, a tester-learner must take some nontrivial action regardless of the input distribution.

The work of [RV23, GKK23] established foundational algorithmic and statistical results for this 36

model and showed that testable learning is in general provably harder than ordinary distribution37

38

specific agnostic learning. As one of their main algorithmic results, they showed tester-learners for

the class of halfspaces over R d that succeed whenever the target marginal is Gaussian (or one of a 39 more general class of distributions), achieving error opt + ϵ in time and sample complexity d ˜ O (1 /ϵ 2 ) . 40 This matches the running time of ordinary distribution-specific agnostic learning of halfspaces over 41 the Gaussian using the standard approach of [KKMS08]. Their testers are simple and label-oblivious, 42 and are based on checking whether the low-degree empirical moments of the unknown marginal 43 match those of the target D ∗ . 44

45

46

47

These works essentially resolve the question of designing tester-learners achieving error opt + ϵ for halfspaces, matching known hardness results for (ordinary) agnostic learning [GGK20, DKZ20, DKPZ21]. Their running time, however, necessarily scales exponentially in 1 /ϵ .

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

A long line of research has sought to obtain more efficient algorithms at the cost of relaxing the optimality guarantee [ABL17, DKS18, DKTZ20a, DKTZ20b]. These works give polynomial-time algorithms achieving bounds of the form opt + ϵ and O ( opt ) + ϵ for the Massart and agnostic setting respectively under structured distributions (see Section 1.1 for more discussion). The main question we consider here is whether such guarantees can be obtained in the testable learning framework.

Our contributions. In this work we design the first tester-learners for halfspaces that run in fully polynomial time in all parameters. We match the optimality guarantees of fully polynomial-time learning algorithms under Gaussian marginals for the Massart noise model (where the labels arise from a halfspace but are flipped by an adversary with probability at most η ) as well as for the agnostic model (where the labels can be completely arbitrary). In fact, for the Massart setting our guarantee holds with respect to any chosen target marginal D ∗ that is isotropic and strongly log-concave, and the same is true of the agnostic setting albeit with a slightly weaker guarantee.

Theorem 1.1 (Formally stated as Theorem 4.1) . Let C be the class of origin-centered halfspaces over R d , and let D ∗ be any isotropic strongly log-concave distribution. In the setting where the labels are corrupted with Massart noise at rate at most η < 1 2 , C can be testably learned w.r.t. D ∗ up to error opt + ϵ using poly( d, 1 ϵ , 1 1 -2 η ) time and sample complexity.

Theorem 1.2 (Formally stated as Theorem 5.1) . Let C be as above. In the adversarial noise or agnostic setting where the labels are completely arbitrary, C can be testably learned w.r.t. N (0 , I d ) up to error O ( opt ) + ϵ using poly( d, 1 ϵ ) time and sample complexity.

Our techniques. The tester-learners we develop are significantly more involved than prior work on testable learning. We build on the nonconvex optimization approach to learning noisy halfspaces due to [DKTZ20a, DKTZ20b] as well as the structural results on fooling functions of halfspaces using moment matching due to [GKK23]. Unlike the label-oblivious, global moment tests of [RV23, GKK23], our tests make crucial use of the labels and check local properties of the distribution in regions described by certain candidate vectors. These candidates are approximate stationary points of a natural nonconvex surrogate of the 0-1 loss, obtained by running gradient descent. When the distribution is known to be well-behaved, [DKTZ20a, DKTZ20b] showed that any such stationary point is in fact a good solution (for technical reasons we must use a slightly different surrogate loss). Their proof relies crucially on structural geometric properties that hold for these well-behaved distributions, an important one being that the probability mass of any region close to the origin is proportional to its geometric measure.

In the testable learning setting, we must efficiently check this property for candidate solutions. Since these regions may be described as intersections of halfspaces, we may hope to apply the momentmatching framework of [GKK23]. Naïvely, however, they only allow us to check in polynomial time that the probability masses of such regions are within an additive constant of what they should be under the target marginal. But we can view these regions as sub-regions of a known band described by our candidate vector. By running moment tests on the distribution conditioned on this band and exploiting the full strength of the moment-matching framework, we are able to effectively convert our weak additive approximations to good multiplicative ones. This allows us to argue that our stationary points are indeed good solutions.

Limitations and Future Work. In this paper we provide the first efficient tester-learners for halfspaces when the noise is either adversarial or Massart. An interesting direction for future work would be to design tester-learners for the agnostic setting whose target marginal distributions may lie within a large family (e.g., strongly log-concave distributions) but still achieve error of O ( opt ) . Another interesting direction is providing tester-learners that are not tailored to a single target distribution, but are guaranteed to accept any member of a large family of distributions.

1.1 Related work 94

We provide a partial summary of some of the most relevant prior and related work on efficient 95 algorithms for learning halfspaces in the presence of adversarial label or Massart noise, and refer the 96 reader to [BH21] for a survey. 97

98

99

100

101

102

103

104

105

106

107

In the distribution-specific agnostic setting where the marginal is assumed to be isotropic and logconcave, [KLS09] showed an algorithm achieving error O ( opt 1 / 3 )+ ϵ for the class of origin-centered halfspaces. [ABL17] later obtained O ( opt ) + ϵ using an approach that introduced the principle of iterative localization , where the learner focuses attention on a band around a candidate halfspace in order to produce an improved candidate. [Dan15] used this principle to obtain a PTAS for agnostically learning halfspaces under the uniform distribution on the sphere, and [BZ17] extended it to more general s -concave distributions. Further works in this line include [YZ17, Zha18, ZSA20, ZL21]. [DKTZ20b] introduced the simplest approach yet, based entirely on nonconvex SGD, and showed that it achieves O ( opt ) + ϵ for origin-centered halfspaces over a wide class of structured distributions. Other related works include [DKS18, DKTZ22].

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

In the Massart noise setting with noise rate bounded by η , work of [DGT19] gave the first efficient distribution-free algorithm achieving error η + ϵ ; further improvements and followups include [DKT21, DTK22]. However, the optimal error opt achievable by a halfspace may be much smaller than η , and it has been shown that there are distributions where achieving error competitive with opt as opposed to η is computationally hard [DK22, DKMR22]. As a result, the distribution-specific setting remains well-motivated for Massart noise. Early distribution-specific algorithms were given by [ABHU15, ABHZ16], but a key breakthrough was the nonconvex SGD approach introduced by [DKTZ20a], which achieved error opt + ϵ for origin-centered halfspaces efficiently over a wide range of distributions. This was later generalized by [DKK + 22].

1.2 Technical overview

Our starting point is the nonconvex optimization approach to learning noisy halfspaces due to [DKTZ20a, DKTZ20b]. The algorithms in these works consist of running SGD on a natural nonconvex surrogate L σ for the 0-1 loss, namely a smooth version of the ramp loss. The key structural property shown is that if the marginal distribution is structured (e.g. log-concave) and the slope of the ramp is picked appropriately, then any w that has large angle with an optimal w ∗ cannot be an approximate stationary point of the surrogate loss L σ , i.e. that ∥∇L σ ( w ) ∥ must be large. This is proven by carefully analyzing the contributions to the gradient norm from certain critical regions of span( w , w ∗ ) , and crucially using the distributional assumption that the probability masses of these regions are proportional to their geometric measures. (See Fig. 3.) In the testable learning setting, the main challenge we face in adapting this approach is checking such a property for the unknown distribution we have access to.

A preliminary observation is that the critical regions of span( w , w ∗ ) that we need to analyze are rectangles, and are hence functions of a small number of halfspaces. Encouragingly, one of the key structural results of the prior work of [GKK23] pertains to 'fooling' such functions. Concretely, they show that whenever the true marginal D X matches moments of degree at most ˜ O (1 /τ 2 ) with a target D ∗ that satisfies suitable concentration and anticoncentration properties, then | E D X [ f ] -E D ∗ [ f ] | ≤ τ for any f that is a function of a small number of halfspaces. If we could run such a test and ensure that the probabilities of the critical regions over our empirical marginal are also related to their areas, then we would have a similar stationary point property.

However, the difficulty is that since we wish to run in fully polynomial time, we can only hope to fool such functions up to τ that is a constant. Unfortunately, this is not sufficient to analyze the probability masses of the critical regions we care about as they may be very small.

The chief insight that lets us get around this issue is that each critical region R is in fact of a very specific form, namely a rectangle that is axis-aligned with w : R = { x : ⟨ w , x ⟩ ∈ [ -σ, σ ] and ⟨ v , x ⟩ ∈ [ α, β ] } for some values α, β, σ and some v orthogonal to w . Moreover, we know w , meaning we can efficiently estimate the probability P D X [ ⟨ w , x ⟩ ∈ [ -σ, σ ]] up to constant multiplicative factors without needing moment tests. Denoting the band { x : ⟨ w , x ⟩ ∈ [ -σ, σ ] } by T and writing P D X [ R ] = P D X [ ⟨ v , x ⟩ ∈ [ α, β ] | x ∈ T ] P D X [ T ] , it turns out that we should expect P D X [ ⟨ v , x ⟩ ∈ [ α, β ] | x ∈ T ] = Θ(1) , as this is what would occur under the structured target distri-

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

bution D ∗ . (Such a 'localization' property is also at the heart of the algorithms for approximately learning halfspaces of, e.g., [ABL17, Dan15].) To check this, it suffices to run tests that ensure that P D X [ ⟨ v , x ⟩ ∈ [ α, β ] | x ∈ T ] is within an additive constant of this probability under D ∗ .

We can now describe the core of our algorithm (omitting some details such as the selection of the slope of the ramp). First, we run SGD on the surrogate loss L to arrive at an approximate stationary point and candidate vector w (technically a list of such candidates). Then, we define the band T based on w , and run tests on the empirical distribution conditioned on T . Specifically, we check that the low-degree empirical moments conditioned on T match those of D ∗ conditioned on T , and then apply the structural result of [GKK23] to ensure conditional probabilities of the form P D X [ ⟨ v , x ⟩ ∈ [ α, β ] | x ∈ T ] match P D ∗ [ ⟨ v , x ⟩ ∈ [ α, β ] | x ∈ T ] up to a suitable additive constant. This suffices to ensure that even over our empirical marginal, the particular stationary point w we have is indeed close in angular distance to an optimal w ∗ .

̸

A final hurdle that remains, often taken for granted under structured distributions, is that closeness in angular distance ∡ ( w , w ∗ ) does not immediately translate to closeness in terms of agreement, P [sign( ⟨ w , x ⟩ ) = sign( ⟨ w ∗ , x ⟩ )] , over our unknown marginal. Nevertheless, we show that when the target distribution is Gaussian, we can run polynomial-time tests that ensure that an angle of θ = ∡ ( w , w ∗ ) translates to disagreement of at most O ( θ ) . When the target distribution is a general strongly log-concave distribution, we show a slightly weaker relationship: for any k ∈ N , we can run tests requiring time d ˜ O ( k ) that ensure that an angle of θ translates to disagreement of at most O ( √ k · θ 1 -1 /k ) . In the Massart noise setting, we can make ∡ ( w , w ∗ ) arbitrarily small, and so obtain our opt + ϵ guarantee for any target strongly log-concave distribution in polynomial time. In the adversarial noise setting, we face a more delicate tradeoff and can only make ∡ ( w , w ∗ ) as small as Θ( opt ) . When the target distribution is Gaussian, this is enough to obtain final error O ( opt ) + ϵ in polynomial time. When the target distribution is a general strongly log-concave distribution, we instead obtain ˜ O ( opt ) + ϵ in quasipolynomial time.

2 Preliminaries

Notation and setup Throughout, the domain will be X = R d , and labels will lie in Y = {± 1 } . The unknown joint distribution over X × Y that we have access to will be denoted by D XY , and its marginal on X will be denoted by D X . The target marginal on X will be denoted by D ∗ . We use the following convention for monomials: for a multi-index α = ( α 1 , . . . , α d ) ∈ Z d ≥ 0 , x α denotes ∏ i x α i i , and | α | = ∑ i α i denotes its total degree. We use C to denote a concept class mapping R d to {± 1 } , which throughout this paper will be the class of halfspaces or functions of halfspaces over R d . We use opt ( C , D XY ) to denote the optimal error inf f ∈C P ( x ,y ) ∼ D XY [ f ( x ) = y ] , or just opt when C and D XY are clear from context. We recall the definitions of the noise models we consider. In the Massart noise model, the labels satisfy P y ∼ D XY | x [ y = sign( ⟨ w ∗ , x ⟩ ) | x ] = η ( x ) , where η ( x ) ≤ η < 1 2 for all x . In the adversarial label noise or agnostic model, the labels may be completely arbitrary. In both cases, the learner's goal is to produce a hypothesis with error competitive with opt .

̸

We now formally define testable learning. The following definition is an equivalent reframing of the original definition [RV23, Def 4], folding the (label-aware) tester and learner into a single tester-learner.

Definition 2.1 (Testable learning, [RV23]) . Let C be a concept class mapping R d to {± 1 } . Let D ∗ be a certain target marginal on R d . Let ϵ, δ > 0 be parameters, and let ψ : [0 , 1] → [0 , 1] be some function. We say C can be testably learned w.r.t. D ∗ up to error ψ ( opt ) + ϵ with failure probability δ if there exists a tester-learner A meeting the following specification. For any distribution D XY on R d ×{± 1 } , A takes in a large sample S drawn from D XY , and either rejects S or accepts and produces a hypothesis h : R d →{± 1 } . Further, the following conditions must be met:

̸

(a) (Soundness.) Whenever A accepts and produces a hypothesis h , with probability at least 1 -δ (over the randomness of S and A ), h must satisfy P ( x ,y ) ∼ D XY [ h ( x ) = y ] ≤ ψ ( opt ( C , D XY )) + ϵ .

(b) (Completeness.) Whenever D XY truly has marginal D ∗ , A must accept with probability at least 1 -δ (over the randomness of S and A ).

̸

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

3 Testing properties of strongly log-concave distributions

In this section we define the testers that we will need for our algorithm. All the proofs from this section can be found in Appendix B. We begin with a structural lemma that strengthens the key structural result of [GKK23], stated here as Proposition A.3. It states that even when we restrict an isotropic strongly log-concave D ∗ to a band around the origin, moment matching suffices to fool functions of halfspaces whose weights are orthogonal to the normal of the band.

Proposition 3.1. Let D ∗ be an isotropic strongly log-concave distribution. Let w ∈ S d -1 be any fixed direction. Let p be a constant. Let f : R d → R be a function of p halfspaces of the form in Eq. (A.2) , with the additional restriction that its weights v i ∈ S d -1 satisfy ⟨ v i , w ⟩ = 0 for all i . For some σ ∈ [0 , 1] , let T denote the band { x : |⟨ w , x ⟩| ≤ σ } . Let D be any distribution such that D | T matches moments of degree at most k = ˜ O (1 /τ 2 ) with D ∗ | T up to an additive slack of d -˜ O ( k ) . Then | E D ∗ [ f | T ] -E D [ f | T ] | ≤ τ.

We now describe some of the testers that we use. First, we need a tester that ensures that the distribution is concentrated in every single direction. More formally, the tester checks that the moments of the distribution along any direction are small.

Proposition 3.2. For any isotropic strongly log-concave D ∗ , there exists some constants C 1 and a tester T 1 that takes a set S ⊆ R d ×{± 1 } , an even k ∈ N , a parameter δ ∈ (0 , 1) and runs and in time poly ( d k , | S | , log 1 δ ) . Let D denote the uniform distribution over S . If T 1 accepts, then for any v ∈ S d -1

Moreover, if S is obtained by taking at least ( d k , ( log 1 δ ) k ) C 1 i.i.d. samples from a distribution whose R d -marginal is D ∗ , the test T 1 passes with probability at least 1 -δ .

Secondly, we will use a tester that makes sure the distribution is not concentrated too close to a specific hyperplane. This is one of the properties we will need to use in order to employ the localization technique of [ABL17].

Proposition 3.3. For any isotropic strongly log-concave D ∗ , there exist some constants C 2 , C 3 and a tester T 2 that takes a set S ⊆ R d ×{± 1 } a vector w ∈ S d -1 , parameters σ, δ ∈ (0 , 1) and runs in time poly ( d, | S | , log 1 δ ) . Let D denote the uniform distribution over S . If T 2 accepts, then

Moreover, if S is obtained by taking at least 100 K 1 σ 2 log ( 1 δ ) i.i.d. samples from a distribution whose R d -marginal is D ∗ , the test T 2 passes with probability at least 1 -δ .

Finally, in order to use the localization idea of [ABL17] in a manner similar to [DKTZ20b], we need to make sure that the distribution is well-behaved also within a band around to a certain hyperplane. The main property of the distribution that we establish is that functions of constantly many halfspaces have expectations very close to what they would be under our distributional assumption. As we show later in this work, having the aforementioned property allows us to derive many other properties that strongly log-concave distributions have, including many of the key properties that make the localization technique successful.

Proposition 3.4. For any isotropic strongly log-concave D ∗ and a constant C 4 , there exists a constant C 5 and a tester T 3 that takes a set S ⊆ R d ×{± 1 } a vector w ∈ S d -1 , parameters σ, τ δ ∈ (0 , 1) and runs in time poly ( d ˜ O ( 1 τ 2 ) , 1 σ , | S | , log 1 δ ) . Let D denote the uniform distribution over S , let T denote the band { x : |⟨ w , x ⟩| ≤ σ } and let F w denote the set {± 1 } -valued functions of C 4 halfspaces whose weight vectors are orthogonal to w . If T 3 accepts, then

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

4 Testably learning halfspaces with Massart noise

In this section we prove that we can testably learn halfspaces with Massart noise with respect to isotropic strongly log-concave distributions (see Definition A.1).

Theorem 4.1 (Tester-Learner for Halfspaces with Massart Noise) . Let D XY be a distribution over R d ×{± 1 } and let D ∗ be an isotropic strongly log-concave distribution over R d . Let C be the class of origin centered halfspaces in R d . Then, for any η < 1 / 2 , ϵ > 0 and δ ∈ (0 , 1) , there exists an algorithm (Algorithm 1) that testably learns C w.r.t. D ∗ up to excess error ϵ and error probability at most δ in the Massart noise model with rate at most η , using time and a number of samples from D XY that are polynomial in d, 1 /ϵ, 1 1 -2 η and log(1 /δ ) .

Algorithm 1: Tester-learner for halfspaces

Input: Training sets S 1 , S 2 , parameters σ , δ , α

Output: A near-optimal weight vector w , or rejection

Run PSGD on the empirical loss L σ over S 1 to get a list L of candidate vectors.

Test whether L contains an α -approximate stationary point w of the empirical loss L σ over S 2 . Reject if no such w exists.

for each candidate w ′ in { w , -w } do

Let B ′ w ( σ ) denote the band { x : |⟨ w ′ , x ⟩| ≤ σ } . Let F ′ w denote the class of functions of at most two halfspaces with weights orthogonal to w ′ . Let δ ′ = Θ( δ ) .

Run T 1 ( S 2 , k = 2 , δ ) to verify that the empirical marginal is approximately isotropic. Reject if T 1 rejects.

Run T 3 ( S 2 , w ′ , σ = σ/ 6 , τ, δ ′ ) and T 3 ( S, w ′ , σ = σ/ 2 , τ, δ ′ ) for a suitable constant τ to verify that the empirical distribution conditioned on B ′ w ( σ/ 6) and B ′ w ( σ/ 2) fools F ′ w up to τ . Reject if T 3 rejects.

Run T 2 ( S 2 , w ′ , σ, δ ′ ) to verify that P S [ B ′ w ] = Θ( σ ) . Reject if T 2 rejects.

Estimate the empirical error of w ′ on S .

If all tests have accepted, output w ′ ∈ { w , -w } with the best empirical error.

To show our result, we revisit the approach of [DKTZ20a] for learning halfspaces with Massart noise under well-behaved distributions. Their result is based on the idea of minimizing a surrogate loss that is non convex, but whose stationary points correspond to halfspaces with low error. They also require that their surrogate loss is sufficiently smooth, so that one can find a stationary point efficiently. While the distributional assumptions that are used to demonstrate that stationary points of the surrogate loss can be discovered efficiently are mild, the main technical lemma, which demostrates that any stationary point suffices, requires assumptions that are not necessarily testable. We establish a label-dependent approach for testing, making use of tests that are applied during the course of our algorithm.

We consider a slightly different surrogate loss than the one used in [DKTZ20a]. In particular, for σ > 0 , we let

where ℓ σ : R → [0 , 1] is a smooth approximation to the ramp function with the properties described in Proposition C.1 (see Appendix C), obtained using a piecewise polynomial of degree 3 . Unlike the standard logistic function, our loss function has derivative exactly 0 away from the origin (for | t | > σ/ 2 ). This makes the analysis of the gradient of L σ easier, since the contribution from points lying outside a certain band is exactly 0 .

The smoothness allows us to run PSGD to obtain stationary points efficiently, and we now state the convergence lemma we need.

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

Proposition 4.2 (PSGD Convergence, Lemmas 4.2 and B.2 in [DKTZ20a]) . Let L σ be as in Equation (4.1) with σ ∈ (0 , 1] , ℓ σ as described in Proposition C.1 and D XY such that the marginal D X on R d satisfies Property (3.1) for k = 2 . Then, for any ϵ > 0 and δ ∈ (0 , 1) , there is an algorithm whose time and sample complexity is O ( d σ 4 + log(1 /δ ) ϵ 4 σ 4 ) , which, having access to samples from D XY , outputs a list L of vectors w ∈ S d -1 with | L | = O ( d σ 4 + log(1 /δ ) ϵ 4 σ 4 ) so that there exists w ∈ L with

In particular, the algorithm performs Stochastic Gradient Descent on L σ Projected on S d -1 (PSGD).

It now suffices to show that, upon performing PSGD on L σ , for some appropriate choice of σ , we acquire a list of vectors that testably contain a vector which is approximately optimal. We first prove the following lemma, whose distributional assumptions are relaxed compared to the corresponding structural Lemma 3.2 of [DKTZ20a]. In particular, instead of requiring the marginal distribution to be 'well-behaved", we assume that the quantities of interest (for the purposes of our proof) have expected values under the true marginal distribution that are close, up to multiplicative factors, to their expected values under some 'well-behaved" (in fact, strongly log-concave) distribution. While some of the quantities of interest have values that are miniscule and estimating them up to multiplicative factors could be too costly, it turns out that the source of their vanishing scaling can be completely attributed to factors of the form P [ |⟨ w , x ⟩| ≤ σ ] (where σ is small), which, due to standard concentration arguments, can be approximated up to multiplicative factors, given w ∈ S d -1 and σ > 0 (see Proposition 3.3). As a result, we may estimate the remaining factors up to sufficiently small additive constants (see Proposition 3.4) to get multiplicative overall closeness to the 'well behaved" baseline. We defer the proof of the following Lemma to Appendix C.1.

Lemma 4.3. Let L σ be as in Equation (4.1) with σ ∈ (0 , 1] , ℓ σ as described in Proposition C.1, let w ∈ S d -1 and consider D XY such that the marginal D X on R d satisfies Properties (3.2) and (3.3) for C 4 = 2 and accuracy τ . Let w ∗ ∈ S d -1 define an optimum halfspace and let η < 1 / 2 be an upper bound on the rate of the Massart noise. Then, there are constants c 1 , c 2 , c 3 > 0 such that if ∥∇ w L σ ( w ) ∥ 2 < c 1 (1 -2 η ) and τ ≤ c 2 , then

Combining Proposition 4.2 and Lemma 4.3, we get that for any choice of the parameter σ ∈ (0 , 1] , by running PSGD on L σ , we can construct a list of vectors of polynomial size (in all relevant parameters) that testably contains a vector that is close to the optimum weight vector. In order to link the zero-one loss to the angular similarity between a weight vector and the optimum vector, we use the following Proposition (for the proof, see Appendix C.2).

̸

Proposition 4.4. Let D XY be a distribution over R d × {± 1 } , w ∗ ∈ arg min w ∈ S d -1 P D XY [ y = sign( ⟨ w , x ⟩ )] and w ∈ S d -1 . Then, for any θ ≥ ∡ ( w , w ∗ ) , θ ∈ [0 , π/ 4] , if the marginal D X on R d satisfies Property (3.1) for C 1 > 0 and some even k ∈ N and Property (3.2) with σ set to ( C 1 k ) k 2( k +1) · (tan θ ) k k +1 , then, there exists a constant c > 0 such that the following is true.

̸

We are now ready to prove Theorem 4.1.

Proof of Theorem 4.1. Throughout the proof we consider δ ′ to be a sufficiently small polynomial in all the relevant parameters. Each of the failure events will have probability at least δ ′ and their number will be polynomial in all the relevant parameters, so by the union bound, we may pick δ ′ so that the probability of failure is at most δ .

The algorithm we run is Algorithm 1, with appropriate selection of parameters and given samples S 1 , S 2 , each of which are sufficiently large sets of independent samples from the true unknown distribution D XY . For some σ ∈ (0 , 1] to be defined later, we run PSGD on the empirical loss L σ over S 1 as described in Proposition 4.2 with ϵ = c 1 (1 -2 η ) σ/ 4 , where c 1 is given by Lemma 4.3. By Proposition 4.2, we get a list L of vectors w ∈ S d -1 with | L | = poly( d, 1 /σ ) such that there exists w ∈ L with ∥∇ w L σ ( w ) ∥ 2 < 1 2 c 1 (1 -2 η ) under the true distribution, if the marginal is isotropic.

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

W

Gc

θ

0-3

A1

A1

1

b

b

G

Having acquired the list L using sample S 1 , we use the independent samples in S 2 to test whether L contains an approximately stationary point of the empirical loss on S 2 . If this is not the case, then we may safely reject: for large enough | S 1 | , if the distribution is indeed isotropic strongly logconcave, there is an approximate stationary of the population loss in L and if | S 2 | is large enough, the gradient of the empirical loss on S 2 will be close to the gradient of the population loss on each of the elements of L , due to appropriate concentration bounds for log-concave distributions as well as the fact that the elements of L are independent from S 2 . For the following, let w be a point such that ∥∇ w L σ ( w ) ∥ 2 < c 1 (1 -2 η ) under theempirical distribution over S 2

In Lemma 4.3 and Proposition 4.4 we have identified certain properties of the marginal distribution that are sufficient for our purposes, given that L contains an approximately stationary point of the empirical (surrogate) loss on S 2 . Our testers T 1 , T 2 , T 3 verify that these properties hold for the empirical marginal over our sample S 2 , and it will be convenient to analyze the optimality of our algorithm purely over S 2 . In particular, we will need to require that | S 2 | is sufficiently large, so that when the true marginal is indeed the target D ∗ , our testers succeed with high probability (for the corresponding sample complexity, see Propositions 3.2, 3.3 and 3.4). Moreover, by standard generalization theory, since the VC dimension of halfspaces is only O ( d ) and for us | S 2 | is a large poly( d, 1 /ϵ ) , both the error of our final output and the optimal error over S 2 will be close to that over D XY . So in what follows, we will abuse notation and refer to the uniform distribution over S 2 as D XY and the optimal error over S 2 simply as opt .

We proceed with some basic tests. Throughout the rest of the algorithm, whenever a tester fails, we reject, otherwise we proceed. First, we run testers T 2 with inputs ( w , σ/ 2 , δ ′ ) and ( w , σ/ 6 , δ ′ ) (Proposition 3.3) and T 3 with inputs ( w , σ/ 2 , c 2 , δ ′ ) and with ( w , σ/ 6 , c 2 , δ ′ ) (Proposition 3.4, c 2 as defined in Lemma 4.3). This ensures that for the approximate stationary point w of the L σ , the probability within the band B w ( σ/ 2) = { x : |⟨ w , x ⟩| ≤ σ/ 2 } is Θ( σ ) (and similarly for B w ( σ/ 6) ) and moreover that our marginal conditioned on each of the bands fools (up to an additive constant) functions of halfspaces with weights orthogonal to w . As a result, we may apply Lemma 4.3 to w and form a list of 2 vectors { w , -w } which contains some w ′ with ∡ ( w ′ , w ∗ ) ≤ c 2 σ/ (1 -2 η )

(where

c

3

is as defined in Lemma 4.3).

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

We run T 1 (Proposition 3.2) with k = 2 to verify that the marginals are approximately isotropic and we use T 2 once again, with appropriate parameters for each w and its negation, to apply Proposition 4.4 and get that { w , -w } contains a vector w ′ with

̸

̸

However, we do not know which of the weight vectors in { w , -w } is the one guaranteed to achieve small error. In order to discover this vector, we estimate the probability of error of each of the corresponding halfspaces (which can be done efficiently, due to Hoeffding's bound) and pick the one with the smallest error. This final step does not require any distributional assumptions and we do not need to perform any further tests.

5 Testably learning halfspaces in the agnostic setting

In this section, we provide our result on efficiently and testably learning halfspaces in the agnostic setting with respect to isotropic strongly log-concave target marginals. We defer the proofs to Appendix D. The algorithm we use is once more Algorithm 1, but we call it multiple times for different choices of the parameter σ , reject if any call rejects and output the vector that achieved the minimum empirical error overall, otherwise. Also, the tester T 1 is called for a general k (not necessarily k = 2 ).

̸

Theorem 5.1 (Efficient Tester-Learner for Halfspaces in the Agnostic Setting) . Let D XY be a distribution over R d ×{± 1 } and let D ∗ be a strongly log-concave distribution over R d (Definition A.1). Let C be the class of origin centered halfspaces in R d . Then, for any even k ∈ N , any ϵ > 0 and δ ∈ (0 , 1) , there exists an algorithm that agnostically testably learns C w.r.t. D ∗ up to error O ( k 1 / 2 · opt 1 -1 k +1 ) + ϵ , where opt = min w ∈ S d -1 P D XY [ y = sign( ⟨ w , x ⟩ )] , and error probability at most δ , using time and a number of samples from D XY that are polynomial in d ˜ O ( k ) , (1 /ϵ ) ˜ O ( k ) and (log(1 /δ )) O ( k ) .

In particular, by picking some appropriate k ≤ log 2 d , we obtain error ˜ O ( opt )+ ϵ in quasipolynomial time and sample complexity, i.e. poly(2 polylog d , ( 1 ϵ ) polylog d ) .

To prove Theorem 5.1, we may follow a similar approach as the one we used for the case of Massart noise. However, in this case, the main structural lemma regarding the quality of the stationary points involves an additional requirement about the parameter σ . In particular, σ cannot be arbitrarily small with respect to the error of the optimum halfspace, because, in this case, there is no upper bound on the amount of noise that any specific point x might be associated with. As a result, picking σ to be arbitrarily small would imply that our algorithm only considers points that lie within a region that has arbitrarily small probability and can hence be completely corrupted with the adversarial opt budget. On the other hand, the polynomial slackness that the testability requirement introduces (through Proposition 4.4) between the error we achieve and the angular distance guarantee we can get via finding a stationary point of L σ (which is now coupled with opt ), appears to the exponent of the guarantee we achieve in Theorem 5.1.

Lemma 5.2. Let L σ be as in Equation (4.1) with σ ∈ (0 , 1] , ℓ σ as described in Proposition C.1, let w ∈ S d -1 and consider D XY such that the marginal D X on R d satisfies Properties (3.2) , (3.3) and (3.4) for w with C 4 = 2 and accuracy parameter τ . Let opt be the minimum error achieved by some origin centered halfspace and let w ∗ ∈ S d -1 be a corresponding vector. Then, there are constants c 1 , c 2 , c 3 , c 4 > 0 such that if opt ≤ c 1 σ , ∥∇ w L σ ( w ) ∥ 2 < c 2 , and τ ≤ c 3 then

We obtain our main result for Gaussian target marginals by refining Proposition 4.4 for the specific case when the target marginal distribution D ∗ is the standard multivariate Gaussian distribution. The algorithm for the Gaussian case is similar to the one of Theorem 5.1, but it runs different tests for the improved version (see Proposition D.1) of Proposition 4.4.

Theorem 5.3. In Theorem 5.1, if D ∗ is the standard Gaussian in d dimensions, we obtain error O ( opt ) + ϵ in polynomial time and sample complexity, i.e. poly( d, 1 /ϵ, log(1 /δ )) .

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

References

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

[DKZ20] Ilias Diakonikolas, Daniel Kane, and Nikos Zarifis. Near-optimal sq lower bounds for agnostically learning halfspaces and relus under gaussian marginals. Advances in Neural Information Processing Systems , 33:13586-13596, 2020. 1

[DTK22] Ilias Diakonikolas, Christos Tzamos, and Daniel M Kane. A strongly polynomial algorithm for approximate forster transforms and its application to halfspace learning. arXiv preprint arXiv:2212.03008 , 2022. 1.1

[GGK20] Surbhi Goel, Aravind Gollakota, and Adam Klivans. Statistical-query lower bounds via functional gradients. Advances in Neural Information Processing Systems , 33:21472158, 2020. 1

[GKK23] Aravind Gollakota, Adam R Klivans, and Pravesh K Kothari. A moment-matching approach to testable learning and a new characterization of rademacher complexity. Proceedings of the fifty-fifth annual ACM Symposium on Theory of Computing , 2023. To appear. (document), 1, 1, 1.2, 3, A.3, A

[KKMS08] Adam Tauman Kalai, Adam R Klivans, Yishay Mansour, and Rocco A Servedio. Agnostically learning halfspaces. SIAM Journal on Computing , 37(6):1777-1805, 2008. 1

[KLS09] Adam R Klivans, Philip M Long, and Rocco A Servedio. Learning halfspaces with malicious noise. Journal of Machine Learning Research , 10(12), 2009. 1.1

[RV23] Ronitt Rubinfeld and Arsen Vasilyan. Testing distributional assumptions of learning algorithms. Proceedings of the fifty-fifth annual ACM Symposium on Theory of Computing , 2023. To appear. (document), 1, 1, 2, 2.1

[SW14] Adrien Saumard and Jon A Wellner. Log-concavity and strong log-concavity: a review. Statistics surveys , 8:45, 2014. A.1, A.2, A

[YZ17] Songbai Yan and Chicheng Zhang. Revisiting perceptron: Efficient and label-optimal learning of halfspaces. Advances in Neural Information Processing Systems , 30, 2017. 1.1

[Zha18] Chicheng Zhang. Efficient active learning of sparse halfspaces. In Conference on Learning Theory , pages 1856-1880. PMLR, 2018. 1.1

[ZL21] Chicheng Zhang and Yinan Li. Improved algorithms for efficient active learning halfspaces with massart and tsybakov noise. In Conference on Learning Theory , pages 4526-4527. PMLR, 2021. 1.1

[ZSA20] Chicheng Zhang, Jie Shen, and Pranjal Awasthi. Efficient active learning of sparse halfspaces with arbitrary bounded noise. Advances in Neural Information Processing Systems , 33:7184-7197, 2020. 1.1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an efficient algorithm for learning halfspaces in the testable learning model in which the tester-learner first applies a test on the training data and if the test succeeds the algorithm produces a hypothesis which is guaranteed to be near-optimal. It is required that if the data comes from a target distribution, then the test must succeed with high probability.\n\nThe paper considers learning halfspaces in the case where the target distribution is Gaussian (or strongly log-concave) and where the labels are subjected to Massart noise or adversarial noise (i.e., agnostic setting). The paper builds on several ideas from previous papers by Diakonikolas et al.\n\nSTRENGTHS:\nLearning halfspaces is a fundamental problem in machine learning. Even though it is one of the simplest tasks, the problem becomes non-trivial in the presence of label noise. Recently, there has been a lot of interest in developing algorithms for learning halfspaces in several settings (e.g., Massart noise, ...). The submitted paper is the first work that presents an efficient algorithm for learning halfspaces in the testable learning model of Rubinfeld and Vasilyan.\n\nI find the paper to be clear and generally well-written, and I find the results novel and interesting.\n\nWEAKNESSES:\nMinor comments/typos:\n- Page 2 line 71: \"an important one being that the probability mass of any region close to the origin is proportional to its geometric measure\" -> \"an important one being that the probability mass of any region close to the origin is roughly proportional to its geometric measure\".\n- Page 5, line 198: Shouldn't the title of the section be \"Testing properties of isotropic strongly log-concave distributions\"?\n- Page 5, line 214: \"and runs and in time poly(...)\" -> \"and runs in time poly(...)\"\n- Page 5, line 235: There should be a comma between \\tau and \\delta.\n- Page 7, line 307: \"Each of the failure events will have probability at least $\\delta'$ \" -> \"Each of the failure events will have probability at most $\\delta'$ \".\n- Page 8, line 322: \"under theempirical distribution\" -> \"under the empirical distribution\".\n\nThe following relevant references seem to be missing:\n- [1] Sitan Chen, Frederic Koehler, Ankur Moitra, Morris Yau, \"Classification Under Misspecification: Halfspaces, Generalized Linear Models, and Connections to Evolvability\", NeurIPS 2020.\n- [2] Rajai Nasser, Stefan Tiegel, \"Optimal SQ Lower Bounds for Learning Halfspaces with Massart Noise\", COLT 2022.\n\nQUESTIONS:\nMinor questions:\n- Page 5: In Algorithm 1, shouldn't T1 also run with $\\delta'$ ?\n\nOptional suggestion:\n- Did the authors consider the Tsybakov noise model? It is harder than the Massart noise but easier than the agnostic setting, and hence it might be possible to get an efficient tester-learner that achieves the information-theoretically optimal error $\\text{opt}+\\epsilon$ for halfspaces with Tsybakov noise (of course, under structured marginal distributions such as Gaussian).\n\nLIMITATIONS:\nNo concerns regarding potential societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers gives an efficient algorithm for learning halfspaces under the testable learning framework of Rubinfeld and Vasilyan (STOC'23) and facing either Massart or agnostic noise. In this setting, the algorithm is given some reference marginal distribution $D^*$ (which is assumed to be isotropic and strongly log-concave), and it may choose to \"reject\" (asserting that the actual marginal is different from $D^*$) instead of outputting a hypothesis. Naturally, the learner needs to satisfy the following two conditions:\n- Completeness: When the marginal is indeed $D^*$, the learner does not reject w.h.p.\n- Soundness: The probability that the learner outputs an insufficiently accurate hypothesis is low. Here, sufficient accuracy means achieving either $\\mathsf{opt} + \\epsilon$ (under Massart noise) or $\\tilde O(\\mathsf{opt}) + \\epsilon$ error (in the agnostic setting), where $\\mathsf{opt}$ is the loss of the optimal halfspace and $\\epsilon > 0$ is a parameter.\n\nThe solution is built upon the nonconvex optimization approach to learning halfspaces under noise in the literature. The key property for this approach to succeed is that when some appropriate loss function is minimized, all the stationary points are reasonably close to the true parameter. The crux of the current work is then to identify certain testable properties of the marginal under which the above argument goes through, so that we either get a good learning guarantee, or obtain a witness for that the marginal is not $D^*$.\n\nSTRENGTHS:\nThe paper studies a fundamental learning theory problem (i.e., learning halfspaces) in the newly introduced testable learning setup. The results are strong and comprehensive, and the solution is nontrivial and requires several novel ideas. The paper is very nicely written and well-strutured, and the main paper contains sufficient details (including the \"technical overview\" section) for the reader to appreciate the high-level ideas behind the work.\n\nDespite the few weaknesses discussed below, I found the paper a strong submission that should be accepted.\n\nWEAKNESSES:\n- The hypothesis class is restricted to homogeneous halfspaces (without a bias).\n- In the agnostic case, the error bound can be higher than the optimal error by a constant factor.\n\nQUESTIONS:\nEchoing the weaknesses part, what are the technical hurdles that prevent the current approach from handling non-homogeneous halfspaces and achieving the \"$\\mathsf{opt} + \\epsilon$\"-type error guarantee?\n\nLIMITATIONS:\nThis is a theory paper and its limitations are in the assumptions made by the problem setting as well as the main results, e.g., the restriction to halfspaces, the noise model, and that a single reference marginal distribution is provided. These are clearly stated in the paper as well as the separate \"Limitations and Future Work\" section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLearning halfspace is a very important problem in machine learning which has been studied extensively. However, generally some distributional assumptions like gaussianity are assumed which in general is difficult to verify. To address this issue, recently Rubinfeld and Vasilyan (STOC 23) have introduced Testable learning framework. Here the primary objective is that if the tester accepts, then the output of the learner is close to OPT + \\epsilon (OPT being the optimal error), and when it satisfies the distributional assumptions, the algorithm accepts with high probability. However, when the Gaussian distributional assumption is taken (let's denote this as D^*), it takes $d^{1/\\epsilon^2}$ samples, which is also tight. Thus often researchers are interested to design algorithms that have better complexity with respect to $1/\\epsilon$, but the error becomes $f(OPT) + \\epsilon$ for some function f.\n\nIn this work, the authors first design a tester when $D^*$ is isotropic log-concave distribution and the labels are corrupted according to Massart noise (the labels are changed by an adversary with a small probability $\\eta$).  Their algorithm runs in polynomial time and has error $OPT + \\epsilon$ (Theorem 4.1). Later they design testers for adversarial noise with respect to Gaussian distribution with error $O(OPT) + \\epsilon$ (Theorem 1.2).\n\nIn Section 4, the authors study the case with Massart noise. The primary idea here is to minimize a non-convex smooth surrogate loss (4.1) such that its stationary points correspond to halfspaces with small error. The authors first run PSGD on this surrogate loss function to get a set of vectors L such that one vector in L is close to the optimal weight vector. Then they apply localization ideas based upon the region that is an axis-aligned rectangle T, and check if the low degree moments of input distribution D conditioned on T match with D^* conditioned on T. This will ensure that the angular distance of a stationary point w is close to the optimal w^* (Lemma 4.3). To convert closeness in angular distance to closeness in 0-1 loss, they use the fact that the distribution is isotropic strongly log-concave (Proposition 4.4).\n\nLater in Section 5, the authors study the agnostic setting where they will call the algorithm from Section 4 several times, each time with different parameters. The idea is that in the agnostic setting, running the algorithm for only once, the algorithm might only consider points that lie within a region with small probability. This finally gives a tester with error $O(OPT) + \\epsilon$  when $D^*$ is isotropic log-concave distribution as well as Gaussian (Theorem 5.1 and Theorem 5.3).\n\nSTRENGTHS:\nThe paper gives the first algorithm for testable learning of halfspaces that runs in poly(d, epsilon). The algorithm is very nice. With the complexity pulled down drastically, a proper implementation and experimental results for this algorithm would be possible and it would be nice to see the relevance of the concept of testable learning in various applications.\n\nAlso the algorithm can handle both adversarial and massard noise.\n\nWEAKNESSES:\nThe usefulness of the testable learning model in real life applications is yet to be understood.\n\nQUESTIONS:\nCan this approach be used to design tester-learners for function classes other than halfspaces.\n\nLIMITATIONS:\nIt is a pure theoretical work in the paradigm of testable learning - a relatively new concept whose importance is not yet fully confirmed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work provides an efficient algorithm for testably learning halfspaces, extending the frontier of the recently introduced testable learning which does not assume anything about the given data distribution. Specificially, the setting is as follows: the target distribution is standard Gaussian (or any fixed strongly log-concave distribution) and the label noise is Massart or adversarial (agnostic).\n\nThe main result is two-fold. \n1. For Massart noise, if the target distribution is strongly log-concave, the paper proves an algorithmic guarantee that testably learns halfspaces up to $opt + \\epsilon$ error and runs in $poly(d,1/\\epsilon,1/(1-2\\eta), \\log(1/\\delta)$ time. \n2. For agnostic learning,  if the target distribution is strongly log-concave, the guarantee is that the algorithm testably learns halfspaces up to $O(k^{1/2} opt^{k/(k+1)} + \\epsilon$ error and runs in \"roughly\" $poly(d^k,1/\\epsilon^k, \\log^k(1/\\delta)$ time (ignoring some logarithmic factors). One can strengthen this result if the target distribution is standard Gaussian. Then the algorithm testably learns halfspaces up to $O(opt) + \\epsilon$ error and runs in $poly(d,1/\\epsilon, \\log(1/\\delta)$ time, a result that matches previous non-testable learning results for halfspaces.\n\nThe methodology borrows two algorithmic ideas and strengthens them. One is the algorithmic idea that runs (nonconvex) SGD on a convex surrogate (ramp function) for the 0-1 loss. Originally, given some distributional assumption, this approach would yield a hypothesis found from an approximate stationary point. In this paper, the authors check if such property is satisfied for the unknown distribution of the testable learning setting, leading to develop a three-stage testing procedure for strongly log-concave distributions. Here, the second algorithmic idea of moment matching kicks in. For the band $T$ s.t. $|\\langle w,x \\rangle| \\le \\sigma$, tests are ran on the empirical distribution conditioned on $T$ to check moments matching those of the target distribution on $T$.\n\nSTRENGTHS:\n- This work extends upon the recently introduced testable learning in one of the fundamental problems of learning, i.e., learning halfspaces. I believe the topic is of good importance as testable learning yields more practicality to learning algorithms. In that sense, the work studies and provides strong algorithmic result to an important problem.\n- The techniques used in the paper utilize two previous algorithmic ideas (convex surrogate SGD + moment-matching to fool) and neatly tie these two ideas together to a testable algorithm. The algorithmic techniques are also practical.\n- The tests are ran on the distribution conditioned on the band $T$. With this \"trick\", the paper manages to change weak additive guarantees to strong multiplicative ones.\n- The results are technically strong and presentation is clear.\n- The agnostic learning algorithm only modifies the Massrt one slightly, which is neat, though this may be more of contribution of previous work than that of this work.\n\nWEAKNESSES:\nNo notable weaknesses, but refer to Questions for a potential undecided one.\n\nQUESTIONS:\n- For agnostic testable learning for strongly log-concave distributions, the guarantees are weaker. Is there any inherent difficulty (e.g., lower bounds or conjectured hardnesses) that may explain the weaker guarantees? What is the intuitive reason why Massart and agnostic differ in results?\n- (Typo) L201 \"stated here as Proposition A.3\": Should this be 3.1 instead of A.3?\n\nLIMITATIONS:\nNo limtation addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-12"}