Title: The Implicit Bias of Gradient Descent on Separable Multiclass Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: gradient descent

multiclass classification

hard-margin SVM

implicit bias
Abstract: Implicit bias describes the phenomenon where optimization-based training algorithms, without explicit regularization, show a preference for simple estimators even when more complex estimators have equal objective values. Multiple works have developed the theory of implicit bias for binary classification under the assumption that the loss satisfies an *exponential tail property*. However, there is a noticeable gap in analysis for multiclass classification, with only a handful of results which themselves are restricted to the cross-entropy loss. In this work, we employ the framework of Permutation Equivariant and Relative Margin-based (PERM) losses [Wang and Scott, 2024] to introduce a multiclass extension of the exponential tail property. This class of losses includes not only cross-entropy but also other losses. Using this framework, we extend the implicit bias result of Soudry et al. [2018] to multiclass classification. Furthermore, our proof techniques closely mirror those of the binary case, thus illustrating the power of the PERM framework for bridging the binary-multiclass gap.

The Implicit Bias of Gradient Descent on Separable Multiclass Data

Abstract

Implicit bias describes the phenomenon where optimization-based training algorithms, without explicit regularization, show a preference for simple estimators even when more complex estimators have equal objective values. Multiple works have developed the theory of implicit bias for binary classification under the assumption that the loss satisfies an exponential tail property . However, there is a noticeable gap in analysis for multiclass classification, with only a handful of results which themselves are restricted to the cross-entropy loss. In this work, we employ the framework of Permutation Equivariant and Relative Margin-based (PERM) losses [Wang and Scott, 2024] to introduce a multiclass extension of the exponential tail property. This class of losses includes not only cross-entropy but also other losses. Using this framework, we extend the implicit bias result of Soudry et al. [2018] to multiclass classification. Furthermore, our proof techniques closely mirror those of the binary case, thus illustrating the power of the PERM framework for bridging the binary-multiclass gap.

1 Introduction

Overparameterized models such as neural networks have shown state-of-the-art performance in many applications, despite having the potential to overfit. Zhang et al. [2021] demonstrate that this potential is indeed realizable by training real-world models to fit random noise. In recent years, there have been several research efforts that aim to understand the impressive performance of overparametrized models despite this ability to overfit. Both the model architecture and the training algorithms for selecting the weights have been investigated in this regard.

Work on implicit bias [Soudry et al., 2018, Ji et al., 2020, Vardi, 2022] has focused on the latter factor. Implicit bias is the hypothesis that gradient-based methods have a built-in preference for models with low-complexity. This hypothesis is perhaps best understood in the setting of (unregularized) empirical risk minimization for learning a linear model under the assumption of linearly separable data. Soudry et al. [2018] showed that in binary classification, implicit bias holds when the loss has the exponential tail property [Soudry et al., 2018, Theorem 3]. The same work also demonstrated implicit bias in the multiclass setting for the cross-entropy loss, but implicit bias for a more broadly defined class of losses in the multiclass case is left open. In this work, we extend the notion of the exponential tail property to multiclass losses and prove that the property is sufficient for implicit bias to occur in the multiclass setting. Toward this end, we employ the framework of permutation equivariant and relative margin-based (PERM) losses [Wang and Scott, 2024].

1.1 Contributions

Multiclass extension of the exponential tail property (Definition 2.2) It is unclear how the exponential tail property for binary margin losses should be extended to the multiclass setting. By using the PERM framework, we provide a multiclass extension that generalizes the exponential tail property to multiclass (Definition 2.2 in Section 2.3). We further verify that this property holds for some common losses.

Sufficiency of the exponential tail property for implicit bias (Theorem 3.4) We prove that the proposed multiclass exponential tail property is sufficient for implicit bias. More precisely, we show in Theorem 3.4 that for almost all linearly separable multiclass datasets, given a convex, ( β -smooth, strictly decreasing) PERM loss satisfying the exponential tail property in Definition 2.2, gradient descent exhibits directional convergence to the hard-margin multiclass SVM.

1.2 Related Work

Soudry et al. [2018] show that gradient descent, applied to unregularized empirical risk minimization, converges to the hard-margin SVM solution at a slow logarithmic rate, provided the loss satisfies the exponential tail property (defined below). Nacson et al. [2019] improve the convergence rate using a specific step-size schedule. Ji and Telgarsky [2019] extend implicit bias to the setting of quasi-complete separation [Candès and Sur, 2020], where the two classes are linearly separated but with a margin of zero. Many works have also considered gradient-based methods beyond gradient descent. For example, Gunasekar et al. [2018] examine the implicit bias effects of mirror descent [Beck and Teboulle, 2003], steepest descent [Boyd and Vandenberghe, 2004], and adaptive gradient descent [Duchi et al., 2011, Kingma and Ba, 2015]. Cotter et al. [2012], Clarkson et al. [2012], Ji et al. [2021] study first order methods that are designed specifically to approach the hard-margin SVM as quickly as possible.

Results for the multiclass setting are more scarce, and are always specific to cross-entropy. Soudry et al. [2018] establish implicit bias for cross-entropy loss. Lyu and Li [2019] focus on homogeneous predictors and prove convergence of GD on cross-entropy loss to a KKT point of the marginmaximization problem. Lyu and Li [2019] proves convergence of gradient flow to a generalized maxmargin classifier for multiclass classification with cross-entropy loss using homogeneous models. 1 In the special case when the model are linear classifiers, the generalized max-margin classifier reduces to the classical hard-margin SVM. Lyu et al. [2021] consider two-layer neural networks and prove convergence of GD on cross-entropy loss to the max-margin solution under an additional assumption on the data, that both x and its negative counterpart -x must belong to the dataset. Wang et al. [2023] prove that in certain overparameterized regimes, gradient descent on squared loss leads to an equivalent solution to gradient descent on cross-entropy loss.

Beyond work establishing (rate of) convergence to the max-margin classifier, there is also a separate line of work [Shamir, 2021, Schliserman and Koren, 2022, 2023] focusing on the generalization aspect of implicit bias. These works examine the binary classification setting, with the exception of Schliserman and Koren [2022] who consider cross-entropy.

1.3 Notations

Let K ≥ 2 and d ≥ 1 denote the number of classes and feature space dimension, respectively. Let [ K ] := { 1 , 2 , . . . , K } . Vectors are denoted by boldface lowercase letters, e.g., v ∈ R K whose entries are denoted by v j for j ∈ [ K ] . Likewise, matrices are denoted by boldface uppercase letters, e.g., W ∈ R d × K . The columns of W are denoted w 1 , . . . , w K . By 0 n and 1 n we denote the n -dimensional vectors of all 0's and all 1's respectively. The n × n identity matrix is denoted by I n .

By ∥ v ∥ we denote the Euclidean norm of vector v . ∥ A ∥ 2 is the spectral norm of matrix A . Given two vectors w , v ∈ R k , we write w ⪰ v (resp. w ≻ v ) if w j ≥ v j (resp. w j > v j ) for all j ∈ [ k ] ; similarly we write w ⪯ v (resp. w ≺ v ) if w j ≤ v j (resp. w j < v j ) for all j ∈ [ k ] . On the other

hand, if A and B are equally-sized symmetric matrices , then by A ⪰ B (resp. A ⪯ B ) we mean that A -B (resp. B -A ) is positive semi-definite, i.e. A -B ⪰ 0 (resp. B -A ⪰ 0 ).

A bijection from [ k ] to itself is called a permutation on [ k ] . Denote by Sym ( k ) the set of all permutations on [ k ] . For each σ ∈ Sym ( k ) , let S σ denote the permutation matrix corresponding to σ . In other words, if v ∈ R k is a vector, then [ S σ v ] j = v σ ( j ) .

2 Multiclass Loss Functions

In multiclass classification, a classifier is typically represented in terms of a class-score function f = ( f 1 , . . . , f K ) : R d → R K , which maps an input x ∈ R d to a vector v := f ( x ) of class scores. For instance, f may be a feed-forward neural network and v in this context is sometimes referred to as the logits. The label set is [ K ] , and a label is predicted as argmax j f j ( x ) . A K -ary multiclass loss function is a vector-valued function L = ( L 1 , . . . , L K ) : R K → R K where L y ( f ( x )) is the loss incurred for outputting f ( x ) when the ground truth label is y .

In binary classification, a classifier is typically represented using a function g : R d → R . The label set is {-1 , 1 } , and labels are predicted as x ↦→ sign( g ( x )) . A binary margin loss is a function of the form ψ : R → R where ψ ( yg ( x )) is the loss incurred for outputting g ( x ) when the ground truth label is y . Margin losses have been central to the development of the theory of binary classification, and the lack of a multiclass counterpart to binary margin losses may have impaired the development of corresponding theory for multiclass classification. To address this issue, Wang and Scott [2024] introduce PERM losses as a bridge between binary and multiclass classification.

2.1 Permutation equivariant and relative margin-based (PERM) losses

Assume the label set is [ K ] . Define 2 the matrix D := [ -I K -1 1 K -1 ] ∈ R ( K -1) × K . Observe that Dv = ( v K -v 1 , v K -v 2 , . . . , v K -v K -1 ) ⊤ for all v ∈ R K .

Definition 2.1 (PERM loss [Wang and Scott, 2024]) . Let K ≥ 2 be an integer, and L be a K -ary multiclass loss function. We say that L is

permutation equivariant if L ( S σ v ) = S σ L ( v ) for all v ∈ R K and σ ∈ Sym ( K ) ,

relative margin-based if for each y ∈ [ K ] there exists a function ℓ y : R K -1 → R so that L y ( v ) = ℓ y ( Dv ) = ℓ y ( v K -v 1 , v K -v 2 , . . . , v K -v K -1) , for all v ∈ R K . We refer to the vector-valued function ℓ := ( ℓ 1 , . . . , ℓ K ) as the reduced form of L .

PERM if L is both permutation equivariant and relative margin-based. In this case, the function ψ := ℓ K is referred to as the template of L .

Wang and Scott [2024] show that PERM losses are characterized by their template ψ . To show this, they introduce the matrix label code , an encoding of labels as matrices. Thus, for each y ∈ [ K -1] , let Υ y be the ( K -1) × ( K -1) identity matrix, but with the y -th column replaced by all -1 's. For y = K , let Υ y be the identity matrix. Note that when K = 2 , this definition reduces to Υ y = ( -1) y , the standard encoding of labels in the binary setting. Observe that (after permutation) Υ y Dv = ( v y -v 1 , v y -v 2 , . . . , v y -v K ) ⊤ ∈ R K -1 , where the v y -v y = 0 entry is omitted. Please see Wang and Scott [2024, Lemma B.2] for a simple proof.

Theorem 2.1 (Wang and Scott [2024]) . Let L : R K → R K be a PERM loss with template ψ , and let v ∈ R K and y ∈ [ K ] be arbitrary. Then ψ is a symmetric function. Moreover,

Conversely, let ψ : R K -1 → R be a symmetric function. Define a multiclass loss function L = ( L 1 , . . . , L k ) : R K → R K according to Eqn. (1) . Then L is a PERM loss with template ψ .

Theorem 2.1 shows that a PERM loss is characterized by its template ψ . The right hand side of Eqn. (1) is referred to as the relative margin form of the loss, which extends binary margin losses to multiclass. As noted by Wang and Scott [2024], an advantage of the relative margin form is that it

a.

b.

C.

6°0

6°0

0.8

0.8

0.7

0.6

0.5

04

1

0

-2

-4

0

-2

0

2

2

4

decouples the labels from the predicted scores, which facilitates analysis. Our results below support this understanding.

Many losses in the literature are PERM losses, including the cross-entropy loss whose template is ψ ( u ) = log(1 + ∑ K -1 i =1 exp( -u i )) , the multiclass exponential loss [Mukherjee and Schapire, 2013] whose template is ψ ( u ) = ∑ K -1 i =1 exp( -u i ) , and the PairLogLoss [Wang et al., 2022] whose template is = ψ ( u ) = ∑ K -1 i =1 log(1 + exp( -u i )) . See Wang and Scott [2024] for other examples.

2.2 Regularity assumptions on loss functions

Let L be a PERM loss with differentiable template ψ . If

i.e., the gradient of the template is entrywise strictly negative, then we say that the PERM loss L is strictly decreasing . In this case, we write ∇ ψ ≺ 0 , where 0 is the 0-vector. If the template is differentiable, then it is convex if:

If ψ is twice-differentiable, this is equivalent to saying that the Hessian is positive-semidefinite:

Finally, the template is said to be β -smooth if its gradient is β -Lipschitz:

If ψ is twice-differentiable, this is equivalent to saying that the maximum eigenvalue of its Hessian is bounded by β :

where ∥ A ∥ 2 is the spectral norm of matrix A .

2.3 Multiclass analogue of exponential tail property

In the binary setting, the exponential tail property defined in prior work (Soudry et al. [2018], Nacson et al. [2019], Ji et al. [2020]) is assumed to hold for the negative derivative of the loss. Similarly, in the multiclass setting we are interested in bounding the negative gradient of the PERM loss template.

Definition 2.2 (Multiclass exponential tail property) . A multiclass PERM loss with template ψ : R K -1 → R has the exponential tail (ET) property if there exist u + , u -∈ R and positive c > 0 such

that for all i ∈ [ K -1] the following holds:

3 Main Result

Consider a dataset { ( x n , y n ) } N n =1 , with x n ∈ R d and class labels y n ∈ [ K ] := { 1 , . . . , K } . The class score function for class k is f k ( x ) = w T k x . Define X ∈ R d × N to be the matrix whose n th column is x n . Define W ∈ R d × K to be the matrix whose k th column is w k . The learning objective is

From Eqn. 1, if L is a PERM loss, then L y ( v ) = ψ ( Υ y Dv ) , and the learning objective becomes

Up to permuting the entries, Υ y i DW ⊤ x i is equal to the ( K -1) -dimensional vector of relativemargins [ ( w y i -w 1 ) ⊤ x i , ( w y i -w 2 ) ⊤ x i , . . . , ( w y i -w K ) ⊤ x i ] ⊤ , where the 0 -valued entry ( w y i -w y i ) ⊤ x i is omitted. This follows from Wang and Scott [2024, Lemma B.2].

We are now ready to state our assumptions on the loss:

Assumption 3.1. The PERM loss's template ψ is convex, β -smooth, strictly decreasing and nonnegative. 3 4

Assumption 3.2. The PERM loss has exponential tail as defined in Definition 2.2.

To optimize Eqn. (2) we employ gradient descent with fixed learning rate η . Define w := vec ( W ) where vec denotes vectorization by column-stacking (See Definition B.1), and let the gradient descent iterate at time t be w ( t ) . Then:

Define the 'matrix-version' of the trajectory W ( t ) ∈ R d × K such that w ( t ) = vec ( W ( t )) . Throughout this work, we frequently work with the risk as a matrix -input scalar-output function R ( W ) , and as a vector -input scalar-output function R ( w ) .

These two formulations will each be useful in different situations. For instances, adopting the matrix perspective can facilitate calculation of bounds, e.g., in Section 4.2. On the other hand, the vectorized formulation is easier for defining the Hessian of the risk ∇ 2 R ( w ) . See Appendix B for detail.

We focus on linearly separable datasets:

Assumption 3.3. The dataset is linearly separable, i.e. there exists w ∈ R dK such that ∀ n ∈ [ N ] , ∀ k ∈ [ K ] \{ y n } : w ⊤ y n x n ≥ w ⊤ k x n + 1 . Equivalently, there exists W ∈ R d × K such that ∀ n ∈ [ N ] , Υ y n DW ⊤ x n ⪰ 1 .

Finally, let ˆ w be the multiclass hard-margin SVM solution for the linearly separable dataset:

̸

Now we state the main result of the paper:

Theorem 3.4. For any PERM loss satisfying Assumptions 3.1 and 3.2, for all linearly separable datasets such that Assumption 4.1 holds, any sufficiently small learning rate 0 < η < 2 β -1 σ -2 max ( X ) , and any initialization w (0) , the iterates of gradient descent will behave as

where the norm of the residual, ∥ ρ ( t ) ∥ , is bounded. This implies a directional convergence behavior:

In Appendix I, we show experimental results demonstrating implicit bias towards the hard margin SVM when using the PairLogLoss, in line with Theorem 3.4.

4 Proof Sketch

In this section we will overview the proof of the result. Along the way, we prove lemmas that extend to the multiclass setting results from Soudry et al. [2018]. The extensions are facilitated by the PERM framework, in particular the relative margin from of the loss.

We adopt the notation of Soudry et al. [2018] where possible throughout this proof. Recalling the notation and definitions from the paper: let us define the standard basis e k ∈ R K such that ( e k ) i = δ ki (where δ is the Kronecker-delta function), and the d -dimension identity matrix I d . Define A k ∈ R dK × d as the Kronecker product between e k and I d , i.e. A k = e k ⊗ I d . We can then relate the original k th -class predictor w k to the long column-vector w as follows: A ⊤ k w = w k . Next define ˜ x n,k := ( A y n -A k ) x n . Using this notation, the multiclass SVM becomes

̸

For each k ∈ [ K ] , define S k = arg min n ( ˆ w y n -ˆ w k ) ⊤ x n = { n : ( ˆ w y n -ˆ w k ) ⊤ x n = 1 } , i.e., the k th class support vectors. From the KKT optimality conditions for Eqn. (5), we have for some dual variables α n,k > 0 that

Finally, define where ˜ w is a solution to

In Soudry et al. [2018], the existence of ˜ w is proven for the binary case for almost all datasets, and assumed in the multiclass case. Here, we also state the existence of ˜ w as an additional assumption:

Assumption 4.1. Eqn. 8 has a solution, denoted ˜ w .

We pose the problem of proving Assumption 4.1 for almost all datasets as a conjecture in Appendix H, where we also show experimentally that on a large number (100 instances for each choice of d ∈ { 2 , 3 , 4 , 5 , 6 } and K ∈ { 3 , 4 , 5 , 6 } ) of synthetically generated linearly separable datasets, Assumption 4.1 indeed holds.

Note that r ( t ) = ρ ( t ) -˜ w , and ˜ w is independent of t , so bounding r ( t ) is equivalent to bounding ρ ( t ) . Following the same steps as Soudry et al. [2018, Appendix E.3]:

The high-level approach is to bound the two terms of the above expansion for r ( t ) and then use a telescoping argument to bound r ( t ) for all t > 0 . Below we provide the main arguments; for a complete proof of the second term's bound, please refer to Appendix F.

4.1 Bounding the First Term

Using log(1 + x ) ≤ x for all x > 0 , we expand the first term as follows:

Obtaining the second inequality requires proving that

We will spend the rest of this subsection going over the complete proof of this inequality.

First we state the following lemma (derived in Appendix B.2) that gives us a useful expression for the gradient of the risk w.r.t. W :

This expression involves weight matrix W . However the inequality we set out to prove (Eqn. (10)) is in terms of w = vec ( W ) . Throughout our main result proof, these two different forms - weight matrix versus vectorization of that matrix - will each be useful in different situations. Thus, to shuttle back and forth between these forms, the following well-known identity is useful:

Lemma 4.3. For equally sized matrices M and N , we have vec ( M ) ⊤ vec ( N ) = tr ( M ⊤ N ) .

Now we can prove our inequality of interest, i.e., Eqn. (10).

Lemma 4.4. (Multiclass generalization of Soudry et al. [2018, Lemma 1]) For any PERM loss that is β -smooth, strictly decreasing, and non-negative, (Assumption 3.1) and Assumption 3.2, and for almost all linearly separable datasets (Assumption 3.3), we have ˆ w ⊤ ∇R ( w ( t )) < 0 .

Proof. Define matrix ˆ W such that ˆ w = vec ( ˆ W ) . Since w ( t ) = vec ( W ( t )) , Lemma 4.3 implies

To see how the PERM framework allows for a simple generalization of binary results, we will compare our multiclass proof side-by-side with the binary proof discussed in Soudry et al. [2018, Lemma 1]. In the binary case, we have R ( w ) = ∑ N i =1 ψ ( y i w ⊤ x i ) = ⇒ ∇R ( w ) = ∑ N i =1 ψ ′ ( y i w ⊤ x i ) y i x i . Thus ˆ w ⊤ ∇R ( w ) = ∑ N i =1 ψ ′ ( y i w ⊤ x i ) y i ˆ w ⊤ x i . In the multiclass case, the analogous quantity is tr ( ˆ W ⊤ ∇R ( W ( t ))) which can be computed as

In the multiclass proof we used the risk gradient from Lemma 4.2 as well as the cyclic property of the trace operator. Then we dropped the trace because ∇ ψ ( Υ y i DW ( t ) ⊤ x i ) ⊤ Υ y i D ˆ W ⊤ x i is a scalar (since ∇ ψ ( · ) ∈ R K -1 , Υ y i DW ( t ) ⊤ x i ∈ R K -1 ). For illustrative purpose, we place the rest of the proof, in both the binary and multiclass setting, side-by-side:

Binary : ˆ w ⊤ ∇R ( w ( t )) .

Focusing on just the i -th term of this sum:

ψ is assumed to be strictly decreasing, i.e. ψ ′ ( y i w ( t ) ⊤ x i ) < 0 . The dataset is linearly separable, so y i ˆ w ⊤ x i ≥ 1 . Thus we obtain a sum (from i = 1 to N ) of negative terms.

Multiclass : tr ( ˆ W ⊤ ∇R ( W ( t ))) .

Focusing on just the i -th term of this sum:

ψ is assumed to be strictly decreasing, i.e. ∇ ψ ( Υ y i DW ( t ) ⊤ x i ) ≺ 0 . The dataset is linearly separable, so Υ y i D ˆ W ⊤ x i ⪰ 1 . Thus we obtain a sum (from i = 1 to N ) of negative terms.

Thus we see how the PERM framework allows us to essentially mirror the binary proof. In Remark 4.5, we elaborate more on the necessity of the relative margin form here.

Lemma 4.4 directly implies the auxiliary inequality we set out to prove (see Eqn. (10)). Thus we obtain:

Remark 4.5. Let us see what happens to our proof if we just used the general risk form in Eqn. (2) without the PERM framework. First, we need an expression for the gradient of the risk: ∇R ( W ) = ∑ N i =1 x i ∇R y i ( W ⊤ x i ) ⊤ . Proceeding similarly to the binary case, we focus on just the i -th term of tr ( ˆ W ⊤ ∇R ( W ) ) :

From here it is not clear how to proceed. The linear separability condition (Assumption 3.3) is not useful anymore- it does not make a statement about the scores in the vector ˆ W ⊤ x i , but rather their relative margins (produced by the multiplication Υ y i D ˆ W ⊤ x i ).

4.2 Bounding the Second Term

In the previous subsection we established a bound on the first term of Eqn. (9). Here we sketch the main arguments required to bound the second term, i.e. ( r ( t +1) -r ( t )) ⊤ r ( t ) . For more details please refer to Appendix F. We state our final bound below as a lemma:

Lemma 4.6. (Generalization of Soudry et al. [2018, Lemma 20]) Define θ to be the minimum SVM margin across all datapoints and classes, i.e. θ = min k [ min n/ ∈S k ˜ x ⊤ n,k ˆ w ] > 1 . Then

A remark is in order on the difference of the above result to Soudry et al. [2018, Lemma 20]: on a high-level, we are able to generalize the argument of Soudry et al. [2018, Lemma 20] to account for both binary and multiclass classification , as well as general PERM ET losses beyond just CE.

We now proceed with the proof sketch. The first step is to rewrite ( r ( t +1) -r ( t )) ⊤ r ( t ) as

We defer the bound on the first term ˆ w ⊤ r ( t ) ( t -1 -log ( 1 + t -1 )) of Equation (14) to the appendix, and instead focus on the second two terms. Using the cyclic property of the trace, the term in the above final line involving the trace can be further simplified as:

Note that for each i ∈ [ N ] , a summand in Equation (15) is an inner product between two ( K -1) -dimensional vectors, i.e., -∇ ψ ( Υ y i DW ( t ) ⊤ x i ) and Υ y i DR ( t ) ⊤ x i . To proceed, to expand this inner product out as

Remark 4.7. Here, J · K k : R K -1 → R is defined as the coordinate projection such that J Υ y i DW ⊤ x i K k = ˜ x ⊤ i,k w . Note that J · K k implicitly depends on i (the ˜ x i,y i 0-entry is omitted). But we abuse notation for brevity. Please see Appendix D for a more precise definition.

Using Equation (6) and Equation (8) we express the last two terms in Equation (14) as

Finally, to upper bound the above expression, we consider a single tuple ( i, k ) case-by-case, depending on the sign of ˜ x ⊤ i,k r ( t ) . This is the step where the upper and lower bounds in Definition 2.2 come in. Lemma D.2 in the appendix essentially applies Definition 2.2 to the relative margins to yield

for all k ∈ [ K ] \ { y i } . We use Definition 2.2's exponential tail bounds by proving that the relative margins ˜ x ⊤ i,k w ( t ) that appear in Lemma D.2 eventually become positive. This is true due to the following lemma (see Appendix E for the proof, which again mirrors the binary case):

Lemma 4.8. (Multiclass generalization of Soudry et al. [2018, Lemma 1]) Consider any linearly separable dataset, and any PERM loss with template ψ that is convex, β -smooth, strictly decreasing, and non-negative. For all k ∈ { 1 , ..., K } , let w k ( t ) be the gradient descent iterates at iteration t for the k th class. Then ∀ i ∈ { 1 , ..., N } , ∀ j ∈ { 1 , ..., K }\{ y i } : lim t →∞ ( w y i ( t ) -w j ( t )) ⊤ x i →∞ .

This lemma lets us use the exponential tail bounds with any finite u ± . To conclude, we apply the upper (18) and lower bounds (19) to the summation in Equation (17), and reduce the problem to that of Soudry et al. [2018, Appendix E], thereby proving Lemma 4.6. See our Appendix F for details.

4.3 Tying It All Together

We use the logic of Soudry et al. [2018, Appendix A.2] to conclude the analysis. Define

In the latter inequality we used Eqn. (12). Thus, C is bounded because from Soudry et al. [2018, Lemma 10], we know that ∑ ∞ t =0 ∥∇R ( w ( t )) ∥ 2 < ∞ . Here we note that Soudry et al. [2018, Lemma 10] requires the ERM objective R ( w ) to be β ′ -smooth for some positive β ′ . It is easy to show that if the loss is β -smooth, then R ( w ) is βσ 2 max ( X ) -smooth. This explains the learning rate condition η < 2 / ( βσ 2 max ( X ) ) in our theorem. Also, a t -p power series converges for any p > 1 .

Recalling the initial expansion of ∥ r ( t +1) ∥ from Eqn. (9):

Combining the bounds in Eqn. (12) and Lemma 4.6 into Eqn. (9), we find

Therefore, ∥ r ( t ) ∥ is bounded, which proves our main theorem.

5 Limitations

Here we describe some of our work's limitations/possible future research directions. We note that these questions have been analyzed for the binary classification setting, but not for multiclass.

Non-ET losses In our paper we only analyze multiclass implicit bias for losses with the ET property. Another possible line of future work is to analyze the gradient descent dynamics for non-ET losses. Nacson et al. [2019] and Ji et al. [2020] prove that in the binary setting, ET and well-behaved super-polynomial tailed losses ensure convergence to the maximum-margin direction, while other losses may converge to a different direction with poor margin. Is such a characterization possible in the multiclass setting?

Other gradient-based methods This paper only analyzes vanilla gradient descent. Another line of work involves exploring implicit bias effects of other gradient-based methods, such as those characterized in Gunasekar et al. [2018]. Nacson et al. [2022] uses similar proof techniques to prove results for SGD, which is prevalent in practice and often generalizes better than vanilla GD ([Amir et al., 2021]).

Non-asymptotic analysis Our result proves that the gradient descent predictors asymptotically do not overfit. However, in the binary classification case, Shamir [2021] goes one step further and proves that for gradient-based methods, throughout the entire training process (not just asymptotically), both the empirical risk and the generalization error decrease at an essentially optimal rate (or remain optimally constant). Does the same phenomenon occur in the multiclass setting?

6 Conclusion

We use the permutation equivariant and relative margin-based (PERM) loss framework to provide an multiclass extension of the binary ET property. On a high level, while the binary ET bounds the negative derivative of the loss, our multiclass ET bounds each negative partial derivative of the PERM template ψ . We demonstrate our definition's validity for multinomial logistic loss, multiclass exponential loss, and PairLogLoss. We develop new techniques for analyzing multiclass gradient descent, and apply these to generalize binary implicit bias results to the multiclass setting. Our main result is that for almost all linearly separable multiclass datasets and a suitable ET PERM loss, the gradient descent iterates directionally converge towards the hard-margin multiclass SVM solution.

Our proof techniques in this paper demonstrate the power of the PERM framework to facilitate extensions of known binary results to multiclass settings and provide a unified treatment of both binary and multiclass classification. Thus it is possible that the binary results discussed in the Limitations section can also be extended using the PERM loss framework. In the future we would like to consider more complex settings that have been analyzed primarily for the binary case, such as non-separable data (Ji and Telgarsky [2019]) and two-layer neural nets (Lyu et al. [2021]).

Acknowledgments and Disclosure of Funding

CS was supported in part by the National Science Foundation under award 2008074, and by the Department of Defense, Defense Threat Reduction Agency under award HDTRA1-20-2-0002. The research of DS was funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of the Schmidt Career Advancement Chair in AI. YW was supported in part by the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship, a Schmidt Futures program.

References

Idan Amir, Tomer Koren, and Roi Livni. Sgd generalizes better than gd (and regularization doesn't help), 2021.

Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, 2003.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge University Press, 2004.

Emmanuel J. Candès and Pragya Sur. The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. The Annals of Statistics , 48(1):27-42, 2020.

Kenneth L Clarkson, Elad Hazan, and David P Woodruff. Sublinear optimization for machine learning. Journal of the ACM (JACM) , 59(5):1-49, 2012.

Andrew Cotter, Shai Shalev-Shwartz, and Nathan Srebro. The kernelized stochastic batch perceptron. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 739-746, 2012.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7), 2011.

Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning , pages 1832-1841. PMLR, 2018.

Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory , pages 1772-1798. PMLR, 2019.

Ziwei Ji, Miroslav Dudik, Robert E Schapire, and Matus Telgarsky. Gradient descent follows the regularization path for general losses. In Conference on Learning Theory , pages 2109-2136. PMLR, 2020.

Ziwei Ji, Nathan Srebro, and Matus Telgarsky. Fast margin maximization via dual acceleration. In International Conference on Machine Learning , pages 4860-4869. PMLR, 2021.

CG Khatri and C Radhakrishna Rao. Solutions to some functional equations and their applications to characterization of probability distributions. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 167-180, 1968.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , volume 6, 2015.

Shuangzhe Liu. Matrix results on the khatri-rao and tracy-singh products. Linear Algebra and its Applications , 289(1-3):267-277, 1999.

Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890 , 2019.

Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems , 34:12978-12991, 2021.

Jan R Magnus and Heinz Neudecker. Matrix differential calculus with applications in statistics and econometrics . John Wiley & Sons, 2019.

Indraneel Mukherjee and Robert E Schapire. A theory of multiclass boosting. Journal of Machine Learning Research , 2013.

Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3420-3428. PMLR, 2019.

Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate, 2022.

Robert R Phelps. Convex functions, monotone operators and differentiability , volume 1364. Springer, 2009.

Matan Schliserman and Tomer Koren. Stability vs implicit bias of gradient methods on separable data and beyond. In Conference on Learning Theory , pages 3380-3394. PMLR, 2022.

Matan Schliserman and Tomer Koren. Tight risk bounds for gradient descent on separable data. arXiv preprint arXiv:2303.01135 , 2023.

Ohad Shamir. Gradient methods never overfit on separable data. The Journal of Machine Learning Research , 22(1):3847-3866, 2021.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1): 2822-2878, 2018. See arxiv.org/abs/1710.10345v7 for the most up-to-date version.

Gal Vardi. On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208.12591 , 2022.

Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign overfitting in multiclass classification: All roads lead to interpolation, 2023.

Nan Wang, Zhen Qin, Le Yan, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, and Marc Najork. Rank4class: A ranking formulation for multiclass classification, 2022.

Yutong Wang and Clayton Scott. Unified binary and multiclass margin-based classification. Accepted to Journal of Machine Learning Research, arXiv:2311.17778, 2024.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

A Discussion of Lyu and Li [2019]

Lyu and Li [2019] allow the class score functions to be linear classifiers, e.g., w ⊤ k x i , but also nonlinear, e.g., 'cubed' linear classifier ( w ⊤ k x i ) 3 . By shifting the cubing operation to the loss, we can view the implicit regularization result of Lyu and Li [2019] as a result for losses beyond the cross entropy. This resulting loss is rather exotic and we are not aware of it being used in the literature; it is interesting nevertheless. However, the optimization problem would become non-convex, so convergence would not necessarily be to a global minimum:

̸

Moreover, the decision region for the k -th class, i.e., the set of x ∈ R d such that ( w ⊤ k x ) 3 > ( w ⊤ j x ) 3 for all j = k , is an intersection of sets constructed via cubic hypersurfaces.

More precisely, the k -th decision region can be written as

̸

Let us define H j := { x ∈ R d : ( w ⊤ k x ) 3 = ( w ⊤ j x ) 3 } . Note that H j ⊆ R d is the zero set of degree 3 polynomials with variables in x , hence, a cubic hypersurface. Now, the set { x ∈ R d : ( w ⊤ k x ) 3 > ( w ⊤ j x ) 3 } is a subset of the set-theoretic complement of H j in R d . Thus, the decision regions are complicated geometric objects, compared to the classical hard-margin SVM.

B Matrix Calculus

This section of the appendix establishes matrix identities that will be useful for us to calculate the gradient/Hessian of the empirical risk objective R ( w ) .

Vector-input scalar-output function . Suppose f : R n → R is a continuously differentiable function. Let x = [ x 1 , · · · , x n ] ⊤ ∈ R n be a vector of variables for differentiation. Define the (column) vector of partial derivatives w.r.t. x : ∂ x := [ ∂ ∂ x i ] i ∈ [ n ] . The gradient of f , denoted ∇ f , is the function

Suppose that f is twice continuously differentiable. The Hessian of f , denoted ∇ 2 f , is the function

Matrix-input scalar-output function . Let f : R m × n → R be a differentiable function. Let X = [ x ij ] i ∈ [ m ] ,j ∈ [ n ] ∈ R m × n be an arbitrary matrix. Define the matrix of partial derivatives w.r.t. X :

Define the gradient of f , denoted ∇ f , to be the function

We do not define the Hessian of a matrix-input scalar-output function f : R m × n → R . Instead, we will define the Hessian for its vectorization vec ( f ) : R mn → R .

Definition B.1 (Vectorization operator) . Let vec denote the vectorization operator by stacking the columns of a vector. In other words, if A ∈ R m × n is a matrix with columns a 1 , . . . , a n ∈ R m , then

Definition B.2 (Vectorization of a matrix-input function) . Let f : R m × n → R be a matrix-input function, we define vec ( f ) : R mn → R to be the vector-input function such that

In particular, if f is already a vector-input function, then vec ( f ) = f .

See [Magnus and Neudecker, 2019, Ch.5-§15]. Below in Lemma B.4, we give a convenient formula to calculate the Hessian of R ( w ) , the vectorization of R ( W ) .

The following relates the vectorization operator with the Kronecker product:

Lemma B.1. Let A ∈ R p × n , B ∈ R n × m , C ∈ R m × q be matrices. Then

Proof. This is [Magnus and Neudecker, 2019, Theorem 2.2].

B.1 Special case of the chain rule for linear functions

Proposition B.2. Let M ∈ R m × n be a matrix. Let f : R m → R be a continuously differentiable function and define g : R n → R by g ( x ) := f ( Mx ) . Then

Proof. See [Magnus and Neudecker, 2019, Ch.9-§13] for the first identity and [Magnus and Neudecker, 2019, Ch.10-§8] for the second identity.

The next two results will be referred to as the 'gradient formula' and the 'Hessian formula', respectively, for the function g ( X ) := f ( AX ⊤ B ) .

LemmaB.3. Let f : R p × q → R be a matrix-input scalar-output differentiable function with Jacobian denoted ∇ f : R p × q → R p × q . Let A ∈ R p × n , X ∈ R m × n , and B ∈ R m × q . Define a function g : R m × n → R by g ( X ) := f ( AX ⊤ B ) . Then

Lemma B.4. Let f : R p → R be a vector-input scalar-output twice differentiable function. Let A ∈ R p × n , X ∈ R m × n be matrices and b ∈ R m be a (column) vector. Let V be another matrix with the same shape as X . Let x := vec ( X ) and v := vec ( V ) . Define g ( X ) := f ( AX ⊤ b ) and let g = vec ( g ) be the vectorization of g . Then we have the following formula for computing v ⊤ ∇ 2 g ( x ) v :

B.2 Proof of the gradient formula: Lemma 4.2

In the notation of Section 2.8.1 of the Matrix Cookbook, define matrix U ∈ R p × q by U := AX ⊤ B . Note that U is a function of X . Then by Eqn. (137) of the Matrix Cookbook, we have for each ( i, j ) ∈ [ m ] × [ n ]

Note that by definition, we have ∂f ( U ) ∂ U = ∇ f ( U ) . Therefore

Next, write U = [ U kℓ ] k ∈ [ p ] ,ℓ ∈ [ q ] in the 'matrix-comprehension' notation. Recall that U kℓ , i.e., the ( k, ℓ ) -th entry of U , is precisely computed by A [ k, :]( X ⊤ B )[: , ℓ ] = A [ k, :] X ⊤ B [: , ℓ ] . For each k, ℓ ∈ [ p ] × [ q ] , we have

where ' [ k, :] ' and ' [: , ℓ ] ' denote taking the k -th row vector and ℓ -th column vector, respectively. Now, by Eqn. (71) of the Matrix Cookbook, we have the following expression of the matrix-partial derivative as an outer product

From this, it follows that computing the entry-wise partial derivative at X ij is simply obtained by indexing at ( i, j ) , i.e., ∂U kℓ ∂X ij = B [ i, ℓ ] A [ k, j ] = A [ k, j ] B [ i, ℓ ] (we emphasize that this is just a product of two scalars). Thus, ∂ U ∂X ij = A [: , j ] B [ i, :] . Consequently,

In other words, ∂ ∂ X f ( AX ⊤ B ) = B ∇ f ( U ) ⊤ A .

For our purposes, we replace f with ψ , A with Υ y i D , X with W , and B with x i . Thus we obtain

as desired.

B.3 Proof of Hessian formula: Lemma B.4

Our goal is to calculate the Hessian of vec ( g ) . First, we note that by definition

Note that the last equality is simply vec ( f )( vec ( AX ⊤ b )) = f ( AX ⊤ b ) , but we work in the more general case of a matrix B right now. We will need to simplify vec ( AX ⊤ b ) . It is more convenient during the first phase of the proof viewing b as a m × 1 matrix and denote it using uppercase letter B . First, applying Lemma B.1 to vec ( AX ⊤ B ) , we get

However, vec ( X ⊤ ) = vec ( X ) in general. However, these two expressions are related using the commutation matrix :

̸

Definition B.3 (Commutation matrix) . Define K m,n to be the permutation matrix in R mn × mn such that K m,n vec ( A ) = vec ( A ⊤ ) for all matrices A ∈ R m × n .

□

See [Magnus and Neudecker, 2019, Ch.3-§7]. Below, we drop the subscripts in Definition B.3 and simply write K := K m,n . Now, we have

Thus

By Proposition B.2, we have

From this, we see that (recall that v = vec ( V ) )

Now, since B = b is just a vector, we have

and

Putting it all together, we get the desired equality.

C PERM Losses That Satisfy Assumptions 3.1 and 3.2

C.1 Cross-Entropy

By Wang and Scott [2024, Example 1], the cross-entropy loss L y ( v ) = -log ( exp( v y ) ∑ K k =1 exp( v k ) ) has template ψ ( u ) = log ( 1 + ∑ K -1 k =1 exp( -u k ) ) . We calculate the partial derivatives:

C.1.1 Convexity

Let us analyze the entries of the Hessian of the template, i.e. ∇ 2 ψ ( u ) . Let [ A ] l,m denote the element of A at the l -th row and m -th column. We get for all i, j ∈ [ K -1] where j = i :

̸

From the definition of C i , this implies that:

̸

Thus, the Hessian is a symmetric diagonally dominant matrix, and hence is positive semi-definite.

̸

□

C.1.2 β -smoothness

For any diagonally dominant matrix B , let | B | be the matrix obtained by taking the absolute value of each element of B , that is:

Additionally, let diag ( · ) : R p → R p × p (for any p ∈ N ) be the function that maps a vector to a diagonal matrix in the obvious way.

Then we have the following lemma:

Lemma C.1. Let B ′ := diag ( | B | 1 ) where 1 is the appropriately-sized vector of all-1's. Then B ⪯ B ′ .

Proof. This can be proven simply by observing that B ′ -B is symmetric and diagonally dominant (and thus positive semi-definite).

Lemma C.1 can be directly applied to analyze the Hessian (and eventually bound its maximum eigenvalue). Define H ′ = diag (∣ ∣ ∇ 2 ψ ( u ) ∣ ∣ 1 ) . In other words, from Eqn. (25):

Thus, by directly applying Lemma C.1, we obtain

So now since H ′ is defined to be a diagonal matrix, all that's left to do is bound the diagonal entries by a positive constant. First note that from the definition of C i , it follows that 1 + ∑ K -1 k =1 e -u k = C i +1+ e -u i . Combining this with Eqn. (26), we get:

We can find a global minimum of the denominator of the above expression and thus arrive at an upper bound for the expression. Differentiating with respect to u i and setting to 0 yields a single critical point at u i = -log ( C i +1) , which produces a value of 4 ( C i +1) when substituted in the denominator (this is a global minimum of the denominator expression). Thus, we get

In the binary i.e. K = 2 case, C i = 0 , so our bound is exactly 1 / 4 . However, in the multiclass case (i.e. K > 2 ), C i can be arbitrarily large. Setting C i = ∞ yields a final upper bound of 1 / 2 .

So our final bound can be summarized as follows:

Thus, β = 1 / 4 for binary cross-entropy (logistic loss), but β = 1 / 2 for K-class cross-entropy.

C.1.3 Exponential Tail

We claim that for the cross-entropy Definition 2.2 holds with u ± = 0 and c = a = 1 . We are interested in analyzing the (negative) gradient of the template. From Eqn. 24:

This proves that the cross-entropy loss satisfies Definition 2.2 with u ± = 0 and c = 1 .

C.2 Multiclass Exponential Loss [Mukherjee and Schapire, 2013]

̸

The multiclass exponential loss L : R K → R can be written as L y ( v ) = ∑ k ∈ [ K ]: k = y exp( -( v y -v k )) . Thus, the template function ψ : R K -1 → R can be expressed as ψ ( u ) = ∑ i ∈ [ K -1] e -u i . The partial derivatives of the template are then simply:

The Hessian is a diagonal matrix with all diagonal entries positive. Hence it is positive definite.

C.2.2 β -'smoothness'

Recall the identity derived in Lemma B.4:

We are interested in the special case where X ← W is a linear classifier (represented as a matrix) and x ← vec ( W ) = w is its vectorization as in Section 3. Moreover, g ( X ) represents the risk R ( W ) , viewed as a matrix-input scalar-output function (defined in Appendix B), while g ( x ) represents the vectorized risk R ( w ) , viewed as a vector-input scalar-output function (defined in Appendix B). We will use the formula in Lemma B.4 to calculate v ⊤ ∇ 2 R ( w ) v , where we substitute in

where ( x i , y i ) is a training sample and ψ is the template of a PERM loss. Since ∇ 2 is linear (i.e., distributive over additions), we have by Lemma B.4 that

Note that ∥ v ∥ = ∥ vec ( V ) ∥ = ∥ V ∥ F by the definitions of the Frobenius norm and vectorization. Thus,

We note that we never defined the Hessian of a matrix -input function, i.e., we do not work with ∇ 2 R ( W ) . Combining the two previous identities, we have proven

Corollary C.2. Let R ( W ) be the risk viewed as a matrix-input scalar-output function defined in Equation (3) . Let R ( w ) be the vectorization of R ( W ) . Then we have

We have

Let vdiag ( · ) be the 'inverse' of diag ( · ) , i.e., vdiag ( · ) takes a diagonal matrix and returns the vector of the diagonal elements.

C.2.1 Convexity

We have

The max eigenvalue of the Hessian of R ( w ) , i.e., Equation (29), is computed below:

In equality 2 we took trace of a scalar (the expression in equality 1 is a scalar, so taking the trace of it will not change the value) and used the cyclic property. For equality 3: as per Equation (28), ∇ 2 ψ is a diagonal matrix. Finally, in the last inequality, we bound each element of the diagonal vector (i.e. exp( -u i ) for all i ∈ [ K -1] ). Dropping the R ( W ) and ' max V ∈ R d × K : ∥ V ∥ F =1 ' from the front:

Note that we applied Cauchy-Schwarz to the inner product space R K × K with inner product ⟨ A , B ⟩ := tr ( A ⊤ B ) . Now, continuing with the calculation:

Therefore we have proven that ∥∇ 2 R ( w ) ∥ 2 ≤ B 2 R ( w ) , where

Now we will also analyze the Euclidean norm of the gradient, for reasons that will become clear later.

Gradient descent is a special case of steepest descent with the Euclidean norm [Boyd and Vandenberghe, 2004]. Thus, we can apply Gunasekar et al. [2018, Lemmas 11 & 12] to see that ∇R ( w ) → 0 even for multiclass exponential loss. Elaborating on this: these lemmas from Gunasekar et al. [2018] assume a convex risk objective (which we have in the case of multiclass exponential loss). Additionally, they assume that ∥∇R ( w ) ∥ ≤ B R ( w ) and ∥ ∥ ∇ 2 R ( w ) ∥ ∥ 2 ≤ B 2 R ( w ) . In the above section we prove these exact results with B as defined in Eqn. 31. Finally, Lemma C.3 below proves that ∥ Υ k D ∥ F = √ 2 K -2 for all k ∈ [ K -1] .

In conclusion, by Gunasekar et al. [2018, Lemma 11], if our learning rate η < 1 B 2 R ( w (0)) , we can use Soudry et al. [2018, Lemma 10].

Finally, we calculate ∥ Υ k D ∥ F which was used in several places above:

Proof. First, if k = K , then Υ K is the identity matrix. In this case, we have Υ k D = D = [ -I K -1 1 K -1 ] is the negative ( K -1) -by-( K -1) identity matrix concatenated with the all-ones vector. Thus,

̸

If k = K , then and so

Now, we recall from the definition of Υ k (Definition 2.4 of Wang and Scott [2024]) that Υ k is obtained by replacing the k -th column of the identity matrix by the 'all-negative-ones' vector. Thus

This proves Lemma C.3, as desired.

C.2.3 Exponential Tail

From Eqn. (27), the negative partial derivative of the template is clearly always positive:

From the above, it is clear that the upper and lower bounds in Definition 2.2 hold when u ± = 0 and c = 1 .

C.3 PairLogLoss [Wang et al., 2022]

Recall that the template of the PairLogLoss is ψ ( u ) = ∑ K -1 k =1 log (1 + exp ( -u k )) . By elementary calculus, we see that

C.3.1 Convexity

We have

The Hessian is a diagonal matrix with all diagonal entries positive. Hence it is positive definite.

C.3.2 β -smoothness

Notice that the partial derivative of the template is exactly the same expression as the derivative of the logistic loss (i.e. binary cross-entropy). Thus, the exact same proof as logistic loss can be used to prove β -smoothness for the PairLogLoss as well. Thus, from Appendix C.1.2, β = 1 / 4 (logistic loss is simply the K = 2 case for cross-entropy).

C.3.3 Exponential Tail

This gives us the desired upper tail. As for the lower tail:

Thus, PairLogLoss satisfies Definition 2.2 with u ± = 0 and c = a = 1 .

D Pseudo-index

Note that Υ y i DR ( t ) ⊤ x i produces a ( K -1) -dimensional vector with entries of the form of ( r y i ( t ) -r k ( t )) ⊤ x i , ∀ k ∈ [ K ] \{ y i } . For k ∈ [ K ] \{ y i } , let us represent the corresponding entry of the vector as J Υ y i DR ( t ) ⊤ x i K k . Note that this indexing is not the same as the k th entry of the vectors, since the y i th entry ( r y i ( t ) -r y i ( t )) ⊤ x i is not present in the vector. Similarly, let us define J -∇ ψ ( Υ y i DW ( t ) ⊤ x i ) K k to be the corresponding entry of -∇ ψ .

This section makes this indexing trick rigorous.

Lemma D.1. Let W ∈ R d × K be arbitrary and w := vec ( W ) be its vectorization. Let ( x i , y i ) be a training sample. Then there exists a bijection that depends only on y i that maps the entries of

to the elements of the set of ' y i -versusk ' relative margins, i.e., { ˜ x ⊤ i,k w ∈ R : k ∈ [ K ] \ { y i }} .

The following definition makes the bijection from Lemma D.1 concrete.

Definition D.1 (Pseudo-index) . In the situation of Lemma D.1, define J · K i,k : R K -1 → R to be the coordinate projection such that J Υ y i DWx i K i,k = ˜ x ⊤ i,k w . In other words, J · K i,k selects the y i -versusk relative margin. When the sample index i is clear from context, we drop i from the subscript and simply write J · K k .

The pseudo-index is useful for working with the exponential tail bounds:

Lemma D.2. In the situation of Lemma D.1, consider ∇ ψ ( Υ y i DW ⊤ x i ) which is the ( K -1) -dimensional vector of partial derivatives of the template evaluated at Υ y i DW ⊤ x i . If ψ satisfies Definition 2.2, then and

for all k ∈ [ K ] \ { y i } .

D.1 Proofs of Lemma D.1 and Lemma D.2

In both lemmas, we work with a fixed sample, i.e., the index i does not change. As such, we simply drop the index and write y ← y i , x ← x i , ˜ x k ← ˜ x i,k , and J · K k ← J · K i,k .

Below, we fix k ∈ [ K -1] throughout the proof. Let v := w ⊤ x = [ v 1 , . . . , v K ] ⊤ . Note that

̸

We prove Lemma D.1 by considering the case y = K and y = K separately. First, let us consider the case when y = K . Then Υ y is the identity matrix and so

Thus, we've proven Lemma D.1 when y = K . In this case, J · K k simply picks out the k -th entry of the input ( K -1) -dimensional) vector. In other words,

By definition, we note that the k -th row of Υ y is

̸

Thus, when y = K

̸

̸

Thus, we've proven Lemma D.1 when y = K as well. In this case, J · K k picks out the k -th element of the input ( K -1) -dimensional) vector when k = y . Otherwise when k = y , we have that J · K k picks out the y -th element. More explicitly,

̸

̸

̸

̸

Next, we prove Lemma D.2 by considering the case y = K and y = K separately. First, assume that we are in the y = K case. From Equation (33), we get that

Now, we have that for u = [ u 1 , . . . , u K -1 ] ⊤ ∈ R K -1 , the upper and lower exponential tail bounds are

̸

̸

Letting u := Υ y i D ⊤ W ⊤ x i , using Lemma D.1 and Equation (33), we immediately prove Lemma D.2 in the case when y = K . This is because we have

and

When y = K , a similar argument proves Lemma D.2 using Equation (33) for the pseudo-index J · K k .

E Proof of Lemma 4.8

Re-stating the lemma:

Lemma 4.8. (Multiclass generalization of Soudry et al. [2018, Lemma 1]) Consider any linearly separable dataset, and any PERM loss with template ψ that is convex, β -smooth, strictly decreasing, and non-negative. For all k ∈ { 1 , ..., K } , let w k ( t ) be the gradient descent iterates at iteration t for the k th class. Then ∀ i ∈ { 1 , ..., N } , ∀ j ∈ { 1 , ..., K }\{ y i } : lim t →∞ ( w y i ( t ) -w j ( t )) ⊤ x i →∞ .

Proof. We know that lim t →∞ ∇R ( w ( t )) = 0 by Soudry et al. [2018, Lemma 10].

This implies that ˆ w ⊤ ∇R ( w ( t )) → 0 . Following the same steps as in the proof of Lemma 4.4, this is equivalent to saying:

However, for linearly separable data we know that Υ y i D ˆ W ⊤ x i ⪰ 1 (since ˆ W here is the hardmargin SVM solution). Thus for the above limit to be true, the limit

must hold. By Proposition G.1, we have

where ∞ is the 'vector' whose entries are all equal to infinity. This is equivalent to

Note that in the binary case, the above 'convergence-to-infinity' condition is for a scalar quanity, where the assumption that the loss be strictly decreasing and non-negative suffices. In the multiclass setting, we must ensure that all entries of the vector Υ y i DW ( t ) ⊤ x i converges to infinity. This is a nontrivial result and is addressed by our Proposition G.1.

F Proof of Lemma 4.6

Let us first re-state the lemma we want to prove.

Lemma 4.6 . (Generalization of Soudry et al. [2018, Lemma 20]) Define θ to be the minimum SVM margin across all data points and classes, i.e., θ = min k [ min n/ ∈S k ˜ x ⊤ n,k ˆ w ] > 1 . Then:

Proof. Proceeding the same way as Soudry et al. [2018], we have

The last equality is a new step required for our multiclass generalization, in which we used Lemma 4.2 and introduced the matrices W ( t ) and R ( t ) , where vec ( W ( t )) = w ( t ) and vec ( R ( t )) = r ( t ) . Let us focus just on the second term of this expansion.

In step (1) we used the fact that for any square matrix M , tr ( M ) = tr ( M ⊤ ) . In step (2) we used the cyclic property of the trace.

Similar to in the proof of Lemma 4.4, the trace's cyclic property has enabled us to convert a matrixproduct into a simple dot product. Since dot products are scalars, we can now drop the trace and rewrite our expression in Eqn. (36) as a dot product:

Using this form, we can rewrite Eqn. (35):

The first term ˆ w ⊤ r ( t ) ( t -1 -log ( t -1 )) is bounded in [Soudry et al., 2018, Eqn. (139)]. We will focus on the second and third terms. Recall by Equation (6) and Equation (8) that

Thus, the third term on the RHS of Equation (37) can be written as

Therefore, the last two terms on the RHS of Equation (37) can be written as

In (1) we used Lemma D.1, which implies that ˜ x ⊤ i,k r ( t ) = J Υ y i DR ( t ) ⊤ x i K k . For (2), from the exponential tail upper/lower bound and Lemma D.2, we have that

We note that Eqn. (38) above is identical to the right hand side of inequality (1) in [Soudry et al., 2018, Eqn. (141)]. Thus, the remainder of the analysis proceeds identically as in Soudry et al. [2018, Lemma 20].

G A structural result on symmetric and convex functions

Proposition G.1. Let ψ : R K -1 → R be the template of a PERM loss that satisfies our Theorem 3.4. Let u t ∈ R K -1 be any sequence, where t = 1 , 2 , . . . , such that

is the zero vector. Then lim t →∞ u t j = ∞ for every j ∈ [ K -1] .

We prove Proposition G.1 by first proving a structural result (Theorem G.2) concerning symmetric and convex function f : R n → R . The proof of Proposition G.1 will be presented in Appendix G.2 as an application of the structural result, where we take f = ψ , the template of a PERM loss, and n = K -1 , number of classes minus one.

Given a vector x ∈ R n and a real number C ∈ R , define x ∨ C ∈ R n to be the vector such that

In other words, x ∨ C 'boosts' entries of x up to C if those entries are smaller than C . Entries of x larger than C are kept as-is.

Define min( x ) = min j ∈ [ n ] x j and argmin( x ) := { i ∈ [ n ] : x i = min( x ) } . We note the following easy-to-prove properties of the ' ∨ ' operation:

min( x ∨ C ) ≥ C with equality if min( x ) ≤ C ,

argmin( x ∨ C ) ⊇ argmin( x ) .

Theorem G.2. Suppose that f : R n → R is a symmetric, convex, and differentiable function. Then for any real number C ∈ R and any x ∈ R n , we have

Before proceeding with the proof (which is in Appendix G.1), we first introduce some necessary preliminary notations and facts. Given a vector x ∈ R n , we define

to be the set of values consistings of the entries of x . For example, if x is the all-ones vector, then val ( x ) = { 1 } . Given v ∈ val ( x ) , we let idx ( v, x ) = { i ∈ x : x i = v } be the set of indices that attains the value v .

Fact 1 : For a convex and differentiable function f : R n → R , we have that

This is a simple and well-known consequence of convexity. See this stackexchange answer for a short proof. When n = 1 , Ineq. (39) is the fact that a convex differentiable function has nondecreasing derivative. Ineq. (39) is also a consequence of [Phelps, 2009, Theorem 3.24].

Fact 2 : For a symmetric and differentiable function f : R n → R , we have that

This fact follows from the chain rule and the definition of a symmetric function. To be precise, let T : R n → R n be the permutation matrix that switches the i and j -th coordinate. Then f ( x ) = f ( Tx ) and moreover ∂f ∂x i ( x ) = ∂f ∂x i ( Tx ) = [ T ∇ f ( Tx )] i = [ ∇ f ( Tx )] j = [ ∇ f ( x )] j = ∂f ∂x j ( x ) .

G.1 Proof of Theorem G.2

Now, to prove the above theorem, we will use induction on ' m ' in the following lemma, which is simply a 'stratification' of Theorem G.2 into cases indexed by the 'parameter' m :

Lemma G.3. Suppose that f : R n → R is a convex, symmetric, and differentiable function. Let m ∈ { 0 , 1 , . . . , n } . Then for any real number C ∈ R and any x ∈ R n with the property that |{ v ∈ val ( x ) : v < C }| = m , we have

Note that if we have proved Lemma G.3 for each m ∈ { 0 , 1 , . . . , n } , then Theorem G.2 holds.

The base step : we prove Lemma G.3 when m = 0 and m = 1 . Strictly speaking, the proof-byinduction technique typically only involve only the base case, which would be the m = 0 case in this instance. But below, we will see that in the induction step, the m = 1 case is helpful.

Note that the m = 0 case holds vacuously, since x ∨ C = x . Below, we focus on the m = 1 case, where there exists a unique v ∈ val ( x ) such that v ≤ C . Let i ∈ argmin( x ) . Note that we have idx ( v, x ) = argmin( x ) . Using Equation (39) (Fact 1), we have that

Definition G.1. Given any set S ⊆ [ n ] , we let χ S ∈ R n denote the characteristic vector on S : χ S is the vector whose j th entry is = 1 if j ∈ S and = 0 otherwise.

By construction, we have

The ' ∂f ∂x i ( · ) ' notation for partial derivatives is a bit cumbersome. Instead, we will write ' [ ∇ f ( · )] i ' from now on. By Equation (40) (Fact 2), we have

and likewise

Thus, by Equation (40), we have

Now, since C > v and | argmin( x ) | > 0 , we must have that [ ∇ f ( x ∨ C )] i -[ ∇ f ( x )] i ≥ 0 , as desired. This proves the base step.

Induction step : Suppose Lemma G.3 holds for every integer m where 0 ≤ m < n , we must show that Lemma G.3 also holds for m + 1 . To this end, let x ∈ R n and C ∈ R be such that |{ v ∈ val ( x ) : v < C }| = m +1 . Let v 1 , . . . , v m +1 ∈ R be all the elements of { v ∈ val ( x ) | v < C } enumerated in increasing order, i.e., v 1 < · · · < v m +1 .

Note by construction, we have that { v ∈ val ( x ) | v < v m +1 } = { v 1 , . . . , v m } and so we immediately get that |{ v ∈ val ( x ) | v < v m +1 }| = m . By the m -th case of Lemma G.3 (i.e., the induction hypothesis) using v m +1 as C , we get

Below fix some i ∈ argmin( x ) arbitrarily. Let x ′ := x ∨ v m +1 . We note that by construction, all the entries of x ′ that are less than C are set to equal to v m +1 . In other words,

is a singleton set. Thus, by the m = 1 case of Lemma G.3 applied to x ′ , we get that

Since argmin( x ′ ) ⊇ argmin( x ) , we have that i ∈ argmin( x ′ ) as well (recall that i was chosen earlier from argmin( x ) arbitrarily). Thus the above inequality implies in particular that

Combined with Equation (41), we get

Finally, we note that x ′ ∨ C = ( x ∨ v m +1 ) ∨ C = x ∨ C . Thus, the above implies

since the choice of i ∈ argmin( x ) was arbitrary.

G.2 Application to our setting

̸

The condition lim t →∞ u t j = ∞ by definition means that for every real number M ∈ R , there exists T such that for all t ≥ T we have u t j > M . Thus, suppose that there exists j ∈ [ K -1] such that lim t →∞ u t j = ∞ , then there exists a real number M ∈ R such that for all T = 1 , 2 , . . . there exists some t ≥ T such that u t j ≤ M . Passing to a subsequence, we assume that u t j ≤ M (and so min( u t ) ≤ M ) for all t = 1 , 2 , . . . . Note that lim t →∞ ∇ ψ ( u t ) = 0 continues to hold.

Below, whenever we say 'for all/every t ', we mean 'for all/every t = 1 , 2 , . . . '.

Onto the proof. First recall the lower bound portion of the exponential tail property:

For all u ∈ R K -1 such that min( u ) > u -, we have

Let C := max { 2 | u -| , M, -log( 1 2( K -1) ) } and v t := u t ∨ C for all t . This choice of C (and v t ) has the following consequences:

The fact that C ≥ -log( 1 2( K -1) ) implies ( 1 -∑ K -1 j =1 exp( -v j ) ) ≥ 1 2 .

If v ∈ R K -1 is such that min( v ) ≥ C , then we have by Equation (42) that

min( u t ) ≤ C . This is true since min( u t ) ≤ M .

Choose i t ∈ argmin( u t ) for every t . Then v t i t = min( v t ) = C for every t . This is simply a consequence of the fact that argmin( u t ) ⊆ argmin( v t ) .

We have lim t →∞ [ ∇ ψ ( u t )] i t = 0 . This follows from the assumption that lim t →∞ ∇ ψ ( u t ) = 0 .

By Theorem G.2, we have for every t that

By plugging in v t for u in Equation (43) above and the fact that v t i t = C , we have

Since -1 2 c exp( -C ) is a constant that doesn't depend on t , it is impossible for lim t →∞ [ ∇ ψ ( v t )] i t = 0 to hold. This proves Proposition G.1.

H On the existence of ˜ w

The goal of this section is to explain the challenge and the current gap in the proof of the existence of ˜ w that satisfies the condition in Equation (8) for almost all linearly separable datasets. To this end, recall Equation (4), the hard-margin SVM formulated as a constrained optimization:

̸

Moreover, recall that S k , the set of support vectors for each k ∈ [ K ] , is defined by

The Lagrangian of the objective in Equation (44) is

̸

where α n,r are the dual variables. Let δ i,j denote the Kronecker delta, i.e., δ i,j = 1 if i = j and δ i,j = 0 otherwise. Taking the gradient of L ( w , α ) with respect to w k , we get

̸

̸

So the KKT conditions satisfied by a stationary point ˆ w (hence globally optimal for Equation (44)) are

̸

where Eqn. (47) (the second line) above is the complementary slackness condition.

The goal of this section is to prove the following result regarding the existence of ˜ w that satisfies the condition in Equation (8), which we restate below:

Conjecture H.1. For almost all linearly separable multiclass datasets, Assumption 4.1 holds, i.e., Eqn. (48) has a solution ˜ w .

Below, we use the word 'generically' to mean 'for linearly separable datasets outside of a set of Lebesgue measure zero'. In order for (48) to have a solution in ˜ w generically, two conditions need to hold (generically).

Condition 1. α n,k > 0 for all k and n such that n ∈ S k := { n : ( ˆ w y n -ˆ w k ) ⊤ x n = 1 } .

Condition 1 is already nontrivial and a gap in proving the Conjecture, as we will see below. For the sake of explaining Condition 2 below, let us assume Condition 1 holds. Then we can rewrite (48) as

Define the vector m n,k obtained by taking the difference between the k -th and y n -th elementary basis vector in R K , i.e.,

Then we can further rewrite (49) as

It is more convenient to pool all the class-specific support vectors S k into a single set: S ≜ { ( n, k ) : ( ˆ w y n -ˆ w k ) ⊤ x n = 1 } . For readability, we linearly order the tuples in S , i.e., we assign to each ( n, k ) ∈ S a unique index i ∈ { 1 , ..., |S|} . In other words, we define n (1) , . . . , n ( |S| ) and k (1) , . . . , k ( |S| ) such that

To reduce notational clutter in the subscript, define x i ≜ x n ( i ) and m i ≜ m n ( i ) ,k ( i ) . Finally, define

with ◦ denoting the Khatri-Rao product, which is, by definition, the matrix obtained by taking the Kronecker product of corresponding columns [Khatri and Rao, 1968]. Note that the Khatri-Rao product is only defined for two matrices that have the same number of columns. See Liu [1999] for a reference. We now state

Condition 2. rank( G ) = |S| generically.

Note that given Condition 2, Eqn. (50) has a solution in ˜ w , while Condition 1 is necessary for the logarithm in (50) to be valid in the first place.

̸

The challenge in proving Condition 2 in the multiclass case is that the column vectors of X may have repeats, i.e., it is possible for n ( i ) = n ( i ′ ) when i = i ′ . It is easy to generate synthetic linearly

separable multiclass datasets satisfying this condition. Nonetheless, we observe that even in such a case, the matrix G has rank |S| , i.e., Condition 2 holds. We verify this experimentally in the Python notebook checking_conjecture_in_Appendix_H.ipynb available at

In the binary case, linear classifiers are parametrized simply as a single vector, rather than the more cumbersome one-vector-per-class parametrization. Under the one-vector parametrization, the M matrix becomes a 1 -by|S| matrix consistings of only ± 1 's, and G reduces to X . Moreover X has no repeats. Thus, Condition 2 holds trivially. In both the multiclass and binary settings, given Condition 2, the proof for Condition 1 can proceed exactly as in Lemma 12 from Soudry et al. [2018] where their X S is replaced by our G .

I Additional Experiments

We provide additional experimental support for our main theoretical result for the PairLogLoss [Wang et al., 2022]. Code for recreating the figures can be found at https://github.com/YutongWangML/neurips2024-multiclass-IR-figures

The code can be ran on Google Colab with a CPU runtime in under one hour.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract clearly introduces the problem considered (implicit bias), identifies a gap in research (few multiclass results, which themselves are only for cross-entropy), and states our contributions (new ET property and implicit bias theorem for new losses). For the sake of brevity we do not state additional assumptions on the loss apart from ET (which we state later in the main text, i.e. smoothness, strictly decreasing, non-negative), because the ET property is a novel contribution and deserves to appear in the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a separate section where we talk about our work's limitations. We highlight 2 natural questions one can ask: non-ET loss characterization, and non-asymptotic analysis (answering whether overfitting occurs after some finite number of (S)GD timesteps).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are stated clearly in bold at the beginning of section 3, and also re-iterated multiple times throughout the paper, The assumption on the learning rate being sufficiently small is mentioned in the theorem statement. Partial proofs are provided in the main text because they highlight salient features of our techniques (namely, simple generalization of binary proof techniques to multiclass). Complete proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We use a simple synthetic setup which can be reproduced easily with Google Colab.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In Appendices H and I, we include a link to our GitHub repo which contains the complete code to reproduce the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details can be found in the GitHub repository.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experiments are used to illustrate the main theoretical result, which is of mathematical nature. All experiments support the convergence behavior that we analyzed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, the experiments can be run with a Google Colab CPU runtime as mentioned in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of multiclass SVM in the realizable (i.e., separable) case.\nThey show that under several assumptions on the loss function, the gradient descent converges in direction toward the (unique) solution of the hard-margin problem.\nFor that, they used the notion of Permutation Equivariant and Relative Margin-based losses (introduced elsewhere) in order to extend the notion of exponential tail (from binary SVM) to multiclass SVM.\nThey also show that three loss functions, including cross-entropy, satisfy the assumptions of the theorem.\n\nSTRENGTHS:\nThis is a strong technical paper (even though I would like to ask several questions to the authors to be convinced of the soundness of the proof; see \"Questions\" section).\n\nWEAKNESSES:\nThe main weakness of the paper in my opinion is the difficulty to understand several technical steps of the proof due to the lack of details/explanations.\nThis is a general comment but I have several specific questions in the \"Questions\" section about steps that I did not understand and made me question the overall soundness of the proof.\nI would be happy to have answers from the authors about these questions.\n\nOther comments:\n- Line 193: Lemma 12 in the other paper does not directly give the result, one needs to browse to Eq 18 to complete the argument.\n- Eq. 20: $\\eta$ missing somewhere.\n\nQUESTIONS:\n- Equation after line 200, first line: the equality seems wrong to me because I would expect a factor $\\log(1+t^{-1})$ multiplying $\\lVert\\hat{w}\\rVert$, but it is not there.\nFurthermore, I do not understand either how we get the inequality simply by showing that the third term is negative, since there is an extra $t^{-2}$ factor multiplying $\\lVert\\hat{w}\\rVert$.\n- Line 262: Why is $u_{\\pm}=0$, while in Def. 2.2, it can be anything?\n- Lines 609 to 611: I understand from the argument that *at least one* component of $u$ must converge to $\\infty$, because indeed $\\nabla\\phi(u)\\prec0$ for all finite $u$, but why do *all* the components of $u$ have to converge to $\\infty$?\n\nLIMITATIONS:\nThe authors acknowledge correctly the limitations of their work.\nHowever, a small discussion somewhere in the paper of how restrictive the assumptions on the loss function are would be welcome.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages the PERM (Permutation Equivariant and Relative Margin-based losses) framework proposed in [Wang and Scott, 2024], and extends the implicit bias result of binary classification to multiclass classification.\n\nSpecifically, the authors extend the exponential tail property to multiclass setting, and proved that for almost all linearly separable datasets with suitable PERM loss, with sufficiently small step size, the gradient descent iterates directionally converge towards the hard-margin multiclass SVM solution.\n\nSTRENGTHS:\nThis paper is strongly motivated, well structured and pleasant to read. It studies the problem of implicit bias for multiclass classification with losses not limited to cross-entropy, bridging the binary-multiclass gap.\n\nThe PERM framework provides an elegant tool for analyzing multiclass classification problems, and in my opinion, this paper is technically solid. I have not gone through the proofs of lemmas in the appendix, but the analysis in the main paper looks correct to me.\n\nWEAKNESSES:\nAs the authors have discussed in the Limitations section, it would also be interesting to consider beyond the asymptomatic setting and end tail property.\n\nSome typos:\n- It has been discussed in 4.3 that $\\mathcal{R}$ is $\\beta\\sigma_{\\textrm{max}}^2(X)$-smooth, and the learning rate should be $\\eta<\\frac{2}{\\beta\\sigma_{\\textrm{max}}^2(X)}$, but the upper bound is stated as $2\\beta^{-1}\\sigma_{\\textrm{max}}^2(X)$ in the statement of Theorem 3.4. \n- In page 3 line 85, the meaning of $[\\textbf{v}]\\sigma(j)$ doesn’t seem clear to me, is it $[\\textbf{v}]_{\\sigma(j)}$?\n- Line 116: from direct calculation ${{\\Upsilon}_y}\\textbf{Dv}$ is not  $(v_y-v_1,v_y-v_2,\\cdots,v_y-v_K)^T$ but its permutation. Although this difference should have no influence in the proofs under PERM loss assumptions, it could still be a bit misleading. \n- Line 200: in the RHS of the equation, $\\left\\|\\hat{\\textbf{w}}\\right\\|$ should be $\\|\\hat{\\textbf{w}}\\|^2t^{-2}$\n- Line 219: $\\hat{\\textrm{w}}^T\\mathcal{R}(\\textbf{w})$ should be $\\hat{\\textrm{w}}^T\\nabla\\mathcal{R}(\\textbf{w})$\n\nQUESTIONS:\n- As the authors have mentioned in Section 5, it has been proved in [Shamir, 2021] that for gradient-based methods in binary classification, throughout entire training process, both empirical risk and generalization error decrease at an optimal rate, I am curious if similar phenomenon has been observed in numerical experiments for multiclass setting?\n\n- It seems to me that the condition learning rate $\\eta<\\frac{2}{\\beta\\sigma_{\\textrm{max}}^2(X)}$ could be strong for dataset $X$ with a large spectral norm. I’m curious if it is more of a necessity for the proof, or the numerical experiments do require this condition to have directional convergence?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

comments: SUMMARY:
This paper investigates the implicit bias of gradient descent on separable multiclass data using a broad class of losses termed Permutation Equivariant and Relative Margin-based (PERM) losses, which include cross-entropy loss, multiclass exponential loss, and PairLogLoss. The main contribution is the extension of the concept of the exponential tail property, commonly used in the analysis of implicit bias in binary classification, to multiclass classification with PERM losses. The proof techniques (and main results) are analogous to those used in the binary case, suggesting that PERM losses can bridge the theoretical gap between binary and multiclass classification.

STRENGTHS:
1. The paper is very well-written and well-organized, making it easy to follow.
2. While the implicit bias of gradient descent has been widely studied for binary classification, its extension to multiclass classification is relatively sparse and mainly restricted to cross-entropy loss. This paper addresses a broad class of PERM losses, extending the concept of the exponential tail through the "template" of the loss. The result is both interesting and significant.
3. The theoretical results are convincing. While the proof largely follows Soudry et al. [2018], the analysis is nontrivial. Additionally, the proof idea is laid out in a clear manner.

WEAKNESSES:
1. The main weakness is the lack of numerical results to verify the theoretical claims. While this is understandable given that the convergence is exponentially slow, some simulations could be presented to illustrate the gap between theory and practice (where networks are not trained to achieve zero loss).

QUESTIONS:
Maybe a minor typo: in Assumption 3.3, line 171, $k$ should not be there.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

comments: SUMMARY:
This paper uses the framework of permutation equivariant and relative margin-based losses of (Wang and Scott, '24) to extend the implicit bias result of (Soudry et al.,'18) to multinomial classification. Namely, the authors prove that when the loss satisfies a multiclass generalisation of the exponential tail property and the dataset is linearly separable, gradient descent with a sufficiently small learning rate converges to the multiclass max-margin solution.

STRENGTHS:
The paper's goal is clearly stated and framed within the relevant literature. The definitions and statements of the main results are easy to follow and, except for a few inaccuracies, so is the proof sketch.

WEAKNESSES:
From a conceptual point of view, the results of the present paper do not provide additional insight into the implicit bias of multiclassification with respect to previous works limited to the cross-entropy loss.

Also at the technical level, as the authors themselves state, using the PERM framework allows for a simple generalisation of the results of (Soudry et al., '18) to the multiclass case, to the point that a large fraction of the proof is identical or almost identical.

QUESTIONS:
-Definition of $\tilde{x}$ in Eq.~(9)?
-There must be some typo in the first equation of section 41;
-The definition of coordinate projection after Eq. (19) in the proof of Lemma 4.5 is unclear;
-The explanation preceding Eq. (20) in the same proof could mention that Eq. (11) is also used.

LIMITATIONS:
The authors have addressed some of the paper's limitations, but not the limited significance with respect to previously existing results on the implicit bias of multiclassification.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-08