SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can check that the RHS of the above has its unique maximizer at  $\alpha = \sqrt{\frac{1 + \gamma^2}{2}}$  for  $|\alpha| \leq \sqrt{\frac{1 + \gamma^2}{2}}$ . Hence  $g(w, b, a) \leq \frac{\sqrt{1 + \gamma^2}}{4}$  in this case. We are now done since any  $(w_1, b)$  satisfying  $\gamma w_1 + b = \sqrt{\frac{1 + \gamma^2}{2}}$  and  $w_1^2 + b^2 \leq 1/2$  has  $b = \frac{1}{\sqrt{2(1 + \gamma^2)}}$ .
Case 2:  $a = -1 / \sqrt{2}$ . Using eqn. (7) we have  $g(w, b, -1 / \sqrt{2}) \leq \phi(b - \gamma w_1) / 2\sqrt{2}$  which for  $b^2 + w_1^2 \leq 1 / 2$  attains its unique maximum  $\sqrt{\frac{\gamma^2 + 1}{4}}$  at  $b = \frac{1}{\sqrt{2(1 + \gamma^2)}}$ .
Finally, note that the weights of the trained network  $(w,b,a)$  are sampled from  $\nu^{*}$ . Hence, the final claim in the theorem about  $f(\nu^{*},Px_{1} + P_{\perp}x_{2})$  follows since the distribution of  $w$  only has a support on  $\mathbf{e}_1$  and  $-\mathbf{e}_1$ .
Note: The proof above also holds for any rotation of the dataset, with the weights corresponding to the first standard basis vector, i.e., the direction of linearly separable coordinate in that basis.
A.1.1 Auxiliary lemmas for rich regime
Lemma A.5. For any symmetric discrete random variable  $X$  with bounded variance, for  $\alpha > 0$ ,
$$
\mathbb {E} [ X \mathbb {I} (X \geq \alpha) ] \leq \sqrt {\frac {1}{2} \min \left(\frac {1}{2} , \frac {V a r (X)}{2 \alpha^ {2}}\right) V a r (X)}.
$$
$$
\mathbb {E} [ X \mathbb {I} (X \geq \alpha) ] = \sum_ {x \geq \alpha} x p (x) = \sum_ {x \geq \alpha} \sqrt {p (x)} \sqrt {p (x)} x \leq \sqrt {p (X \geq \alpha) \sum_ {x \geq \alpha} x ^ {2} p (x)}, \tag {8}
$$
where the last inequality is by Cauchy-Schwartz. Also by Chebyshev's inequality,  $p(|X| \geq \alpha) \leq Var(X) / 2\alpha^2$ . Combining this with eqn. (8) and using symmetry of  $X$  and non-negativity of  $\alpha$  gives the required lemma.
A.1.2 OrthoP method on IFM
Here, we theoretically establish that  $f$  and  $f_{\mathrm{proj}}$  obtained via OrthoP rely on different features for any dataset within IFM. Consequently, by the definition of IFM,  $f$  and  $f_{\mathrm{proj}}$  have independent logits conditioned on the class.
Proposition A.6. Consider any IFM dataset as described in Section 4.1. Let  $f$  be the model described in Theorem 3.1 and  $f_{proj}$  be the second model obtained by OrthoP. Then, the outputs  $f$  and  $f_{proj}$  on  $x$  i.e.,  $f(x)$  and  $f_{proj}(x)$  depend only on  $x_1$  and  $\{x_2, \dots, x_d\}$  respectively. Let the model obtained in Theorem 3.1 be denoted by  $f$ . Consider the projection matrix  $P$  along the top singular vector of the first layer weight matrix of  $f$ . Then, the dataset obtained by projecting the input through  $P_\perp$  is not separable along the linear coordinate.
Proof. As shown in Theorem 3.1, the final distribution of the weights is given by  $\nu^{*} = 0.5\delta_{\theta_{1}} + 0.5\delta_{\theta_{2}}$  where  $\theta_{1} = (\frac{\gamma}{\sqrt{2(1 + \gamma^{2})}}\mathbf{e}_{1},\frac{1}{\sqrt{2(1 + \gamma^{2})}},1 / \sqrt{2}),\theta_{2} = (-\frac{\gamma}{\sqrt{2(1 + \gamma^{2})}}\mathbf{e}_{1},\frac{1}{\sqrt{2(1 + \gamma^{2})}}, - 1 / \sqrt{2})$  and  $\mathbf{e}_1\stackrel {\mathrm{def}}{=}$ $[1,0,\dots ,0]$  denotes first standard basis vector.
As the first layer weight matrix only has support along the  $\mathbf{e}_1$  direction, therefore its top singular vector also points along the  $\mathbf{e}_1$  direction. Hence,  $P = \mathbf{e}_1\mathbf{e}_1^\top$  and  $P_{\perp} = I - \mathbf{e}_1\mathbf{e}_1^\top$ , where  $I$  denotes the identity matrix. Thus, the dataset obtained by projecting the input through  $P_{\perp}$  has value 0 for the linear coordinate, for both  $y = +1$  and  $y = -1$ . Hence, it is not separable along the linear coordinate. Thus, the second model  $f_{proj}$  relies on other coordinates for classification.
A.2 Lazy regime
Theorem 4.2 is a corollary of the following more general theorem.
Theorem A.7. Consider a point  $x \in D$ . For sufficiently small  $\epsilon > 0$ , there exist an absolute constant  $N$  such that for all  $d > N, \gamma < \epsilon \sqrt{d}$  and  $\gamma \geq 7$ , for the joint training of both the layers of 1-hidden layer FCN with ReLU activation in the NTK regime, the prediction of any point of the form  $(\zeta, x_{2:d})$  satisfies the following:
1. For  $\zeta \geq 0.73$ , the prediction is positive.
2. For  $\zeta \leq -0.95\gamma$ , the prediction is negative.
The above theorem establishes that perturbing  $x_{1}$  by  $O(\gamma)$  changes  $pred(f(x))$  for  $x \in D$  (whereas a classifier exists that achieves a margin of  $\Omega (\sqrt{d})$  on  $D$ , as  $D$  has margin 1 for coordinates  $\{2\cdots d\}$ ). As  $\gamma = o(d)$ , this shows that the learned model is adversarially vulnerable.
Proof of Theorem A.7. The idea of the proof is to obtain an analytical expression for  $f(x)$  using KKT conditions for the max-margin SVM for the NTK kernel (as in Theorem 3.2).
We begin with some preliminaries. We will refer to the first coordinate of the instance as the 'linear' coordinate, and to the rest as 'non-linear' coordinates. Also, henceforth we append an extra coordinate with value 1 to all our instances (corresponding to bias term) - as is standard for working with unbiased SVM without loss of generality.
Explicit expression for  $f$ . Using representer theorem for max margin kernel SVM, we know that  $f$  can be expressed as
$$
f (x) = \sum_ {(x ^ {(t)}, y ^ {(t)}) \in D} \lambda_ {t} y ^ {(t)} K (x, x ^ {(t)}),
$$
for some  $\lambda_t \geq 0$  (that are known as Lagrange multipliers). Further by KKT conditions, a function possessing such a representation (that correctly classifies  $D$ ) has maximum margin if  $y^{(t)}f(x^{(t)}) = 1$  whenever  $\lambda_t > 0$  (training points  $t$  satisfying  $\lambda_t > 0$  are called support vectors).
We begin with a useful claim.
Claim A.8. The max margin kernel SVM for  $D$  with the NTK kernel has all points in  $D$  as support vectors.
Proof. By the above discussion, it suffices to show that the (unique) solution  $\alpha \in \mathbb{R}^{|D|}$  to  $K\alpha = y$  satisfies  $\mathrm{sign}(\alpha_i) = y^{(i)}$  for all  $i$ , where  $K$  is the  $|D|\times |D|$  Gram matrix with  $(i,j)$ th entry  $K(x^{(i)},x^{(j)})$  and  $y_{i} = y^{(i)}$  (the Lagrange multipliers  $\lambda_{i}$  are then given by  $y_{i}\alpha_{i}$ ).
Structure of Gram matrix. Order  $D$  so that the positive instances appear first. Then the Gram matrix  $K$  has a block structure of the form  $\left( \begin{array}{cc} B & C \\ C^T & R \end{array} \right)$  where  $B \in \mathbb{R}^{2^{d-1} \times 2^{d-1}}$  and  $R \in \mathbb{R}$  are the Gram matrices for the positive and negative instances respectively, and  $C \in \mathbb{R}^{2^{d-1} \times 1}$  represents the  $K(x^{(i)}, x^{(|D|)})$  values for  $i < |D|$ .
Recall that for the NTK kernel,  $K(x^{(i)},x^{(j)})$  has the form  $\| x^{(i)}\| \| x^{(j)}\| \kappa (\langle x^{(i)},x^{(j)}\rangle)$ . Note all the positive instances have the same norm (denoted by  $\rho_{1} = \sqrt{d + \gamma^{2}}$ ) and the inner product between two positive instances depends only on the number  $i$  of non-matching non-linear coordinates (denoted by  $\beta_{i}$  for  $0\leq i\leq d - 1$ ). Hence, the rows of  $B$  are permutations of each other, with the entry  $\rho_1^2\beta_i$  appearing  $\binom{d-1}{i}$  times. Similarly, the entries in  $C$  are all equal and are denoted by  $\rho_{1}\rho_{2}\beta_{d}$  where  $\beta_{d}$
denotes  $\kappa(x^{(t)}, x^{|D|})$  for any  $t < |D|$  and  $\rho_2 = \|x^{|D|}\| = \sqrt{1 + \gamma^2}$ . The only entry in  $R$  is  $\rho_2^2\kappa(1)$ . In particular,
$$
\beta_ {i} = \kappa \left(\frac {d - 2 i + \gamma^ {2}}{d + \gamma^ {2}}\right) \text {f o r} i \in [ | D | - 1 ], \qquad \text {a n d} \qquad \beta_ {d} = \kappa \left(\frac {1 - \gamma^ {2}}{\sqrt {d + \gamma^ {2}} \sqrt {1 + \gamma^ {2}}}\right).
$$
Now we are ready to solve  $K\alpha = y$ . By symmetry in the structure of  $\mathbf{K}$ ,  $\alpha$  looks like  $[a, a, \dots, b]$ , where the first  $|D| - 1$  entries are the same.
Expanding  $K\alpha = y$  , we get two equations given by
$$
a \rho_ {1} ^ {2} \left(\sum_ {i = 0} ^ {d - 1} \binom {d - 1} {i} \beta_ {i}\right) + b \rho_ {1} \rho_ {2} \beta_ {d} = 1 \qquad \text {a n d} \qquad 2 ^ {d - 1} a \rho_ {1} \rho_ {2} \beta_ {d} + \rho_ {2} ^ {2} \kappa (1) b = - 1.
$$
Solving, we get
$$
a = \frac {\rho_ {2} \kappa (1) + \rho_ {1} \beta_ {d}}{\rho_ {1} ^ {2} \rho_ {2} \sum_ {i = 0} ^ {d - 1} \left(\binom {d - 1} {i} [ \kappa (1) \beta_ {i} - \beta_ {d} ^ {2} ]\right)} \qquad \text {a n d} \qquad b = \frac {- 1 - 2 ^ {d - 1} a \rho_ {1} \rho_ {2} \beta_ {d}}{\rho_ {2} ^ {2} \kappa (1)}.
$$
We now show that  $a > 0$  and  $b < 0$ . Note that for sufficiently large  $d$ ,  $\beta_d$  can be made arbitrarily close to  $\kappa(0) = 1 / \pi$  (since  $\kappa$  is smooth around 0). Hence,  $a > 0$  implies  $b < 0$ . We in fact give the following estimate for  $a$ :
$$
a = 2 ^ {1 - d} \cdot \frac {\rho_ {2} \kappa (1) + \rho_ {1} \beta_ {d}}{\xi \rho_ {1} ^ {2} \rho_ {2}} \quad \text {w h e r e} \quad \frac {2}{\pi} - \frac {1}{\pi^ {2}} + O \left(\frac {1}{d}\right) \leq \xi \leq 2 + O \left(\frac {1}{d}\right). \tag {9}
$$
For the lower bound on  $\xi$ , write
$$
\begin{array}{l} \sum_{i = 0}^{d - 1}\binom {d - 1}{i}[\kappa (1)\beta_{i} - \beta_{d}^{2}] = \kappa (1)\sum_{i = 0}^{\lfloor d / 2\rfloor}\binom {d - 1}{i}(\beta_{i} + \beta_{d - 1}) - 2^{d - 1}\beta_{d}^{2} \\ \geq \kappa (1)\sum_{i = 0}^{\lfloor d / 2\rfloor}\binom {d - 1}{i}2\beta_{d / 2} - 2^{d - 1}\beta_{d}^{2}\geq 2^{d - 1}\left(\kappa (1)\kappa (0) - \kappa^{2}(0) + O\left(\frac{1}{d}\right)\right), \\ \end{array}
$$
where for the first inequality we used convexity of  $\kappa$  and for the second inequality we used  $\beta_{d/2} = \kappa(0) + O(1/d), \beta_d = \kappa(0) + O(1/\sqrt{d})$ . For the upper bound on  $\xi$ , write
$$
\begin{array}{l} \sum_ {i = 0} ^ {d - 1} \binom {d - 1} {i} [ \kappa (1) \beta_ {i} - \beta_ {d} ^ {2} ] \leq \kappa (1) \sum_ {i = 0} ^ {d - 1} \binom {d - 1} {i} \kappa \left(1 - \frac {2 i}{d + \gamma^ {2}}\right) \\ \leq \kappa (1) \sum_ {i = 0} ^ {d - 1} \binom {d - 1} {i} \left(2 - \frac {2 i}{d + \gamma^ {2}}\right) = \kappa (1) 2 ^ {d} - \frac {\kappa (1) (d - 1) 2 ^ {d - 1}}{d + \gamma^ {2}}, \\ \end{array}
$$
where for the second inequality we used  $\kappa(u) \leq 1 + u$  (which holds by convexity and  $\kappa(-1) = 0, \kappa(1) = 2$ ).
Now we analyze predicted labels for points of the form  $(\zeta, x_{2:d+1})$  where  $x \in D$ . We make two cases depending on the label of  $x$ .
Predicted label for point  $(\zeta, x_{2:d+1}^{(t)})$  where  $x^{(t)} \in D$  has positive label
Our point (denoted by  $x$ ) has the form  $(\zeta, \zeta_1, \zeta_2, \ldots, \zeta_d, 1)$  where  $\zeta_i \in \pm 1$ . The idea of the proof is to write  $f$  explicitly as a function of  $\zeta$  and work with its first order Taylor expansion around  $\zeta = \gamma$ , with some additional work to take care of non-smoothness of  $f$ .
Explicit form for  $f$ . Let  $\tau_i \stackrel{\mathrm{def}}{=} \langle x, x' \rangle / (\|x\| \|x'\|)$  for a positive instance  $x' \in D$ , where  $x$  and  $x'$  have exactly  $i$  non-matching non-linear coordinates (for  $0 \leq i \leq d - 1$ ). Similarly denote by  $\tau_d$  the quantity  $\langle x, x^{|D|} \rangle / (\|x\| \|x^{|D|}\|)$ . In particular,
$$
\tau_ {i} = \left(\frac {d - 2 i + \gamma \zeta}{\rho_ {1} \| x \|}\right) \qquad \text {a n d} \qquad \tau_ {d} = \left(\frac {1 - \gamma \zeta}{\rho_ {2} \| x \|}\right).
$$
By the above discussion, we have
$$
f(x) = a\left(\sum_{t = 1}^{|D| - 1}K(x,x^{(t)})\right) + bK(x,x^{|D|}) = a\rho_{1}\| x\| \left(\sum_{i = 0}^{d - 1}\binom {d - 1}{i}\kappa (\tau_{i})\right) + b\rho_{2}\| x\| \kappa (\tau_{d}).
$$
Substituting  $b$  and denoting  $f(x) / \| x\|$  by  $g(\zeta)$  we get
$$
g (\zeta) = a \rho_ {1} \left[ \sum_ {i = 0} ^ {d - 1} \binom {d - 1} {i} \kappa \left(\tau_ {i} (\zeta)\right) - \frac {2 ^ {d - 1} \beta_ {d}}{\kappa (1)} \kappa \left(\tau_ {d} (\zeta)\right) \right] - \frac {\kappa \left(\tau_ {d} (\zeta)\right)}{\rho_ {2} \kappa (1)}. \tag {10}
$$
Now try to expand  $g(\zeta)$  using the Taylor series around  $\zeta = \gamma$  (note that  $g(\gamma) = 1 / \rho_1$ ). Note that  $\kappa'$  can however be unbounded around  $-1$  and  $1$ . To get around this, write  $g = h + q$ , where  $h$  has bounded first and second derivative, and  $q$  has lower order than  $h$  for  $\zeta$  of interest. In particular,
$$
h (\zeta) = a \rho_ {1} \left[ \sum_ {i = d / 4} ^ {3 d / 4} \binom {d - 1} {i} \kappa (\tau_ {i} (\zeta)) - \frac {2 ^ {d - 1} \beta_ {d}}{\kappa (1)} \kappa (\tau_ {d} (\zeta)) \right] - \frac {\kappa (\tau_ {d} (\zeta))}{\rho_ {2} \kappa (1)} \quad \text {a n d}
$$
$$
q(\zeta) = a\rho_{1}\left[\sum_{i:|d / 2 - i| > d / 4}\binom {d - 1}{i}\kappa (\tau_{i}(\zeta))\right].
$$
Observe that  $q(\zeta) = o(c^{d})$  for  $c < 1$  using the estimate eqn. (9) for  $a$  and concentration for sums of independent Bernoullis. By Taylor's theorem,
$$
g (\zeta) = h (\gamma) + h ^ {\prime} (\gamma) (\zeta - \gamma) + \frac {h ^ {\prime \prime} (\theta) (\zeta - \gamma) ^ {2}}{2} + q (\zeta), \tag {11}
$$
for some  $\theta \in [\gamma, \zeta]$ , where  $h(\gamma) \approx 1 / \sqrt{d}$ . It will turn out that  $|h'(\gamma)| = \Theta(1 / \sqrt{d})$ ,  $|h''(\zeta)| = o(1 / \sqrt{d})$ . This will allow us to complete the proof using the linear approximation of  $g(\zeta)$  by neglecting the second order term and  $q(\zeta)$ . We now compute  $h', h''$ , treating  $\|x\| = \sqrt{d + \zeta^2}$  as a constant for exposition (the proof works without this approximation or the reader may think of  $\gamma$  as  $o(\sqrt{d})$ ). Using  $\tau_i'(\zeta) \approx \frac{\gamma}{\rho_1 \|x\|}, \tau_d'(\zeta) \approx \frac{-\gamma}{\rho_2 \|x\|}$ ,
$$
h ^ {\prime} (\zeta) \approx a \rho_ {1} \left[ \sum_ {i = 0} ^ {d - 1} \binom {d - 1} {i} \kappa^ {\prime} (\tau_ {i} (\zeta)) \frac {\gamma}{\rho_ {1} \| x \|} + \frac {2 ^ {d - 1} \beta_ {d}}{\kappa (1)} \kappa^ {\prime} (\tau_ {d} (\zeta)) \frac {\gamma}{\rho_ {2} \| x \|} \right] + \frac {\kappa^ {\prime} (\tau_ {d} (\zeta))}{\rho_ {2} \kappa (1)} \frac {\gamma}{\rho_ {2} \| x \|}
$$
$$
h ^ {\prime \prime} (\zeta) \approx a \rho_ {1} \left[ \sum_ {i = 0} ^ {d - 1} \binom {d - 1} {i} \kappa^ {\prime \prime} (\tau_ {i} (\zeta)) \frac {\gamma^ {2}}{\rho_ {1} ^ {2} \| x \| ^ {2}} - \frac {2 ^ {d - 1} \beta_ {d}}{\kappa (1)} \kappa^ {\prime \prime} (\tau_ {d} (\zeta)) \frac {\gamma^ {2}}{\rho_ {2} ^ {2} \| x \| ^ {2}} \right] - \frac {\kappa^ {\prime \prime} (\tau_ {d} (\zeta))}{\rho_ {2} \kappa (1)} \frac {\gamma^ {2}}{\rho_ {2} ^ {2} \| x \| ^ {2}}.
$$
Plugging  $\| x\| \approx \rho_1\approx \sqrt{d}$  and substituting  $a$  from eqn. (9),
$$
h ^ {\prime} (\zeta) = \frac {(1 + \beta_ {d} ^ {2} / \xi) \kappa^ {\prime} (\tau_ {d} (\zeta)) \gamma}{\rho_ {2} ^ {2} \kappa (1) \sqrt {d}} + o \left(\frac {1}{\sqrt {d}}\right) \qquad \mathrm {a n d} \qquad h ^ {\prime \prime} (\zeta) = O \left(\frac {1}{d}\right),
$$
which substituted in eqn. (11) with  $\tau_d(\zeta)\approx 0,\beta_d\approx \kappa (0),\kappa '(\tau_d(\zeta))\approx \kappa '(0)$  gives
$$
g (\zeta) = \frac {1}{\sqrt {d}} \left(1 + \frac {(1 + \kappa^ {2} (0) / \xi) \kappa^ {\prime} (0) \gamma}{\kappa (1) \rho_ {2} ^ {2}} (\zeta - \gamma)\right) + o \left(\frac {1}{\sqrt {d}}\right),
$$
Hence,  $g(\zeta) > 0$  whenever the coefficient of  $1 / \sqrt{d}$  above is bounded above zero, and a similar condition holds for  $g(\zeta) < 0$ . Using the estimates of  $\xi$  from eqn. (9) and  $\kappa'(0) = 1$ ,  $\kappa(0) = 1 / \pi$ ,  $\kappa(1) = 2$ ,  $\rho_2^2 = 1 + \gamma^2$  in the above gives that  $g(\zeta) > 0$  for  $\zeta > -0.68\gamma - 1.68/\gamma$  and  $g(\zeta) < 0$  for  $\zeta < -0.905\gamma - 1.905/\gamma$ .
Predicted label for point  $(\zeta, x_{2:d+1}^{(t)})$  where  $x^{(t)} \in D$  has negative label
Following the same plan, write our point (denoted by  $x$ ) as  $(\zeta, 0, \dots, 0, 1)$ .
Explicit form for  $f$ . Begin by finding
$$
\tau_ {i} = \left(\frac {1 + \gamma \zeta}{\rho_ {1} \| x \|}\right) \qquad \text {a n d} \qquad \tau_ {d} = \left(\frac {1 - \gamma \zeta}{\rho_ {2} \| x \|}\right).
$$
eqn. (10) now gives
$$
g (\zeta) = 2 ^ {d - 1} a \rho_ {1} \left[ \kappa (\tau_ {0} (\zeta)) - \frac {\beta_ {d} \kappa (\tau_ {d} (\zeta))}{\kappa (1)} \right] - \frac {\kappa (\tau_ {d} (\zeta))}{\rho_ {2} \kappa (1)}.
$$
Expanding  $\kappa (\tau_0(\zeta))$  using Taylor series around  $\zeta = -1 / \gamma$
$$
\kappa (\tau_ {0} (\zeta)) = \kappa (0) + \kappa^ {\prime} (\tau_ {0} (\theta)) \tau_ {0} ^ {\prime} (\theta) (\zeta + \frac {1}{\gamma}),
$$
for some  $\theta \in [-1, 1]$ . For large  $d$ ,  $\tau_0(\theta) \approx 0$  and  $\tau_0'(\theta) = O(1 / \sqrt{d})$ . Hence we have
$$
\begin{array}{l} g (\zeta) = \frac {\rho_ {2} \kappa (1) + \rho_ {1} \beta_ {d}}{\xi \rho_ {1} \rho_ {2}} \left[ \kappa (0) + O \left(\frac {1}{\sqrt {d}}\right) - \frac {\beta_ {d} \kappa (\tau_ {d} (\zeta))}{\kappa (1)} \right] - \frac {\kappa (\tau_ {d} (\zeta))}{\rho_ {2} \kappa (1)} \\ = \frac {1}{\rho_ {2}} \left(\frac {\kappa^ {2} (0)}{\xi} - \left(\frac {\kappa^ {2} (0)}{\xi \kappa (1)} + \frac {1}{\kappa (1)}\right) \kappa (\tau_ {d} (\zeta))\right) + o (1). \\ \end{array}
$$
As before  $g(\zeta) > 0$  whenever the coefficient of  $1 / \rho_2$  above is bounded above zero which happens for  $\zeta \geq 0.73$  (for  $\gamma \geq 3$ ). Similarly,  $g(\zeta) < 0$  for  $\zeta \leq 0$ .
In this section, we provide experimental details, including hyperparameter tuning setup and some additional experiments.
B.1 Details on the experimental setting
We will first describe the four datasets that have been used in this work.
1. Imagenette (FastAI, 2021): This is a subset of 10 classes of Imagenet, that are comparatively easier to classify.
2. b-Imagenette: This is a binarized version of Imagenette, where only a subset of two classes (tench and English springer) is used.
3. Waterbirds-Landbirds (Sagawa et al., 2020a): This is a majority-minority group dataset, consisting of waterbirds on water and land background, as well as landbirds on land and water background. This dataset serves as a baseline for checking the dependence of model on the spurious background feature when predicting the bird class, as most of the training examples have waterbirds on water and landbirds on land background.
4. Imagenet (Deng et al., 2009): This is the standard benchmark for large scale image classification.
5. MNIST-CIFAR (Shah et al., 2020): This is a collage dataset, created by concatenating MNIST and CIFAR images along an axis. This is a synthetic dataset for evaluating the simplicity bias of a trained model.
Setup Throughout the paper, we work with the pretrained representations of the above datasets, obtained by using anImagenet pretrained Resnet 50. We finetune a 1-hidden layer FCN with a hidden dimension of 100 (8000 forImagenet) on top of these representations (keeping the backbone fixed) using SGD with a momentum of 0.9. Every model is trained for 20000 (100000 forImagenet) steps with a warmup and cosine decay learning rate scheduler. For each of the runs, we tune the batch size, learning rate and weight decay using validation accuracy. Below are the hyperparameter tuning details:
- Batch size  $\in$  {128, 256}
- Learning rate:
- Rich regime:  $\in \{0.5, 1.0\}$  (for imagenet,  $\in \{5.0, 10.0\}$  as learning rate in rich regime needs to scale up with the hidden dimension)
- Lazy regime:  $\in$  {0.01, 0.05}
- Weight decay:  $\in \{0, 1e^{-4}\}$
The final numbers reported are averaged across 3 independent runs with the selected hyperparameters.
Evaluation For Imagenette, b-Imagenette, Imagenet and MNIST-CIFAR, we report the standard test accuracy in all the experiments. For waterbirds, we report train-adjusted test accuracy, as reported in Sagawa et al. (2020a). Precisely, accuracy for each group present in the test data is individually calculated and then weighed by the proportion of the corresponding group in the train dataset.
In this section, we present a few additional experimental results.
Results on Imagenet The evolution of effective rank of the first layer weight matrix is shown in Figure 5. As can be seen, the weight matrix becomes sufficiently low rank in three rich regimes as the training progresses.
Singular value decay. In Figure 6, we provide the singular value decay of the weight matrix for the first model trained in rich regime. As can be seen, the top few singular values capture most of the Frobenius norm of the matrix.
(b) Lazy regime
Figure 5: Evolution of effective rank of first layer weight matrix (dimension -  $2048 \times 2000$ ) for Imagenet dataset in rich and lazy regime.
Figure 6: Fraction of Frobenius norm captured by the top  $i^{th}$  singular value i.e.,  $\sigma_i^2 / \sum_{j=1}^d \sigma_j^2$  vs  $i$  of the first layer weight matrix trained in rich regime for various datasets.
MNIST-CIFAR In Figure 7, we show that an ensemble of  $f$  and  $f_{\mathrm{proj}}$  has better gaussian robustness than an ensemble of  $f$  and  $f_{\mathrm{ind}}$  on MNIST-CIFAR dataset.
Non-linearity of decision boundary Figure 8 shows the decision boundary of  $f$  and  $f_{\mathrm{proj}}$  on 2-dimensional subspace spanned by top two singular vectors of the weight matrix. We observe that the decision boundary of the second model is more non-linear compared to that of the first model. We also report a quantitative measure of non-linearity of the decision boundary along the top two singular vectors for  $f$  and  $f_{\mathrm{proj}}$ . Basically, we fit a linear classifier to the decision boundary and report its accuracy. As shown in Table 6, the test accuracy obtained by the linear classifier for  $f_{\mathrm{proj}}$  is less than  $f$ .
Variation of LD-SB with depth In Figure 9 and 10, we show the evolution of effective rank of weight matrices for depth-2 and 3 ReLU networks. As can be seen, the rank still decreases with training, however the effect is less pronounced for the initial layers. Note that the initialization used in these runs was the feature learning initialization as proposed in Yang & Hu (2021).
Comparison of Mistake diversity with Evading SB Table 7 shows the results of comparing OrthoP with Evading-SB (Teney et al., 2022), with regards to diversity metrics. As can be seen, OrthoP is either better or comparable to Evading-SB.
Figure 7: Variation of test accuracy with the standard deviation of Gaussian noise added to the pretrained representations of MNIST-CIFAR dataset. Model 1 is kept fixed, and values for both the ensembles are averaged across 3 runs.
(a) b-Imagenette  $(f)$
Figure 8: Decision boundaries for  $f$  and  $f_{\mathrm{proj}}$  for B-Imagenette and Waterbirds datasets, visualized in the top 2 singular directions of the first layer weight matrix. The decision boundary of  $f_{\mathrm{proj}}$  is more non-linear compared to that of  $f$ .
(b) b-Imagenette  $(f_{\mathrm{proj}})$
(c) Waterbirds  $(f)$
(d) Waterbirds  $(f_{\mathrm{proj}})$
Table 6: Quantitative measurement of non-linearity of decision boundary – accuracy of fitted linear classifier to the decision boundary
Dataset	Linear-Classifier-Acc(f)	Linear-Classifier-Acc(fproj)
b-Imagenette	96.12	95.28 ± 0.2
Waterbirds	97.28	93.24 ± 0.24
(a) Layer 1
Figure 9: Evolution of effective rank of the weight matrices for a depth-2 ReLU network on Resnet-50 pretrained representations of the dataset
(b) Layer 2
Figure 10: Evolution of effective rank of the weight matrices for a depth-3 ReLU network on Resnet-50 pretrained representations of the dataset
(c) layer 3