SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Settings: We conduct the experiments on two datasets, MNIST and CIFAR-10. For MNIST, we trained a 3-hidden layer fully-connected neural network (width = 1000). First, we train the network until convergence (loss function value smaller than  $10^{-4}$ ), and record the parameters of this trained network as the optimal solution  $\bar{w}$ . For CIFAR-10, we train a two-layer CNN for 4000 iterations, and record the final parameter setting as  $\bar{w}$ . Then, for each experiment, we took a second run, and at each iteration  $w_{t}$  we compute the estimate of the aiming condition coefficient using the equation mentioned above.
Results: Figure 4 show the plots of against the iteration number. In both cases, we observe that the estimate stays positive, which suggests that aiming condition holds.
E Aiming Condition vs. quasar-convexity
Following [15], a nonnegative function  $\mathcal{L}$  is called quasar-convex relative to a point  $\bar{w}$  with  $\mathcal{L}(\bar{w}) = 0$  if the inequality  $\langle \nabla \mathcal{L}(w), w - \bar{w} \rangle \geq \theta \cdot \mathcal{L}(w)$  holds for all  $w$  near  $\bar{w}$ . Aiming, in contrast, stipulates the analogous inequality but with  $\bar{w}$  crucially replaced by  $\operatorname{proj}_S(w)$ , where  $S$  is the set of zero loss solutions. We argue that in the interpolation setting, quasar-convexity is a very restrictive condition because it requires the set of zero-loss solutions to be an affine subspace, which is usually not the case (see for example [24]). To see this, note that quasar-convexity implies that  $S$  is star-convex relative to  $\bar{w}$  (Observation 3 in [15]). That is, there exists  $\epsilon > 0$  such that for any  $w \in B_{\epsilon}(\bar{w}) \cap S$ , the line segment joining  $w$  and  $\bar{w}$  is fully contained in  $S$ . This is a very strong conclusion. For example, a smooth-manifold  $S$  is star convex near  $\bar{w}$  if and only if it coincides with a linear subspace around  $\bar{w}$ . The PL-condition in turn implies by Theorem 2.16 in [35] that  $S$  is a smooth manifold. Therefore, if loss functions for training wide neural networks were quasar-convex, then the set of interpolating solutions would form a linear space on any compact set, which is certainly not true. In contrast, as we have proved, any  $C^3$ -smooth function satisfying PL automatically satisfies the aiming inequality near a point  $\bar{w} \in S$ . In summary, quasar-convexity does not hold when training wide neural networks (since otherwise it would imply that interpolating solutions form a linear subspace), while aiming provably holds—one of our main results.