SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As expected for the MSE loss setup, the weight decay is the main driving force for the occurrence of DNC. Therefore, predictably, the weight decay has a direct influence on the speed of convergence towards both DNC and global optimality. This is clearly demonstrated in Figure 6, where we plot the training dynamics of 4-DUFM with width 64 and weight decays of sizes 0.0001, 0.0005, 0.0009, 0.0013, 0.0017. By increasing the value of the weight decay, the loss as well as DNC metrics converge much faster. However, there is no qualitative change in the result of the training for different values of weight decay.
C.5 Interconnection of DNC and optimality of DUFM
As an interesting bonus, we show a single, cherry-picked run for the 4-DUFM with width 10 (although such runs occur quite often at this width). This training first got stuck at an almost-optimal solution, then after a long stay in the saddle point, it jumped to the optimum. The DNC metrics showed a clear decrease at exactly that point of the training, as shown in Figure 7. This run clearly shows how interconnected the deep neural collapse is with optimality.
D Pre-trained DUFM experiments on CIFAR-10
Here, we specify the training conditions from Section 4.2 in more detail, and add results of further runs to provide more robust evidence. We use three different architectures - ResNet20, DenseNet121 and VGG13. We only impose one change to these - the standard backbone is extended with one extra fully-connected layer with output dimension 64 to enable free features and reasonable size of the DUFM. The weights are initialized with He initialization [9] with scaling 1. The DUFMs in question are 3- and 5-layered of constant width 64. We train with weight decay 0.00001 and with learning rate 0.001 on 1000000 full GD steps. The backbone was then trained using stochastic gradient descent with batch size 128, learning rate 0.00001 and with 10000 epochs on classes 0 and 1 of CIFAR-10. While in Figure 1 we plot results of a single training of the 3-DUFM model followed by 7 independent trainings of independent initializations of the ResNet20, here we add two more independent trainings of the DUFM averaged over further 7 independent runs of ResNet20 training. The results are qualitatively the same as in the main body, which suggests that the ability of ResNet20 to make unconstrained features suitable for the emergence of DNC is robust w.r.t. the well-trained DUFM head. The results of these two runs are depicted in Figure 8.
As a side note, we remark that unlike end-to-end training or DUFM training, the ResNet20 training with pre-trained DUFM head is more prone to undesired behaviours. First of all, we need a rather small learning rate to prevent divergence and even with that, a minority of runs never jumps from  $50\%$  training accuracy. Therefore, in our results, we only report those runs, which converged to near  $100\%$  training accuracy.
Figure 5: Losses and DNC metric progressions as a function of training time. The unlabeled loss curves correspond to all the regularization losses. Each row represents a different width, in increasing order ranging through  $\{2, 5, 10, 20, 64, 1000\}$ . As the width increases, the training gets more stable. For width 2, no training converges to optimum. For width 5, the training only rarely converges to optimum. For width 10, the training usually converges to optimum, but often after encountering saddle points. From width 20, we see a stable training behavior with almost no difference between different high-width trainings.
Figure 6: Losses and DNC metric progressions as a function of training time. The unlabeled loss curves correspond to all the regularization losses. Each row represents a different value of weight decay in increasing order through a set  $\{0.0001, 0.0005, 0.0009, 0.0013, 0.0017\}$ . By increasing the weight decay, the training dynamics get faster.
Figure 7: Losses and DNC metric progressions as a function of training time. The unlabeled loss curves correspond to all the regularization losses. The plot shows a single run of the training of 4- DUFM with width 10. After staying in a close-to-optimal saddle point, the model achieved optimum, the DNC metrics rapidly decreasing.
Figure 8: Loss functions and deep neural collapse metrics as a function of training progression. The rows correspond to different pre-training runs of the 3-DUFM model. On the left, the training loss function and its decomposition are displayed. Next, the DNC1-3 metrics are displayed. The ResNet20 recovers the DNC metrics for DUFM, in accordance with our theory. This validates unconstrained features as a modeling principle for neural collapse.
Figure 9: Loss functions and deep neural collapse metrics as a function of training progression. The first row corresponds to VGG13 experiments, the second row corresponds to DenseNet121 experiments with the pre-trained 3-DUFM model. On the left, the training loss function and its decomposition are displayed. Next, the DNC1-3 metrics are displayed. Both architectures recover the DNC metrics for DUFM, in accordance with our theory. This validates unconstrained features as a modeling principle for neural collapse.
Figure 10: Loss functions and deep neural collapse metrics as a function of training progression. The head considered is a pre-trained 5-DUFM model. On the left, the training loss function and its decomposition are displayed. Next, the DNC1-3 metrics are displayed. The ResNet20 recovers the DNC metrics for DUFM, in accordance with our theory. This validates unconstrained features as a modeling principle for neural collapse.
We further present the results of the same experiment but with 5-DUFM attached to ResNet20 as well as 3-DUFM attached to DenseNet121 and VGG13, to even further validate the robustness of our results. These results are presented in Figures 9 and 10, and they are in full accordance with the ResNet20 experiments.