SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {\mathbf {A}} _ {d, c, n} (\mathbf {Q}) = \mathbf {Q} ^ {\top} \mathbf {A} _ {d, c, n} \mathbf {Q}. \tag {80}
$$
Since  $\mathbf{A}_n$  is arbitrary, this relaxation allows arbitrary non-zero values of non-diagonal kernels  $\Theta_{k,s}^{h}$  with  $k\neq s$ . The following observations justify the consistency of the above assumption:
- Since  $\mathbf{W} \in \mathbb{R}^{C \times n}$ ,  $\mathbf{R} \in \mathbb{R}^{C \times N}$  and  $N > n > C$ ,  $\mathbf{R}^{\top} \mathbf{W}$  has a non-empty kernel (possibly time-dependent).
- The dynamics is rotation invariant under the assumption, i.e., the equation (75) holds.
- The expression of the assumption is rotation invariant, in a sense that  $\tilde{\mathbf{A}}_{d,c}(\mathbf{Q}) = \kappa_{d,c}\mathbb{I}_n + \tilde{\mathbf{N}}_{d,c}(\mathbf{Q})$ , where  $\tilde{\mathbf{N}}_{d,c}^{\top} \in \ker (\mathbf{R}^{\top}\mathbf{W}\mathbf{Q})$  for any orthogonal  $\mathbf{Q}$ .
Under the above assumption, the derivative in 77 becomes zero, so the dynamics has an invariant of the form  $\mathbf{E} \coloneqq \mathbf{W}^{\top}\mathbf{W} - c_{1}\mathbf{H}_{1}\mathbf{H}_{1}^{\top} - c_{2}\mathbf{H}_{2}\mathbf{H}_{2}^{\top}$ . Moreover, the statement and the proof of our main Theorem 5.2 remains unchanged. Thus, DNNs satisfying the conditions of Theorem 5.2 display NC under Assumption D.2.
D.3 Discussion
The analysis of the DNNs dynamics is simplified significantly by assuming that  $\Theta^h$  has a block structure. However, formulating a reasonable and consistent assumption on the NTK and its components is non-trivial. The Assumption 3.2 that we used in the main text is justified by the empirical results but may not capture all the relevant properties of the NTK. We believe that studying DNNs' dynamics under a more general or a more reasonable assumption on the NTK is a promising future work direction. The relaxed block structure assumption proposed in this section is the first step into this direction.
Figure 3: VGG11 trained on MNIST. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
Figure 4: VGG11 trained on FashionMNIST. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
Figure 5: VGG16 trained on CIFAR10. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
Figure 6: ResNet20 trained on MNIST. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
Figure 7: ResNet20 trained on FashionMNIST. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
$\| H_1H_1^T\| _F$
$\| H_2H_2^T\| _F$
$\| \langle h\rangle \langle h\rangle^T\| _F$
$\langle \Theta /\| \Theta \| _F,YY^T /||YY^T ||_F\rangle$
$\langle \Theta^h /\| \Theta^h\| _F,YY^T /\| YY^T\| _F\rangle$
Figure 8: ResNet20 trained on CIFAR10. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
$\| H_1H_1^T\| _F$
$\| H_2H_2^T\| _F$
$\| \langle h\rangle \langle h\rangle^T\| _F$
$\langle \Theta /\| \Theta \| _F;YY^T / / \| YY^T\| _F\rangle$
$\langle \Theta^h /\| \Theta^h\| _F,YY^T /\| YY^T\| _F\rangle$
Figure 9: DenseNet40 trained on MNIST. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
$\| H_1H_1^T\| _F$
$\| H_2H_2^T\| _F$
$\| \langle h\rangle \langle h\rangle^T\| _F$
$\langle \Theta /\| \Theta \| _F,YY^T /\| YY^T\| _F\rangle$
$\langle \Theta^h /\| \Theta^h\| _F,YY^T /\| YY^T\| _F\rangle$
Figure 10: DenseNet40 trained on FashionMNIST. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
Figure 11: DenseNet40 trained on CIFAR10. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
$\| H_1H_1^T\| _F$
$\| H_0H^T\| _F$
11211F
1  $\| \langle h\rangle \langle h\rangle^t\| _F$
$\langle \Theta / \| \Theta \|_F, YY^T / \| YY^T \|_F \rangle$
$\langle \Theta^h /\| \Theta^h\| _F,YY^T /\| YY^T\| _F\rangle$
Figure 12: ResNet20 trained on MNIST with  $CE$  loss. See Figure 2 for the description of panes a-h. i) Norms of matrices  $\mathbf{H}_1\mathbf{H}_1^\top$ ,  $\mathbf{H}_2\mathbf{H}_2^\top$ , and  $\langle h\rangle \langle h\rangle^\top$  at the end of training. j) Alignment of kernels  $\Theta$  and  $\Theta^h$  at the end of training. The color in panes i-j is the color of the same model in panes a-e.
$\| H_1H_1^T\| _F$
$\| H_2H_T^T\| _F$
$\| \langle h\rangle \langle h\rangle^T\| _F$
$\langle \Theta / \| \Theta \|_F; YY^T / \| YY^T \|_F \rangle$
$\langle \Theta^h /\| \Theta^h\| _F,YY^T /\| YY^T\| _F\rangle$
Figure 13: NTK block structure of VGG11 trained on MNIST. LeCun normal initialization, initial learning rate 0.131. The kernel is computed on a random data subset with 4 samples from each class. See Figure 1 for the description of panes.
Figure 14: NTK block structure of VGG11 trained on FashionMNIST. LeCun normal initialization, initial learning rate 0.049. The kernel is computed on a random data subset with 4 samples from each class. See Figure 1 for the description of panes.
Figure 15: NTK block structure of VGG11 trained on CIFAR10. LeCun normal initialization, initial learning rate 0.131. The kernel is computed on a random data subset with 4 samples from each class. See Figure 1 for the description of panes.
Figure 16: NTK block structure of ResNet20 trained on FashionMNIST. LeCun normal initialization, initial learning rate 0.094. The kernel is computed on a random data subset with 12 samples from each class. See Figure 1 for the description of panes.
Figure 17: NTK block structure of ResNet20 trained on CIFAR10. LeCun normal initialization, initial learning rate 0.068. The kernel is computed on a random data subset with 12 samples from each class. See Figure 1 for the description of panes.
Figure 18: NTK block structure of DenseNet40 trained on MNIST. LeCun normal initialization, initial learning rate 0.049. The kernel is computed on a random data subset with 12 samples from each class. See Figure 1 for the description of panes.
Figure 19: NTK block structure of DenseNet40 trained on FashionMNIST. LeCun normal initialization, initial learning rate 0.094. The kernel is computed on a random data subset with 12 samples from each class. See Figure 1 for the description of panes.
Figure 20: NTK block structure of DenseNet40 trained on CIFAR10. LeCun normal initialization, initial learning rate 0.094. The kernel is computed on a random data subset with 12 samples from each class. See Figure 1 for the description of panes.