SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recall from the gradient flow in Eqn 21, by using the method of Euler discretization, after  $k$  update steps starting from the initial  $\pmb{u}(x,0) = \pmb{v}(x)$  with adaptive stepsize  $\Delta t = 1 / \int_{\Omega} (k(x,y) + k(y,x)) dy$  and by choosing  $\lambda = \tilde{\lambda} / \Delta t(x)$ , we obtain the following
$$
\begin{array}{l} \boldsymbol {u} (x, k \Delta t (x)) = \boldsymbol {u} (x, (k - 1) \Delta t (x)) - \Delta t (x) \nabla_ {\boldsymbol {u}} J - \lambda \Delta t (x) (\boldsymbol {u} (x, (k - 1) \Delta t (x)) - \boldsymbol {f} (x)) \\ = \int_ {\Omega} \frac {K (x , y) \boldsymbol {u} (y , (k - 1) \Delta t (x))}{\int_ {\Omega} K (x , y ^ {\prime}) d y ^ {\prime}} d y + \tilde {\lambda} (\boldsymbol {f} (x) - \boldsymbol {u} (x, (k - 1) \Delta t (x))). \tag {27} \\ \end{array}
$$
Discretizing  $\pmb{u}(x, k\Delta t(x))$  and using Monte-Carlo approximation for the integrals in 27, we obtain
$$
\boldsymbol {u} ^ {(k)} (i) = \sum_ {j = 1} ^ {N} \mathbf {A} _ {i j} \boldsymbol {u} ^ {(k - 1)} (j) + \tilde {\lambda} (\boldsymbol {f} (i) - \boldsymbol {u} ^ {(k - 1)} (i)), \tag {28}
$$
where  $\mathbf{A}_{ij}$  is computed using the keys and queries as either  $\mathrm{softmax}\left(\boldsymbol {k}(i)^{\top}\boldsymbol {k}(j) / \sqrt{D_{qk}}\right)$  or  $\mathrm{softmax}\Bigl (\pmb {q}(i)^{\top}\pmb {k}(j) / \sqrt{D_{qk}}\Bigr)$ .
Suppose that  $\pmb{u}^{(k)}(i)$ , defined as Eqn. 28, converges to a constant vector  $\bar{\pmb{u}}$  as  $k\to \infty$ . We have
$$
\begin{array}{l} \boldsymbol {u} ^ {(k + 1)} (i) - \boldsymbol {u} ^ {(k + 1)} (j) \\ = \sum_ {l = 1} ^ {N} \mathbf {A} _ {i l} \boldsymbol {u} ^ {(k)} (l) - \sum_ {l = 1} ^ {N} \mathbf {A} _ {j l} \boldsymbol {u} ^ {(k)} (l) + \tilde {\lambda} (\boldsymbol {u} ^ {(k)} (j) - \boldsymbol {u} ^ {(k)} (i)) + \tilde {\lambda} (\boldsymbol {f} (i) - \boldsymbol {f} (j)) \\ = \left(\sum_ {l = 1} ^ {N} \mathbf {A} _ {i l} \boldsymbol {u} ^ {(k)} (l) - \boldsymbol {u} ^ {(k)} (i) \sum_ {l = 1} ^ {N} \mathbf {A} _ {i l}\right) - \left(\sum_ {l = 1} ^ {N} \mathbf {A} _ {j l} \boldsymbol {u} ^ {(k)} (l) - \boldsymbol {u} ^ {(k)} (j) \sum_ {l = 1} ^ {N} \mathbf {A} _ {j l}\right) \tag {29} \\ + (\tilde {\lambda} - 1) \left(\boldsymbol {u} ^ {(k)} (j) - \boldsymbol {u} ^ {(k)} (i)\right) + \tilde {\lambda} (\boldsymbol {f} (i) - \boldsymbol {f} (j)) \\ = \sum_ {l = 1} ^ {N} \mathbf {A} _ {i l} (\boldsymbol {u} ^ {(k)} (l) - \boldsymbol {u} ^ {(k)} (i)) - \sum_ {l = 1} ^ {N} \mathbf {A} _ {j l} (\boldsymbol {u} ^ {(k)} (l) - \boldsymbol {u} ^ {(k)} (j)) + (\tilde {\lambda} - 1) (\boldsymbol {u} ^ {(k)} (j) - \boldsymbol {u} ^ {(k)} (i)) \\ + \tilde {\lambda} (\boldsymbol {f} (i) - \boldsymbol {f} (j)) \\ \end{array}
$$
Since  $\pmb{u}^{(k)}(i)\rightarrow \bar{\pmb{u}}$  for  $i = 1,2,\ldots ,N$  as  $k\to \infty$  we have  $\left\{ \begin{array}{l}(\pmb{u}^{(k + 1)}(i) - \pmb{u}^{(k + 1)}(j))\to \mathbf{0}\\ (\pmb{u}^{(k)}(l) - \pmb{u}^{(k)}(i))\to \mathbf{0}\\ (\pmb{u}^{(k)}(l) - \pmb{u}^{(k)}(j))\to \mathbf{0}\\ (\pmb{u}^{(k)}(j) - \pmb{u}^{(k)}(i))\to \mathbf{0} \end{array} \right.$
as  $k \to \infty$ . This is a contradiction since while the LHS of 29 approaches 0, its RHS approaches  $\tilde{\lambda}(\pmb{f}(i) - \pmb{f}(j))$ , which is not 0 in general. Thus, we obtain the conclusion of Proposition 1.
C Derivation of Gradient of  $\mathbf{E}$  as Given in Eqn. 21
Taking the gradient of  $E(\pmb{u}, \pmb{f})$  with respect to  $\pmb{u}$ , we obtain
$$
\nabla_ {\boldsymbol {u}} E = \nabla_ {\boldsymbol {u}} J + \left[ \frac {\partial G}{\partial u _ {1}}, \frac {\partial G}{\partial u _ {2}}, \dots , \frac {\partial G}{\partial u _ {D}} \right] ^ {T}. \tag {30}
$$
The partial derivative  $\partial G / \partial u_j$ ,  $j = 1,2,\ldots ,D$ , is defined through its dot product with an arbitrary function  $h_j\in L^2 (\Omega)$  as follows
$$
\begin{array}{l} \frac {\partial G}{\partial u _ {j}} \cdot h _ {j} (x) = \frac {d}{d \tau} G (u _ {j} + \tau h _ {j}) \big | _ {\tau = 0} \\ = \frac {\lambda}{2} \left(\frac {d}{d \tau} \int_ {\Omega} (u _ {j} (x) - f _ {j} (x) + \tau h _ {j} (x)) ^ {2} d x\right) \Bigg | _ {\tau = 0} \\ = \lambda \int_ {\Omega} \left(u _ {j} (x) - f _ {j} (x)\right) h _ {j} (x) d x. \\ \end{array}
$$
Thus, the Frechet derivative of  $\mathbf{F}$  with respect to  $u_{j}$  is given by
$$
\frac {\partial G}{\partial u _ {j}} = \lambda \left(u _ {j} (x) - f _ {j} (x)\right) \tag {31}
$$
Substituting the formula for  $\partial G / \partial u_{j}$  in Eqn. 31 into Eqn. 30 for  $\nabla_{\pmb{u}}E(\pmb {u},\pmb {f})$  , we obtain the following gradient flow
$$
\frac {d \boldsymbol {u} (x , t)}{d t} = - \nabla_ {\boldsymbol {v}} E (\boldsymbol {u}, \boldsymbol {f}) = - \nabla_ {\boldsymbol {u}} J (\boldsymbol {u}) (x) + \lambda (\boldsymbol {f} (x) - \boldsymbol {u} (x)), \tag {32}
$$
where  $t$  is a dummy time variable and  $-\nabla_{\pmb{u}}J(\pmb{u})$  is defined as in 9.
D Results of Symmetric Setting
In this section, we show that NeuTRENO significantly improves the performance of a symmetric transformer baseline, which utilizes symmetric self-attention. We refer to the DeiT with symmetric attention, defined in 14, as Sym-DeiT and the Sym-DeiT combined with our NeuTRENO method as Sym-NeuTRENO DeiT.
Object classification on Imagenet To further illustrate the advantage of our NeuTRENO method, we compare Sym-NeuTRENO DeiT with the Sym-DeiT baseline on the ImageNet image
Table 5: Top-1 and Top-5 accuracy (%) of Sym-NeuTRENO DeiT vs. Sym-DeiT on the ImageNet classification task. The Sym-NeuTRENO DeiT models significantly outperform the Sym-DeiT in terms of accuracy, indicating the benefit of NeuTRENO method.
Model/Metric	Top-1 Acc (%)	Top-5 Acc (%)
Sym-DeiT	71.14	90.54
Sym-NeuTRENO DeiT	72.07	91.22
Table 6: Single-scale (SS) MIoU and multi-scale (MS) MIoU of the Sym-NeuTRENO DeiT vs. Sym-DeiT. The Sym-NeuTRENO DeiT model is beneficial since they significantly outperform the Sym-DeiT.
Model/Metric	SS M IoU	MS M IoU (%)
Sym-DeiT	35.18	36.00
Sym-NeuTRENO DeiT	35.68	36.39
classification task. Our Sym-NeuTRENO DeiT outperforms the Sym-DeiT baseline, as shown in Table 5. Notably, the Sym-NeuTRENO DeiT achieves higher performance in terms of both top-1 accuracy and top-5 accuracy than Sym-DeiT baseline. These results further confirm the benefits of our proposed NeuTRENO model.
Image Segmentation on ADE20K dataset We also compare the performance of the Segmenter models [56] using the Sym-NeuTRENO DeiT backbone with models using the Sym-DeiT backbone on ADE20K image segmentation [72], as shown in Table 6. The results demonstrate the substantial performance improvements achieved by utilizing the Sym-NeuTRENO DeiT backbone compared to the Sym-DeiT backbone in terms of both single-scale (SS) MIoU and multi-scale (MS) MIoU metrics. This result further validates the advantages of our NeuTRENO models in enhancing image segmentation performance in the symmetric setting.
E Additional Details on the Empirical Analysis in Section 5
In this section, we provide the details for the empirical analysis in Section 5.
E.1 Average Cosine Similarity between Gradient Approximations
To produce the results in Table 4, we derive the approximation for the gradient  $\nabla_{\pmb{u}}J(\pmb{u})$ , from Eqn 9, at time  $t = 0$ :
$$
\nabla_ {\boldsymbol {u}} J (\boldsymbol {u}) = \int_ {\Omega} \left(\boldsymbol {u} (x, 0) - \boldsymbol {u} (y, 0)\right) K (x, y) d y = \int_ {\Omega} \left(\boldsymbol {v} (x) - \boldsymbol {v} (y)\right) K (x, y) d y,
$$
where  $K(x,y) \coloneqq k(x,y) + k(y,x)$ . Using Monte-Carlo approximation for the integral and choosing  $K(x,y) = \exp \bigl (k(x)^T k(y) / \sqrt{D_{qk}}\bigr)$ , the symmetric approximation of the gradient is derived as  $\sum_{j = 1}^{N}\bigl (\pmb {v}(i) - \pmb {v}(j)\bigr)\exp \bigl (\pmb {k}(i)^T\pmb {k}(j) / \sqrt{D_{qk}}\bigr)$ . Otherwise, by choosing  $K(x,y) = \exp \bigl (q(x)^T k(y) / \sqrt{D_{qk}}\bigr)$ , the asymmetric approximation of the gradient is derived as  $\sum_{j = 1}^{N}\bigl (\pmb {v}(i) - \pmb {v}(j)\bigr)\exp \bigl (\pmb {q}(i)^T\pmb {k}(j) / \sqrt{D_{qk}}\bigr)$ . In this analysis, we take the dot product between the symmetric and asymmetric approximation of the gradient  $\nabla_{\pmb{u}}J(\pmb{u})$  and average these dot products over positions. We finally report the average cosine similarity over 1000 training data and 1000 test data, as shown in Table 4.
E.2 Average Value of Function
In order to report the average value of function  $J(\pmb{u})$  in Fig. 3, we follow the process of computing  $J(\pmb{u})$  for 1000 data points for each transformer block. Subsequently, the average value is reported for each layer. This procedure is carried out for both the training and test datasets.
E.3 Over-smoothing Analysis
The average cosine similarity between all pairs of token's representations  $(\pmb{x}_i, \pmb{x}_j)$  in a sequence is computed as
$$
\frac {1}{N (N - 1)} \sum_ {i \neq j} \frac {\boldsymbol {x} _ {i} ^ {T} \boldsymbol {x} _ {j}}{\| \boldsymbol {x} _ {i} \| _ {2} \| \boldsymbol {x} _ {j} \| _ {2}}.
$$
The result is then averaged over 1000 randomly chosen test data in ImageNet and ADE20K. The result is then reported for each layer, as in Fig. 1.
Table 7: Top-1 and Top-5 accuracy (\%) of NeuTRENO DeiT-small vs. DeiT-small on the ImageNet benchmark. The NeuTRENO DeiT-small significantly outperform the DeiT-small in terms of accuracy. We also compare NeuTRENO DeiT-small with DeiT plusFeatScale, a vision transformer model that addresses over-smoothing, showing the advantage of NeuTRENO. The accuracies reported in [59] for DeiT-small and [65] for DeiT-small plusFeatScale, respectively, are in parentheses.
Model/Metric	Top-1 Acc (%)	Top-5 Acc (%)
DeiT-small	79.97 (79.9)	95.05 (95.0)
DeiT-small +FeatScale	79.96 (80.9)	95.06
NeuTRENO DeiT-small	80.68	95.30
Table 8: Accuracy of NeuTRENO vs.Kernel Transformerr on the CIFAR-10 dataset [37]. The NeuTRENO model significantly outperforms the in terms of accuracy.
Model/Metric	Accuracy (%)
Kernel Transformer	75.89
NeuTRENO	76.75
F.1 Object classification on Imagenet with DeiT-small baseline
In this section, we show the advantages of our method when we further scale up the model by doubling the model dimension and the number of heads compared to that of the DeiT-tiny. In particular, the NeuTRENO DeiT-small achieves better results in both Top-1 Accuracy and Top-5 Accuracy, as shown in Table 7. Our method also outperforms DeiT plusFeatScale. Here, we did our best to reproduce the results of DeiT-small plusFeatScale [65]. In Table 7, we include our reproduced results and the results reported in [59] for DeiT-small and [65] for DeiT-small plusFeatScale, respectively.
F.2 Beyond Softmax-Attention
We show that NeuTRENO can be combined with other baseline attention mechanisms other than softmax attention. In particular, our NeuTRENO significantly improves transformer-based models with kernel attention [54, 61], on the CIFAR-10 image classification task [37], as shown in Table 8. This further confirms the benefits of our model. Here, both models share the same configuration regarding training, the model's size, and the model's depth (12 layers).
G Additional Empirical Analysis Results
This section provides extra empirical analysis to further demonstrate the benefits of NeuTRENO models in mitigating over-smoothing.
G.1 Visualizing Attention Matrices
Fig. 4 displays the 3-head attention matrices obtained from layer [1, 6, 12] of both the pre-trained NeuTRENO DeiT-tiny and the DeiT-tiny baseline models, using a random sample from the ImageNet dataset.
G.2 Head Redundancy between Layers
NeuTRENO mitigates head redundancy between layers, particularly in the final transformer layers where over-smoothing is most pronounced. Fig. 5 shows the average cosine similarity of attention matrices between two successive layers, over 1000 randomly sampled data. The trained NeuTRENO DeiT obtains lower cosine similarity than that of the trained DeiT as the model depth increases.
G.3 NeuTRENO Inherently Mitigates Over-smoothing, even without Training the Models
Randomly-initialized NeuTRENO DeiT-tiny significantly reduces the average cosine similarity between token representations of 12-layer randomly-initialized DeiT-tiny model, as shown in Fig. 6, on the Imagenet classification task. This observation highlights the ability of our NeuTRENO models in mitigating over-smoothing.
G.4 Efficiency Analysis
We report the ratios of the floating-point operations per second (FLOPs), the inference memory, and the inference real-time running of NeuTRENO DeiT vs. DeiT per sample on the ImageNet dataset, which are 1.00005, 1.000002, 1.00013, respectively. This indicates that the significant gain in the performance of NeuTRENO does not come with the cost of efficiency.
Figure 4: Plot of attention matrices attained from layer [1, 6, 12] of both the pretrained DeiT-tiny baseline (Left) and the NeuTRENO DeiT-tiny (Right) models, for each head, using a random sample from the Imagenet dataset.
Figure 5: The average cosine similarity of attention matrices between two successive layers, over 1000 randomly sampled data, of the trained NeuTRENO DeiT and trained DeiT models on theImagenet classification task.
Figure 6: The average cosine similarity between token representations of 12-layer randomly-initialized NeuTRENO DeiT and DeiT models, on theImagenet classification task. Here, 1000 data are randomly sampled for the analysis.
G.5 Stability and Significance of NeuTRENO
To further confirm the stability and significance of NeuTRENO's performance, we provide the standard deviations from five runs for both the NeuTRENO and baseline models for each experiment (in the main text) in Tables 9, 10, 11.
Table 9: Means and standard deviations over five runs with different random seeds of models trained on the Imagenet Classification task.
Model/Metric	Top-1 Acc (%)	Top-5 Acc (%)
Softmax DeiT-Tiny	72.17 ± 0.07	91.02 ± 0.04
NeuTRENO DeiT-Tiny	73.01 ± 0.09	91.56 ± 0.05
NeuTRENO Adaptation	72.63 ± 0.07	91.38 ± 0.03
DeiT-Tiny +FeatScale	72.346 ± 0.06	91.22 ± 0.04
NeuTRENO DeiT-Tiny +FeatScale	73.23 ± 0.08	91.73 ± 0.05
Table 10: Means and standard deviations over five runs with different random seeds of models trained on the ADE20K image segmentation task
Metric/Model	Pretrained Softmax DeiT-Tiny	Pretrained NeuTRENO DeiT-Tiny
SS M IoU	35.72 ± 0.57	37.24 ± 0.62
MS M IoU	36.68 ± 0.42	38.06 ± 0.54
Table 11: Means and standard deviations over five runs with different random seeds of models trained on the WikiText-103 language model task.
Metric/Model	Softmax Transformer	NeuTRENO
Valid PPL	33.15 ± 0.07	32.60 ± 0.08
Test PPL	34.29 ± 0.09	33.70 ± 0.07
Table 12: Evaluation of NeuTRENO DeiT-Tiny vs. Softmax DeiT-Tiny on the ImageNet-C (mean corruption error mCE), Imagenet-A (Accuracy), and Imagenet-R (Accuracy) datasets.
Model/Dataset (Metric)	Imagenet-C (mCE)	Imagenet-A (Accuracy %)	Imagenet-R (Accuracy %)
Softmax DeiT-Tiny	71.6	6.9	32.83
NeuTRENO-DeiT-Tiny	70.1	8.2	33.82
Table 13: Ablation study of different values hyperparameter  $\tilde{\lambda}$  of NeuTRENO DeiT-Tiny on the ADE20K Image Segmentation task.
Metric/Model	Baseline	λ=0.1	λ=0.2	λ=0.4	λ=0.5	λ=0.6	λ=0.8	λ=1.0	λ=2.0
SS MIoU	35.72	34.94	35.60	37.54	37.38	37.24	36.71	36.37	26.25
MS MIoU	36.68	35.53	36.45	38.62	38.22	38.06	37.82	37.26	27.2
G.6 Robustness of NeuTRENO
In addition to the standard metrics, we evaluate the robustness of our NeuTRENO model compared to the baseline transformer model, particularly under adversarial examples and for out-of-distribution generalization. Table 12 demonstrates that NeuTRENO DeiT-Tiny is consistently more robust than the DeiT-Tiny baseline on theImagenet-C (common data corruption and perturbations, such as adding noise and blurring the images) [29],Imagenet-A (adversarial examples) [30], andImagenet-R (out of distribution generalization) [28] datasets, which are widely used to test the model's robustness.
G.7 NeuTRENO in Incremental Learning
In an incremental learning setting [34], our 8-layer NeuTRENO achieves  $1.97\%$  higher accuracy on the sentiment classification task [36] than the 8-layer baseline transformer.
G.8 Ablatation study on the choice of  $\tilde{\lambda}$
We also conduct an ablation study on the impact of the hyperparameter. In particular, on the ADE20K image segmentation task, we train NeuTRENO with different values. We summarize our results in Table 13. Our findings reveal that within the range of [0.2, 1], NeuTRENO consistently outperforms the softmax baseline. However, when values become small or big (below 0.2 or above 1, respectively), NeuTRENO's performance declines.
G.9 Scalability of NeuTRENO
To demonstrate the scalability of our proposed model, we conduct additional experiments to show that our NeuTRENO method can effectively mitigate the oversmoothing issue in the BERT-base model. In particular, in Figure 7 (Left), we plot the cosine similarity between token representations across layers of a pre-trained BERT-base model [17] on the SQuAD v1.1 question answering task [48] and observe
Figure 7: The average cosine similarity between token representations of 12-layer trained (Left) and randomly-initialized (Middle) BERT-base and NeuTRENO BERT-base on the SQuAD question answering task. We also plot the same cosine similarity scores for the trained SimCSE and NeuTRENO SimCSE models (Right) on the STS-12 semantic textual similarity task. Here, 1000 and 500 data are randomly sampled for the analysis on the SQuAD and STS-12 datasets, respectively.
the presence of the oversmoothing issue as the model gets deeper, causing tokens to become identical. We then apply NeuTRENO on the same pre-trained BERT model, and without any fine-tuning, we observe a significant reduction in the cosine similarity between token embeddings in each layer (see Figure 7 (Left)), indicating that NeuTRENO effectively mitigates the oversmoothing problem in BERT. Additionally, our NeuTRENO BERT finetuned on the task yields better accuracy than the finetuned BERT (81.39 exact match score and 88.62 F1-score vs. 80.77 exact match score and 88.12 F1-score). Moreover, we have conducted the same analysis for a randomized BERT-base model and a randomized NeuTRENO BERT-base model and obtained the same encouraging results (see Figure 7 (Middle)). These results further suggest that NeuTRENO helps alleviate the over-smoothing issue in large-scale transformer models.
We also obtain additional results and show that our NeuTRENO SimCSE, after fine-tuned on the STS-12 semantic textual similarity task [1], gains a significant improvement over the baseline SimCSE [22], which is also fine-tuned on the same task (77.32% vs. 75.29% Spearman's correlation. Here, the higher correlation, the better). This additional result further verifies that decreasing the cosine dissimilarity between tokens within trained transformer-based models leads to improved empirical performance.