SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In general,  $\pi_i$  can be found by finding the left eigenvector of  $\mathbf{A}$  corresponding to the dominant eigenvalue  $1$ .
From the Lemma 1 and Lemma 2, we see that, for all  $i = 1,2,\dots ,N$
$$
\boldsymbol {u} ^ {(k)} (i) = \mathbb {E} \left[ \mathbf {B} ^ {(k)} (i) \right] = \sum_ {j = 1} ^ {N} \boldsymbol {v} (j) \mathbb {P} \left(\mathbf {B} ^ {(k - 1)} (i) = \boldsymbol {v} (j)\right)\rightarrow \sum_ {j = 1} ^ {N} \pi_ {j} \boldsymbol {v} (j) =: \bar {\boldsymbol {v}}. \tag {20}
$$
as  $k \to \infty$ . This shows that when  $k$  increases,  $\pmb{u}(i)^{(k)}$  converges to a constant vector, indicating that  $\pmb{u}(x)$ , under the dynamic in 9, suffers from over-smoothing.
Figure 2: Our proposed NeuTRENO model adds a proportion of the difference between the values of the first and that of the current layer to the self-attention's output at each layer.
3 NeuTRENO: Mitigating the Over-smoothing in Transformers via Minimizing a Regularized Functional
In Section 2.1, we have shown that self-attention implicitly performs a gradient descent step to minimize the nonlocal functional  $J(\pmb{u})$  in Eqn. 5, which results in the diffusive characteristics of  $\pmb{u}$  and causes the over-smoothing phenomenon in transformers, as proved in Section 2.2. Fortunately, our objective is not to minimize  $J(\pmb{u})$  but the energy/regularized functional  $E(\pmb{u},\pmb{f})$  defined by Eqn. 5. This regularized functional consists of not only  $J(\pmb{u})$  but also the convex fidelity term  $G(\pmb{u},\pmb{f}) = \frac{\lambda}{2}\int_{\Omega}\|\pmb{u}(x) - \pmb{f}(x)\|_2^2 dx$ . This fidelity term aims to preserve the relevant information in the observed noisy signal  $\pmb{f}(x)$  by penalizing solution  $\pmb{u}(x)$  that deviates significantly from  $\pmb{f}(x)$ , thereby mitigating the effects of over-smoothing caused by minimizing  $J(\pmb{u})$ .
In this section, we will derive our Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO) by minimizing the regularized functional  $E(\pmb{u}, \pmb{f})$ . We then provide a theoretical result to prove that NeuTRENO does not suffer from over-smoothing. Recall from Eqn. 5 that  $E(\pmb{u}, \pmb{f})$  is given by
$$
E (\boldsymbol {u}, \boldsymbol {f}) = J (\boldsymbol {u}) + G (\boldsymbol {u}, \boldsymbol {f}) = J (\boldsymbol {u}) + \frac {\lambda}{2} \int_ {\Omega} \sum_ {j = 1} ^ {D} (u _ {j} (x) - f _ {j} (x)) ^ {2} d x
$$
Following a similar derivation as in Section 2.1 (see Appendix C for the detailed derivation), we obtain the following gradient flow when minimizing  $E(\pmb{u}, \pmb{f})$  using gradient descent
$$
\frac {d \boldsymbol {u} (x , t)}{d t} = - \nabla_ {\boldsymbol {u}} E (\boldsymbol {u}, \boldsymbol {f}) = - \nabla_ {\boldsymbol {u}} J (\boldsymbol {u}) - \lambda (\boldsymbol {u} (x) - \boldsymbol {f} (x)), \tag {21}
$$
NeuTRENO-attention is an Euler Discretization of the Gradient Flow Given in 21. Following the similar derivation in Section 2.1, we discretize the gradient flow in Eqn. 21 using the Euler method [21] with step size  $\Delta t(x) = 1 / \int_{\Omega} (k(x,y) + k(y,x)) dy$  and initializing  $\mathbf{u}$  at  $t = 0$  with  $\mathbf{v}(x)$ , i.e.,  $\mathbf{u}(x,0) = \mathbf{v}(x)$ . Choosing  $\lambda = \lambda / \Delta t(x)$ , we obtain the following update
$$
\begin{array}{l} \boldsymbol {u} (x, \Delta t (x)) = \boldsymbol {u} (x, 0) - \Delta t (x) \nabla_ {\boldsymbol {u}} J - \lambda \Delta t (x) (\boldsymbol {u} (x, 0) - \boldsymbol {f} (x)) \\ = \int_ {\Omega} \frac {K (x , y) \boldsymbol {v} (y)}{\int_ {\Omega} K \left(x , y ^ {\prime}\right) d y ^ {\prime}} d y + \tilde {\lambda} \left(\boldsymbol {f} (x) - \boldsymbol {v} (x)\right). \tag {22} \\ \end{array}
$$
We choose the observed noisy signal  $\pmb{f}(x) = \pmb{v}^0(x)$  where  $\pmb{v}^0(x)$  is  $\pmb{v}(x)$  at the first layer in the transformer model. The update in Eqn. 22 becomes
$$
\boldsymbol {u} (x, \Delta t (x)) = \int_ {\Omega} \frac {K (x , y) \boldsymbol {v} (y)}{\int_ {\Omega} K (x , y ^ {\prime}) d y ^ {\prime}} d y + \tilde {\lambda} \left(\boldsymbol {v} ^ {0} (x) - \boldsymbol {v} (x)\right). \tag {23}
$$
Applying the Monte-Carlo method to approximate the integrals in Eqn. 23 and discretizing  $\pmb{u}(x, \Delta t(x))$ ,  $\pmb{v}(x)$ , and  $\pmb{v}^0(x)$  on a 1-D grid, we attain the following new formula for calculating symmetric self-attention:
$$
\boldsymbol {u} (i) = \sum_ {j = 1} ^ {N} \operatorname {s o f t m a x} \left(\boldsymbol {k} (i) ^ {\top} \boldsymbol {k} (j) / \sqrt {D _ {q k}}\right) \boldsymbol {v} (j) + \tilde {\lambda} \left(\boldsymbol {v} ^ {0} (i) - \boldsymbol {v} (i)\right), i = 1, \dots , N. \tag {24}
$$
Table 1: Top-1 and Top-5 accuracy (\%) of NeuTRENO DeiT vs. DeiT on the ImageNet benchmark. We also present the performance of adapting NeuTRENO to the pre-trained DeiT baseline, NeuTRENO Adaptation. In addition, we compare NeuTRENO withFeatScale [65] and incorporate our method withFeatScale model.
Model/Metric	Top-1 Acc (%)	Top-5 Acc (%)
Softmax DeiT	72.17	91.02
NeuTRENO-DeiT	73.01	91.56
NeuTRENO Adaptation	72.63	91.38
DeiT +FeatScale	72.346	91.22
NeuTRENO DeiT +FeatScale	73.23	91.73
Its corresponding asymmetric self-attention is obtained by replacing the key vectors  $\pmb{k}(i)$  with the query vectors  $\pmb{q}(i), i = 1, \dots, N$ , and given by
$$
\boldsymbol {u} (i) = \sum_ {j = 1} ^ {N} \operatorname {s o f t m a x} \left(\boldsymbol {q} (i) ^ {\top} \boldsymbol {k} (j) / \sqrt {D _ {q k}}\right) \boldsymbol {v} (j) + \tilde {\lambda} \left(\boldsymbol {v} ^ {0} (i) - \boldsymbol {v} (i)\right), i = 1, \dots , N. \tag {25}
$$
Leveraging Eqn. 25, we define the Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO) as follows.
Definition 1 (Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO)). Given a set of key and value vectors  $\{\pmb{k}^{\ell}(j),\pmb{v}^{\ell}(j)\}_{j = 1}^{N}$  in each layer  $\ell$ ,  $\ell = 1,\dots,L$ , for each query vector  $\pmb{q}^{\ell}(i)$ ,  $i = 1,\dots,N$ , in the same layer, the self-attention unit at layer  $\ell$  in a Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO) computes the corresponding output vector  $\pmb{u}^{\ell}(i)$  of the query  $\pmb{q}^{\ell}(i)$  by the following attention formula:
$$
\boldsymbol {u} ^ {\ell} (i) = \sum_ {j = 1} ^ {N} \operatorname {s o f t m a x} \left(\boldsymbol {q} ^ {\ell} (i) ^ {\top} \boldsymbol {k} ^ {\ell} (j) / \sqrt {D _ {q k}}\right) \boldsymbol {v} ^ {\ell} (j) + \tilde {\lambda} (\boldsymbol {v} ^ {0} (i) - \boldsymbol {v} ^ {\ell} (i)), i = 1, \dots , N. \tag {26}
$$
where  $\pmb{v}^0 (1),\dots \pmb{v}^0 (N)\in \mathbb{R}^D$  are the value vectors in the first layer of NeuTRENO.
Fig. 2 illustrates the architecture of NeuTRENO.
Proposition 1. The evolution of  $\pmb{u}(x)$  under the dynamic in 21 does not converge to a constant vector.
Proposition 1 indicates that our NeuTRENO mitigates the over-smoothing issue, suggesting the benefit of our method. The proof for Proposition 1 is given in Appendix B.3.
In this section, we empirically demonstrate the advantages of our proposed NeuTRENO approach across various tasks, including ImageNet classification [15], ADE20K image segmentation [73], and language modeling on the WikiText-103 [42]. Our aim to show: (i) NeuTRENO significantly outperforms the transformer baseline with softmax-attention defined in 2 across various tasks; moreover, NeuTRENO surpass FeatScale, a vision transformer that addresses over-smoothing, combining NeuTRENO with FeatScale is beneficial; (ii) the advantages of incorporating our proposed method with pre-trained models. We also demonstrate the benefits of our NeuTRENO in the symmetry setting and we point to Appendix D for the results. Throughout our experiments, we compare the performance of our proposed models with baselines of the same configuration. For additional details regarding datasets, models, and training procedures, please refer to Appendix A.
Object classification on ImageNet. To demonstrate the advantage of our NeuTRENO method, we compare it with the DeiT baseline [59] on the ImageNet image classification task. Our NeuTRENO DeiT surpasses the DeiT baseline, as shown in Table 1. Notably, our NeuTRENO DeiT achieves significantly higher performance in terms of both Top-1 Accuracy and Top-5 Accuracy. We also compare our method with FeatScale [65], a vision transformer model addressing over-smoothing (see Table 1). Our NeuTRENO significantly outperforms FeatScale, and combining NeuTRENO with FeatScale leads to substantial improvements. These results confirm the benefits of our model.
Image Segmentation on ADE20K dataset. To further validate the advantages of our proposed methods, we compare the performance of the Segmenter models [56] using the NeuTRENO DeiT
Table 2: Single-scale (SS) MIoU and multi-scale MIoU (MS) of the NeuTRENO DeiT vs. the DeiT on the ADE20K image segmentation.
Model/Metric	SS M IoU	MS M IoU (%)
Softmax DeiT	35.72	36.68
NeuTRENO DeiT	37.24	38.06
Table 3: Test and valid perplexity (Test PPL and Valid PPL) on WikiText-103 of NeuTRENO compared to the softmax transformer. Our proposed method achieves a significantly better performance PPL than the baseline.
Method/Metric	Valid PPL	Test PPL
Softmax Transformer	33.15	34.29
NeuTRENO	32.60	33.70
Figure 3: The average value of functional  $J(\pmb{u})$  over 1000 training (Left) samples and test (Right) samples. When softmax attention is applied, the functional decreases as the depth of the trained DeiT increases. and DeiT backbones the on ADE20K image segmentation task [72], as shown in Table 2. The results demonstrate the substantial performance improvements achieved by utilizing the NeuTRENO DeiT backbone over the DeiT backbone, in terms of both single-scale (SS) MIoU and multi-scale (MS) MIoU metrics. These results strongly emphasize the effectiveness of our NeuTRENO approach in enhancing image segmentation performance.
Language Model on WikiText-103. In addition to computer vision tasks, we also evaluate the effectiveness of our model on a large-scale natural language processing application, specifically language modeling on WikiText-103. Our NeuTRENO language model demonstrates better performance in terms of both test perplexity and valid perplexity when compared to the softmax transformer language model [68]. These findings, combined with the results obtained across various tasks, empirically confirm the significant benefits of our NeuTRENO models.
Combine with pre-trained models. Furthermore, our proposed method is also beneficial to combine with pre-trained models. To empirically demonstrate that we incorporate NeuTRENO with pre-trained DeiT and fine-tune on the ImageNet dataset with one-third number of epochs that are used in training. The result is presented in Table 1, showing that combined with our method improves both the Top-1 and Top-5 accuracies of the pre-trained models.
Applying Softmax-Attention Reduces the functional  $J(u)$ . We present evidence supporting that the employment of softmax attention minimizes the functional  $J(u)$ . Initially, we observe that the average cosine similarity between the numerical approximation of  $\nabla_{\boldsymbol{u}}J(\boldsymbol{u})$  using symmetric or asymmetric kernel  $K(x,y)$  for both the trained Sym-DeiT (using symmetric self-attention 14) and DeiT models, closed 1, as shown in Table 4. This suggests that reversing the direction of the asymmetric approximation effectively decreases  $J(u)$ . Considering that softmax attention takes steps in this reversed direction numerically, its application leads to a reduction in  $J(u)$ . This is further substantiated by Fig. 3, which demonstrates a decrease in  $J(u)$  as the depth of the trained DeiT increases when softmax attention is employed. More details of this analysis are in Appendix E
Over-smoothing Analysis. We empirically illustrate the effectiveness of NeuTRENOs in mitigating the over-smoothing problem in transformers. Fig. 1 compares the cosine similarity between token representations across layers for both NeuTRENO and softmax baseline models, specifically focusing on the Imagenet classification task (Left) and ADE20K image segmentation (Right). The token
Table 4: The average cosine similarity between the numerical approximation of  $\nabla J(\pmb{u})(x)$  using symmetric or asymmetric kernel  $K(x,y)$ , for the trained Sym-DeiT and softmax DeiT models. The metric is evaluated on 1000 training and 1000 test data samples. The average score close to 1 shows a strong alignment between symmetric and asymmetric gradient approximations, suggesting that reversing the direction of the asymmetric approximation effectively reduces the functional  $J(\pmb{u})$ .
Model	Training data	Test data
Sym-DeiT	0.982	0.976
Softmax DeiT	0.973	0.964
features extracted by NeuTRENOs exhibit significantly lower similarity, particularly in the final layers. This finding highlights the ability of NeuTRENOs to address the over-smoothing issue and improve the diversity of token representations. We provide more details of this analysis in Appendix E.
Over-smoothing in Transformers. Over-smoothing in deep transformers has been observed in various domains and applications from natural language processing [55] to computer vision [65, 18]. In vision tasks, [74] observes that the performance of the vision transformer (ViT [20]) quickly saturates as more layers are added to the model. Moreover, experiments in [74] show that the 32-layer ViT underperforms the 24-layer ViT, indicating the difficulty of ViTs in gaining benefits from deeper architectures. The authors point out that over-smoothing results in this phenomenon by causing the token representations to become identical when the model grows deeper. Based on this observation, the authors propose a cross-head communication method that helps enhance the diversity of both token representations and attention matrices. Furthermore, it has been shown in [60] that the training of ViT models encounters instability with greater depths. [25] proposes that this instability arises from the over-smoothing, where token representation for patches within an image becomes progressively alike as the model's depth increases. To explain this issue, [65] finds out that self-attention acts as a low-pass filter and smoothens the token representations in ViTs. This leads to the proposal of the FeatScale method [65], which regulates feature frequencies, whether low or high, to counteract the consequences of over-smoothing.
In addition, [55] observes the phenomenon in BERT [16], a deep language model, and explores over-smoothing through the graph perspective. The work utilizes hierarchical fusion strategies by preserving the output of self-attention through all layers, which is memory-costly. On the other hand, [65, 18] investigate over-smoothing in the image domain through the lens of Fourier spectrum, showing that self-attention are low-pass filters, retaining only low-frequency, causing over-smoothed outputs. Our work is an orthogonal explanation of the previous work. We focus on developing a variational denoising framework to understand the self-attention of transformers as a gradient descent approximation of a functional. Our new finding explains the over-smoothing issue of transformers due to self-attention minimizing a functional and inspires us to derive the novel NeuTRENO method to overcome over-smoothing.
Nonlocal Functionals for Image Processing. Total variation [51] is well-known as an image-denoising technique. It denoises a noisy image by solving a constraint optimization problem. The method is also related to PDE-flow-based image-denoising techniques [24], namely isotropic and anisotropic diffusion [67] models. The method is edge preserving, meaning to avoid over-blurring edges' information [7]. Nonlocal functionals [35, 24] is considered as an extension of total variation to a nonlocal scale. Nonlocal functional and edge preservation properties are the motivation of our work to explain and overcome over-smoothing in transformers.
In this paper, we establish a nonlocal variational denoising framework for self-attention. From this variational perspective, we explain over-smoothing in self-attention, which hinders the representation capacity of transformer models. We also derive the novel Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO) to alleviate the over-smoothing. We empirically verify the benefits of NeuTRENO with a wide range of large-scale applications including ImageNet object classification, ADE20K object segmentation, and WikiText-103 language modeling. A limitation of our paper is that the privacy-preserving of NeuTRENO has not been addressed. It is interesting to explore if regularized nonlocal functional can also help improve the privacy-preserving of transformer models. We leave this exciting research idea as future work.
RGB acknowledges support from the NSF grants CCF-1911094, IIS-1838177, and IIS-1730574; ONR grants N00014-18-12571, N00014-20-1-2534, and MURI N00014-20-1-2787; AFOSR grant FA9550-22-1-0060; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047. TMN acknowledges support from his start-up grant at the National University of Singapore (Grant Number: A-0009807-00-00).
Supplement to "Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals"
A Additional Details on the Experiments in Section 4 16
A.1 Image Classification on Imagenet 17
A.2 Image Segmentation on ADK20 dataset 17
A.3 Language Modeling on WikiText-103 17
B Technical Proofs 17
B.1 Proof of Lemma 1 17
B.2 Proof of Lemma 2 18
B.3 Proof of Proposition 1 18
C Derivation of Gradient of  $\mathbf{E}$  as Given in Eqn. 21 19
D Results of Symmetric Setting 19
E Additional Details on the Empirical Analysis in Section 5 20
E.1 Average Cosine Similarity between Gradient Approximations 20
E.2 Average Value of Function 20
E.3 Over-smoothing Analysis 20
F Additional Experimental Results 21
F.1 Object classification on Imagenet with DeiT-small baseline 21
F.2 Beyond Softmax-Attention 21
G Additional Empirical Analysis Results 21
G.1 Visualizing Attention Matrices 21
G.2 Head Redundancy between Layers 21
G.3 NeuTRENO Inherently Mitigates Over-smoothing, even without Training the Models 21
G.4 Efficiency Analysis 21
G.5 Stability and Significance of NeuTRENO 22
G.6 Robustness of NeuTRENO 23
G.7 NeuTRENO in Incremental Learning 23
G.8 Ablatation study on the choice of  $\tilde{\lambda}$  23
G.9 Scalability of NeuTRENO 23
A Additional Details on the Experiments in Section 4
This section provides datasets, models, and training details for experiments in Section 4. The code to reproduce our experimental results is included in our Supplementary Material submission.
A.1 Image Classification on Imagenet
Datasets and Metrics. The ImageNet dataset [15, 52] comprises 1.28 million training images and 50,000 validation images, encompassing the classification of 1000 categories. The evaluation metrics used for performance assessment are the top-1 and top-5 accuracies.
Models and Baselines. Our baseline model is the DeiT-tiny model [59], which consists of 12 transformer layers, 3 attention heads per layer, and a model dimension of 192. For model setting and setting and configuration, we follow [59]. Their implementation is available at https://github.com/facebookresearch/deit. The  $\tilde{\lambda}$  used for our NeuTRENO method is 0.6.
A.2 Image Segmentation on ADK20 dataset
Datasets and Metrics. The ADE20K dataset is recognized for its inclusion of challenging scenes with fine-grained labels, making it one of the most demanding semantic segmentation datasets. The training set consists of 20,210 images encompassing 150 semantic classes. Additionally, there are 2,000 images in the validation set and 3,352 images in the test set. This in task the Single-scale mean Intersection over Union (SS mIoU) and the Multi-scale (MS mIoU).
Models and baselines. The training configuration and setting for our models are followed by [56]. The baseline model is finetuned with the pretrained DeiT-tiny backbone while our segmenter model used the pretrained NeuTRENO DeiT-tiny, with  $\tilde{\lambda} = 0.6$ .
A.3 Language Modeling on WikiText-103
Datasets and Metrics. The WikiText-103 dataset consists of articles extracted from Wikipedia and is specifically designed to capture long contextual dependencies. The training set comprises approximately 28,000 articles, totaling 103 million running words. Each article contains text blocks consisting of approximately 3,600 words. The validation and test sets contain 218,000 and 246,000 running words, respectively, with each set consisting of 60 articles and approximately 268,000 words. Our experiment follows the standard setting [42, 53], which involves dividing the training data into independent long segments of  $L$  words. For evaluation, we employ a batch size of 1 and process the text sequence using a sliding window of size  $L$ . When computing perplexity (PPL), we consider only the last position, except for the first segment where all positions are evaluated, following the approach in [2, 53].
Models and baselines. For our language modeling implementation, we rely on the publicly available code https://github.com/IDSIA/lmtool-fwp developed by [53]. In our experiments, we set the dimensions of keys, values, and queries to 128, while the training and evaluation context length is set to 256. In this experiment,  $\tilde{\lambda} = 0.4$  yields the best performance of NeuTRENO language model.
For all  $i = 1, \ldots, N$ , we have  $\mathbb{E}[\mathbf{B}^{(0)}(i)] = \pmb{v}(i)$ . Assume that  $\mathbb{E}[\mathbf{B}^{(k)}(i)] = \pmb{u}^{(k)}(i)$ , then
$$
\begin{array}{l} \mathbb {E} [ \mathbf {B} ^ {(k + 1)} (i) ] = \sum_ {j = 1} ^ {N} \boldsymbol {v} (j) \mathbb {P} (\mathbf {B} ^ {(k + 1)} (i) = \boldsymbol {v} (j)) \\ = \sum_ {j = 1} ^ {N} \boldsymbol {v} (j) \sum_ {l = 1} ^ {N} \mathbb {P} (\mathbf {B} ^ {(k + 1)} (i) = \boldsymbol {v} (j) | \mathbf {B} ^ {(1)} (i) = \boldsymbol {v} (l)) \mathbb {P} (\mathbf {B} ^ {(1)} (i) = \boldsymbol {v} (l)) \\ = \sum_ {j = 1} ^ {N} \boldsymbol {v} _ {j} \sum_ {l = 1} ^ {N} \mathbb {P} (\mathbf {B} ^ {(k)} (l) = \boldsymbol {v} (j)) \mathbb {P} (\mathbf {B} ^ {(1)} (i) = \boldsymbol {v} (l) | \mathbf {B} ^ {(0)} (i) = \boldsymbol {v} (i)) \\ = \sum_ {j = 1} ^ {N} \boldsymbol {v} (j) \sum_ {l = 1} ^ {N} \mathbf {A} _ {i l} \mathbb {P} (\mathbf {B} ^ {(k)} (l) = \boldsymbol {v} (j)) \\ = \sum_ {l = 1} ^ {N} \mathbf {A} _ {i l} \mathbb {E} [ \mathbf {B} ^ {(k)} (l) ] = \sum_ {l = 1} ^ {N} \mathbf {A} _ {i l} \boldsymbol {u} ^ {(k)} (l) \\ = \boldsymbol {u} ^ {(k + 1)} (i). \\ \end{array}
$$
Thus, by induction, we obtain the conclusion of the lemma.
Since the transition matrix  $\mathbf{A} \in \mathbb{R}^{N \times N}$  is right-stochastic, its largest eigenvalue is 1 (see Theorem 4.1 in [5]). Also,  $\mathbf{A}$  is a regular positive matrix since its elements are positive. Thus, the Perron-Frebenius theorem [8] implies the existence of a unique probability distribution  $\pi$ , which is a positive left eigenvector of the transition matrix  $\mathbf{A}$  associated with its largest eigenvalue 1. In particular, in the case of symmetricity constraint,  $\pi$  can be chosen as follows
$$
\boldsymbol {\pi} = \left(\frac {d _ {1}}{\sum_ {j = 1} ^ {N} d _ {j}}, \frac {d _ {2}}{\sum_ {j = 1} ^ {N} d _ {j}}, \dots , \frac {d _ {n}}{\sum_ {j = 1} ^ {N} d _ {j}}\right),
$$
where  $d_{i} = \sum_{j = 1}^{N}\exp \Bigl {(}\pmb {k}(i)^{\top}\pmb {k}(j) / \sqrt{D_{qk}}\Bigr)$ . It is easy to see that
$$
\begin{array}{l} \sum_ {i = 1} ^ {N} \pi_ {i} \mathbf {A} _ {i j} = \sum_ {i = 1} ^ {N} \frac {d _ {i}}{\sum_ {l = 1} ^ {N} d _ {l}} \frac {\exp \left(\boldsymbol {k} (i) ^ {\top} \boldsymbol {k} (j) / \sqrt {D _ {q k}}\right)}{d _ {i}} \\ = \frac {\sum_ {i = 1} ^ {N} \left(\exp \left(\boldsymbol {k} (i) ^ {\top} \boldsymbol {k} (j) / \sqrt {D _ {q k}}\right)\right)}{\sum_ {l = 1} ^ {N} d _ {l}} \\ = \frac {d _ {j}}{\sum_ {l = 1} ^ {N} d _ {l}} = \pi_ {j}. \\ \end{array}
$$
As a consequence,  $\pi$  must be the unique stationary distribution of the random walk  $\{\mathbf{B}^{(k)}(i)\}_{k\in K}$ . This concludes the proof.