SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Extension to Other SSL Methods. As replacing the 2-factor contrast  $f(x)^{\top}f(x^{\prime})$  with the 3-factor contrast  $f(x)^{\top}Sf(x^{\prime})$  is a simple operation and not constrained to the special form of spectral contrastive loss, we extend triCL to other contrastive frameworks. We first take another representative contrastive learning method SimCLR [4] as an example. Comparing the InfoNCE loss in SimCLR and the spectral contrastive loss, we find they are quite similar and the only difference is that they push away the negative pairs with different loss functions ( $l_{2}$  loss v.s. logsumexp loss). So we propose the tri-InfoNCE loss by changing the terms of negative samples in triCL (Eq. 9) to a tri-logsumexp term, i.e.,
$$
\mathcal {L} _ {\mathrm {t r i N C E}} (f) = - \mathbb {E} _ {x, x ^ {+}} \log \frac {\exp (f (x) ^ {\top} S f (x ^ {+}))}{\mathbb {E} _ {x ^ {-}} \exp (f (x) ^ {\top} S f (x ^ {-}))} + \| \mathbb {E} _ {x} f (x) f (x) ^ {\top} - I \| ^ {2}. \tag {16}
$$
Besides the InfoNCE loss used in SimCLR, we present more types of tri-factor contrastive loss in Appendix C. In summary, triCL can be applied in most of the contrastive frameworks by replacing the  $f(x)^{\top}f(x^{\prime})$  term with a tri-term  $f(x)^{\top}Sf(x^{\prime})$  when calculating the similarity, where the importance matrix  $S$  is a diagonal matrix capturing the feature importance.
Extension to the Multi-modal Domain. Different from the symmetric contrastive learning frameworks like SCL [18] and SimCLR [4], the multi-modal contrastive frameworks like CLIP [28] have asymmetric networks to encode asymmetric image-text pairs. So we extend the tri-factor contrastive loss to a unified asymmetric form:
$$
\begin{array}{l} \mathcal {L} _ {\text {t r i C L I P}} \left(f _ {A}, f _ {B}, S\right) = - 2 \mathbb {E} _ {x _ {a}, x _ {b}} f _ {A} \left(x _ {a}\right) ^ {\top} S f _ {B} \left(x _ {b}\right) + \mathbb {E} _ {x _ {a} ^ {-}, x _ {b} ^ {-}} \left(f _ {A} \left(x _ {a} ^ {-}\right) ^ {\top} S f _ {B} \left(x _ {b} ^ {-}\right)\right) ^ {2} \tag {17} \\ + \left\| \mathbb {E} _ {x _ {a}} f _ {A} (x _ {a}) f _ {A} (x _ {a}) ^ {\top} - I \right\| ^ {2} + \left\| \mathbb {E} _ {x _ {b}} f _ {B} (x _ {b}) f _ {B} (x _ {b}) ^ {\top} - I \right\| ^ {2}. \\ \end{array}
$$
where  $f_{A}, f_{B}$  are two different encoders with the same output dimension  $k$ . Uniformly, we denote that the positive pairs  $(x_{a}, x_{b}) \sim \mathcal{P}_{O}$  and the negative samples  $x_{a}^{-}, x_{b}^{-}$  are independently drawn from  $\mathcal{P}_{A}, \mathcal{P}_{B}$ . With different concrete definitions of  $\mathcal{P}_{O}, \mathcal{P}_{A}, \mathcal{P}_{B}$ , different representation learning paradigms can be analyzed together. For single-modal contrastive learning, the symmetric process in Section 3 is a special case of the asymmetric objective. For multi-modal learning, we denote  $\mathcal{P}_{A}, \mathcal{P}_{B}$  as the distributions of different domain data and  $\mathcal{P}_{O}$  as the joint distribution of semantically related pairs (e.g., semantically similar image-text pairs in CLIP). We theoretically prove that the asymmetric tri-factor contrastive learning still obtains the exact feature identifiability and guaranteed generalization performance in Appendix A.
In this section, we provide empirical evidence to support the effectiveness of tri-factor contrastive learning. In section 6.1, we empirically verify that the tri-factor contrastive learning can enable the exact feature identifiability on the synthetic dataset. In section 6.2, we empirically analyze the properties of the importance matrix and demonstrate the advantages of automatic feature importance discovery on various tasks. Furthermore, in section 6.3, we empirically compare the generalization performance of tri-factor contrastive learning with different baselines including SimCLR [4] and spectral contrastive learning (SCL) [18] on CIFAR-10, CIFAR-100 and ImageNet-100.
6.1 The Verification of the Identifiability on the Synthetic Dataset
Due to the randomness of optimization algorithms like SGD, we usually can not obtain the optimal encoder of the contrastive loss. So we consider verifying the feature identifiability on the synthetic dataset. To be specific, we first construct a random matrix  $\bar{A}$  with size  $5000 \times 3000$  to simulate the augmentation graph. Then we compare two different objectives:  $\| \bar{A} - FG^{\top} \|_F^2$ ,  $\| \bar{A} - FSG^{\top} \|_F^2$ , where  $F$  is a matrix with size  $5000 \times 256$ ,  $G$  is a matrix with size  $3000 \times 256$  and  $S$  is a diagonal with size  $256 \times 256$ . With the analysis on [18], these two objectives are respectively equivalent to the spectral contrastive loss and tri-factor contrastive loss. According to the Eckart-Young Theorem [12], we obtain the optimal solutions of them with the SVD algorithms. For two objectives, we respectively obtain 10 optimal solutions and we calculate the average Euclidean distance between 10 solutions. We investigate that the optimal solutions of the trifactorization objective are equal (average distance is 0) while there exist significant differences between optimal solutions of bifactorization (average distance is 101.7891 and the variance of distance is 17.2798), which empirically verifies the optimal solutions of contrastive learning obtain the freedom while triCL can remove it. More details can be found in Appendix B.
6.2 The Automatic Discovery of Feature Importance
Except for the visualization example in Figure 1, we further quantitatively explore the properties of ordered representations learned by triCL.
The Distribution of the Importance Matrix. We first observe the distribution of the diagonal values in the learned importance matrix  $S$ . Specifically, we pretrain the ResNet-18 on CIFAR-10, CIFAR-100 and ImageNet-100 [8] by triCL. Then we normalize the importance matrices and ensure the sum of the diagonal values is 1. In Figure 2(a), we present the diagonal values of the importance
(a) The distributions of diagonal values on the im-(b) K-NN accuracy on adjacent 10 dimensions or importance matrix learned on different datasets. ordered by the importance matrix learned on CIFAR-10, CIFAR-100 and ImageNet-100.
Figure 2: The distributions of the discovered feature importance and the verifications on whether the order of feature importance indicated by the importance matrix is accurate.
matrices trained on different datasets and we find they are highly related to the properties of the datasets. For instance, CIFAR-100 and ImageNet-100, which possess a greater diversity of features, exhibit more active diagonal values in the top 50 dimensions compared to CIFAR-10. While for the smallest diagonal values, the different importance matrices are quite similar as most of them represent meaningless noise.
The K-NN Accuracy on Selected Dimensions. With the ResNet-18 and the projector pretrained on CIFAR-10, CIFAR-100 and ImageNet-100, we first sort the dimensions according to the importance matrix  $S$ . Then we conduct the k-NN evaluation for every 10 dimensions. As shown in Figure 2(b), the k-NN accuracy decreases slowly at first as these dimensions are related to the ground-truth labels and they share a similar significance. Then the k-NN accuracy drops sharply and the least important dimensions almost make no contributions to clustering the semantically similar samples. Note that the k-NN accuracy drops more quickly on CIFAR-10, which is consistent with the fact it has fewer types of objects.
Linear Evaluation on Selected Dimensions. We first train the ResNet-18 with triCL and spectral contrastive learning (SCL) on CIFAR-10. Then we select 20 dimensions from the learned representations (containing both the backbone and the projector). For triCL, we sort the dimensions according to the descending order of the importance matrix and select the largest 20 dimensions (1-20 dimensions), middle 20 dimensions (119-138 dimensions), and smallest 20 dimensions (237-256 dimensions). And for SCL, we randomly choose 20 dimensions. Then we train a linear classifier following the frozen representations with the default settings of linear probing. As shown in Figure 3(a), the linear accuracy decreases in the descending order of dimensions in triCL and the linear accuracy of the largest 20 dimensions of triCL is significantly higher than random 20 dimensions of SCL, which verifies that triCL can discover the most important semantic features related to the ground-truth labels.
Image Retrieval. We conduct the image retrieval on ImageNet-100. For each sample, we first encode it with the pretrained networks and then select dimensions from the features. For the methods learned by triCL, we sort the dimensions according to the values in the importance matrix  $S$  and select the largest dimensions. For SCL, we randomly choose the dimensions. Then we find 100 images that have the largest cosine similarity with the query image and calculate the mean average precision (mAP) that returned images belong to the same class as the query ones. As shown in Figure 3(b), we observe the 50 dimensions of triCL show the comparable performance as the complete representation while the performance of SCL continues increasing with more dimensions, which further verifies that triCL can find the most important features that are useful in downstream tasks.
Out-of-Distribution Generalization. Besides the in-domain downstream tasks, we also examine whether the importance matrix can sort the feature importance accurately with out-of-domain shifts. To be specific, we pretrain the ResNet-18 with SCL and triCL on ImageNet-100 and then conduct
(a) Linear evaluation results of selected dimensions of triCL and SCL on CIFAR-10.
(b) The mean average precision in the image retrieval on the selected dimension of triCL and SCL on ImageNet-100.
Figure 3: With the automatic discovery of feature importance, Tri-factor contrastive learning (triCL) obtains superior performance than spectral contrastive learning (SCL) on downstream tasks that use selected representations.
(c) Transfer accuracy of selected dimensions of triCL and SCL on stylized ImageNet-100.
Table 1: The linear probing accuracy and finetune accuracy of ResNet-18 pretrained by tri-factor contrastive learning and self-supervised baselines on CIFAR-10, CIFAR-100, ImageNet-100. TriCL averagely obtains a comparable performance as the original contrastive learning. Moreover, triCL achieves significant improvements on some tasks, such as finetuning on CIFAR-100.
Linear Accuracy	Finetune Accuracy
CIFAR10	CIFAR100	ImageNet100	CIFAR10	CIFAR100	ImageNet100
SCL	88.4	60.3	72.3	92.3	66.1	76.4
tri-SCL	88.3	60.8	71.2	92.7	67.8	76.3
SimCLR	87.9	60.2	73.5	92.2	72.5	75.9
tri-SimCLR	87.9	59.8	74.1	92.3	73.3	75.6
linear probing with 40 dimensions of learned representations on the out-of-domain dataset stylized ImageNet-100 [13]. During the downstream evaluation process, we sort the dimensions according to the descending order of the importance matrix and select the largest 40 dimensions (1-40 dimensions), middle 40 dimensions (237-276 dimensions), and smallest 40 dimensions (473-512 dimensions) of the features learned by triCL. And for SCL, we randomly select 40 dimensions. As shown in Figure 3(c), we observe that the transfer accuracy of the most important features learned by triCL shows significantly superior performance, which shows the robustness and effectiveness of the importance matrix.
6.3 Transfer Learning on Benchmark Datasets
**Setup.** During the pretraining process, we utilize ResNet-18 [19] as the backbone and train the models on CIFAR-10, CIFAR-100 and ImageNet-100 [8]. We pretrain the model for 200 epochs On CIFAR-10, CIFAR-100, and for 400 epochs on ImageNet-100. We select two self-supervised methods as our baselines and apply tri-factor contrastive learning on them, including spectral contrastive learning (SCL) [18], SimCLR [4]. When implementing the tri-factor contrastive learning, we follow the default settings of the baseline methods. During the evaluation process, we consider two transfer learning tasks: linear evaluation and finetuning. During the linear evaluation, we train a classifier following the frozen backbone pretrained by different methods for 50 epochs. During the finetuning process, we train the whole network (including the backbone and the classifier) for 30 epochs.
Results. As shown in Table 1, we find that triCL shows comparable performance as the original contrastive learning methods in transfer learning on different real-world datasets, which is consistent with our theoretical analysis. Meanwhile, we observe that triCL shows significant improvements on some tasks, e.g., triCL improves the finetune accuracy of SCL by  $1.7\%$  on CIFAR-100. Compared to the original contrastive learning methods, the representations learned by triCL keep comparable transferability with stronger identifiability and interpretability.
In this paper, we propose a new self-supervised paradigm: tri-factor contrastive learning (triCL) which replaces the traditional 2-factor contrast with a 3-factor form  $z_{x}^{\top} S z_{x}^{\prime}$  when calculating the similarity in contrastive learning, where  $S$  is a learnable diagonal matrix named importance matrix. With the importance matrix  $S$ , triCL enables the exact feature identifiability. Meanwhile, the diagonal values in the importance matrix reflect the importance of different features learned by triCL, which means triCL obtains the ordered representations. Moreover, we theoretically prove that the generalization performance of triCL is guaranteed. Empirically, we verify that triCL achieves the feature identifiability, automatically discovers the feature importance and achieves the comparable transferability as current contrastive learning methods.
[1] Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. arXiv preprint arXiv:2106.09943, 2021.
[2] Han Bao, Yoshihiro Nagano, and Kento Nozawa. Sharp learning bounds for contrastive unsupervised representation learning. arXiv preprint arXiv:2110.02501, 2021.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[5] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[6] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
[7] Jingyi Cui, Weiran Huang, Yifei Wang, and Yisen Wang. Rethinking weak supervision in helping contrastive learning. In ICML, 2023.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[9] Chris Ding, Tao Li, Wei Peng, and Haesun Park. Orthogonal nonnegative matrix t-factorizations for clustering. In SIGKDD, 2006.
[10] Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. arXiv preprint arXiv:2010.14407, 2020.
[11] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021.
[12] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218, 1936.
[13] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231, 2018.
[34] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
[35] Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR, 2023.
[36] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
[37] Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In NeurIPS, 2022.
[38] Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML, 2023.
[39] Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. In ICLR, 2023.
A.1 Proof of Theorem 4.5
Proof. We first rewrite  $\mathcal{L}_{\mathrm{tri}}$  as a matrix decomposition objective
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {t r i}} (f) = - 2 \mathbb {E} _ {x, x ^ {+}} f (x) ^ {\top} S f (x ^ {+}) + \mathbb {E} _ {x} \mathbb {E} _ {x ^ {-}} \left(f (x) ^ {\top} S f (x ^ {-})\right) ^ {2} \\ = \sum_ {x, x ^ {\prime}} \left(\frac {A _ {x x ^ {\prime}} ^ {2}}{D _ {x x} D _ {x ^ {\prime} x ^ {\prime}}} + D _ {x x} D _ {x ^ {\prime} x ^ {\prime}} \left(f (x) ^ {\top} S f \left(x ^ {\prime}\right)\right) ^ {2} - 2 A _ {x x ^ {\prime}} f (x) ^ {\top} S f \left(x ^ {\prime}\right)\right) + c o n s t \tag {18} \\ = \left\| \bar {A} - F S F ^ {\top} \right\| ^ {2}. \\ \end{array}
$$
According to the Eckart-Young Theorem [12], the optimal solutions  $F^{\star}, S^{\star}$  satisfy
$$
F ^ {\star} S ^ {\star} \left(F ^ {\star}\right) ^ {\top} = U ^ {k} \Sigma \left(V ^ {k}\right) ^ {\top},
$$
where  $\Sigma \in \mathbb{R}^{k\times k}$  is a diagonal matrix with the  $k$ -largest eigenvalues of  $\bar{A}$  and  $U\in \mathbb{R}^{N\times k}$  contains the corresponding eigenvectors of the  $k$ -largest eigenvalues. When the regularizer  $\mathcal{L}_{\mathrm{Dec}}$  is minimized,  $F^{\star}$  satisfy  $(F^{\star})^{\dagger}F^{\star} = I$ . In the next step, we prove the uniqueness of the optimal solution.
We denote  $H = F^{\star}\Sigma (F^{\star})^{\top}$ . As  $(F^{\star})^{\top}F^{\star} = I$ , we obtain  $HH^{\top} = F^{\star}S^{\star}(S^{\star})^{\top}(F^{\star})^{\top}$ . If  $\zeta, \sigma$  are a pair of eigenvector and eigenvalue of  $HH^{\top}$ , we have
$$
H H ^ {\top} \zeta = F ^ {\star} S ^ {\star} (S ^ {\star}) ^ {\top} (F ^ {\star}) ^ {\top} \zeta = \sigma \zeta ,
$$
$$
S ^ {\star} \left(S ^ {\star}\right) ^ {\top} \left(F ^ {\star}\right) ^ {\top} \zeta = \sigma \left(F ^ {\star}\right) ^ {\top} \zeta , \tag {19}
$$
$$
S ^ {\star} \left(S ^ {\star}\right) ^ {\top} \left(\left(F ^ {\star}\right) ^ {\top} \zeta\right) = \sigma \left(\left(F ^ {\star}\right) ^ {\top} \zeta\right).
$$
So the eigenvalues of  $HH^{\top}$  are the eigenvalues of  $S^{\star}(S^{\star})^{\top}$ . As the positive eigenvalues of  $HH^{\top}$  are uniquely determined and  $S^{\star}$  has a descending order,  $S^{\star}$  is also determined and  $S^{\star} = \sum$ .
We note that  $HH^{\top} = F^{\star}S^{\star}(S^{\star})^{\top}(F^{\star})^{\top}$ , i.e.,  $HH^{\top}F^{\star} = F^{\star}S^{\star}(S^{\star})^{\top}$ , which means that the  $k$  columns of  $F^{\star}$  are the eigenvectors of  $HH^{\top}$  and the corresponding eigenvalues are  $\sigma_{1}\dots \sigma_{k}$ . As  $HH^{\top}$  only has  $k$  different non-negative eigenvalues  $\sigma_{1},\dots ,\sigma_{k}$ , the eigenspace of each eigenvalue is one-dimensional. When we consider the real number space, any two eigenvectors  $\zeta_i,\zeta_i'$  of the same eigenvalue  $\sigma_{i}$  satisfy  $\zeta_i = c\zeta_i'$ . As  $(F^{\star})^{\top}F^{\star} = I$ , we obtain  $c = \pm 1$ . As  $f(x) = \frac{1}{\sqrt{D_{xx}}} F_x$ , we obtain
$$
f _ {j} ^ {\star} (x) = \pm \frac {1}{\sqrt {D _ {x x}}} \left(U _ {x} ^ {k}\right) _ {j}, S ^ {*} = \operatorname {d i a g} \left(\sigma_ {1}, \dots , \sigma_ {k}\right), \tag {20}
$$
Proof. The following analysis mainly follows the proofs in [37].
We denote  $d_x$  as the  $x$ -th row of  $D$ . And we denote  $F_m$  as the matrix composed of encoder features on selected dimensions, i.e.,  $(F_m)_x = \sqrt{d_x} f^{(m)}(x)$ . Recall that  $A_{x,x^+} = \mathcal{A}(x,x^+) = \mathbb{E}_{\bar{x} \sim \mathcal{P}_u}[\mathcal{A}(x|\bar{x})\mathcal{A}(x^+|\bar{x})]$  and  $\bar{A}$  is the normalized form of  $A$ , i.e.,  $\bar{A}_{x,x^+} = \frac{\mathcal{A}(x,x^+)}{\sqrt{d_x \cdot d_{x^+}}}$ . Then we reformulate the downstream error,
$$
\begin{array}{l} \mathbb {E} _ {x, y} \| y - W _ {f} f ^ {(m)} (x) \| ^ {2} = \sum_ {(x, y _ {x})} d _ {x} \| y _ {x} - W _ {f} f ^ {(m)} (x) \| ^ {2} \\ = \left\| D ^ {1 / 2} Y - F _ {m} W _ {f} \right\| ^ {2} \\ = \| D ^ {1 / 2} Y - \bar {A} C + \bar {A} C - F _ {m} W _ {f} \| ^ {2}, \\ \end{array}
$$
where  $C_{x,j} = \sqrt{(d_i)\mathbb{1}_{y_x = j}}$ . Then we consider the relationship between the downstream error and the augmentation graph, we element-wise consider the matrix  $(D^{1 / 2}Y - \bar{A} C)$ ,
$$
\left(D ^ {1 / 2} Y\right) _ {x, j} = \sqrt {\left(d _ {x}\right) \mathbb {1} _ {y _ {x} = j}}, \quad (\bar {A} C) _ {x, j} = \sum_ {x ^ {+}} \frac {A _ {x , x ^ {+}}}{\sqrt {d _ {x}} \cdot \sqrt {d _ {x ^ {+}}}} \sqrt {\left(d _ {x ^ {+}}\right) \mathbb {1} _ {y _ {x ^ {+}} = j}}. \tag {21}
$$
So when  $j = y_{x}$
$$
\begin{array}{l} (D ^ {1 / 2} Y - \bar {A} C) _ {x, j} = \sqrt {(d _ {x}) \mathbb {1} _ {y _ {x} = j}} - \sum_ {x ^ {+}} \frac {\mathcal {A} (x , x ^ {+})}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} = j} \\ = \sqrt {d _ {x}} - \sum_ {x ^ {+}} \frac {\mathcal {A} (x , x ^ {+})}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} = j} \\ = \sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} - \sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} = j} \tag {22} \\ = \sum_ {x ^ {+}} \frac {\mathcal {A} (x , x ^ {+})}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} \neq j} \\ = \sum_ {x ^ {+}} \frac {\mathcal {A} (x , x ^ {+})}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} \neq y _ {x}}. \\ \end{array}
$$
When  $j\neq y_{x}$
$$
\begin{array}{l} (D ^ {1 / 2} Y - \bar {A} C) _ {x, j} = \sqrt {(d _ {x}) \mathbb {1} _ {y _ {x} = j}} - \sum_ {x ^ {+}} \frac {\mathcal {A} (x , x ^ {+})}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} = j} \\ = 0 - \sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} = j} \tag {23} \\ = - \sum_ {x ^ {+}} \frac {\mathcal {A} (x , x ^ {+})}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} = j}. \\ \end{array}
$$
We define  $\beta_{x} = \sum_{x^{+}}\mathcal{A}(x,x^{+})\mathbb{1}_{y_{x + }\neq y_{x}}$  , and we have
$$
\begin{array}{l} \| (D ^ {1 / 2} Y - \bar {A} C) _ {x} \| ^ {2} = \left(\sum_ {x ^ {+}} \frac {\mathcal {A} (x , x ^ {+})}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} \neq y _ {x}}\right) ^ {2} + \sum_ {j \neq y _ {x}} \left(\sum_ {x ^ {+}} \frac {\mathcal {A} (x , x ^ {+})}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} = j}\right) ^ {2} \\ \leq \left(\sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} \neq y _ {x}}\right) ^ {2} + \left(\sum_ {j \neq y _ {x}} \sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} = j}\right) ^ {2} \\ \leq \left(\sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} \neq y _ {x}}\right) ^ {2} + \left(\sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} \sum_ {j \neq y _ {x}} \mathbb {1} _ {y _ {x +} = j}\right) ^ {2} \tag {24} \\ = \left(\sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} \neq y _ {x}}\right) ^ {2} + \left(\sum_ {x ^ {+}} \frac {\mathcal {A} \left(x , x ^ {+}\right)}{\sqrt {d _ {x}}} \mathbb {1} _ {y _ {x +} \neq y _ {x}}\right) ^ {2} \\ = \frac {2 \beta_ {x} ^ {2}}{d _ {x}}. \\ \end{array}
$$