SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With that, we obtain  $\| D^{1 / 2}Y - \bar{A} C\| = \sum_x\frac{2\beta_x^2}{d_x}$ . As we assume that  $E_{\bar{x}\sim \mathcal{P}_u}(\mathcal{A}(x|\bar{x})\mathbb{1}_{y_x\neq \bar{y}}) = \alpha$ , so
$$
\begin{array}{l} \sum_ {x, x ^ {+}} \mathcal {A} (x, x ^ {+}) \mathbb {1} _ {y _ {x +} \neq y _ {x}} = \sum_ {x, x ^ {+}} E _ {\bar {x}} (\mathcal {A} (x | \bar {x}) \mathcal {A} (x _ {x ^ {+}} | \bar {x}) \mathbb {1} _ {y _ {x +} \neq y _ {x}}) \\ \leq \sum_ {x, x ^ {+}} E _ {\bar {x}} \left(\mathcal {A} (x | \bar {x}) \mathcal {A} \left(x _ {x ^ {+}} | \bar {x}\right) \left(\mathbb {1} _ {y _ {i} \neq \bar {y}} + \mathbb {1} _ {y _ {x ^ {+}} \neq \bar {y}}\right)\right) \tag {25} \\ = 2 E _ {\bar {x} \sim \mathcal {P} _ {d}} \left(\mathcal {A} (x | \bar {x}) \mathbb {1} _ {y _ {x} \neq \bar {y}}\right) \\ = 2 \alpha . \\ \end{array}
$$
$$
\begin{array}{l} \left\| D ^ {1 / 2} Y - \bar {A} C \right\| = \sum_ {x} \frac {2 \beta_ {x} ^ {2}}{d _ {x}} \\ \leq \sum_ {x} 2 \beta_ {x} \quad (d _ {x} = \sum_ {x ^ {+}} \mathcal {A} (x, x ^ {+}) \geq \sum_ {x ^ {+}} \mathcal {A} (x, x ^ {+}) \mathbb {1} _ {y _ {x} \neq y _ {x +}}) \\ = 2 \sum_ {x, x ^ {+}} \mathcal {A} (x, x ^ {+}) \mathbb {1} _ {y _ {x} \neq y _ {x +}} \quad (\text {d e f i n i t i o n o f} \beta_ {x}) \\ \leq 4 \alpha . \tag {Equation(25)} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {x, y} \| y - W _ {f} f ^ {(m)} (x) \| ^ {2} \\ = \| D ^ {1 / 2} Y - \bar {A} C + \bar {A} C - F _ {m} W _ {f} \| ^ {2} \\ \leq 2 \| \bar {A} C - F _ {m} W _ {f} \| ^ {2} + 8 \alpha \quad (\| A + B \| ^ {2} \leq 2 \| A \| ^ {2} + 2 \| B ^ {2} \|) \\ = 2 \| \left(\bar {A} - F _ {m} F _ {m} ^ {T} + U U ^ {T}\right) C - F _ {m} W _ {f}) \| ^ {2} + 8 \alpha \\ = 2 \| \left(\bar {A} - F _ {m} F _ {m} ^ {T}\right) C + F _ {m} \left(F _ {m} ^ {T} C - W _ {f}\right) \| ^ {2} + 8 \alpha \\ \leq 4 \left(\| \left(\bar {A} - F _ {m} F _ {m} ^ {T}\right) C \| ^ {2} + \| F _ {m} \left(F _ {m} ^ {T} C - W _ {f}\right) \|\right) ^ {2} + 8 \alpha \quad \left(\| A + B \| ^ {2} \leq 2 \left(\| A \| ^ {2} + \| B \| ^ {2}\right)\right) \\ \leq 4 \left(\| \left(\bar {A} - F _ {m} F _ {m} ^ {T}\right) \| ^ {2} \| C \| ^ {2} + \| F _ {m} \| ^ {2} \| \left(F _ {m} ^ {T} C - W _ {f}\right) \| ^ {2}\right) + 8 \alpha \quad (\| A B \| \leq \| A \| \| B \|) \\ \leq 4 \left(\| \left(\bar {A} - F _ {m} F _ {m} ^ {T}\right) \| ^ {2} + 8 \alpha \right. \tag {26} \\ \end{array}
$$
In the next step, we analyze the prediction error. We denote  $\bar{y}$  as the ground-truth label of original data  $\bar{x}$ . We first define an ensembled linear predictor  $p_f'$ . For an original sample, the predictor ensembles the results of all different views and chooses the label predicted the most. With the definition,  $\bar{y} \neq p_f'(\bar{x})$  only happens when more than half of the views predict wrong labels. So
$$
\begin{array}{l} \Pr (\bar {y} \neq p _ {f} ^ {\prime} (\bar {x})) \leq 2 \Pr (\bar {y} \neq p _ {f} (x)) \\ \leq 4 \mathbb {E} _ {\bar {x} \sim \mathcal {P} _ {d} (x), x \sim \mathcal {M} _ {1} (x | \bar {x})} \| \bar {y} - W _ {f} f ^ {(m)} (x) \| ^ {2} \\ \leq 8 \left(\mathbb {E} _ {x, y} \| y - W _ {f} f ^ {(m)} (x) \| ^ {2} + \mathbb {E} _ {\bar {x} \sim \mathcal {P} _ {d} (x), x \sim \mathcal {M} _ {1} (x | \bar {x})} \| y - \bar {y} \| ^ {2}\right) \\ \leq 8 \left(\mathbb {E} _ {x, y} \| y - W _ {f} f (x) \| ^ {2} + 2 \alpha\right) \\ \leq 3 2 \left\| \left(\bar {A} - F _ {m} F _ {m} ^ {T}\right) \right\| ^ {2} + 6 4 \alpha + 1 6 \alpha . \\ \end{array}
$$
Then we plug the optimal solutions of SCL (Lemma 4.1) in to  $F_{m}$ , and we obtain
$$
\mathcal {E} \left(\left(f _ {S C L} ^ {\star (m)}\right)\right) \leq 3 2 \left(\left(1 - \frac {m}{k}\right) \sum_ {i = 1} ^ {k} \sigma_ {l _ {i}} ^ {2} + \sum_ {i = k + 1} ^ {N} \sigma_ {i} ^ {2}\right) + 8 0 \alpha . \tag {27}
$$
When we plug the optimal solutions of triCL (Theorem 4.5) into  $F_{m}$ , we obtain:
$$
\mathcal {E} \left(\left(f _ {\text {t r i C L}} ^ {\star (m)}\right)\right) \leq 3 2 \sum_ {i = m + 1} ^ {N} \sigma_ {i} ^ {2} + 8 0 \alpha . \tag {28}
$$
A.3 Feature Identifiability of Asymmetric Tri-Factor Contrastive Learning
We first extend the augmentation graph to an asymmetric form. The asymmetric augmentation graph is defined over the set of all samples with its adjacent matrix denoted by  $P_O$ . In the augmentation graph, each node corresponds to a sample, and the weight of the edge connecting two nodes  $x_A$  and  $x_B$  is equal to the probability that they are selected as a positive pair, i.e.,  $(P_O)_{x_a,x_b} = \mathcal{P}_O(x_a,x_b)$ . And we denote  $\bar{P}_O$  as the normalized adjacent matrix of the augmentation graph, i.e.,  $(\bar{P}_O)_{x_a,x_b} = \frac{\mathcal{P}_O(x_a,x_b)^2}{\mathcal{P}_A(x_a)\mathcal{P}_B(x_b)}$ .
Similar to the symmetric form, we then rewrite  $\mathcal{L}_{\mathrm{tri}}$  as a matrix decomposition objective
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {t r i}} (f _ {A}, f _ {B}, S) = - 2 \mathbb {E} _ {x _ {a}, x _ {b}} f _ {A} (x _ {a}) ^ {\top} S f _ {B} (x _ {b}) + \mathbb {E} _ {x _ {a} ^ {-}, x _ {b} ^ {-}} \left(f _ {A} (x _ {a} ^ {-}) ^ {\top} S f _ {B} (x _ {b} ^ {-})\right) ^ {2} \\ = \sum_ {x _ {a}, x _ {b}} \left(\frac {\mathcal {P} _ {O} (x _ {a} , x _ {b}) ^ {2}}{\mathcal {P} _ {A} (x _ {a}) \mathcal {P} _ {B} (x _ {b})} + \mathcal {P} _ {A} (x _ {a}) \mathcal {P} _ {B} (x _ {b}) \left(f _ {A} (x _ {a}) ^ {\top} S f _ {B} (x _ {L})\right) ^ {2} \right. \\ - 2 \mathcal {P} _ {O} \left(x _ {a}, x _ {b}\right) f _ {A} \left(x _ {a}\right) ^ {\top} S f _ {B} \left(x _ {L}\right)) + c o n s t \\ = \| \bar {P} _ {O} - F _ {A} S F _ {B} ^ {\top} \| ^ {2}. \\ \end{array}
$$
According to the Eckart-Young Theorem [12], the optimal solutions  $F_A^\star, S^\star, F_B^\star$  satisfy
$$
F _ {A} ^ {\star} S ^ {\star} \left(F _ {B} ^ {\star}\right) ^ {\top} = U ^ {k} \Sigma \left(V ^ {k}\right) ^ {\top},
$$
where  $\Sigma \in \mathbb{R}^{k\times k}$  is a diagonal matrix with the  $k$ -largest eigenvalues of  $\bar{P}_O$  and  $U\in \mathbb{R}^{N_A\times k}$  contains the corresponding eigenvectors of the  $k$ -largest eigenvalues. When the regularizer  $\mathcal{L}_{\mathrm{Dec}}$  is minimized,  $F_{A}^{\star}$  and  $F_{B}^{\star}$  satisfy  $(F_{A}^{\star})^{\top}F_{A}^{\star} = I$ ,  $(F_B^\star)^\top F_B^\star = I$ . In the next step, we prove the uniqueness of the optimal solution.
We denote  $H = F_A^\star \Sigma F_B^\star$ , and we obtain  $HH^\top = F_A^\star S^\star (S^\star)^\top (F_A^\star)^\top$ . If  $\zeta, \sigma$  are a pair of eigenvector and eigenvalue of  $HH^\top$ , we have
$$
H H ^ {\top} \zeta = F _ {A} ^ {\star} S ^ {\star} (S ^ {\star}) ^ {\top} (F _ {A} ^ {\star}) ^ {\top} \zeta = \sigma \zeta ,
$$
$$
S ^ {\star} \left(S ^ {\star}\right) ^ {\top} \left(F _ {A} ^ {\star}\right) ^ {\top} \zeta = \sigma \left(F _ {A} ^ {\star}\right) ^ {\top} \zeta , \tag {29}
$$
$$
S ^ {\star} \left(S ^ {\star}\right) ^ {\top} \left(\left(F _ {A} ^ {\star}\right) ^ {\top} \zeta\right) = \sigma \left(\left(F _ {A} ^ {\star}\right) ^ {\top} \zeta\right).
$$
So the eigenvalues of  $HH^{\top}$  are the eigenvalues of  $S^{\star}(S^{\star})^{\top}$ . As the positive eigenvalues of  $HH^{\top}$  are uniquely determined and  $S^{\star}$  has an increasing order,  $S^{\star}$  is also determined and  $S^{\star} = \sum$ .
We note that  $HH^{\top} = F_{A}^{\star}S^{\star}(S^{\star})^{\top}(F_{A}^{\star})^{\top}$ , i.e.,  $HH^{\top}F_{A}^{\star} = F_{A}^{\star}S^{\star}(S^{\star})^{\top}$ , which means that the  $k$  columns of  $F_{A}^{\star}$  are the eigenvectors of  $HH^{\top}$  and the corresponding eigenvalues are  $\sigma_1\cdots\sigma_k$ . As  $HH^{\top}$  only has  $k$  different non-negative eigenvalues  $\sigma_1,\dots,\sigma_k$ , the eigenspace of each eigenvalue is one-dimensional. When we consider the real number space, any two eigenvectors  $\zeta_i,\zeta_i'$  of the same eigenvalue  $\sigma_i$  satisfy  $\zeta_i = c\zeta_i'$ . As  $(F_{A}^{\star})^{\top}F_{A}^{\star} = I$ , we obtain  $c = +1$ . Then we eliminate the ambiguity of the sign following Eq 11 and  $F_{A}^{\star}$  is unique. Similarly,  $F_{B}^{\star}$  is also unique. So the optimal solution of  $\mathcal{L}_{\mathrm{triCLIP}}$  is unique.
B.1 Experiment Details of Section 6.1
We first generate a random matrix  $A$  with size  $5000 \times 3000$ , and make sure that it does not contain multiple eigenvectors (which is easy to satisfy). For the matrix factorization problem  $\| A - FG^{\top} \|_F^2$ , we apply off-the-shelf algorithms and repeat this process ten times. We then calculate the mean and variance of the  $l_2$  pairwise distance between the obtained solutions of  $F$ . For the trifactorization objective  $\| A - FSG^{\top} \|_F^2$ , we use SVD to obtain an initial solution and apply the sign identification procedure to determine the sign of each eigenvector. Similarly, we also repeat this process ten times and calculate the mean and variance of the  $l_2$  pairwise distance between different solutions.
B.2 Experiment Details of Section 6.2
Pretraining Setups. For different evaluation tasks (k-NN, linear evaluation, image retrieval), we use the same pretrained models. We adopt ResNet-18 as the backbone. For CIFAR-10 and CIFAR-100, the projector is a two-layer MLP with hidden dimension 2048 and output dimension 256. And for ImageNet-100, the projector is a two-layer MLP with hidden dimension 4096 and output dimension 512. We pretrain the models with batch size 256 and weight decay 0.0001. For CIFAR-10 and CIFAR-100, we pretrain the models for 200 epochs. While for ImageNet-100, we pretrain the models for 400 epochs. We use the cosine anneal learning rate scheduler and set the initial learning rate to 0.4 on CIFAR-10, CIFAR-100, and 0.3 on ImageNet-100.
As the importance matrix is learned on the projection layer, we conduct the downstream tasks on the features encoded by the complete networks (containing both the backbones and the projectors).
The Distribution of the Importance Matrix. When observing the distribution of feature importance discovered by the importance matrix  $S$ , we first apply the softplus activation functions on the diagonal values of  $S$  and sort different rows of  $S$  by the descending order of corresponding diagonal values in  $S$ . We denote the non-negative ordered diagonal values of  $S$  as  $(s_1, \dots, s_k)$ . When we present the distribution of them in Figure 2(a), we normalize the diagonal values and obtain
$$
\left(s _ {1} / \sum_ {i = 1} ^ {k} s _ {i}, \dots , s _ {k} / \sum_ {i = 1} ^ {k} s _ {i}\right).
$$
The K-NN Accuracy on Selected Dimensions. For k-NN evaluation on 10 selected dimensions, we do not finetune the models. We sort the dimensions of  $f(x)$  by the descending order of corresponding diagonal values in the importance matrix. The k-NN is conducted on the standard split of CIFAR-10, CIFAR-100 and ImageNet-100 and the predicted label of samples is decided by the 10 nearest neighbors.
Linear Evaluation on Selected Dimensions. We train the linear classifier on 20 dimensions of the frozen networks for 30 epochs during the linear evaluation. We set batch size to 256 and weight decay to 0.0001. For triCL, we sort the dimensions by descending order of the importance matrix. And for SCL, we randomly choose 20 dimensions.
C More Extensions of Tri-factor Contrastive Learning
In this section, we apply tri-factor contrastive learning to another representative contrastive learning objective: the non-contrastive loss [15, 5].
Besides contrastive learning, non-contrastive learning is another popular self-supervised framework that throws the negative samples in contrastive learning and learns the meaningful representations only by aligning the positive pairs. Taking the state-of-the-art algorithm BYOL [15] as an example, they use an MSE loss:
$$
\mathcal {L} _ {\mathrm {M S E}} (f, g) = 2 - 2 \cdot \mathbb {E} _ {x. x ^ {+}} \frac {g (x) ^ {\top} f \left(x ^ {+}\right)}{\| g (x) \| _ {2} \cdot \| f \left(x ^ {+}\right) \| _ {2}}, \tag {30}
$$
where  $g(x)$  and  $f(x)$  are two different networks to avoid the feature collapse. Then we consider adapting the tri-term loss to the non-contrastive learning, i.e.,
$$
\mathcal {L} _ {\mathrm {t r i M S E}} (f, g) = 2 - 2 \cdot \mathbb {E} _ {x. x ^ {+}} \frac {g (x) ^ {\top} S f \left(x ^ {+}\right)}{\| g (x) \| _ {2} \cdot \| f \left(x ^ {+}\right) \| _ {2}} + \left\| \mathbb {E} _ {x} g (x) g (x) ^ {\top} - I \right\| ^ {2}. \tag {31}
$$
It is noticed that BYOL utilizes the stop-gradient technique on the target network  $f$  and it is updated by exponential moving average. So we only calculate the feature decorrelation loss on the online network  $g$ .