SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yang Qin $^{1}$ , Yuan Sun $^{1}$ , Dezhong Peng $^{1,3,4}$ , Joey Tianyi Zhou $^{2}$ , Xi Peng $^{1}$ , Peng Hu $^{1*}$
<sup>1</sup> College of Computer Science, Sichuan University, Chengdu, China.
<sup>2</sup> Centre for Frontier AI Research (CFAR) and Institute of High Performance Computing (IHPC), A*STAR, Singapore.
<sup>3</sup> Chengdu Ruibei Yingte Information Technology Co., Ltd, Chengdu, China.
Sichuan Zhiqian Technology Co., Ltd, Chengdu, China.
{qinyang.gm, joey.tianyi.zhou,pengx.gm,penghu.ml}@gmail.com, sunyuan_work@163.com, pengdz@scu.edu.cn
Recently, image-text matching has attracted more and more attention from academia and industry, which is fundamental to understanding the latent correspondence across visual and textual modalities. However, most existing methods implicitly assume the training pairs are well-aligned while ignoring the ubiquitous annotation noise, a.k.a noisy correspondence (NC), thereby inevitably leading to a performance drop. Although some methods attempt to address such noise, they still face two challenging problems: excessive memorizing/overfitting and unreliable correction for NC, especially under high noise. To address the two problems, we propose a generalized Cross-modal Robust Complementary Learning framework (CRCL), which benefits from a novel Active Complementary Loss (ACL) and an efficient Self-refining Correspondence Correction (SCC) to improve the robustness of existing methods. Specifically, ACL exploits active and complementary learning losses to reduce the risk of providing erroneous supervision, leading to theoretically and experimentally demonstrated robustness against NC. SCC utilizes multiple self-refining processes with momentum correction to enlarge the receptive field for correcting correspondences, thereby alleviating error accumulation and achieving accurate and stable corrections. We carry out extensive experiments on three image-text benchmarks, i.e., Flickr30K, MS-COCO, and CC152K, to verify the superior robustness of our CRCL against synthetic and real-world noisy correspondences. Code is available at https://github.com/QinYang79/CRL.
Image-text matching aims to search the most relevant samples across different modalities, which is fundamental for most cross-modal tasks [1, 2, 3, 4, 5, 6]. The core of image-text matching is how to accurately measure the similarity between distinct modalities, however, which is challenging due to the visual-textual discrepancy. To tackle the challenge, numerous deep methods are presented to learn the visual-semantic associations of image-text pairs and achieve remarkable progress, thanks to the powerful representation ability of Deep Neural Networks (DNNs) and some well-designed similarity inference architectures [7, 8, 4, 9]. They could be roughly divided into two groups, i.e., global-level methods [7, 9] and local-level methods [8, 4], which aim at learning the image-to-sentence and region-to-word correlation to infer the cross-modal similarity, respectively. Although these methods
achieved promising matching performance, most of them implicitly require large-scale well-aligned data for training, which is expensive or even impossible to collect due to ubiquitous noise in real-world scenarios [10, 11]. Therefore, there is inevitably imperfect alignment luring in the data, i.e., noisy correspondence (NC) [12], resulting in inferior performance.
To handle the NC problem, some prior arts are presented to alleviate the adverse impact of NC in various tasks, e.g., partially view-aligned clustering [13, 14, 15, 16], video-text retrieval [17], visible-infrared person re-identification [18], and image-text matching [12, 19, 20]. Specifically, inspired by learning with noisy labels [21, 22], some works [12, 23, 24] are proposed to alleviate the negative impact brought by NC. These works attempt to leverage the memorization effect of DNNs [25] to gradually distinguish the noisy image-text pairs for robust learning in a co-teaching manner. Furthermore, the predicted soft correspondence is used to recast a soft margin to replace the scalar margin of triplet ranking loss [7], which helps to avoid misleading the model by mismatched pairs. However, the soft-margin ranking loss is experimentally found to provide only limited robustness against NC, especially under high noise (as shown in Table 1), due to unstable division based on inaccurate predictions. In contrast, some works [19, 20] aim to enhance the robustness of cross-modal methods against NC by starting with a robust loss function, i.e., avoiding over-amplification of wrong supervision information to reduce misleading risk. However, the lack of explicitly mitigating the effect of easily separable noise makes them hard to further improve the performance.
To address the aforementioned issues, we propose a generalized robust framework, dubbed CRCL, for learning with noisy correspondences. CRCL could be easily migrated into existing image-text matching methods to enhance their robustness against NC. Our framework introduces a novel Active Complementary Loss (ACL) that applies active and complementary learning to mutually boost robustness against NC. Specifically, we present a robust complementary learning loss that employs complementary pairs, such as "input image-text pairs are irrelevant", to conduct indirect cross-modal learning with exponential normalization. Due to the low likelihood of selecting incorrect complementary pairs, robust complementary learning could reduce the risk of providing incorrect supervision and smooth the losses, thus embracing robustness against NC. However, the robust loss will face the underfitting problem, leading to suboptimal performance. To overcome this issue, a weighted Active learning loss is proposed to enforce the model focus on more reliable positive pairs in addition to only complementary pairs. In addition, we propose a Self-refining Correspondence Correction paradigm (SCC) to obtain stable and accurate correspondence correction. SCC utilizes Momentum Correction (MC) to aggregate historical predictions for stable and accurate correspondence corrections. By combining multiple Self-Refining processes (SR) throughout the entire training process, we alleviate over-memorization for NCs. In summary, the key contributions and innovations of this work are as follows:
- We propose a generalized Cross-modal Robust Contrastive Learning framework (CRCL) to address a pressing and widely-exist problem in image-text matching, i.e., noisy correspondence. CRCL empowers existing methods with strong robustness through the perspectives of robust loss and correction techniques.
- A novel Active Complementary Loss (ACL) is presented to balance active and complementary learning, mutually enhancing robustness against NC while encapsulating correct cross-modal associations in the latent common space.
- We design an effective Self-refining Correspondence Correction paradigm (SCC) to achieve accurate and stable soft correspondences, which enables the prediction-based corrections to perceive larger fields and self-refine from the historically learned knowledge.
- Extensive experiments verify the effectiveness and superiority of our framework on three benchmark image-text datasets: Flickr30K, MS-COCO, and CC152K. Additionally, comprehensive ablation studies and insightful analyses demonstrate the reliability and practicability of the proposed CRCL.
2 The Proposed Method
2.1 Preliminaries and Problem Statement
To be specific, we first provide some definitions of instance-level image-text matching so as to conveniently study noisy correspondence. Let  $\mathcal{D} = \{\mathcal{I},\mathcal{T},\mathcal{V}\}$  be an image-text dataset, where  $\mathcal{I} =$
$\{I_i\}_{i=1}^N$  and  $\mathcal{T} = \{T_i\}_{i=1}^N$  are the training image and text set with size of  $N$ . The correspondence label space is defined as  $\mathcal{V} = \{y_{ij} | i = 1, \dots, N; j = 1, \dots, N\}$ , where  $y_{ij}$  represents the correspondence of pair  $(I_i, T_j)$ , i.e., if  $I_i$  and  $T_j$  are matched (i.e., positive pair),  $y_{ij} = 1$  otherwise  $y_{ij} = 0$ . We assume each pair with the same indices has matched correspondence, i.e.,  $y_{ii} = 1, i = 1, \dots, N$ . However, due to the ubiquitous noise during data collection, some negative pairs are mismatched as positives, a.k.a noisy correspondence (NC) [12, 19], which would introduce wrong supervisory information and misguide model training, leading to performance degradation. Mathematically, we define NC as shown in Definition 1.
Definition 1. Due to the existence of NC, the learner only has access to the noisy training data  $(\mathcal{D}_{\eta})$  instead of clean data  $(\mathcal{D})$ . Thus, the correspondence label for pair  $(I_i, T_j)$  is reconsidered as
$$
\tilde {y} _ {i j} = \left\{ \begin{array}{l l} y _ {i j} & \text {w i t h p r o b a b i l i t y} (1 - \eta_ {i j}), \\ 1 - y _ {i k} & \text {w i t h p r o b a b i l i t y} \bar {\eta} _ {i k}, \forall k \neq j. \end{array} \right. \tag {1}
$$
For all pairs, conditioned on that if  $i = j$  then  $y_{ij} = 1$  else  $y_{ij} = 0$ , we have  $\sum_{j \neq k} \bar{\eta}_{ik} = \eta_{ij}$ . Similar to the definitions of noisy labels [26], we assume that NC is uniform, i.e.,  $\eta_{ij} = \eta$  and  $\bar{\eta}_{ik} = \frac{\eta}{N - 1}, \forall k \neq j$ , where  $\eta$  is a constant to represent the noise rate.
The key to learning with noisy correspondence is to alleviate the misleading impact and rectify the mismatched pairs. One direct solution is to enhance the robustness of loss function  $\mathcal{L}$  against noisy pairs, which can help prevent overfitting on mismatched pairs. The second aims at using the memorization effect of DNNs [25] to discriminate the mismatched pairs, thus removing unreliable supervision from the training data.
For image-text matching, images and texts are first projected into a shared representation space by two modality-specific networks, denoted as  $v$  and  $g$ , respectively. The cross-modal similarity of a given pair  $(I_i, T_j)$  is then computed as  $S(v(I_i), g(T_j))$ , where  $S(*)$  could be the cosine function [7, 9] or a relevance inference module [8, 4]. For brevity,  $S(v(I_i), g(T_j))$  is denoted as  $S(I_i, T_j)$  or  $S_{ij}$  in the following. The computed similarities could be considered as supporting evidence for retrieved results. Thus, the learning objective of image-text matching is to maximize the cross-modal similarities of positive pairs while minimizing those of negatives in the latent common space, which is commonly achieved by using contrastive learning [7, 8, 4].
The widely-used triplet ranking loss [7] has shown excellent performance in cross-modal contrastive learning tasks [7, 27]. However, recent research [19] has demonstrated that this loss function fails to perform well in image-text data with NCs, especially when using the hardest negative samples as comparison items. To address this issue, some works proposed an adaptive soft margin approach to improve the robustness of the ranking loss [12, 24, 23], which is defined as follows:
$$
\mathcal {L} _ {\text {s o f t}} \left(I _ {i}, T _ {i}\right) = \left[ \hat {\alpha} _ {i} - S \left(I _ {i}, T _ {i}\right) + S \left(I _ {i}, \hat {T} _ {h}\right) \right] _ {+} + \left[ \hat {\alpha} _ {i} - S \left(I _ {i}, T _ {i}\right) + S \left(\hat {I} _ {h}, T _ {i}\right) \right] _ {+}, \tag {2}
$$
where  $\hat{T}_h$  and  $\hat{I}_h$  denote the hardest cross-modal samples in a mini-batch,  $\hat{\alpha}_i$  is a soft margin adaptively computed by  $\hat{\alpha}_i = \frac{m^{\hat{y}_{ii}} - 1}{m - 1}\alpha$ ,  $\alpha$  is a constant margin,  $m$  is a curve parameter, and  $\hat{y}_{ii}$  is the rectified correspondence between  $I_i$  and  $T_i$ . However, this approach has two disadvantages: 1) The margin setting  $\alpha$  may not be consistent with the empirical setting under NC scenarios. 2) The inaccurate prediction  $\hat{y}_{ii}$  can still easily produce the risk of misleading gradient, which can cause trivial solutions to fail, especially in the case of high noise (e.g., the results of NCR [12] and BiCro [23] on Flickr30K with  $80\%$  noise). To overcome these limitations, we propose a novel Active Complementary Loss (ACL) under the risk minimization theory [28, 26] to provide noise tolerance for noisy pairs while ensuring discriminative learning.
2.2 Active Complementary Loss
For the image-text dataset  $\mathcal{D}$ , our goal is to learn a cross-modal model  $(\mathcal{M})$  that can discriminatively identify the positive (matched) and negative (unmatched) pairs well for retrieval, which is intuitively equivalent to maximizing bidirectional matching probabilities of positives. The bidirectional matching probabilities for pair  $(I_i, T_j)$  are defined as:
$$
p _ {i j} ^ {\circ} = f \left(I _ {i}, T _ {j}\right) = \frac {e ^ {\left(S _ {i j} / \tau\right)}}{\sum_ {l = 1} ^ {N} e ^ {\left(S _ {i l} / \tau\right)}}, \quad p _ {i j} ^ {\diamond} = f \left(T _ {j}, I _ {i}\right) = \frac {e ^ {\left(S _ {i j} / \tau\right)}}{\sum_ {l = 1} ^ {N} e ^ {\left(S _ {l j} / \tau\right)}}, \tag {3}
$$
where  $\tau$  is a temperature parameter [29, 30, 31],  $f$  is regarded as the cross-modal decision function, and “ $\circ, \diamond$ ” means the two retrieval directions, i.e., image-to-text and text-to-image, respectively. For any loss function  $\mathcal{L}$ , the matching risk of  $f$  for image-text matching can be defined as
$$
R _ {\mathcal {L}} (f) = \mathbb {E} _ {(I _ {i}, y _ {i \cdot}) \sim \mathcal {D}} [ \mathcal {L} (f (I _ {i}, T,), y _ {i \cdot}) ] + \mathbb {E} _ {(T _ {i}, y _ {i \cdot}) \sim \mathcal {D}} [ \mathcal {L} (f (T _ {i}, I,), y _ {i \cdot}) ], \tag {4}
$$
where  $\mathbb{E}[\cdot ]$  represents the expectation operator. Considering noisy correspondences, the risk of  $f$  in noisy data  $\mathcal{D}_{\eta}$  could be formulated as follows:
$$
R _ {\mathcal {L}} ^ {\eta} (f) = \mathbb {E} _ {\left(I _ {i}, \tilde {y} _ {i .}\right) \sim \mathcal {D} _ {\eta}} \left[ \mathcal {L} \left(f \left(I _ {i}, T _ {i} , \tilde {y} _ {i .}\right)\right) \right] + \mathbb {E} _ {\left(T _ {i}, \tilde {y} _ {i .}\right) \sim \mathcal {D} _ {\eta}} \left[ \mathcal {L} \left(f \left(T _ {i}, I _ {i} , \tilde {y} _ {i .}\right)\right) \right], \tag {5}
$$
where  $\tilde{y}_{ij}$  is the noisy correspondence label as shown in Definition 1. Thus, the cross-modal learning objective is to learn a model  $\mathcal{M}_{\mathcal{L}}^{*}$  with a global minimizer  $f_{\eta}^{*}$  of  $R_{\mathcal{L}}^{\eta}(f)$ . To achieve robustness,  $f_{\eta}^{*}$  should be also the global minimizer of  $R_{\mathcal{L}}(f)$  on the noise-free data.
Inspired by complementary contrastive learning [20], we propose to optimize the matching probabilities of all negative pairs for learning with noisy data indirectly, thereby avoiding fast overfitting to NC. Simultaneously, to further improve the noise tolerance, we introduce an exponential normalization to smooth the complementary loss. Hence, the robust complementary loss for pair  $(I_i, T_i)$  could be formulated as:
$$
\begin{array}{l} \mathcal {L} _ {r} \left(I _ {i}, T _ {i}, q\right) = \mathcal {L} _ {r} ^ {\circ} \left(I _ {i}, T _ {i}, q\right) + \mathcal {L} _ {r} ^ {\diamond} \left(T _ {i}, I _ {i}, q\right) \\ = \sum_ {j \neq i} ^ {N} \tan \left(p _ {i j} ^ {\circ}\right) / \left(\sum_ {k = 1} ^ {N} \tan \left(p _ {i k} ^ {\circ}\right)\right) ^ {q} + \sum_ {j \neq i} ^ {N} \tan \left(p _ {j i} ^ {\circ}\right) / \left(\sum_ {k = 1} ^ {N} \tan \left(p _ {k i} ^ {\circ}\right)\right) ^ {q}, \tag {6} \\ \end{array}
$$
where  $\tan (\cdot)$  is the tan function and  $q\in [0,1]$  is a regulatory factor. Theoretically, for any input  $(I_i,T_i)$  under noise rate  $\eta \leq \frac{N - 1}{N}$ , we can show (see proofs in supplementary material)
$$
C \leq R _ {\mathcal {L} _ {r}} \left(f ^ {*}\right) - R _ {\mathcal {L} _ {r}} \left(f _ {\eta} ^ {*}\right) \leq 0, \tag {7}
$$
where  $C = 2\eta (A_{\min}^{(1 - q)} - A_{\max}^{(1 - q)}) / (1 - \frac{N\eta}{N - 1}) \leq 0$ .  $C$  increases as  $q$  increases and when  $q = 1$ ,  $C$  takes the maximum value 0.  $A_{\min}$  and  $A_{\max}$  are the maximum and minimum values of  $\sum_{j=1}^{N} \tan(p_{ij})$  under the condition  $\sum_{j=1}^{N} p_{ij} = 1$ , where  $1 < A_{\min} < A_{\max}$ , and  $0 \leq p_{ij} \leq 1$  ( $p_{ij} = p_{ij}^{\circ}$  or  $p_{ij}^{\diamond}$ ).  $f^*$  and  $f_{\eta}^*$  are the global minimizers of  $R_{\mathcal{L}_r}(f)$  and  $R_{\mathcal{L}_r}^{\eta}(f)$ , respectively.
Analysis: The larger the  $q$  is,  $C \to 0$ , the tighter the bound of Equation (7) is. When  $q = 0$ ,  $\mathcal{L}_r$  is a standard complementary contrastive loss [20]. In the extreme case of  $q = 1$ ,  $R_{\mathcal{L}_r}(f^*) = R_{\mathcal{L}_r}(f_\eta^*)$ , i.e.,  $\mathcal{L}_r$  is noise tolerant since  $f_\eta^*$  and  $f^*$  are simultaneously the minimizers of  $R_{\mathcal{L}_r}(f)$  (It can also be obtained from Lemma 1 that  $\mathcal{L}_r$  is robust under  $q = 1$ ). Put another way, as  $q$  approaches 1, the optimum  $f_\eta^*$  of the noisy risk will be close to  $f^*$  on the clean data more likely, which implies noise tolerance.
Like most robust learning methods [32, 33], we should focus more on reliable data, while less on unreliable data. In other words, a smaller value of  $q$  is used for more convincing pairs (with larger  $\hat{y}_{ii}$ ), while a larger value of  $q$  is for less convincing pairs (with smaller  $\hat{y}_{ii}$ ). Thus, we could empirically utilize the soft corrected label  $\hat{y}_{ii}$  to recast  $q$  like the soft margin used in NCR [12], i.e.,  $q = 1 - \hat{y}_{ii}$ . However, indirect learning will face the underfitting problem, resulting in suboptimal/insufficient performance. To address this issue, we introduce a weighted active learning loss  $\mathcal{L}_d$  to make the model pay more attention to positive/matched pairs, i.e.,  $\mathcal{L}_d(I_i,T_i,\tilde{y}_{ii}) = -\tilde{y}_{ii}\left(\log p_{ii}^{\circ} + \log p_{ii}^{\diamond}\right)$ . This positive learning will mine discrimination from direct supervision, which complements the complementary learning loss. By combining active and complementary learning losses, our active complementary loss is defined as:
$$
\mathcal {L} _ {a c l} \left(I _ {i}, T _ {i}, \hat {y} _ {i i}\right) = \mathcal {L} _ {d} \left(I _ {i}, T _ {i}, \hat {y} _ {i i}\right) + \lambda \mathcal {L} _ {r} \left(I _ {i}, T _ {i}, 1 - \hat {y} _ {i i}\right), \tag {8}
$$
where  $\lambda$  is a scale factor to prevent  $\mathcal{L}_d$  from dominating the cross-modal training and quickly overfitting NC. As shown in Equation (8), when  $(I_i, T_i)$  is a noisy pair and  $\hat{y}_{ii}$  ideally approaches 0, the loss emphasizes robust complementary learning, thus mitigating overfitting to NC. Conversely, when  $(I_i, T_i)$  is a clean pair and  $\hat{y}_{ii}$  ideally approaches 1, the loss focuses on discriminative learning, thereby facilitating the accurate acquisition of visual-semantic associations. However, due to computational resource constraints, we cannot use the entire training set to perform cross-modal learning. Therefore,
we relax  $N$  to the size  $K$  of the mini-batch  $\mathbf{x}$  by Monte Carlo sampling. Without loss of generality, the final loss for cross-modal learning is given by:
$$
\mathcal {L} _ {a c l} (\mathbf {x}) = \frac {1}{K} \sum_ {i = 1} ^ {K} \mathcal {L} _ {a c l} \left(I _ {i}, T _ {i}, \hat {y} _ {i i}\right). \tag {9}
$$
Lemma 1. In an instance-level cross-modal matching problem, under uniform NC with noise rate  $\eta \leq \frac{N - 1}{N}$ , when  $q = 1$ ,  $\mathcal{L}_r$  is noise tolerant.
Proof. The proofs of Equation (7) and Lemma 1 can be found in the supplementary material.
2.3 Self-refining Correspondence Correction
Another key to solving NC is how to obtain accurate correspondence estimations so as to reduce the adverse effects of NC. To this end, we propose an effective Self-refining Correspondence Correction paradigm (SCC). SCC leverages Momentum Correction (MC) to aggregate historical predictions, providing stable and accurate correspondence estimations while alleviating the over-memorization to NC. To eliminate the error accumulation against NC, we combine multiple independent Self-Refining (SR) in the entire training process. Specifically, the MC for the correspondence of  $(I_i, T_i)$  at the  $t$ -th epoch is defined as follows:
$$
y _ {i i} ^ {t} = \beta y _ {i i} ^ {t - 1} + (1 - \beta) \hat {p} ^ {t} \left(I _ {i}, T _ {i}\right), \tag {10}
$$
where  $\beta \in (0,1)$  represents the momentum coefficient,  $\hat{p}^t (I_i,T_i) = (p_{ii}^\circ +p_{ii}^\diamond) / 2$  denotes the average matching probability at the  $t$ -th epoch. Through adequate cross-modal training, our CRCL will obtain a more stable and accurate soft correspondence label by smoothly evolving and expanding the receptive field of correction based on historical predictions with MC. Notably, as training progresses, some pairs would always be incorrectly distinguished as clean or noisy ones, resulting in the error accumulation of the estimated labels (see Figure 1c). Additionally, even though the updates performed by MC help reduce the negative influence of NC, the initial correspondence label  $(y_{ii}^{0})$  still greatly affects the quality of subsequent smooth corrections. In other words, providing more accurate initial correspondences weakens the DNN's memorization against NC, thereby reducing the risk of error accumulation.
Algorithm 1: The pseudo-code of CRCL
Input: A noisy training dataset  $\mathcal{D}_{\eta}$  , image-text matching model  $\mathcal{M}(\Theta)$  Initialize:  $\Theta$    
for  $e^j$  in  $[e_1,e_2,\dots ,e_m]$  do for t in [1,2,,ej] do for x in batches do Obtain the bidirectional matching probabilities of  $\mathbf{x}$  with Equation (3); Update the correspondence labels with Equation (11); Obtain the corrected labels with Equation (12); Compute the overall loss  $\mathcal{L}_{acl}(\mathbf{x})$  with Equation (9);  $\Theta =$  Optimizer  $(\Theta ,\mathcal{L}_{acl}(\mathbf{x}))$  end end Re-initialize  $\Theta$  .   
end   
Output: The learned parameters  $\hat{\Theta}$
To achieve accurate initial correspondence estimations, SCC refines updated correspondence labels historically in epochs using MC through multiple concatenated SR pieces during the entire training process. Subsequent SR pieces could gradually aggregate the learned correspondence knowledge from previous pieces, thus improving the quality of estimated correspondences progressively. Furthermore, each SR piece is trained from scratch, which aims to clear accumulated error/noise that has been memorized, thus providing more accurate correspondence predictions for subsequent training. Mathematically, SCC consists of multiple SR pieces  $\left([e_1,\dots ,e_m]\right)$  based on MC, where each piece
undergoes robust learning for  $e_j$  ( $j \in \{1, \dots, m\}$ ) epochs. Thus, during the  $j$ -th SR training, the estimated soft label of the  $i$ -th pair at  $t$ -th epoch is reconsidered as follows:
$$
y _ {i i} ^ {(j, t)} = \left\{ \begin{array}{l l} y _ {i i} ^ {(j - 1, e _ {j - 1})}, & \text {i f} t \leq e _ {f}, \\ \hat {p} ^ {(j, t - 1)} \left(I _ {i}, T _ {i}\right), & \text {i f} j = 1 \text {a n d} t = \left(e _ {f} + 1\right), \\ \beta y _ {i i} ^ {(j, t - 1)} + (1 - \beta) \hat {p} ^ {(j, t - 1)} \left(I _ {i}, T _ {i}\right), & \text {o t h e r w i s e}, \end{array} \right. \tag {11}
$$
where  $\hat{p}^{(j,t - 1)}(*)$  is the average matching probability of pair  $(I_i,T_i)$  at the  $(t - 1)$ -th epoch during the  $j$ -th training piece,  $e_f$  denotes the number of epochs to freeze the correspondence label, preventing insufficient model training in the early stage from affecting the correction quality. In our experiments, we set all initial labels to 1, assuming that all training pairs are matched at the beginning of the first SR piece. In practice, we assign the label of the confident noisy pair as 0 to reduce the risk of producing erroneous supervision information. Therefore, the final corrected correspondence label used for  $\mathcal{L}_{acl}$  is defined as: