SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {y} _ {i i} = \left\{ \begin{array}{l l} 0, & \text {i f} y _ {i i} ^ {(j, t)} <   \epsilon , \\ y _ {i i} ^ {(j, t)}, & \text {o t h e r w i s e}, \end{array} \right. \tag {12}
$$
where  $\epsilon = 0.1$  is a fixed threshold used to filter the confident noisy pairs in experiments.
In this section, comprehensive experiments are conducted on three widely used benchmarks to demonstrate the robustness and effectiveness of our CRCL under multiple scenarios, including synthetic noise, real-world noise, and well-annotated correspondence.
3.1 Datasets and Protocols
Datasets: For an extensive evaluation, we use three benchmark datasets (i.e., Flickr30K [34], MS-COCO [35] and CC152K [12]) in our experiments. More specifically, Flickr30K is a widely-used image-text dataset collected from the Flickr website, which comprises 31,000 images and each one has 5 corresponding textual descriptions. Following [36], 30,000 images are employed for training, 1,000 images for validation, and 1,000 images for testing in our experiments. MS-COCO is a large-scale image-text dataset, which has 123,287 images, and 5 captions are given to describe each image. We follow the split of [36, 8] to carry out our experiments, i.e., 5000 validation images, 5000 test images, and the rest for training. CC152K is a subset of Conceptual Captions (CC) [10] collected in the real world, which is selected by [12]. Due to the absence of manual annotation, there are about  $3\% \sim 20\%$  incorrect correspondences in CC, i.e., real-world noisy correspondences. CC152K contains 150,000 image-text pairs for training, 1,000 pairs for validation, and 1,000 pairs for testing.
Evaluation Protocols: Recall at K (R@K=1, 5, and 10) is used to measure the performance of bidirectional retrievals, which is defined as the proportion of the queries with the correct item in the top K retrieved results. Besides, flowing [19], we also take the sum of all Recalls to evaluate the overall performance, i.e., rSum.
3.2 Implementation Details
Our CRCL is a generalized robust framework that could extend existing methods to confront noisy correspondences. To demonstrate the effectiveness and robustness of CRCL, we extend two representative methods, i.e.,  $\mathrm{VSE\infty[9]}$  and SGRAF (SGR and SAF) [4], to perform robust image-text matching, respectively. Specifically, the shared hyper-parameters are set as the same as the original works [4, 9], e.g., the batch size is 128, the word embedding size is 300, and the joint embedding dimensionality is 1,024. More specific hyper-parameters and implementation details are given in our supplementary material due to the space limitation.
3.3 Comparison with State-of-the-Arts
In this section, we evaluate our CRCL by comparing it with 7 state-of-the-art methods on three benchmarks, i.e., SCAN (ECCV'18) [8], SGRAF (SGR and SAF, AAAI'21) [4], VSE∞ (CVPR'21) [9], NCR (NeurIPS'21) [12], DECL (ACM MM'22) [19], BiCro (CVPR'23) [23] and MSCN
(CVPR'23) [24]. For a fair comparison, all tested approaches adopt the same visual features (BUTD features) [8] and textual backbone Bi-GRU [37]. To comprehensively investigate the robustness of our method, we artificially inject synthetic false correspondence of different ratios by proportionally shuffling the captions on Flickr30K and MS-COCO like [12], i.e.,  $20\%$ ,  $40\%$ ,  $60\%$ , and  $80\%$  noise rates. In addition to synthetic noise, we also evaluate the robustness of tested methods against the real-world noisy correspondences on CC152K. Due to the space limitation, we only provide the results on MS-COCO 5K under well-annotated correspondences in Table 2. For fairness, like [19, 23], the ensemble results of CRCL-SGRAF are reported in the paper. More extensive comparison results are provided in the supplementary material to fully demonstrate the superiority of CRCL.
Table 1: Performance comparison (R@K(%) and rSum) of image-text matching on Flickr30K and MS-COCO 1K. The highest scores are shown in **. means robust methods.
Flickr30K	MS-COCO 1K
Image → Text	Text → Image	Image → Text	Text → Image
Noise	Methods	R@1	R@5	R@10	R@1	R@5	R@10	rSum	R@1	R@5	R@10	R@1	R@5	R@10	rSum
20%	SCAN	56.4	81.7	89.3	34.2	65.1	75.6	402.3	28.9	64.5	79.5	20.6	55.6	73.5	322.6
SAF	51.8	79.5	88.3	38.1	66.8	76.6	401.1	41.0	78.4	89.4	38.2	74.0	85.5	406.5
SGR	61.2	84.3	91.5	44.5	72.1	80.2	433.8	49.1	83.8	92.7	42.5	77.7	88.2	434.0
VSE∞	69.0	89.2	94.8	48.8	76.3	83.8	461.9	73.5	93.3	97.0	57.4	86.5	92.8	500.5
NCR*	76.7	93.9	96.9	57.5	82.8	89.2	497.0	77.0	95.6	98.1	61.5	89.3	95.1	516.6
DECL*	75.6	93.8	97.4	58.5	82.9	89.4	497.6	77.1	95.9	98.4	61.6	89.1	95.2	517.3
BiCro*	78.1	94.4	97.5	60.4	84.4	89.9	504.7	78.8	96.1	98.6	63.7	90.3	95.7	523.2
MSCN*	77.4	94.9	97.6	59.6	83.2	89.2	501.9	78.1	97.2	98.8	64.3	90.4	95.8	524.6
CRCL*	77.9	95.4	98.3	60.9	84.7	90.6	507.8	79.6	96.1	98.7	64.7	90.6	95.9	525.6
40%	SCAN	29.9	60.5	72.5	16.4	38.5	48.6	266.4	30.1	65.2	79.2	18.9	51.1	69.9	314.4
SAF	34.3	65.6	78.4	30.1	58.0	68.5	334.9	36.0	74.4	87.0	33.7	69.4	82.5	383.0
SGR	47.2	76.4	83.2	34.5	60.3	70.5	372.1	43.9	78.3	89.3	37.0	72.8	85.1	406.4
VSE∞	30.2	58.3	70.2	22.3	49.6	62.7	293.3	53.3	84.3	92.1	31.4	63.8	75.0	399.9
NCR*	75.3	92.1	95.2	56.2	80.6	87.4	486.8	76.5	95.0	98.2	60.7	88.5	95.0	513.9
DECL*	72.5	93.1	97.0	55.8	81.2	88.1	487.7	77.1	95.7	98.3	61.5	89.2	95.3	517.1
BiCro*	74.6	92.7	96.2	55.5	81.1	87.4	487.5	77.0	95.9	98.3	61.8	89.2	94.9	517.1
MSCN*	74.4	94.4	96.9	57.2	81.7	87.6	492.2	74.8	94.9	98.0	60.3	88.5	94.4	510.9
CRCL*	77.8	95.2	98.0	60.0	84.0	90.2	505.2	78.2	95.7	98.3	63.3	90.3	95.7	521.5
60%	SCAN	16.9	39.3	53.9	2.8	7.4	11.4	131.7	27.8	59.8	74.8	16.8	47.8	66.4	293.4
SAF	28.3	54.5	67.5	22.1	47.3	59.0	278.7	28.2	63.9	79.4	31.1	65.6	80.5	348.7
SGR	28.7	58.0	71.0	23.8	49.5	60.7	291.7	37.6	73.3	86.3	33.8	68.6	81.7	381.3
VSE∞	18.0	44.0	55.7	15.1	38.5	51.8	223.1	33.4	64.8	79.1	26.0	60.1	76.3	339.7
NCR*	68.7	89.9	95.5	52.0	77.6	84.9	468.6	72.7	94.0	97.6	57.9	87.0	94.1	503.3
DECL*	69.4	89.4	95.2	52.6	78.8	85.9	471.3	73.8	94.7	97.7	59.6	87.9	94.5	508.2
BiCro*	67.6	90.8	94.4	51.2	77.6	84.7	466.3	73.9	94.4	97.8	58.3	87.2	93.9	505.5
MSCN*	70.4	91.0	94.9	53.4	77.8	84.1	471.6	74.4	95.1	97.9	59.2	87.1	92.8	506.5
CRCL*	73.1	93.4	95.8	54.8	81.9	88.3	487.3	76.3	95.1	97.9	60.8	89.0	95.1	514.2
80%	SCAN	5.1	18.1	27.3	3.9	13.1	19.1	86.6	22.2	51.9	67.5	13.8	41.1	58.6	255.1
SAF	12.2	32.8	48.4	11.8	30.5	41.5	177.2	24.2	57.5	74.1	24.7	57.1	73.0	310.6
SGR	13.7	35.1	47.6	12.1	30.9	41.9	181.3	26.7	60.7	75.6	25.3	58.2	72.6	319.1
VSE∞	8.1	23.1	34.7	7.4	22.6	31.8	127.7	25.4	55.1	70.6	19.2	50.5	68.0	288.8
NCR*	1.4	7.1	11.7	1.5	5.4	9.3	36.4	21.6	52.6	67.6	15.1	38.1	49.8	244.8
DECL*	60.7	84.6	91.2	42.1	69.6	78.6	426.8	65.6	91.6	96.6	52.0	83.0	91.3	480.1
BiCro*	3.6	13.9	20.5	1.7	7.5	13.0	60.2	40.0	72.6	84.7	22.6	53.0	67.2	340.1
MSCN*	1.0	4.4	9.1	0.4	1.4	2.5	18.8	66.8	91.6	96.2	52.7	83.0	90.9	481.2
CRCL*	62.3	86.8	92.8	46.0	73.6	82.2	443.7	72.7	93.5	97.6	57.5	86.8	93.7	501.8
3.3.1 Results under Synthetic Noisy Correspondences
For quantitative evaluation under specific noise levels, we conduct all tested methods under four different noise rates (i.e.,  $20\%$ ,  $40\%$ ,  $60\%$ , and  $80\%$ ) of synthetic noisy correspondences on the Flickr30K and MS-COCO datasets. Quantitative results on Flickr30K and MSCOCO 1K test set are shown in Table 1. For MS-COCO, the results are computed by a veraging over 5 folds of 1K test images. From the results, one can see that our CRCL could remarkably outperform the robust baselines (NCR, DECL, BiCro, and MSCN) on most of the metrics, which demonstrates the superior robustness of CRCL against NC. Moreover, our CRCL not only performs well in low noise but also achieves the best performance under high noise, especially  $80\%$  noise, which provides strong evidence for the stability and robustness of our method.
Table 2: Performance comparison on CC152K and MS-COCO 5K.
CC152K	MS-COCO 5K
Image → Text	Text → Image	Image → Text	Text → Image
Methods	R@1	R@5	R@10	R@1	R@5	R@10	rSum	R@1	R@5	R@10	R@1	R@5	R@10	rSum
SCAN	30.5	55.3	65.3	26.9	53.0	64.7	295.7	44.7	75.9	86.6	33.3	63.5	75.4	379.4
VSE∞	34.0	64.5	77.0	12.9	19.2	21.6	229.2	56.6	83.6	91.4	39.3	69.9	81.1	421.9
SGRAF	32.5	59.5	70.0	32.5	60.7	68.7	323.9	58.8	84.8	92.1	41.6	70.9	81.5	429.7
NCR*	39.5	64.5	73.5	40.3	64.6	73.2	355.6	58.2	84.2	91.5	41.7	71.0	81.3	427.9
DECL*	39.0	66.1	75.5	40.7	66.3	76.7	364.3	59.2	84.5	91.5	41.7	70.6	81.1	428.6
MSCN*	40.1	65.7	76.6	40.6	67.4	76.3	366.7	-	-	-	-	-	-	-
BiCro*	40.8	67.2	76.1	42.1	67.6	76.4	370.2	59.0	84.4	91.7	42.4	71.2	81.7	430.4
CRCL*	41.8	67.4	76.5	41.6	68.0	78.4	373.7	61.3	85.8	92.7	43.5	72.6	82.7	438.6
3.3.2 Results under Real Noisy Correspondences
In addition to synthetic noise, we carry the comparison experiments on the real-world noisy dataset CC152K. The quantitative results are shown in Table 2. From the results, one can see that our CRCL is superior to all baselines with the best overall performance of  $373.7\%$ , which indicates that the proposed method is robust against real-world noise. Specifically, CRCL outperforms the best baseline BiCro, with absolute performance improvement of  $1.0\%$ ,  $0.2\%$ ,  $0.4\%$ ,  $0.4\%$ , and  $2.0\%$  across different metrics, except for R@1 in text-to-image retrieval.
3.3.3 Results under Well-annotated Correspondences
Besides noisy correspondences, we also evaluate the tested methods trained on the well-aligned MS-COCO dataset for a comprehensive comparison. The results on MS-COCO 5K are shown in Table 2, wherein the results of all baselines are reported by the original papers for a fair comparison, except for the reproduced results of BiCro. From the table, our CRCL remarkably outperforms all baselines in terms of rSum. Specifically, CRCL prevails over the best baselines by  $8.2\%$  on overall performance (i.e., rSum) absolutely, which shows that our CRCL is not only suitable for noisy cases but also performs well in well-aligned ones.
3.4 Comparison to pre-trained model
In this section, we compare our CRCL to the pre-trained model CLIP [38] to further evaluate its effectiveness in handling NC. CLIP is a well-known large pre-trained model that is trained from scratch on a dataset of 400 million image-text pairs collected from the Internet, which includes a large number of training pairs with real NCs. More specifically, following [12], we report the zero-shot results and fine-tuning results under  $20\%$  noise and compare them with that of CRCL-SGRAF in Table 3. From the results, CLIP shows a significant performance drop during fine-tuning under NC. On the contrary, the performance of our CRCL under  $20\%$  noise is even better than the zero-shot result of CLIP (ViT-L/14), which shows the strength and potential of our CRCL in dealing with NC.
Table 3: Comparison with CLIP on MS-COCO 5K.
Image → Text	Text → Image
Noise Rate	Methods	R@1	R@5	R@10	R@1	R@5	R@10	rSum
0%, Zero-Shot	CLIP (ViT-L/14)	58.4	81.5	88.1	37.8	62.4	72.2	400.4
CLIP (ViT-B/32)	50.2	74.6	83.6	30.4	56.0	66.8	361.6
20%, Fine-tune	CLIP (ViT-L/14)	36.1	61.3	72.5	22.6	43.2	53.7	289.4
CLIP (ViT-B/32)	21.4	49.6	63.3	14.8	37.6	49.6	236.3
Our CRCL	59.3	85.2	91.9	42.9	71.9	82.1	433.3
3.5 Ablation Study
In this section, we present ablation studies conducted on Flickr30K with  $60\%$  noise, as shown in Table 4. From the table, one can find that the full version of our CRCL achieves the best performance, which indicates that each component contributes to our method for performance improvement. By recasting  $q$  with the rectified label  $\hat{y}_{ii}$  using SCC,  $\mathcal{L}_r$  shows higher potential against NC. Moreover,
through the combination of active learning and robust complementary learning, there is further performance improvement, which indicates the effectiveness of our ACL. Note that these results are obtained from a single model for a comprehensive evaluation, and are not ensemble results as shown in Tables 1 and 2.
Table 4: Ablation studies on Flickr30K with  $60\%$  noise.
Configuration	Image → Text	Text → Image
loss	SCC	R@1	R@5	R@10	R@1	R@5	R@10	rSum
\( \mathcal{L}_{acl} \)	✓	70.5	91.3	95.6	52.5	79.4	86.8	476.1
\( \mathcal{L}_d \)	✓	69.3	91.1	95.2	51.4	78.2	86.0	471.2
\( \mathcal{L}_r \)	✓	67.7	91.2	95.0	52.3	79.0	86.4	471.6
\( \mathcal{L}_r(q=1) \)	25.2	53.9	65.6	20.3	45.0	57.2	267.2
\( \mathcal{L}_r(q=0) \)	63.5	89.7	93.8	48.2	74.5	82.0	451.7
\( \mathcal{L}_d \)	65.6	87.5	93.2	47.7	74.6	82.7	451.3
3.6 Visualization Analysis
To further investigate the generalization and robustness of CRCL, we visually study the retrieval performance of our CRCL-VSE $\infty$ , CRCL-SAF, and CRCL-SGR on Flickr30K and MS-COCO with varying noise rates in Figure 1(a,d). From the figure, one can see that CRCL can enhance the robustness of the original methods and perform well even under high noise. In addition, we provide visualization to intuitively showcase the effectiveness of the proposed SCC. Specifically, we visualize the distributions of both cross-modal similarities and corrected correspondences for noisy training pairs using CRCL-SGR on Flickr30K with  $60\%$  NCs. For comparison, we also visualize the results without SCC, where we replace  $\hat{y}_{ii}$  with the prediction  $\hat{p}^t (I_i,T_i)$  for the current iteration. The visualizations clearly indicate that SCC prevents the over-accumulation of errors in correspondence correction, thus verifying its effectiveness in mitigating the impact of NC.
(a) Flickr30K
(b) Similarities w/o SCC
(c) Correspondences w/o SCC
(d) MS-COCO
(e) Similarities with SCC
(f) Correspondences with SCC
Figure 1: (a,d) The performance on Flickr30K and MS-COCO with varying noise rates; (b,c/e,f) The similarities and corrected correspondences of training pairs after learning without/with SCC.
In this paper, we propose a generalized robust framework CRCL to endow traditional methods with robustness against noisy correspondences. To alleviate the harmful effects brought by NC, we present
an Active Complementary Loss (ACL) and a Self-refining Correspondence Correction technique (SCC). These techniques enable effective learning of visual-semantic associations and progressive correction of correspondences based on historical predictions, thus boosting the robustness of image-text matching. Extensive experiments demonstrate that our CRCL achieves state-of-the-art robustness to both synthetic and real-world NCs. Furthermore, ablation studies and visualization analyses further verify the effectiveness and reliability of the proposed components.
5 Limitations and Broader Impact Statement
Despite the promising performance of our proposed CRCL, there are some limitations that should be acknowledged. First, we only study the NC problem between two modalities, i.e., image and text. Second, we did not take into account category-level noise. The proposed CRCL likely impacts various applications that require robust image-text matching, e.g., multimedia retrieval, and image annotation. We encourage further study to understand and mitigate the biases and risks potentially brought by image-text matching.
This work was supported by the National Natural Science Foundation of China (U19A2078, U21B2040, 62372315, 62176171, and 62102274), Sichuan Science and Technology Planning Project (2023YFQ0020, 2023YFG0033, 2023ZHCG0016, 2022YFQ0014, 2022YFH0021), Chengdu Science and Technology Project (2023-XT00-00004-GX), the SCU-LuZhou Sciences and Technology Cooperation Program (2023CDLZ-16), and Fundamental Research Funds for the Central Universities under Grant YJ202140.
[1] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE international conference on computer vision, pages 1-9, 2015.
[2] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164, 2015.
[3] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[4] Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1218-1226, 2021.
[5] Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. arXiv preprint arXiv:2303.12501, 2023.
[6] Peng Hu, Hongyuan Zhu, Jie Lin, Dezhong Peng, Yin-Ping Zhao, and Xi Peng. Unsupervised contrastive cross-modal hashing. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2022.
[7] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612, 2017.
[8] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pages 201-216, 2018.
[9] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15789-15798, 2021.
[27] Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. Dual encoding for zero-example video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9346-9355, 2019.
[28] Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics, 43(3):1146-1151, 2013.
[29] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018.
[30] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
[31] Peng Hu, Xi Peng, Hongyuan Zhu, Liangli Zhen, and Jie Lin. Learning cross-modal retrieval with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5403-5413, 2021.
[32] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International conference on machine learning, pages 6543-6553. PMLR, 2020.
[33] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[34] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[35] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[36] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137, 2015.
[37] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673-2681, 1997.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.