SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yulei Qin $^{1}$  Xingyu Chen $^{2}$  Yunhang Shen $^{1}$  Chaoyou Fu $^{1}$
Yun Gu³ Ke Li¹ Xing Sun¹ Rongrong Ji⁴
Tencent YouTu Lab 2ByteDance
$^{3}$ Shanghai Jiao Tong University  $^{4}$ Xiamen University
yuleiqin@tencent.com
Webly supervised learning has attracted increasing attention for its effectiveness in exploring publicly accessible data at scale without manual annotation. However, most existing methods of learning with web datasets are faced with challenges from label noise, and they have limited assumptions on clean samples under various noise. For instance, web images retrieved with queries of "tiger cat" (a cat species) and "drumstick" (a musical instrument) are almost dominated by images of tigers and chickens, which exacerbates the challenge of fine-grained visual concept learning. In this case, exploiting both web images and their associated texts is a requisite solution to combat real-world noise. In this paper, we propose Cross-modality Aligned Prototypes (CAPro), a unified prototypical contrastive learning framework to learn visual representations with correct semantics. For one thing, we leverage textual prototypes, which stem from the distinct concept definition of classes, to select clean images by text matching and thus disambiguate the formation of visual prototypes. For another, to handle missing and mismatched noisy texts, we resort to the visual feature space to complete and enhance individual texts and thereafter improve text matching. Such semantically aligned visual prototypes are further polished up with high-quality samples, and engaged in both cluster regularization and noise removal. Besides, we propose collective bootstrapping to encourage smoother and wiser label reference from appearance-similar instances in a manner of dictionary look-up. Extensive experiments on WebVision1k and NUS-WIDE (Web) demonstrate that CAPro well handles realistic noise under both single-label and multi-label scenarios. CAPro achieves new state-of-the-art performance and exhibits robustness to open-set recognition. Codes are available at https://github.com/yuleiqin/capro.
Large-scale annotated datasets (e.g., ImageNet [1]) were the driving force behind the revolution of computer vision in the past decade, but the expensive and time-consuming collection process now becomes the bottleneck of model scaling. Consequently, researchers seek to crawl web images, where search queries and user tags are directly used as labels. However, the large proportion of noise in web datasets (e.g.,  $20\%$  in JMT-300M [2],  $34\%$  in WebVision1k [3], and  $32\%$  in WebFG496 [4]) impedes learning visual concepts. Many studies on Webly Supervised Learning (WSL) are conducted to reduce the negative impact of noise and effectively explore web data [5, 6, 7, 8, 9, 10].
Figure 1: (a) We explore cross-modality alignment to select clean examples and generate visual prototypes with correct semantics. (b) Collective bootstrapping provides consistent label references and regularization from visual dictionary. (c) Compared to 765 unambiguous classes, our advantage is much more highlighted on 235 classes where semantic noise prevails due to polysemy concepts.
Early WSL methods claim that simply scaling up datasets with standard supervised learning suffices to overcome web noise [3, 11, 2, 12]. Such a solution comes at the cost of huge computation resources, and the supervision source from noisy labels is proved suboptimal [13, 10, 14]. Therefore, various techniques are designed to reduce noise [15], such as neighbor density [16], guidance of clean samples [17, 18, 19], confidence bootstrapping [20, 21, 22], and side information [23, 24].
Despite the promising improvement, the above-mentioned methods still face challenges. First, most of them address certain types of noise such as label-flipping noise and out-of-distribution (OOD), neglecting the critical-yet-under-explored semantic noise. To clarify, semantic noise is caused by the misalignment between image contents and the associated texts when the search query (e.g., class) has multiple and ambiguous interpretations and the retrieved images do not correspond to the correct semantics. Without proper contextual information, it is rather difficult to pinpoint clean examples in polysemy classes. Second, the dominant idea of bootstrapping labels and discarding incorrect data is prone to noise overfitting [13]. The model predictions on individual images vary sharply over training epochs and such inconsistency also makes WSL inefficient. Some methods [4, 25, 26] also maintain peer models and require alternative steps to improve the reliability of bootstrapping. However, their complicated training procedures restrict scalability in practice.
To this end, we propose CAPro: Cross-modality Aligned Prototypes for robust representation learning from web data. Compared with previous prototypical methods [19, 27, 28], CAPro is able to handle label-flipping noise, OOD, and especially the semantic noise that remains unexplored (see Fig. 1).
First, CAPro exploits web data across modalities to formulate semantically-correct textual and visual prototypes. Since visual prototypes simply formed with images suffer from semantic ambiguity, we propose text matching to leverage textual prototypes to establish their noise-robust estimation. Motivated by successful language models [29, 30, 31], we extract textual knowledge to imply the extent to which one instance is aligned with its textual prototype. Specifically, we prepare descriptive texts (e.g., definitions) of each category and project them into the embedding space as textual prototypes via the pre-trained language model. For each sample, its affiliated texts of the website title, caption, and tags are also encoded into the same embedding space. The similarity between a sample and its prototype indicates "cleanness". Since incomplete and mismatched image-text pairs introduce additional noise [32], we bring in text enhancement with text guidance from visually-similar neighbors to mitigate the effect of noisy texts on clean sample selection. We consecutively construct image and text graphs and rerank neighbor candidates for text matching and enhancement. Samples that exactly match the target semantics are chosen as anchors for the initialization of visual prototypes. These visual prototypes are continuously polished up by high-quality web images to improve generalizability and discriminability. During representation learning, the intra-class distance between prototypes and instances is minimized by contrastive learning for regularization. By means of class-representative visual prototypes, various kinds of noise can be filtered out.
Second, we propose collective bootstrapping (CB) to provide smoother label reference by extending bootstrapping with collective knowledge. For each sample, instead of bootstrapping its target independently [33, 34], CAPro keeps bootstrapping the entire dynamic dictionary and provides label reference in the mode of dictionary look-up. The dictionary keys are the learned representations from the sampled data while the query is the current encoded sample. We aggregate model predictions on all keys and use their weighted combination as the pseudo target, where weights are determined by the matching scores of query-key pairs. By penalizing deviation from such targets, The proposed
Figure 2: Overview of CAPro. Images  $\mathbf{x}_i$  and texts  $\mathbf{t}_i$  are respectively fed into the image and text encoders for features  $\mathbf{v}_i$  and  $\mathbf{s}_i$ . Then,  $\mathbf{v}_i$  is projected into the embedding space as  $\mathbf{z}_i$ , followed by the reconstruction from  $\mathbf{z}_i$  to  $\tilde{\mathbf{v}}_i$ . Visual prototypes  $\mathbf{z}^c$  are initialized with anchor instances that are selected by matching enhanced texts  $\tilde{\mathbf{s}}_i$  to textual prototypes  $\mathbf{s}^c$  for semantic alignment. They are constantly polished up by clean images and engage in contrastive learning to constrain cluster distribution. Collective bootstrapping exploits visual dictionary for regularization on the auxiliary classifier output  $\mathbf{q}_i$ , where each key embedding is matched to the query for the reference  $\mathbf{b}_i$ . Web labels  $y_i$  are simultaneously refined as  $\tilde{y}_i$  for "denoised" supervision on the classifier output  $\mathbf{p}_i$ .
CB achieves two advantages: 1) It encourages consistent performance among the query and visually similar keys. 2) Unstructured noise is suppressed by referring to the dictionary for label regularization. Our CB can also be viewed as encoding the neighborhood structure of data in the low-dimensional space, where the closely matched keys are neighbors of the query. Inductive propagation of self-labels is implicitly realized through such a structure, which draws on the assumption of manifold regularization [35] that close samples probably belong to the same class.
In summary, our contributions are summarized as:
- We propose CAPro, a prototypical learning framework to efficiently handle various noises including label-flipping noise, OOD, and semantic noise for webly supervised learning.
- We integrate class prototypes across text and image modalities to align with unambiguous semantics. We verify that text enhancement by visual guidance is the key to handling noisy texts for clean sample selection, which in turn improves visual prototypes.
- We investigate collective bootstrapping as label regularization by matching one query to all keys in a dynamic dictionary for reference. We show that scaling up the nearest neighbors from the mini-batch to the entire dictionary better leverages the visual data structure.
Experiments on WebVision1k and NUS-WIDE (Web) confirm the competitiveness of CAPro with prior state-of-the-art methods. CAPro performs robustly against real-world noise under single-label and multi-label scenarios and demonstrates superiority in open-set recognition.
Webly Supervised Learning (WSL) WSL aims at utilizing abundant but weakly labeled web data. It serves a range of tasks, such as recognition [36, 37, 38, 39, 40, 41], detection [42, 43], and segmentation [44, 45]. In this study, we focus on visual representation learning [46, 11, 2, 3, 47, 12]. To combat noise, previous studies combine web labels with the pseudo labels generated by the model. With respect to the pseudo labels, Hinton et al. [34] adopts soft targets in a fashion of distillation.
Tanaka et al. [21] considers both soft and hard targets and proposes an alternative optimization framework. Yang et al. [22] estimates the correctness of hard targets on a case-by-case basis and dynamically balances the two supervision sources with label confidence. Recently, the idea of prototypical learning [48] has been applied for WSL. Han et al. [49] predicts self-labels by prototype voting and uses a constant ratio to combine web and pseudo labels. Momentum Prototype (MoPro) [28] improves prototypes with the momentum update policy [50] for smooth label adjustment.
The differences between CAPro and the closely related MoPro exactly highlight our contributions. First, we take advantage of both textual and visual prototypes to handle semantic misalignment noise. MoPro neglects such noise and its prototypes could be overwhelmed by irrelevant samples, which impairs the subsequent sample correction. Second, MoPro does not refer to appearance-similar samples for self-labels, which is prone to real-world noise that causes unreasonable label updates. In contrast, CAPro adopts bootstrapping wisely by performing dictionary look-up: the model prediction of the current query sample refers to its matching keys in the dictionary, where the matching score by visual similarity determines the contribution of each key. Third, MoPro only tackles single-label representation learning while CAPro extends prototypical contrastive learning for the multi-label scenario, which is non-trivial due to the intricate nature of noisy multi-labeled data. In that case, CAPro maintains prototypes in subspaces of the shared embedding space, which not only resolves the inter-class contrastive conflicts but also fosters implicit exploitation of label dependency.
Noise-Robust Learning from Neighbors Several approaches attempt to correct labels with neighborhood consensus. Both Wang et al. [51] and Guo et al. [16] measure data complexity using local neighbor density for sample selection. Huang et al. [52] progressively discovers neighbors to form decision boundaries in an unsupervised manner. Bahri et al. [53] filters out training data whose label collides with the predictions of its  $k$ -NN. Wu et al. [54] employs topology to only keep the largest connected component of a  $k$ -NN graph. Neighborhood collective estimation [55] evaluates model confidence on the "cleanness" of each candidate by its neighbors. Ortego et al. [56] identifies correct examples by comparing original labels with the soft labels from their neighbors. Neighbors are also involved in consistency regularization [57, 58] to encourage similar outputs on samples within the  $k$ -neighborhood. Such inductive label propagation allows correct supervision to transfer directly to mislabeled data via the neighborhood structure.
Unfortunately, the aforementioned methods are not scalable due to the huge complexity of updating a global  $k$ -NN graph frequently. Both Li et al. [57] and Iscen et al. [58] only consider neighbors within a mini-batch for on-the-fly graph construction. However, web data tends to be sparsely distributed and the graph built within mini-batch samples hardly provides reliable neighbors. To deal with such a trade-off, CAPro pinpoints all potential neighbors in the dictionary by matching representations without explicit graph building. We maintain the dictionary as a queue whose length is much larger than the batch size, enabling collective bootstrapping from appropriate neighbors.
Nearest neighbors play a vital role throughout our CAPro, from text enhancement to matching and collective bootstrapping. Compared with previous methods, our mechanism differs in that: 1) We acquire guidance from cross-modality neighbors, where noisy texts are enhanced by image neighbors to alleviate the mismatch problem. In contrast, existing studies investigate neighbors of one modality. 2) We exploit reciprocal structures to filter nearest neighbors for pertinent text matching, while most works neglect those top-ranked false positive neighbors. 3) We resort to neighbors for on-line collective bootstrapping in a manner of dictionary look-up instead of explicit graph construction.
Learning with Visual-Semantic Alignment Various tasks seek to learn visual representations for semantic concepts, including retrieval [59, 60, 61, 62, 63], caption [64, 65, 66], matching [67, 68], visual question answer [69], and zero-shot learning [70, 71]. Recently, learning unified embeddings has been studied for foundation models by language-image pre-training [72, 73, 74, 75, 76, 77].
In WSL, textual metadata such as titles and hashtags are too scarce to carry out language-image pre-training. Few studies harness both images and texts to learn semantically-correct representations. Zhou et al. [23] designs a co-training scheme to extract semantic embeddings to transfer knowledge from head to tail classes. Cheng et al. [24] builds visual and textual relation graphs to choose prototypes by graph-matching. Yang et al. [78] builds a visual-semantic graph and uses a graph neural network for label refinement. Nevertheless, these methods do not take noisy texts into serious consideration and underestimate their negative effect on seed selection. We also find that image outliers are wrongly kept even with textual concept matching. For example, images of a rugby team
are ranked top for the class "tiger-cat" just because the team name "tiger-cat" is frequently mentioned. On the contrary, CAPro introduces text enhancement by smoothing and reranking to improve its robustness to noise. Furthermore, the prototypes in [24, 78] are fixed during training, while CAPro keeps polishing up prototypes with clean examples for better domain generalizability.
Noisy Correspondence Rectification One paradigm similar to WSL is noisy correspondence rectification or calibration [60, 65, 62, 66, 61, 63, 68, 79]. It tackles the mismatched image and text pairs and aims to simultaneously learn aligned visual and textual embeddings for improved cross-modal retrieval. Huang et al. [65] utilizes the memorization effect of neural networks to partition clean and noisy data and then learns to rectify correspondence. Hu et al. [61] derives a unified framework with contrastive learning to reform cross-modal retrieval as an N-way retrieval. Han et al. [66] proposes a meta-similarity correction network to view the binary classification of correct/noisy correspondence as the meta-process, which facilitates data purification. Our CAPro differs in two aspects: 1) We focus on the label noise where images are wrongly-labeled by keywords or hashtags. Noisy correspondence emphasizes the instance-level mismatch between an image and its associated text. 2) We aim to learn visual representations with categorical labels while most methods on noisy correspondence align image and text embeddings to improve cross-modal retrieval.
3.1 Problem Definition and Framework Architecture
Given an interested class  $y_{i} \in \{1, \dots, C\}$ , web data are collected as  $D = \{(\mathbf{x}_i, \mathbf{t}_i, y_i)\}_{i=1}^N$ , where  $\mathbf{x}_i$  and  $\mathbf{t}_i$  respectively denote the image and textual metadata. Due to the noise issues,  $y_{i}$  might not equal to the ground-truth  $y_{i}^{*}$ . We aim to optimize a deep model  $\mathcal{F}(\theta_e; \theta_c)$  with parameters of an encoder  $\theta_e$  and a classifier  $\theta_c$ . Existing WSL studies often neglect  $\mathbf{t}_i$  and seldom consider the intervention between images and texts. Comparatively, our CAPro unearths  $\mathbf{t}_i$  for aligning visual representations with semantic concepts, which facilitates correction of various kinds of noise.
CAPro consists of the following components (see Fig. 2). Siamese image encoders extract features  $\mathbf{v}_i, \mathbf{v}_i' \in \mathbb{R}^{d_v}$  from inputs  $\mathbf{x}_i$  and their augmented counterparts  $\mathbf{x}_i'$ . Following MoCo [50], parameters of the first query encoder  $\theta_e^1$  are updated by back-propagation and those of the second key encoder  $\theta_e^2$  are updated by the momentum method. A text encoder generates embeddings  $\mathbf{s}_i, \mathbf{s}^c \in \mathbb{R}^{d_t}$  respectively from the instance  $\mathbf{t}_i$  and the category  $\mathbf{t}^c$ . Any off-the-shelf language model can be used with its pre-trained encoder frozen. A classifier, via a fully-connected (FC) layer, maps  $\mathbf{v}_i$  to predictions  $\mathbf{p}_i \in \mathbb{R}^C$  over  $C$  classes. A projector distills discriminative low-dimensional embeddings  $\mathbf{z}_i \in \mathbb{R}^{d_p}$  from  $\mathbf{v}_i$ . It has two FC layers, followed by  $\ell_2$ -normalization for unit-sphere constraint on  $\mathbf{z}_i$ . A constructor, symmetric to the projector, recovers  $\tilde{\mathbf{v}}_i$  from  $\mathbf{z}_i$  to be close to  $\mathbf{v}_i$ . An auxiliary classifier, of the same structure as the classifier, outputs predictions  $\mathbf{q}_i \in \mathbb{R}^C$  on  $\mathbf{z}_i$ . A dictionary, implemented as a queue of size  $Q \times d_p$ , records keys for both contrastive learning and collective bootstrapping. The latest embeddings  $\mathbf{z}_i'$  are dequeued while the oldest are dequeued.
Image encoders and classifiers are trained with a cross-entropy loss. Since features  $\mathbf{v}_i$  contain redundant description that is vulnerable to image corruption and domain gap, we emphasize class-indicative contents by learning a low-dimensional embedding space. Inspired by denoising autoencoders [80, 27], a projector and a reconstructor are designed to optimize the projection from  $\mathbf{v}_i$  to  $\mathbf{z}_i$ . An auxiliary classifier helps retain the representation capacity of  $\mathbf{z}_i$ .
$$
\mathcal {L} _ {i} ^ {\mathrm {c l s}} = - \log \left(\mathbf {p} _ {i \left(y _ {i}\right)}\right), \mathcal {L} _ {i} ^ {\mathrm {p r j}} = \| \tilde {\mathbf {v}} _ {i} - \mathbf {v} _ {i} \| _ {2} ^ {2} - \log \left(\mathbf {q} _ {i \left(y _ {i}\right)}\right). \tag {1}
$$
3.2 Cross-Modality Alignment
Text Encoding For raw texts in metadata, we remove all html tags, file format extensions, punctuation, digits, and stop words. Then, tokenization is performed in accordance with the language model in use. After that, we obtain the pre-processed metadata  $\mathbf{t}_i$  and use the text encoder to extract  $\mathbf{s}_i$ .
Text Enhancement To handle missing and mismatched texts, we assume that similar images should share similar texts, and consider text enhancement with guidance from visual data structure. One simple way of encoding visual structure is to build a global  $k$ -NN graph on  $\mathbf{v}_i$  [78]. However, our
preliminary experiments show that the top-ranked neighbors may not be pertinent due to the noise and domain gap. To detect truly matched neighbors, we construct a  $k$ -reciprocal-NN graph [81]  $\mathcal{G} = \{\mathcal{V}, \mathcal{E}\}$  and use the re-ranking technique to evaluate neighbor relationship. Each node in  $\mathcal{V}$  denotes an image and the edge connectivity from  $\mathcal{E}$  is represented as the adjacency matrix  $A$ .
$$
A _ {i j} = \left\{ \begin{array}{l l} 1 - d \left(\mathbf {v} _ {i}, \mathbf {v} _ {j}\right) & , \text {i f} \mathbf {x} _ {i} \in \mathcal {R} \left(\mathbf {x} _ {j}, k\right) \text {o r} \mathbf {x} _ {j} \in \mathcal {R} \left(\mathbf {x} _ {i}, k\right), \\ 0 & , \text {o t h e r w i s e}, \end{array} \right. \tag {2}
$$
where  $\mathcal{N}(\mathbf{x}_i,k)$  and  $\mathcal{R}(\mathbf{x}_i,k) = \{\mathbf{x}_j|\mathbf{x}_j\in \mathcal{N}(\mathbf{x}_i,k)\wedge \mathbf{x}_i\in \mathcal{N}(\mathbf{x}_j,k)\}$  respectively denote  $k$  -NN and  $k$  -reciprocal-NN of  $\mathbf{x}_i$ . The cosine distance is used here:  $d(\mathbf{v}_i,\mathbf{v}_j) = 1 - \frac{\mathbf{v}_i\cdot\mathbf{v}_j}{\|\mathbf{v}_i\|_{\|}\mathbf{v}_j\|}$ . Neighbor re-ranking is achieved by re-calculating the pairwise distance. The vanilla cosine distance only weighs relative priority by measuring features, overlooking the context information of overlapped reciprocal neighbors. Hence, Jaccard Distance [81, 82, 83] is introduced to measure the intersection between reciprocal neighbors. The refined distance  $d^{*}(\mathbf{v}_{i},\mathbf{v}_{j}) = \frac{1}{2} (d(\mathbf{v}_{i},\mathbf{v}_{j}) + d_{J}(\mathbf{v}_{i},\mathbf{v}_{j}))$
$$
d _ {J} \left(\mathbf {v} _ {i}, \mathbf {v} _ {j}\right) = 1 - \frac {\sum_ {k = 1} ^ {N} \min  \left(V _ {\mathbf {v} _ {i} , \mathbf {v} _ {k}} , V _ {\mathbf {v} _ {j} , \mathbf {v} _ {k}}\right)}{\sum_ {k = 1} ^ {N} \max  \left(V _ {\mathbf {v} _ {i} , \mathbf {v} _ {k}} , V _ {\mathbf {v} _ {j} , \mathbf {v} _ {k}}\right)}, V _ {\mathbf {v} _ {i}, \mathbf {v} _ {j}} = \left\{ \begin{array}{l l} \exp (- d (\mathbf {v} _ {i}, \mathbf {v} _ {j})) & , \text {i f} \mathbf {x} _ {j} \in \mathcal {R} (\mathbf {x} _ {i}, k) \\ 0 & , \text {o t h e r w i s e .} \end{array} \right. \tag {3}
$$
Given  $\mathcal{G}$ , smoothing is performed on  $\mathbf{S} = (\mathbf{s}_1, \mathbf{s}_2, \dots, \mathbf{s}_N) \in \mathbb{R}^{N \times d_t}$  via graph convolution [84]:  $\hat{\mathbf{S}} = \tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}\mathbf{S}, \tilde{A} = A + I_N, \tilde{D}_{ii} = \sum_j\tilde{A}_{ij}$ , where  $\tilde{A}$  refers to the adjacency matrix with self-connection and  $\tilde{D}$  is the diagonal degree matrix.