CAPro: Webly Supervised Learning with Cross-modality Aligned Prototypes
Authors: Yulei Qin
Xingyu Chen
Yunhang Shen
Chaoyou Fu
Yun Gu
Ke Li
Xing Sun
Rongrong Ji
Venue: NeurIPS 2023 poster (2023)
Keywords: webly supervised learning
representation learning
visual-semantic alignment
collective bootstrapping

Abstract: Webly supervised learning has attracted increasing attention for its effectiveness in exploring publicly accessible data at scale without manual annotation. However, most existing methods of learning with web datasets are faced with challenges from label noise, and they have limited assumptions on clean samples under various noise. For instance, web images retrieved with queries of ”tiger cat“ (a cat species) and ”drumstick“ (a musical instrument) are almost dominated by images of tigers and chickens, which exacerbates the challenge of fine-grained visual concept learning. In this case, exploiting both web images and their associated texts is a requisite solution to combat real-world noise. In this paper, we propose Cross-modality Aligned Prototypes (CAPro), a unified prototypical contrastive learning framework to learn visual representations with correct semantics. For one thing, we leverage textual prototypes, which stem from the distinct concept definition of classes, to select clean images by text matching and thus disambiguate the formation of visual prototypes. For another, to handle missing and mismatched noisy texts, we resort to the visual feature space to complete and enhance individual texts and thereafter improve text matching. Such semantically aligned visual prototypes are further polished up with high-quality samples, and engaged in both cluster regularization and noise removal. Besides, we propose collective bootstrapping to encourage smoother and wiser label reference from appearance-similar instances in a manner of dictionary look-up. Extensive experiments on WebVision1k and NUS-WIDE (Web) demonstrate that CAPro well handles realistic noise under both single-label and multi-label scenarios. CAPro achieves new state-of-the-art performance and exhibits robustness to open-set recognition. Codes are available at https://github.com/yuleiqin/capro.

CAPro: Webly Supervised Learning with Cross-Modality Aligned Prototypes
Yulei Qin
Xingyu Chen²
Yunhang Shen
Chaoyou Fu'
Yun Gu
Ke Li!
Xing Sun'
Rongrong Ji
Teneent YouTu Lab
ByttDancc
Shanghai Jiao Tong University "Xiamen University
yuleiqin@tencent.cm
Abstract
Webly supervised learning has attracted increasing attention for its effectiveness exploring publicly accessible data scale without manual annotation. However, most existing methods of learning with web datasets are faced with challenges from label noise, and they have limited assumptions on clean samples under various noise. For instance, web images retrieved with queries of "liger cat" (a cat species) and "drumstick" (a musical instrument) are almost dominated by images of tigers and chickens, which exacerbates the challenge of fine-grained visual concept learning. In this case, exploiting both web images and their associated texts is a requisite solution to combat real-world noise. In this paper, we propose Cross- Cross-modality Aligned Prototypes (CAPro), unified prototypical contrastive learning framework to learn visual representations with correct semantics. For one thing, we leverage textual prototypes, which stem from the distinct concept definition of classes, to select clean images by text matching and thus disambiguate the formation of visual prototypes. For another, handle missing and mismatched noisy texts, we resort to the visual feature space to complete and enhance individual texts and thereafter improve text matching. Such semantically aligned visual prototypes are further polished up with high-quality samples, and engaged in both cluster regularization and noise removal. Besides, we propose collective bootstrapping to encourage smoother and wiser label reference from appearance-similla instances in a manner of dictionary look-up. Extensive experiments on Web kk and NUS-WIDE (Web) demonstrate that CAPro well handles realistic noise under both single-label and multi-labe scenarios. CAPro achieves new state-of-the-un performance and exhibits robustness open-set recognition. Codes available at hhttps://githul cachuu..commyaaeee coom/yuleiqin/aan capro
Introduction
Large-scale annotated datasets (e.g., ImageNet (1) were the driving force behind the revolution of computer vision in the past decade, but the expensive and time-consuming collection process now becomes the bottleneck of model scaling. Consequently, researchers seek crawl web images, where search queries and user tags are directly used as labels. However, the large proportion of noise in web datasets (e.g.. 20% in JMT-300M [2), 34% in WebVision [3]. and 32% WebFG496 [ impedes learning visual concepts. Many studies on Webly Supervised Learning (WSL) are conducted to reduce the negative impact of noise and effectively explore web data
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Drumstick
Image Test
Reference
Effect Visual Proiceypes
"caedces
Look-un
WuaVidastt
73.5
Drumstick
Image &
Modaalydalllyy Visual Sersaticc
Alignntest
Boctstropiin
Collective
67.8
71a
06.1
71.9
64.2
08.0
FRALE
Reference
61.6
Visual Dictionary
Look-up
"icooret somantics polysemy classes
Vanilla
MoPro
CAProfoars)
(b)
(c)
Figure 1: (a) We explore cross- mods--mo aligmment to select clean examples and generate visual prototypes with correct semantics. (b) Collective bootstrapping provides consistent label references and regularization from visual dictionary. (c) Compared to 765 unambiguous classes, our advantage is much more highlighted on 235 classes where semantic noise prevails due to polysemy concepts.
Early WSL methods claim that simply scaling up datasets with standard supervised learning suffices to overcome web noise resources, and the supervision source from noisy labels is proved suboptimal [13] 10, 14]. Therefore,
12]. Such a solution comes at the cost of huge computation
various techniques are designed to reduce noise [15], such as neighbor density [66.. guidance of
clean samples [7 [8 19]. confidence bootstrapping (00 21 22], and side information 23 24]
Despite the promising improvement, the above-mentioned methods still face challenges. First, most of them address certain types of noise such as tabel-flippin noise and out- -ff-----iitiiuttion (OOD), neglecting the setiiccl-yy--udde--epprrere semantic noise. To clarify, semantic noise is caused by the misalignment between image contents and the associated texts when the search query (e.g., class) has multiple and ambiguous interpretations and the retrieved images do not correspond to the correct semantics. Without proper contextual information, it is rather difficult to pinpoint clean examples in polysemy classes. Second, the dominant idea of bootstrapping labels and discarding incorrect data is prone to noise overfitting [ The model predictions epochs and such inconsistency also makes WSL inefficient. Some methods [4] 25 266 also maintain peer models and require alternative steps to improve the reliability of bootstrapping. However, their complicated training procedures restrict scalability in practice.
individual images vary sharply over training
To this end, we propose CAPro: from web data. Compared with previous prototypical methods 19 27 28), CAPro able to handle label-fiiping noise, OOD, and especially the semantic noise that remains unexplored (see Fig. D.
Cross-modality Aligned Prototypes for robust representation learning
First, CAPro exploits web data across modalities to formulate semantically-coreet correct textual and visual prototypes. Since visual prototypes simply formed with images suffer from semantic ambiguity, we propose text matching leverage textual prototypes establish their noise-robust estimation. Motivated by successful language models [29 30 31], we extract textual knowledge to imply the extent to which one instance is aligned with its textual prototype. Specifically, we prepare descriptive texts (e.g., definitions) of each category and project them into the embedding space as textual prototypes via the pre-trained language model. For each sample, its affiliated texts of the website title, caption, and tags are also encoded into the same embedding space. The similarity between a sample and its prototype indicates "cleanness"" Since incomplete and mismatched image-text pairs introduce additional noise (32, we bring in text enhancement with text guidance from visually- similar neighbors to mitigate the effect of noisy texts on clean sample selection. We consecutively construct image and text graphs and rerank neighbor candidates for text matching and enhancement. Samples that exactly match the target semantics are chosen as anchors for the initialization of visual prototypes. These visual prototypes are continuously polished up by high-quality web images to improve generalizability and discriminability. During representation learning, the intra-class distance between prototypes and instances is minimized by contrastive learning for regularization. By means of class-represed reass-representttvee visual prototypes, various kinds of noise can be filtered out.
Second, we propose collective bootstrapping (CB) to provide smoother label reference by extending bootstrapping with collective knowledge. For each sample, instead of bootstrapping its target independently [33] 34], CAPro keeps bootstrapping the entire dynamic dictionary and provides label reference in the mode of dictionary look-up. The dictionary keys are the learned representations from the sampled data while the query is the current encoded sample. We aggregate model predictions on all keys and use their weighted combination as the pseudo target, where weights are determined by the matching scores of query-key pairs. By penalizing deviation from such targets, The proposed
2

---

Off-ise
Text Medalit
Test Enceder
Textull Pronctypce
Teak divegee
Ine Manhing
2001
5001
Teat Eaooder
Ensaee Iacoder
Projecer
Mazal Prosuyypc
G (USd)
Tighr Dusniic
miiied (WSLL)
Disurtaiie
Irauge Modalley
Sncoder
Crus-Modalit Aligurment with Di
On line
seal Preoottpps
Lusaat Eacoder
Projeckor
Projectox Staeed
Image Tinasder
X) Cestraaiive loes
Visual D
Dielionary
Menoeiimi
X Collective Bootstrgping
Staree
QQ Visui Dichioary
Classifeer
Paple Beocstopping less
Aus. Clas
WVVisa Proteigpea
Stare
Sturts
Cho-elferr R.
-hellsg callen
Neise Remtoval
9, - Recoratraction hoa
Recomntraadr
Di
Figure 2: Overview CAPro. Images x, and texts t; are respectively fed into the image and text encoders for features V; and s;. Then, V; is projected into the embedding space 2,, followed by the reconstruction from z; to Vi. Visual prototypes z" are initialized with anchor instances that selected by matching enhanced texts to textual prototypes s for semantic alignment. They are constantly polished up by clean images and engage in contrastive learning to constrain cluster distribution. Collective bootstrapping exploits visual dictionary for regularization classifier output q, where each key embedding is matched to the query for the reference b,- Web labels y, are simultaneously refined y; for "denoised" supervision on the classifier output p;.
the auxiliary
CB achieves two advantages: 1) It encourages consistent performance among the query and visually similar keys. 2) Unstructured noise is suppressed by referring to the dictionary for label regularization. Our CB can also be viewed as encoding the neighborhood structure of data in the low-dimensional space, where the closely matched keys are neighbors of the query. Inductive propagation of self- labels is implicitly realized through such structure, which draws the assumption of manifold regularization 35 that close samples probably belong to the same class.
In summary, our contributions are summarized as:
We propose CAPro, prototypical learning framework to efficiently handle various noises including label-fliping noise, OOD, and semantic noise for webly supervised learning.
We integrate class prototypes across text and image modalities to align with unambiguous semantics. We verify that text enhancement by visual guidance the key to handling noisy texts for clean sample selection, which in turn improves visual prototypes.
We investigate collective bootstrapping as label regularization by matching one query all keys in a dynamic dictionary for reference. We show that scaling the nearest neighbors from the mini-batch to the entire dictionary better leverages the visual data structure.
Experiments Web Visionso and LUS-WIDE (Web) confirm the competitiveness of CAPro with prior state-of-the-a methods. CAPro performs robustly against real-world noise under single-label and multi-label scenarios and demonstrates superiority in open-set recognition.
2 Related Work
Webly Supervised Learning (WSL) WSL aims at utilizing abundant but weakly labeled web data. It serves range of tasks, such as recognition [36 37
and To combat noise, previous studies combine web labels with the pseudo labels generated by model. With respect to the pscudo labels, Hinton et al. [34] adopts soft targets in fashion of distillation.
detection segmentation 44 45 In this study, we focus on visual representation learning 46
40
12

---

Tanaka et al. [21] considers both soft and hard targets and proposes an alternative optimization framework. Yang et al. [22] estimates the correctness of hard targets on a case-by-case basis and dynamically balances the two supervision sources with label confidence. Recently, the idea of prototypical learning [8] has been applied for WSI.. Han et al. [49] predicts self-labels by prototype voting and uses a constant ratio to combine web and pseudo labels. Momentum Prototype (MoPro) 28 improves prototypes with the momentum update policy [50] for smooth label adjustment.
The differences between CAPro and the closely related MoPro exactly highlight our contributions. First, we take advantage of both textual and visual prototypes to handle semantic misalignment noise. MoPro neglects such noise and its prototypes could be overwhelmed by irrelevant samples, which impairs the subsequent sample correction. Second. MoPro does not refer to appearance-sing similar samples for self-labels, which is prone to real-world noise that causes unreasonable label updates. In contrast, CAPro adopts bootstrapping wisely by performing dictionary look-up: the model prediction of the current query sample refers to its matching keys in the dictionary, where the matching score by visual similarity determines the contribution of each key. Third, MoPro only tackles single-label representation learning while CAPro extends prototypical contrastive learning for the multi-label scenario, which is non-trivial due to the intricate nature of noisy multi-labeled data. In that case, CAPro maintains prototypes in subspaces of the shared embedding space, which not only resolves the inter-class contrastive conflicts but also fosters implicit exploitation of label dependency.
Noise-Robust Learning from Neighbors Several approaches attempt to correct labels with neigh- borhood consensus. Both Wang et al. [51] and Guo et al. [16] measure data complexity using local neighbor density for sample selection. Huang et al. [52] progressively discovers neighbors to form decision boundaries in unsupervised manner. Bahri el al.
[33 filters out training data whose label collides with the predictions of its k-NN. Wu et al. [54] employs topology to only keep the largest connected component of a k-NN graph. Neighborhood collective estimation [55] evaluates model confidence on the "cleanness" " of each candidate by its neighbors. Ortego et al. [56] identifies correct examples by comparing original labels with the soft labels from their neighbors. Neighbors also involved in consistency regularization (57, (8))58) to encourage similar outputs on samples within the k-neighborhood. Such inductive label propagation allows correct supervision to transfer directly to mislabeled data via the neighborhood structure.
Unfortunately, the aforementioned methods are not scalable due to the huge complexity of updating global k-NN graph frequently. Both Li el al. [57] and Iscen et al. (58) only consider neighbors within for on-the-fly fly graph construction. However, web data tends to be sparsely distributed and the graph built within mini-batch samples hardly provides reliable neighbors. To deal with such trade-off, CAPro pinpoints all potential neighbors in the dictionary by matching representations without explicit graph building. We maintain the dictionary as a queue whose length is much larger than the batch size, enabling collective bootstrapping from appropriate neighbors.
Nearest neighbors play a vital role throughout our CAPro, from text enhancement to matching and collective bootstrapping. Compared with previous methods, our mechanism differs in that: 1) We acquire guidance from cross-modality neighbors, where noisy texts are enhanced by image neighbors to alleviate the mismatch problem. In contrast, existing studies investigate neighbors of one modality. 2) We exploit reciprocal structures to filter nearest neighbors for pertinent text matching, while most works neglect those top-ranked false positive neighbors. 3) We resort to neighbors for on-line collective bootstrapping in a manner of dictionary look-up instead of explicit graph construction.
Learning with Visual-Semantic Alignment Various tasks seek to learn visual representations for semantic concepts, including retrieval (59 60 61 62 63]. caption (64 65. (66, matching [6 68]
visual question answer (69], and zero-shot learning 71. Recently, learning unified embeddings
has been studied for foundation models by language-image pre-training 772 74
75 76
In WSL, textual metadata such as titles and hashtags are too scarce to carry out language-image representations. Zhou et al. [23] designs co-training scheme to extract semantic embeddings to transfer knowledge from head to tail classes. Cheng et al. [24] builds visual and textual relation graphs to choose prototypes by graph-matching. Yang et al. (78) builds a visual-semanttc graph and uses a graph neural network for label refinement. Nevertheless, these methods do not take noisy texts into serious consideration and underestimate their negative effect on seed selection. We also find that image outliers are wrongly kept even with textual concept matching. For example, images of rugby team
pre-training. Few studies harness both images and texts to learn semantically

---

are ranked top for the class "tiger-cat" just because the team name "tiger-cat" is frequently mentioned. On the contrary, CAPro introduces text enhancement by smoothing and reranking to improve its robustness to noise. Furthermore, the prototypes in [24] 78 are fixed during training, while CAPro keeps polishing up prototypes with clean examples for better domain generalizability.
Noisy Correspondence Rectification One paradigm similar to WSL is noisy correspondence
rectification or calibration [60, 65 62 66
61
63
68, [7]. tackles the mismatched image and
text pairs and aims to simultancously learn aligned visual and textual embeddings for improved cross-modal retrieval. Huang er al. [65] utilizes the memorization effect of neural networks to partition clean and noisy data and then learns to reetify correspondence. Hu et al. [6]] derives unified framework with contrastive learning to reform cross-modal retrieval as an N-way retrieval. Han et al. [66] proposes meta-similarity correction network to view the binary classification of correct/noisy correspondence as the meta-process, which facilitates data purification. Our CAPro differs in two aspects: 1) We focus the label noise where images wrongly-labeled by keywords or hashtags. Noisy correspondence emphasizes the instance-level mismatch between an image and its associated text. 2) We aim to learn visual representations with categorical labels while most methods on noisy correspondence align image and text embeddings to improve cross-modal retrieval.
3 Method
3.1 Problem Definition and Framework Architecture
Given interested class yi E (I,..., C). web data are collected as D = {(x,, t, ())); where x; and t; respectively denote the image and textual metadata. Due to the noise issues, Y; might not equal to the ground y We aim to optimize deep model F 02) with parameters of an encoder 0g and classifier de- Existing WSL studies often neglect t; and seldom consider the intervention between images and texts. Comparatively, our CAPro unearths t, for aligning visual representations with semantic concepts, which facilitates correction of various kinds of noise.
CAPro consists of the following components (see Fig. D Siamese image encoders extract features Vi. v E IRd from inputs and their augmented counterparts x/. Following MoCo 500 parameters of the first query encoder 07 are updated by back-propagation and those of the second key encoder 02 are updated by the momentum method. A text encoder generates embeddings s;, s IRd, respectively from the instance t; and the category ts. Any off-the-shelf language model can be used with its pre-trained encoder frozen. A classifier, via a fully-connected (FC) layer, maps V; to predictions P, E RCC over C' classes. A projector distills discriminative low-dimensional embeddings z, E RR from Vi. It has two FC layers, followed by fz-oormalization for unit-sphere constraint 2;. A reconstructor, symmetric to the projector, recovers v; from z, to be close to v;- An auxiliary classifier, of the same structure as the classifier, outputs predictions qi E IR C on 2,- A dictionary, implemented as a queue of size Q x dp. records keys for both contrastive learning and collective bootstrapping. The latest embeddings z are enqueued while the oldest are dequeued.
Image encoders and classifiers are trained with cross-entropy loss. Since features V; contain redun- dant description that is vulnerable to image corruption and domain gap. we emphasize -aass-indieetive contents by learning
diwdeeensional embedding space. Inspired by denoising autoencoders [80 27). a projector and reconstructor are designed to optimize the projection from v; to z, An auxiliary classifier helps retain the representation capacity of 2,.
cols log(P)).)) CPdH) = || -vll2 -log(
(1)
3.2 Cross-Modality Alignment
Text Encoding For raw texts in metadata, we remove all html tags, file format extensions, punctua- tions, digits, and stop words. Then, tokenization is performed in accordance with the language model in use. After that, we obtain the pre-processed metadata t; and use the text encoder extract s,,
Text Enhancement
To handle missing and mismatched texts, we assume that similar images should share similar texts, and consider text enhancement with guidance from visual data structure. One simple way of encoding visual structure is to build global k-NN graph on v, (78). However, our
5

---

preliminary experiments show that the top-ranked neighbors may not be pertinent due to the noise and domain gap. To detect truly matched neighbors, we construct a k-reciprocal-NN graph [8]] G = (V. {) and use the re-ranking technique to evaluate neighbor relationship. Each node in V denotes an image and the edge connectivity from represented as the adjaceney matrix A.
{0 d(vi, vz)
if R((x; k)) or xj R((x; k),
A.j
(2)
oothewwise,
where (xx, k) and R(x; k) = (xylx, E (x,, k) Ax; E Mxx,, k)} respectively denote k-NN ViV; ||v; Neighbor re-ranking is achieved by re-calculating the pairwise distance. The vanilla cosine distance only weighs relative priority by measuring features, overlooking the context information of overlapped reciprocal neighbors. Hence, Jaccard Distance 8T 82 83 is introduced to measure the intersection between reciprocal neighbors. The refined distance d' (Vi, V2) (d((v, V2) + dd(vs, V,)):
and k-reciprocal-NN of x, The cosine distance is used here: d(v,, vj)
dv(v,,v))=1 min( Viai Vegva) k= max Veave Ve,,va)
exp( Texp(-d(ve) d(; v2)) ifx E R(xi, oothewiise.
Vvnv,
Given G. smoothing is performed on S (81, (81,,),...,NN) E IR Nxd via graph convolution [84] = カー A &s, = A TN. Dis E, Aij: where Á refers to the adjacency matrix with
self-connection and D is the diagonal degree matrix.
Textual Prototypes To establish textual prototypes, do not estimate s from instances in one class (e.g., average) considering the insufficient, noisy nature of metadata. Instead, we refer to WordNet [85] for the vocabulary hierarchy [86] 78, 24 For the e-th class, extract its definition in WordNet and expand context with its siblings (synonyms), children (hyponyms), and parents (hypernyms). Then, we get t and encode for sf, Such prototypes s have two advantages: I) enriches semantic representations of classes. For instance, the comprehensive text of the class "tiger cat" is cat having a striped coat; domestic cat, house cat. felis domesticus, felis catus: any domesticated member of the genus Felis. It provides disambiguation to web instances of "tiger-cat" (medium-sizze sized wildcat in Central South America) and "tiger, cat" (large feline of forests in most of Asia having tawny coat with black stripes; endangered). 2) It reveals the underlying inter-class relationship by language encoding. The structural information of class hierarchy is hard infer from metadata instances but can be directly indexed in WordNet.
Text Matching With textual prototypes as queries, web instances with correct semantics can be retrieved by matching queries to their embeddings as keys. To improve precision, the same distance measurement in Eq. (3) for k-reciprocal-NV encoding adopted to rerank the matched candidates. We sort samples by distance in an ascending order, and select the top-K as clean set Dx.
DK D U D U... Dj ((x;, t. vi) (2: c) A (d (B, ok)),
(4)
where of denotes the K-th smallest distance in the c-th class.
Visual Prototypes D& plays an anchoring role in shaping visual prototypes. We initialize the e-th prototype z" by averaging instances in D 2 Ex.EDg z,
| Given such a good starting point, visual prototypes are consistently polished up by trustworthy web examples with a momentum coefficient mp: 2 m2z + (1 - m,, z = T ls We perform instance instance-prototype contrastive learning to pull instances around their prototypes and push apart different class clusters. Instance-level discrimination also encouraged improve separation across classes.
2n
2"
C. log exp(z, th /T)
exp(z; ZL/T)
clus log
(5)
exp(z, z" 2//T)
Σラ exp(z, z =
where is temperature coefficient.

---

Noise Removal Noisy instances can be filtered out by self seff-prediction and instance protttyye similarity. We refer to MoPro (88 for rules of label adjustment by O, IR:
exp(z; z" /T)
op: (1 o) ((( > = explzi x² ///T)
(6)
where a balances two terms. Given threshold S S I, the pseudo-label ÿt estimated by:
1/1
if E DK, else if max, O((c) > 7 else if Oi(ss) 1/C, otherwise.
arg max, Oz(e)
V/
Null (OOD)
The above control flow guarantees continuous guidance from Dr on cluster separation. If the highest score of o; is above 7, the label will be changed accordingly. To prevent aggressive elimination of hard examples, we keep an instance till the next epoch so long as its confidence above average. Otherwise, it is removed as OOD. The refined label y; successively affects Eqs. (I) and (5)
3.3 Collective Bootstrapping
Due to memorization [87, as the training epoch increases, deep models will be prone overfit noise even with the carefully designed logic of noise removal. We assume that overfitting occurs less dramatically when majority can be consulted for the model to avoid over-confident decision on one single instance. With regard to the consultancy basis, the fow-dimensional embedding is good choice because its distilled description about visual contents is robust enough. Therefore, we propose to exploit the large dictionary, which is originally set up for instance-wiss contrastive learning, to realize collective bootstrapping by dictionary look-up. The matching scores of the current query 2; to in the dictionary act as the weights for the bootstrapped representations b; IR..
all keys zi
exp(z: 2//TT)
exp(z): zz /T)
b; Σ waylaaq +1 -a)r)) Wij Ee exp(z; 2//TT
(8)
((k E exp(z) x 2²/T)
We minimize the difference between predictions and bootstrapping targets via KL-divergence loss.
hts (Ki(q: b.)
Σ lie) log
qite) ba(e)
It not only allows collaborative contribution individual soft label estimation, but also encourages consistent performance on visually similar examples. Note that such regularization is imposed on the auxiliary classifier qı- Compared with pa, constraints on qi coincide with our contrastive learning setting without potential conflicts with the hard label assignment in Eq. (7). The total objective is:
C. = Ei
== (1 C
bts Cbts + XrI Lpr) Xpro L. + Cins.
4
Experiments
4.1
Experimental Setup
We evaluate CAPro WebVisionlk 3 (Google500 78 and JUS-WIDE (Web) 88 for single-
label and multi-label representation learning, respectively. They contain image-text pairs which are in line with WSL setting. All datasets under investigation are described in Sec. A We perform ablation studies on Google500 and NUS- WIDE for low cost without losing generalization [22] 78). The R50 M (L6) [30] are used as image/text encoders by default. Exhaustive details about hyper-pann parameeers,, implementation, and training are elaborated in Secs. B and Algo.
4.2 Comparison with the SOTA
Table 1 reports the topp/ accuracy of WebVision 1 and Google500. Results of the SOTA methods trained and evaluated on the same datasets are quoted here. Due to different choices of image encoders and training strategies, prior methods may not be directly comparable. For example, VSGraph adopts
7

---

Table Results on Web and Google500. Best/2nd best are marked bold/underlined.
Back-
WebVision
ImageNerlk Google500 ImageNer500
Method
bone
Top1
Top5
Topl
Top5 Topl Top5
Top1
Top5
MentorNet
17
IRV2 [90
72.6
88.9
64.2
84.8
Curriculum 16
IV2 191
72.1
89.1
64.8
84.9
Multimodal
IV3 [93]
73.2
89.7
Vanilla [2
R50D [94]
75.0
89.2
67.2
84.0
75.4
88.6
68.8
84.6
SCC [22]
R50D
75.3
89.3
67.9
84.7
76.4
89.6
69.7
85.3
Vanilla 78
R50
74.2
89.8
68.2
86.2
66.9
82.6
61.5
78.8
CoTeach 20
R50
67.6
84.0
62.1
80.9
VSGraph 78
R50
75.4
90.1
69.4
87.2
68.1
84.4
63.1
81.4
Vanilla 28
R50
72.4
89.0
65.7
85.1
SOMNer 8
R50
72.2
89.5
65.0
85.1
Curriculum [6
R50
70.7
88.6
62.7
83.4
CleanNet 18
R50
70.3
87.7
63.4
84.5
SINet 24
R50
73.8
90.6
66.8
85.9
NCR 58
R50
73.9
NCR 58
R50
75.7
Mil.c 26
R50
75.2
90.3
67.1
85.6
MoPro 28
R50
73.9
90.0
67.8
87.0
Vanilla (ours)
R50
72.6
89.7
67.0
86.8
69.9
86.5
64.5
83.1
CAPro (ours)
R50
74.2
90.5
68.0
87.2
76.0
91.3
72.0
89.2
Results on WebVision Ik are under optimized training settings with hatch size of 1024.
the same R50, but is trained with batch-size of 1024. The benefits of a larger batch size have been validated in NCR, where the same method achieves 75.7% and 73.9% in topl accuracy respectively for the batch size of 1024 and 256. We believe batch size the reason that vanilla baseline [78] surpasses most SOTAs. Due to the limited budget, training with a batch size of 1024 is currently affordable, but we will experiment in the future. In this case, methods within each row group of Table are fairly comparable with each other, including the vanilla trained by a cooss-y loss.
CAPro achieves quite competitive performance WebVisionlk, with improvement of 1.6% (topl accuracy) over our vanilla. Although SCC and VSGraph respectively opt for stronger backbones (e.g., R50D) and longer training epochs (e.g.. 150) with larger batch size (e.g.. 1024), CAPro still excels in terms of the top5 accuracy. Furthermore, our gain of 1% (top1 accuracy) on ImageNetlk demonstrates that CAPro is robust to the domain gap between web and real-world datasets. Web data include advertisements, artworks, and renderings that differ from realistic photographs. On Google500 and ImageNet500, CAPro outperforms existing methods despite our disadvantages.
Table 2: Results on NUS-WIDE (Web).
Table 3: Results on open-set recognition.
Back-
NUS-WIDE
Web Wisiin
ImageNet
Method
Method
bone
C-F1
O-F1
mAP
C-F1
C-F1
Vanilla 78
R50
37.5
39.6
43.9
Vanilla 78
50.5
46.4
VSGraph 778
R50
38.6
40.2
44.8
CoTeach 20 78
$1.0
47.7
MCPL 995
R101
22.5
17.2
47.4
VSGraph 78
$7.2
52.8
Vanilla (ours)
R50
37.8
42.4
38.3
Vanilla (ours)
54.6
48.3
CAPro (ours)
R50
39.3
45.4
48.0
CAPro (ours)
62.2
57.8
Table reports per-Class Fl (C-F1), Overall F1 (O-F1), and mean Average Precision (mAP) on NUS- WIDE. Most prior multi-label methods are developed for ground-truth labels, while we are concerned with noisy WSL settings. Under this cireumstance, CAPro compared with methods that are trained on NUS- WIDE (Web) and evaluated on clean testing set. Following [96] 78), the top three categories of an image by prediction confidence are chosen for metric calculation. CAPro reaches the SOTA with significant increase of 1.5% (C-F1), 3.0% (O-F1), and 9.7% (mAP) over our vanilla.
4.3 Discussion on Open-Set Recognition
To verify if CAPro can identify outliers of unknown categories, we conduct experiments on open- set recognition. Specifically, we train CAPro on Google500 and validate on the testing sets of WebVision and ImageNetlk. Images from the remaining 500 classes all belong to one open-set

---

category. We follow 98 78 to classify an image as open-set if its highest prediction confidence is below threshold. The average C-FI adopted reflect whether a model can discriminate between base and novel classes (501 in total). Table 3 confirms our superiority over existing methods, showing that CAPro is capable of detecting semantic novelty. More analysis can be found in Sec.
Table 4: Ablation study on text encoding, enhancement, and reference provider.
Text
Text Enhan-
Reference Provider
Google500
ImageNet500
NUS-WIDE
Encoding
cement
Topl
Top5
Topl
Top5
C-F1
O-FI
mAP
X
X
71.5
87.8
66.5
84.6
37.2
42.4
46.2
Minil.
VSCraph
72.0
88.0
66.9
85.4
39.2
44.4
46.8
MiniLM
V (ours)
75.5
91.0
71.5
88.8
39.3
44,9
47.4
XLNet
VSGraph
78
71.6
87.8
66.8
84.8
38.6
43,4
47.6
XLNet
v (ouns)
75.4
91.0
71.5
88.8
39.3
45.1
47.5
GPT-Neo
VSGraph
X
72.0
88.0
67.2
85.3
39.2
45.0
47.4
GPT-Neo
V (ours)
X
75.7
91.1
71.6
88.5
39,2
45.1
47.6
MiniLM V (ours)
Mix-up (MU)
75.7
90.9
71.4
88.6
38.7
45.3
47.2
MiniLM M
V (ours)
Bootstrap [33
75.5
90.8
71.3
88.4
38.1
43.2
46.0
MiniLM
V (ours)
Label smooth 100
75.4
90.8
71.2
88.4
36.9
42.1
46.8
MiniLM
V (ours)
SCC [2
73.8
89.9
70.2
88.0
35.6
41.3
45.0
Minil. V (ours)
NCR 58
75.5
91.1
71.5
88.8
37.6
43.4
46.8
MiniLM V (ours)
v CB (ours)
76.0
91.3
72.0
89.2
39.3
45.4
48.0
MiniLM V (ours)
CB (ours) MU
76.5
91.1
71.9
88.8
40.4
46.7
49.9
GPT-Neo V (ours)
V CB (ours)
76.1
91.4
72.1
89.4
39.3
44.9
47.7
GPT-Noo V (ours)
v CB (ours) + MU
76.5
91.2
72.0
88.8
40.7
45.2
50.0
4.4 Ablation Study
Text Encoding and Enhancement Table 4 reveals the benefit from cross-modality alignment. For the method without text encoding and enhancement, we sample K examples randomly from each category. These instances barely provide reliable prototypes with semantic correctness. With respect to the text encoder, we additionally validate XLNet (basc) [29] and GPT-Neo (I.3B) [31]. MiniLM surpasses XLNet by minor margin, but both exhibit similar performance with our enhancement. GPT-Neo displays its power even with the plain k-NN-based smoothing in VSGraph (78), implying that advanced Large Language Model (LLM) would boost CAPro. Note that encoders in CLIP [76] are not applicable here to avoid visual data leakage. All methods with text encoding and enhancement outperforms the vanilla one, validating the guidance from textual knowledge. Besides, we notice an increase up to 3.8%/4.7% on Google500/lmageee50000 by our text enhancement, showing that proper noise suppression indispensable. Fig. 1.s33pressse presents a comparison on the top-K K matched instances. Enhancement in VSGraph can filter out OOD to certain degree, but can do nothing with lexical confusion (e.g., food in Drumstick and players in Tiger cat). More qualitative results are in Sec. D.I
Reference Provider Our collective bootstrapping is compared against commonly used regulariza- tion methods which provide reference on targets. Surprisingly, most existing methods bring about no or even negative impact. In one respect, bootstrapping and label smoothing are already implicitly inherited in our framework as Eqs. (6) and 77. Therefore, no further gains can be seized. As for SCC, its estimated confidence may not comply with our Eq. (6) which leads to incompatible optimization. NCR has two drawbacks: 1) The chance of similar instances appear in one mini-batch is fairly small for large web datasets. 2) It only counts on self- reff-prediction as reference source which is fragile to noise. In contrast, CAPro enlightened by MoCo to maintain the dictionay as a queue, which enlarges the number of reference objects beyond instances in mini-batei Our enriched sources from both self- prediction and instance- prottttype similarity expedite a steady learning progress. Moreover, mix-up improves CAPro in topl but lowers top5. It adopts convex combinations for both inputs and targets, enforcing a stronger regularization than our CB where we only recombine targets. For Web lk examples with noisy labels still resemble prototypes and therefore neighbor knowledge brings useful reference. Mix-up does not consider appearance similarity and causes over-regularization.
bts and top-K Fig. shows that an increasing b triggers off worse results on Google500 and ImageNet500. This suggests that collective knowledge should not overwhelm individual instance decision. With regard top- K on prototype initialization, there exists a trade-off between domain- variety and class-purity. The rise of K increases diversity but also the risk of noise.

---

More ablation studies on the threshold 7, the update frequency of visual prototypes, and the noise removal policy can be found in Secs. D.2 D.3 and D.4 respectively. Empirical guidelines on prper-parameter tuning are concluded in Sec. We also provide analysis of failure cases in Sec. Our computation cost with respect to performance gains can be found in Sec.
Tiger cat (domestic cat)
Drumstick (musical instrument)
(b)
(c)
Figure 3: Top-matched WebVisionIk Ik instances are chosen: (a) without text enhancement, (b) with text enhancement in VSGraph (78), and (c) with our text enhancement.
2 Google5U0
2 NUS WIDE
Top-K on Google500
Top-K on NUS- WIDE
75
76
75.5
76.2
45.4
45.3
47.8
45.3
47.6
45.2
76
75.9
76.0
76.1
46
46
48.1
45.5
48,0
45.4
45.0
47.4
72
71.5
718
71.9
71.6
40
42
39.2
73
71.6
72.0
71.9
39,4
39,3
39.6
38.7
05
0
0.5
0.5
07
25
25
100
-00ogg00000 -mmgeNee15000
C-FI
--0-1
-AP
Gongle:000
ImageNec500
O.F.FF
--
Figure 4: Impact of hyper- prrper-prramtter Xbts and top- K on CAPro.
Conclusion
CAPro utilizes web datasets to learn visual representations that aligned with correct semantics. Cross-modality alignment and collective bootstrapping are corroborated as two keys to improve WSL. The benefits of building prototypes descriptions can be efficiently removed by simply measuring the distance between an instance and its prototype in the embedding space. 2) The inter-class relationship can be statistically studied by comparing each instance to all class prototypes, which may shed light on visual similarity for species. Three potential drawbacks should be considered: 1) the limited intra-class diversity with less tolerance to the minority in one category. Images crawled from websites follow the long-tailed distribution, which means that the more common or typical one instance is, the greater the likelihood that it gets exposed online. Over-emphasis on the purity of class prototypes leads to false negatives on the recognition of atypical samples. One possible solution is introduce randomness into the initialization and update of prototypes to improve generalization. 2) the noteworthy domain gap and bias of web data. Even image contents are correct, their styles (e-g., advertising photos, artworks, and rendering) are different from the realistic datasets. When it comes to modalities such as infrared or medical tomography images, there exist very few images online. Therefore, it is encouraged to prepare realistic images for guided-training and evaluation, where early-stopping and normalization techniques can used to avoid overfitting. 3) the accuracy of prior knowledge about class hierarchy. Since definitions of class prototypes rely on the systematic understanding of concepts, improper, coarse-grann grasned,, or even wrong descriptions would devalue the semantic alignment. A thorough analysis on class concepts is requisite to developing prototypes. Future work includes extension to other modalities (e.g., audio and video clips) and to other tasks (e.g., semi-supervised learning).
two-fold: 1) Noisy web data whose visual and textual
Broader Impact CAPro manoeuvres language models for visual concept learning, where LLMs (e.g., GPT-Neo) can deliver promising results for future development, Regardless data sources, we showcase a cost-efficient and practical way to utilize cross-modality data. It also promotes rethinking the key-value matching mechanism for creative usages. For example, the visual dictionary originally built for instance- wise contrastive learning is re-discovered for our collective bootstrapping.
10

---

Acknowledgements
The authors would like to express gratitude to the anonymous reviewers who greatly help improve the manuseript. In addition, the authors thank Hao Cheng from the University of California, Santa Cruz for the valuable comments.
References
[1] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 248-255. IEEE, 2009.
[2] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 843-852, 2017.
[3] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862, 2017.
[4] Zeren Sun, Yazhou Yao, Xiu-Shen Wei, Yongshun Zhang, Fumin Shen, Jianxin Wu, Jian Zhang, and Heng Tao Shen. Webly supervised fine- ane-grained recognition: Benchmark datasets and an approach. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10602-10611, 2021.
Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei. The unreasonable effectiveness of noisy data for fine- grained recognition. In European Conference Computer Vision, pages 301-320 Springer, 2016.
[6] Parneet Kaur, Karan Sikka, and Ajay Divakaran. Combining weakly and webly supervised learning for classifyyng food images. arXiv preprint 17X1v:171..08730, 2017.
[7] Chuanyi Zhang, Yazhou Yao, Huafeng Liu, Guo-Sen Xie, Xiangbo Shu, Tianfei Zhou, Zheng Zhang, Fumin Shen, and Zhenmin Tang. Web-supervised network with softly update-drop training for fine-grained visual classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12781-12788, 2020.
[8] Yi Tu, Li Niu, Junjie Chen, Dawei Cheng, and Liqing Zhang. Learning from web data with self self-organizing memory module. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12846 22846-1285,, 2020.
[9] Huafeng Liu, Haofeng Zhang, Jianfeng Lu, and Zhenmin Tang. Exploiting web images for fine-grained visual recognition via dynamic loss correction and global sample selection. IEEE Transactions on Multimedia, 24: 10111105,,,11111 2021.
[10] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning (still) requires rethinking generalization. Communications of the ACM, 64 13):107-115, 2021.
[11] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. European Conference on Computer Vision, pages 181-196, 2018
[12] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv preprint arXiv: 13770, 2(8), 2019.
[13] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.

---

[14] Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia. Sudanthi learning with noisy labels. In
Wijewickrema, and Janes Bailey. Dimensionality-drinn International Conference on Machine Learning, pages 3355-3364. PMLR, 2018.
[15] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Gil Lee. Learning from noisy labels with deep neural networks: survey. IEEE Transuctions on Neural Networks and Learning Systems, 2022.
[16] Sheng Guo, Weilin Huang. Haozhi Zhang. Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In European Conference on Computer Vision, pages 135-150. 2018.
[17] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pages 2304-2313. PMLR, 2018.
[18] Kuang- Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5447-5456, 2018
[19] Yulei Qin, Xingyu Chen, Chao Chen, Yunhang Shen, Bo Ren, Yun Gu, Jie Yang, and Chunhua Shen. Fopro: Few-shot guided robust webly-supervised prototypical learning. arXiv preprint (XXiv22222000400 2022.
[20] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems, 31, 2018.
[21] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5552-5566 2018
[22] Jingkang Yang, Litong Feng, Weirong Chen, Xiaopeng Yan, Huabin Zheng, Ping Luo, and Wayne Zhang. Webly supervised image classification with self-contained confidence. In European Conference on Computer Vision, pages 779-795. Springer. 2020.
[23] Xiangzeng Zhou, Pan Pan, Yun Zheng, Yinghui Xu, and Rong Jin. Large scale long-tailed product recognition system alibaba. In Proceedings of ACM International Conference on Information & Knowledge Management, pages
2020.
[24] Lele Cheng, Xiangzeng Zhou, Liming Zhao, Dangwei Li, Hong Shang, Yun Zheng, Pan Pan, and Yinghui Xu. Weakly supervised learning with side information for noisy labeled images. In European Conference on Computer Vision, pages 306-321. Sprinerr 2020.
[25] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2019.
[26] Sai Rajeswar, Pau Rodriguez, Soumye Singhal, David Vazquez, and Aaron Courville. Multi- label iterated learning for image classification with label ambiguity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4783-4793, 2022.
[27] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2020.
[28] Junnan Li, Caiming Xiong, and Steven Hoi. Mopro: Webly supervised learning with momen- tum prototypes. In International Conference on Learning Representations, 2020.
[29] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems, 32, 2019.
[30] Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for tank-agi compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
12

---

[31] Sid Black. Leo Gao, Phil Wang, Connor Leaby, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorfloww March 2021. URLhttps //doi 1r.ol.or///0..58//eeoooo metadata.
If you use this software, please cite it using these
[32] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[33] Scott E Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In International Conference on Leurning Representations (Workshop), 2015.
[34] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0233 2(7), 2015.
[35] Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7(11), 2006.
[36] Hilde Kuchne, Ahsan Iqbal, Alexander Richard, and Juergen Gall. Mining youtube-a dataset for learning ane-grained action concepts from webly supervised video data. arXiv preprint arXiv:1906.01000 2019.
[37] Alessandro Bergamo and Lorenzo Torresani. Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach. Advances in Neural Information Processing Systems, 23, 2010.
[38] Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, and Dahua Lin. Omni-sourced webly-supervised learning for video recognition. In European Conference on Computer Vision, pages Springer, 2020.
[39] Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Nge: unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 62-71, 2021.
[40] Yazhou Yao, Jian Zhang, Fumin Shen, Xiansheng Hua, Jingsong Xu, and Zhenmin Tang. Exploiting web images for dataset construction: A domain robust approach. IEEE Transactions on Multimedia, 1978):77711177 2017.
[41] Yazhou Yao, Xiansheng Hua, Guanyu Gao, Zeren Sun, Zhibin Li, and Jian Zhang. Bridging the web data and fine-grained visual recognition via alleviating label noise and domain mismatch. In Proceedings of the ACM International Conference on Multimedia, pages 1735-1744, 2020.
[42] Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. Learning everything about anything: Webly-supervised visual concept learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3270-322 00700-327..00 2014.
[43] Yunhang Shen, Rongrong Ji, Zhiwei Chen, Xiaopeng Hong, Feng Zheng, Jianzhuang Liu, Mingliang Xu, and Qi Tian. Noise-aware fully webly supervised object detection. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11326 1133,, 2020.
[44] Tong Shen, Guosheng Lin, Chunhua Shen, and lan Reid. Bootstrapping the performance of webly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1363-1371, 2018.
[45] Bin Jin, Maria V Ortiz. Segovia, and Sabine Susstrunk. Webly supervised semantic seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 0626636333 2017.
[46] Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. European Conference on Computer Vision, pages 67-84. Springer, 2016.
13

---

[47] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5051 2019.
[48] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30, 2017.
[49] Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5138-5147, 2019.
[50] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729 9738, 2020.
[51] Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu- Tao Xia. Iterative learning with open-set noisy labels. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 8688-8698 2018.
[52] Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neighbourhood discovery. In International Conference on Machine Learning, pages 2849- 2858. PMLR, 2019.
[53] Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Conference on Machine Learning, pages 540-550. PMLR, 2020.
[54] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. Advances in Neural Information Processing Systems, 33:21382-2133 2020.
[55] Jichang Li, Guanbin Li, Feng Liu, and Yizhou Yu. Neighborhood collective estimation for noisy label identification and correction. In European Conference Computer Vision, pages 128 145. Springer, 2022.
[56] Diego Ortego, Eric Arazo, Paul Albert, Noel E O' D'Connor, and Kevin McGuinness. Multi- objective interpolation training for robustness to label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6606-6615, 2021.
[57] Junnan Li, Caiming Xiong, and Steven CH Hoi. Learning from noisy data with robust repre- sentation learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9485-9494, 2021.
[58] Ahmet Iscen, Jack Valmadre, Anurag Amab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4672-4681, 2022.
[59] Raul Gomez, Lluis Gomez, Jaume Gibert, and Dimosthenis Karatzas. Learning to learn from web data through deep semantic embeddings. In European Conference on Computer Vision Workshops, pages
2018.
[60] Peng Hu, Xi Peng, Hongyuan Zhu, Liangli Zhen, and Jie Lin. Learning cross cross-modal retrieval with noisy labels. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5403-5413, 2021.
[61] Peng Hu, Zhenyu Huang, Dezhong Peng, Xu Wang, and Xi Peng. Cross-modal retrieval with partially mismatched pairs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[62] Yang Qin, Dezhong Peng, Xi Peng, Xu Wang, and Peng Hu. Deep evidential learning with noisy correspondence for cross-modal retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4948 4956, 2022.
[63] Huaiwen Zhang, Yang Yang, Fan Qi, Shengsheng Qian, and Changsheng Xu. Robust video-text retrieval via noisy pair calibration. IEEE Transactions Multimedia, 2023.
14

---

[64] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3128-3137, 2015.
[65] Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. Advances in Neural Informa- tion Processing Systems, 34:29406-29444 2021.
[66] Haochen Han, Kaiyao Miao, Qinghua Zheng, and Minnan Luo. Noisy correspondence learning with meta similarity correction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 7517-7526, 2023.
[67] Yijie Lin, Mouxing Yang. Jun Yu, Peng Hu, Changqing Zhang, and Xi Peng. Graph matching with bi-level noisy correspondence. arXiv preprint arXiv:2212.04085, 2022.
[68] Zheng Li, Caili Guo, Zerun Feng, Jenq-Neng Hwang, and Zhongtian Du. Integrating language guidance into image-text matching for correcting false negatives. IEEE Transactions on Multimedia, 2023.
[69] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123:32-73, 2017.
[70] Shafin Rahman, Salman Khan, and Nick Barnes. Improved visual-semantic alignment for zero-shot object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11932-11939, 2020.
[71] Rui Gao, Xingsong Hou, Jie Qin, Yuming Shen, Yang Long, Li Liu, Zhao Zhang, and Ling Shao. Sesual-semantic aligned bidirectional network for zero-shot learning. IEEE Transactions on Multimedia, 2022.
[72] Hao Wu, Jiayuan Mao, Yufeng Zhang. Yuning Jiang, Lei Li, Weiwei Sun, and Wei- Ying Ma. Unified
embeddings: Bridging vision and language with structured meaning representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6609-6618, 2019.
[73] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for langun-language tasks. In European Conference Computer Vision, pages 121-137. Springer, 2020.
[74] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Confer- ence on Computer Vision, pages 104-120. Springer, 2020.
[75] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le, Yun- Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pages 9904-4916 PMLR, 2021.
[76] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages $748-8763. PMLR, 2021.
[77] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[78] Jingkang Yang, Weirong Chen, Litong Feng, Xiaopeng Yan, Huabin Zheng, and Wayne Zhang Webly supervised image classification with metadata: Automatic noisy label correction via visual-semantic graph. In Proceedings of ACM International Conference on Multimedia, pages
83-91, 2020.
15

---

[79] Shuo Yang, Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, and Min Xu. Bicro:
Noisy correspondence rectification for modal similarity consistency. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 19883-19892, 2023.
data via bi-directional cross-
[80] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning, pages 1096-1100 103, 2008.
[81] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identification with k-reciprocal encoding. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1318-1327, 2017.
[82] Song Bai and Xiang Bai. Sparse contextual activation for efficient visual re-ranking. IEEE Transactions on Image Processing, 25(3): 1056-190,,,, 2016.
[83] Mang Ye, Chao Liang, Yi Yu, Zheng Wang, Qingming Leng, Chunxia Xiao, Jun Chen, and Ruimin Hu. Person reidentification via ranking aggregation of similarity pulling and dissimilarity pushing. IEEE Transactions on Multimedia, 18(1 18(12):5533-2666 2016.
[84] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2016.
[85] George A Miller. WordNet: An electronic lexical database. MIT press, 1998.
[86] Tingting Wei, Yonghe Lu, Huiyou Chang, Qiang Zhou, and Xianyu Bao. A semantic approach for text clustering using wordnet and lexical chains. Expert Systems with Applications, 42(4): 2264-2275, 2015.
[87] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pages 233-242. PMLR, 2017.
[88] Tat-Seng Chua, Jinhui Tang, Richang Hong. Haojie Li, Zhiping Luo, and Yantao Zheng. NUS- WIDE: real-world web image database from national university of singapore. In Proceedings of the ACM International Conference on Image and Video Retrieval, pages 1-9, 2009.
[89] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 770-778, 2016.
[90] Christian Szegedy, Sergey Ioffe. Vincent Vanhoucke, and Alexander Alemi. Inception- -4, inception-resme reenett and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[91] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 1-9, 2015.
[92] Manan Shah, Krishnamurthy Viswanathan. Chun Ta Lu, Ariel Fuxman, Zhen Li, Aleksei Timofeev, Chao Jia, and Chen Sun. Inferring context from pixels for multimodal image classification. In Proceedings of the ACM International Conference on Information and Knowledge Management, pages 189-198, 2019.
[93] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2826, 2016.
[94] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition, pages 558-567, 2019.
16

---

[95] Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep convnet for multi-label classification with partial labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 647-657, 2019.
[96] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu, Cnn-rnn: A unified framework for multi-label image classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2285-2294 2016.
[97] Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pages 2898-28 2906, 2019.
[98] Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Nae- mura. Clasification-rcoosstuttioo learning for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4016 4025, 2019.
[99] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
[100] Rafacl Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in Neural Information Processing Systems, 32, 2019.
[101] Jorge Ramón Fonseca Cacho, Ben Cisneros, and Kazem Taghva. Building a wikipedia n-gram corpus. In Proceedings of SAI Intelligent Systems Conference, pages 277-294. Springer, 2021.
17

---

A Datasets Details
A.1 Web Visionlk
contains 2.4M web images collected from Google and Flickr, which share the same 1K category names with ImageNet1k []. For each example, we use all available description, title, and tag in its metadata for raw text preparation. Besides, we follow [22 78 to use the subset of Web Google500 for ablation studies in consideration of lower GPU resource and time consumption without losing generalization. contains 0.48M images from Google with randomly chosen 500 categories. The testing set of ImageNetIk and its subset ImageNet500 involved well for evaluation.
A.2 NUS-WIDE (Web)
contains 0.26M web images from Flickr with 5k unique user tags. Each example is manually annotated with multiple labels within 81 concepts that are filtered out of the 5k tags. It also provides weak labels (an official web version) by checking each of the 81 category name appears in user tags of every example. Almost 50% of the web labels are wrong and 50% of the true labels are missing in tags [8]. Since tags contain phrases without delimiters, we split phrases based on unigram frequencies [101 for raw text preparation. We follow [78] to train models with weakly-labela training set (a.k.a., NUS- WIDE Web) and validate them on the clean testing set.
B Mathmatical Notations
In this section, we present the description of all math notations in the manuscript (see Table 5).
C
Implementation and Training Details
C.1 General Settings
In consideration of performance and efficiency, we adopt ResNet-50 [89] and MiniLM-L6 [30] as image and text encoders by default. The training settings are listed in Table 6 In general, we refer to 28 for: batch size 256; optimizer SGD; learning rate linearly rises with 10 warm-up epochs and decays by cosine schedule. Specifically, although the benefit of an increased batch size (e.g.. 1024) has been validated (78 58,, we follow the standard batch size setting of 256 on WebVision for two reasons: I) comparability with most of the previous methods; 2) limited computing resources with 8 GPUs (a mini-batch size of 32 on each GPU for batch size of 256 in total).
We empirically set APrj = 1, APro = 1, Ains 1, dts = 0.1, m, = 0.999, d, = 128, 7 0.1, 0.5, and K 50 by default. Their optimal values require meticulous fine-tuning of each hyper-parameter, which is beyond consideration of the present study.
Data augmentation (random cropping, resacling, and horizontal flipping) is applied on the inputs to the query encoder while stronger augmentation, including color jittering and blurring (50)), is added on those the key encoder.
Experiments are conducted on a CentOS 7 workstation with an Intel 8255C CPU, 377 GB Mem, and 8 NVIDIA V100 GPUs. The training of CAPro Web Visionl,, Google500, and NUS-WIDE respectively costs about ten days, three days, and one day under the environment mentioned above.
C.2 Wehiinnnkk--ong Implementation
We refer to 28 for Q 8192 and 7 = 0.6. The learning rate is 0. and the number of epochs is 120. Given the complexity of graph construction, we use k for both text denoising and matching.
C.3 NUS-WIDE (Web)-only Implementation
In view of the dataset scale, we set Q = 2048 with a learning rate of 10-3 and total of 100 epochs. & = 10 is chosen since user tags noiser and sparser in NUS-WIDS
18

---

Table List of symbols.
Symbol
Description a web image indexed with
X;
t;
textual metadata associated with x;
web label associated with X; ground-truth label associated with X;
VI
N
the total number of images in web dataset the total number of categories a web dataset parameters of image encoder parameters of classifier
C
D
ee
Oe
F(0c; 0c)
a deep model with image encoder and classifier dimension of the visual features
du
visual features of extracted by image encoder, E IR dimension of the textual embeddings
de
textual embeddings of t, extracted by text encoder, 8; E IR
s,
to
category definition of the e-th class
s
textual embeddings of to extracted by text encoder, sf E IR
predictions V, from classifier, p, E IR. Pi(k) denotes its k-th element dimension of the visual embeddings low-dimensional embeddings of v; after projection, 2, E RR reconstructed visual features from 24, V, E RRd predictions on 2, from auxiliary classifier, qi E IR the size of dictionary, namely the length of queue number of nearest neighbors (NN) in k-NN and k-reciprocal-NN vertices, nodes edges
P
dp
2;
V;
q
Q
k
V
E
9 A
graph, G EV. adjacency matrix
(xx,
k-NN sets of x;
R(x;,k) k)
k-reciprocal-Ne sets of x; distance between V; and Vj refined distance between V; and v k-reciprocal feature encoding the distance between v, and V
d(v., Vj)
d' (v..,V))
S
concatenated textual embeddings, (s1 S2, $2, sy), IR Nxd, identity matrix
IN
A
adjacency matrix A with self-connection IN degree matrix of Á
Da
S
denoised after smoothing denoised 8; after smoothing
S,
K
top-K selected examples with visual-semantic alignment
of
the K-th smallest distance d' (5..s)) in the e-th class top-K set of the c-th class sets of top-K examples from all classes unnormalized visual prototype of the e-th class normalized visual prototype of the e-th class temperature coefficient
D
DR
2
2
weight between self- eeffprediction and prototype-instance similarity fused, comprehensive output for label correction, E IRC prototype-instanee similarity, r, IR. P((K)) denotes its k-th element threshold for label correction momentum coefficient
r;
7
mp
b;
collective bootstrapping target on Z; weight of contribution from z in the dictionary for b;
Wij
weight for collective bootstrapping loss
Aprj
weight for projection and reconstruction losses
Arro
weight for prototypical contrastive loss
X inn
weight for instance contrastive loss

---

Table List of for training settings.
Settings
WebVision I/Google500 NUS-WIDE
Optimizer
SGD
Optimizer momentum
0.9
Optimizer weight decay
1 x 10
Batch size
256
Stepl pre-training scheduler
Cosine decay with linear warm-up
Stepl pre-training warm-up epochs
5/10
10
Stepl pre-training learning rate
1 x 10-1
2 x 10-3
Stepl pre-training epochs with encoders frozen
Stepl pre-training epochs total
120
100
Step2 training scheduler
Cosine decay with linear warm-up
Step2 training warm-up epochs
5/10
10
Step2 training learning rate
1 x 10
2 x 10
Step2 training epochs with encoders frozen
5/10
10
Step2 training epochs in total
60/1 20
100
Step3 tine-tuning scheduler
Cosine decay
Step3 fine-tuning warm-up epochs
Step3 fine-tuning learning rate
1 x 10
2 x 10-5
Step3 fine-tune fine-tunin epochs with encoders frozen
15/20
20
Step3 fine-tuning epochs in total
15/20
20
Image encoder (by default)
ResNet-50 89
Text encoder (by default)
MiniLM-Lo
30
Arro
X ins
X bts
0.1
mp
0.999
dp
128
t
0.1
0.5
top-K
50
8192
2048
0.6
0.9
S-NN/k-reciprocal-NNN
10
To support multi-label learning, is necessary, but not yet enough, to simply replace the softmax- based cross-entropy losses with sigmoid-based based binary cross-entropy losses. The premise of prototypical contrastive learning does not hold true anymore because one instance can simultaneously engaged in formation of multiple clusters, which violates the exclusivity inherited in softmax activation. Our experiments demonstrate that, only by projecting V; into compact subspaces specific to each class, properly learn the decision boundary to continue prototypical learning. Technically, we set additional fully-connected (FC) layers after the projector to respectively map z; into Zie E RR = 2, ..., For the c-th class, both positive and negative prototypes že z shaped accordingly for the contrast against 2,.c- Such operation be viewed as magnifying class-indicas contents via recombination the shared embedding z,, Another minor modification is required noise removal, where the output Og is fused independently for the c-th class for binary separation. Considering the overwhelming negative instances, we set ? = 0.9 to avoid deviation by majorities in label rectification. Discussion the hyper-parameter 7 be found in Sec. D.2
C.4 Training Steps
CAPro adopts three-step training pipeline. It employs a pre- pre-training step to learn common visual patterns from original web dataset D. Then, the training starts with instance-prototype and instance-wise contrastive learning, collective bootstrapping, and on-line noise removal. Finally, given the trained model, off-line noise removal performed on D for data cleaning and we fine-tune the classifier alone with the cleaned dataset.
20

---

Step1 pre-training We perform visual pre-training on ResNet-50 with cross-entropy and projection- reconstruction losses. At this time, we only use original web labels train the model learn common visual description, which lays foundation for the subsequent prototype initialization in step2. Note that the pre-trained model is also the vanilla method our experiments.
Step2 training All components initialized with the pre-trained parameters in stepl. As shown in Table we keep the encoders frozen with warm-up epochs. This helps stabilize training to avoid the prototypical embeddings, which initialized at the beginning by averaging top- semantically- correct examples, being perturbed drastically. In this step, we re-train the model with losses. Apart from classification, perform instance-wise and instance-prototype contrastive learning, collective bootstrapping, noise removal, and prototype update.
Step3 fine-tuning We follow MoPro 28 to perform noise removal on the training dataset D. The trained model in step2 is used to correct labels or discard OOD examples with our control flow. Then, we keep encoders frozen and fine-tune the classifier alone with the cleaned set for better performance. Note that such fine-tuned model is exactly the CAPro in experiments.
Algorithm 1: CAPro's 's training procedure.
Data: Web images and their associated texts and labels D {(x;, t,, y.) Step1 pre-training
for (x,, 91) E D do
L = Cels + C,")
Update image encoder, classifier, projector, and reconstructor to minimize C;;
5 end
Step2 training
7 for (x,, (i) E D do
Extract V, from x, via the image encoder;
Extract s; from t; via the text encoder;
10 end
11 Build k-reciprocal-No graph g = (V,E), with {v.)i Enhance text embeddings from to &; via graph convolution on 9;
13 for E 21,2.... C) do
Extract s from t via the text encoder;
15
for i E {1, 2,,,,,., ..., Nly; c} do
Match textual instances s; to prototypes sf obtain visual anchors Di: 17 end
16
18 end
Initialize visual prototypes with Dx:
20
for (X;, E D do
L: = (1 X bb*) + bts chtts + x c + X c + Xins Cins;
21
22 Update image encoder, classifier, and projector to minimize C,;
23
Refine yi to ia to remove noise;
24 end
Step3 fine-tuning
26
for (x,, 0:) E D do
27
=
28
Update classifier to minimize L;;
29
end
We also prepare Algo. to explicitly explain the entire training process.
21

---

Spotlight (lamp)
X
Stage (platform)
Egyptian cat (domestic cat)
dele
Nail (metal fastener)
FEEA
Figure 5: Top-matched Web instances are chosen: (a) without text enhancement, (b) with text enhancement in VSGraph (78). and (c) with our text enhancement.
Airport (airdrome)
Bridge (span)
Car (auto)
(8)
Figure 6: Top-matched NUS-WIDE (Web) instances are chosen: (a) without text enhancement, (b)
with text enhancement VSGraph (78), and (c) with our text enhancement.
22

---

D Ablation Study
D.1 Effect of Text Enhancement
Figs. and present additional qualitative comparison for selecting instances with potentially- correct semantics. For WebVision, noiser categories are chosen validate the effectiveness text enhancement by smoothing and reranking. We can observe that due to the problem of polysemy, majority of the retrieved images are irrelevant to the correct semantics and simple 6-NN-based smoothing in [78] can hardly handle such situation. In contrast, our text enhancement helps pinpoint, not perfect but comparatively reliable, web instances that share similar semantics (e.g.. metalwork in Nail). Besides, we also sample three categories from the noiser NUS-WIDE to double-check the effectiveness of text enhancement. For example, in the category of airport, direct matching user tag embeddings to the textual prototype returns a few close-up images of warcrafts, which has nothing to do with Airport. On the contrary, our text enhancement helps to select the truly matched instances.
Table 7: Effect of 7 on CAPro without collective bootstrapping.
Reference
Google500
ImageNet500
NUS- WIDE
Provider
Top1
Top5
Top1
Top5
C-FI
O-FI
mAP
0.6
X
72.0
88.0
66.9
85.4
8.3
9.1
6.9
0.8
71.2
87.7
65.9
84.8
0.9
X
39.2
44.4
46.8
D.2 Effect of y on Noise Removal
Table rreports the influence of when collective bootstrapping is not applied. We follow MoPro 28 to validate two y candidates: 7 0.6 and 7 0.8. We find that y 0.6 works the best Google500. As increases, it becomes more difficult for the model to correct labels and thereafter label-flipping emoos may still exist. As suggested MoPro, the optimum 7 is related the percentage of noise in web datasets. For noiser datasets, 7 should be decreased for the model to correct wrong labels at an early stage. Otherwise, overfitting might occur and weaken prototypical representation learning. The fine-tuning of 7 requires elaborate experiments, which is beyond the scope of the present study.
For multi-label learning NUS-WIDE, the optimum 7 is not only related to the noise level but also to the ratio the number of positive examples to that negative examples. Since the negative instances in each class exceeds positive ones by more than order of magnitude, decreasing y will easily make the model to classify any instance as negative. Once the model overfits the overwhelming negative examples, valid positive supervision sources would only come from the top-K K matched examples with cross-modality alignment, which degrades generalizability greatly. In this case, we should keep stricter threshold 0.9 only allow confident label rectification.
Table 8: Effect of prototype update frequency on CAPro. By default, we update visual prototypes every epoch using high-quality examples in each mini-batch. For 0-epoch per update, we do not introduce additional high-quality web examples to polish prototypes, but only update them with the top-K matched semantically-corree examples with their latest visual embeddings.
# Epochs
Google500)
ImageNet500
NUS-WIDE:
per update
Top1
Top5
Topl
Top5
C-F1
O-F1
mAP
0
75.5
91.1
71.6
88.8
39.2
44.4
47.2
(by default)
76.0
91.3
72.0
89.2
39.3
45.4
48.0
5
75.9
91.2
71.8
89.2
39.6
45.0
47.6
10
76.0
91.2
71.7
89.1
39.3
45.8
48.2
D.3 Effect of Prototype Update Frequency
By default, we update prototypes by examples mini-batch for every epoch. We additionally perform ablation study on the update frequency where comparison is conducted between: 1) 0- epoch, 2) 1-epoch (by default), 3) 5-epoch, and 4) 10-epoch. For O-epoch update, we do not update prototypes with embeddings from other high-quality web examples. Instead, prototypes renewed
23

---

with the latest embeddings only from the top-K K matched examples to avoid being out-of-date For 5-epoch and 10-epoch update, we polish prototypes every 5 and 10 epochs, respectively.
Table 8 reports the effect of prototype update frequency on CAPro. On the Google500 and NUS- WIDE datasets, if prototypes are only formed by limited clean examples, their generalization across domain becomes poor and thus causes performance drop of 0.45% (topl) and 0.6% on average, respectively. For Google500, reduced update frequency generally causes lower performance, meaning that the prototypes should keep refreshed for large-scale datasets. For NUS-WIDE, if the update frequency decreases a bit (5-epoch), the model improves on C-F1 but underperforms a little on O-FI and mAP. With the 10-epoch frequency, we surprisingly find that results are improved on all evaluation metries. One possible explanation is that the delayed prototype update can help stabilize training at an early stage, but the optimal frequency might be subject to dataset scale and noise level.
Table 9: Effect of noise removal policy on CAPro. We compare with MoPro to show the effectiveness of keeping labels of top-K K matched semantically-coree examples unchanged.
Noise Removal
Google500
ImageNet500
NUS-WIDE
policy
Top1
Top5
Top1
Top5
C-F1
O-F1
mAP
MoPro
28
75.8
91.1
71.7
89.0
38.8
42.2
47.2
CAPro (ours)
76.0
91.3
72.0
89.2
39.3
45.4
48.0
D.4 Effect of Noise Removal Policy
Table reports the effect of noise removal policy on CAPro. Our label adjustment policy is inspired from the Eq. (5) of MoPro [28] but differs in that we keep labels of the top-K matched examples selected in cross- pross-modality alignment unchanged. Therefore, if we replace our noise removal policy with the MoPro one, we actually allow the deep model to get rid of the guidance from top- K examples. These selected top-K examples can be altered or even discarded by the MoPro-style policy.
turns out that without such enforcement, the performance dropped under both single-label and multi-label scenarios. The possible reason behind is that, due to the overwhelming noise (e.g. the semantic-misann mistllgmeett noise) in certain categories, the model itself cannot keep representation learning robust to noise even with good start. The labels of top- K samples will be prone to the noisy majority, which invalidates prototypical learning. Besides, the superiority of CAPro over MoPro-style update also substantiates the purity and correctness of the selected examples.
E Analysis on Open-Set Recognition
We provide detailed analysis on CAPro for open-set recognition. Fig. presents the per-class F1 (C-F1), per-class Precision (C-P), and per-class Recall (C-R). We compare five methods for ablation study including: 1) the vanilla method, 2) CAPro without text enhancement (TE) & collective bootstrapping (CB), 3) CAPro without CB, 4) CAPro, and 5) CAPro with mix-up (MU).
We vary the confidence threshold from to with an interval of 0.01 to comprehensively measure the performance of CAPro. For each example, if the highest model prediction confidence is below the threshold, the example will be classified as the open-set category. Otherwise, the example will be classified into one of the known categories. We train methods on the Google500 training set and validate them on Web Visionsoo Ik and ImageNetlk testing sets, Examples from the remaining 500 novel categories should all fall into the open-set category.
can be observed that compared with vanilla method, CAPro enjoys a much higher recall but a relatively lower precision. It means that our CAPro is more confident about its prediction and a threshold around 0.6 would end up with an optimal C-FI over 66%. The precision-recall curve reflects that each key component does improve open-set recognition. Note that due to the limited sampling of confidence threshold, the precision-recal curve is not spanning across the entire axes. However, the tendency of curves confirms the effectiveness of our components.
24

---

C.5
60
58
Proposed
Propored
Proposed
00
Proposed
Priposed
Proposed
Proposed
75
60
50
45
20
80
300
20
80
300
80
70
Threshold
60
50
50
40
30
Wrill
Venila
20
Proposed wro CS
Proposed wrb TE
Proposie wro CB
Proposed WTD &
Proposed
Proposed
10
Propesed MU
Proposed wi MU
03
05
0.8
0.9
0.2
03
0.6
077
0.8
0.9
Threshold
(a)
(b)
Figure 7: Effect of threshold for open-set recognition on (a) WebVision kk and (b) ImageNet TE, CB, and MU respectively refer text enhancement, collective bootstrapping. and mix-up. Examples with prediction confidence lower than the threshold will be classified as open-set category.
25

---

F Guidelines on Tuning of Hyper-Parameters
Loss weights of X Apri, A and X First, for the total objective, we follow MoPro 28 to use A 1 and A 1. Out of simplicity, we also use A = 1 as default.
Second, we would like explain the effect of A A and APD on regularzation. A larger a may pull instances too close totheir prototypes, which "shrinks" class clusters in the embedding space. larger A will enforce stronger visual discriminability between two instances. It may cause two examples from the same category to differ greatly and thereafter downgrades the visual prototype update and class cluster regularization. larger X improves the reconstruction quality of v,, which encourages to retain more information of v; in the embedding space. The perectiionrcoonstuuctiin loss only involved in the pre-training stage (see Algo. I]. and therefore APD will not affect the prototypical and instance-wist contrastive learning in the following stage.
Third, for one's s custom web datasets, suggest that \ Ains and A should be tuned according to the performance results under three settings: 1)APto = 0 vs. an 1; 2) = 0 vs. A 1; 3)AP4 = 0 vs. API =
According to our experiments on both single-label and multi-label datasets, the default settings of A = 1, 1, and A should work well most cases.
For bbs suggest 0.1 would achieve a balance between the individual and collective label references. A much larger value may cause over smoothing and over-regularization on visual learning.
Threshold 7 For 7 on single-label datasets, its value is related to the percentage of noise in datasets. For Web Visionkk and Google500 (34% noise (3. 7 0.6 works better than 0.8. For one's own web dataset, if the noise ratio is larger, 7 should be tuned lower so that wrong labels could be corrected at an earlier stage before overfitting. For y multi-label datasets, its value is related to both the percentage of noise and the number ratio positive-negative samples. For NUS-WIDD (50% noise (78) and 0.02 avg. ratio of positive-negative examples),
0.9 works better than 7 0.6. For one's own web dataset, if the noise ratio is smaller and the positive over negative ratio is smaller, y should be tuned higher that hard positive samples will not be easily discarded to avoid underfitting.
Prototype Update Frequency For the update frequency, its value is related to the dataset scale and noise level. For WehVision and Google500, visual prototypes should be updated per epoch to improve their diversity, which better handles the domain gap between web and realistic datasets. For NUS-WIDE, the update frequency could be reduced stabilize training, where the prototypes can be prone to the overwhelming negative examples in each category.
Top-K For top-K, its value is related the percentage of noise. If the noise ratio less than 30%, K should be set higher than 50 to include more diverse examples.
The current settings of other dataset, we believe these values can be set as starting points and finetuned accordingly. Among all syper-parameters, our ablation results show that for b 7. and top-K, their values do affect performance and should be set following the rules mentioned above (such as the dataset scale, the noise ratio, and the positive-negatiee ratio). For the remaining hyper-parametert such as the prototype update frequency, do not observe significant fluctuation. In other words, the model is robust to these
Others
(see Table 6) work well. For one's own
G Failure Cases
We provide failure cases of our CaPro on the summarized as the following.
dataset (see Fig. Our observations are
First, CAPro can handle fine-grat tne-grained categories WeiVisi The introduction of atypicals increases the risk noise. For generalization on anomalies or rarities, one solution is to choose both top-K and randomly sampled instances.
Second, for Web Ik both MoPro and CAPro underperform the vanilla baseline (optimized only by the cross-entropy loss) on a total of 387 and 312 classes, respectively. Top5 failures of classes
26

---

n04152593 Synsct( screen.n.03)
m04356056
n02097658
n02089867
Synset( sungets(sunlasss...00 n.01))
Synset(silky (easssssiittteeren.... Jemier.n.01))
Synsct( (wwlker wwakeer_houdd.nn __..nddduon))
Figure 8: Failure cases of our CAPro on certain classes where the simple vanilla baseline achieves better performance.
include screen, sunGlasses, hellCote, ballPlayer, and popBottle. For ImageNetlk, MoPro and CAPro underperform the vanilla total of 450 and 358 classes, respectively. Top-5 failures of classes include silky Terrier, walkerHound, academicGown, standardSchnauzer, and bellCote.
We also provide interesting findings below:
First, the domain gap exists between web and realistic datasets the top failure the WebVisionIk and ImageNetlk testing sets are quite different.
Second, the vanilla method tends to overfit the training concepts such as screen vs. monitor and sunGlasses vs. sunGlass.
that it outperforms highly similar
Third, mistakes on silky vs. yorkshire Terrier and walker vs. englishFox hound are ascribed to over-regularization. The inter-class relationship might be used for class-wise adjustment.
H Computational Complexity
Table 10: The parameters and GFLOPs of different encoders.
Encoders
Number of Parameters
GFLOPs
R50 [89
25M
3.8
MiniL. 30
22M
4.7
XL.Net 29
110M
29
GPT-Neo 31
1.3B
3400
First, we present the number of parameters and GFLOPs for the image and text encoders in Table [1
Table 11: The cost of the text enhancement of our CAPro with respect to its performance gains.
Text
Text
Google500
ImageNet500
Cost
Encoding
Enhancement
Top1
Top1
VSGraph 78
(ON,,
72.0
66.9
MiniLM 33
Ours
+3.5
+4.6
VSGraph 78
ONN',,)O(N,, Nkdd)
71.6
66.8
XL.Net 29
Ours
(0(3NNdd)+40(4)))00AA))))) 4k))
+3.8
+4.7
VSGraph 78
ONN4,))O((Nd, Nkd,))
72.0
67.2
GPT-Neo 31
Ours
+0(3)
+3.7
+4.4
Second, we present the cost of the text enhancement of our CAPro with respect to its performance gains. Here, N denotes the number of all nodes in the visual graph. & is the number of neighbors per node and d, is the dimension of the feature v. With à: = 5 and = respectively for WebVisionlk improvement over VSGraph is worthy the expense of such low
and NUS-WIDE,
Third, we present the cost of the reference provider of our CAPro with respect to its performance gains. Here, is the batch size, d, is the dimension of is the size of dictionary, d, is the
27

---

Table 12: The cost of the reference provider of our CAPro with respect to its performance gains.
Text
Reference
Google500)
ImageNet500
Cost
Encoding
Provider
Topl
Topl
Mix-up [99]
75.7
71.4
MiniLM (30
NCR 55
(d+
-0.2
+0.1
Our CB
O(mQ(d, +
+0.3
+0.6
Bootstrap 33
75.5
71.3
MiniLM 30
NCR [58
(d + +C)
+0
+0.2
Our CB
O(mQ(d, + C))
+0.5
+0.7
LabelSmooth 100
75.4
71.2
MiniLM 30
NCR [58]
Om + +
+0.1
+0.3
Our CB
O(mQ(dp CC)
+0.6
+0.8
SCC 22
73.8
70.2
MiniLM 30
NCR
Om + +C)
+1.7
+1.3
Our CB
OmmQ(dp + C))
+2.2
+1.8
dimension of 2, and C' the number of classes. The common reference provider techniques such as the Mix-up, Bootstrapping, label smoothing, and SCC do incur significant overhead. Operations of our CB are fast to compute for moderate m 256, Q 8192, dp 128, and C' 1000 since PyTorch supports efficient matrix multiplication on GPUs. Besides, compared with NCR, our 256 is 4x smaller than = 1024 in our cost 20.37x smaller than NCR. It is reasonable to conclude that our CB is more efficient and effective than NCR.
dp = 128 is 16x smaller than d,,2008 in NCR, and our NCR. For WebVision kk, our cost is 1 1.35x smaller than NCR. For
Finally, it is noted that the encoding and text enhancement methods are performed off-line and executed only once. They do not participate in network optimization. Besides, the pretrained text encoders only used for inference under VI00 GPU, Therefore, the additional cost is acceptable in return for semantically-corres web images.
28


Reviews:
[{"review_id": "IjO0CgFms1", "reviewer": "Reviewer_HQEo", "summary": "The authors propose a prototypically-aligned contrastive learning framework for vision and language in order to enable better web-scraping of fine-grained, rare concepts that are easily confused or mapped to other more common concepts when either vision or language is considered in isolation (what they term “semantic noise”). Their goal is to improve webly-supervised learning, which is often plagued by label noise, particularly for fine-grained categories. They call their method CAPro, for “Cross-modality Aligned Prototypes.” Their method is built from a few simple ideas: 1) use text prototypes, well-defined in the literature, to scrape a cleaner set of visual prototypes for fine-grained classes, 2) Use visual features to fill gaps or correct errors within text prototypes for fine-grained concepts, generating “semantically-aligned visual prototypes,” 3) an additional cleaning step to further reduce noise between the visual and textual prototypes that uses cluster regularization, 4) “collective bootstrapping” to further smooth concepts or “label references” via essentially an adapted/aggregated/bootstrapped dictionary lookup over the entire dynamic concept dictionary that reduces the effect of examples divergent from the average when making predictions, particularly helpful for visually similar classes.  Their method is highly similar to MoPro, but with the addition of the above described semantic noise correction and regularization across modalities.\n\nAfter reading the authors rebuttal and discussing with them, as well as reading the comments and discussion with other reviewers, I will increase my score to a 6.", "questions": "I’m curious how this method would handle visual dimorphism in fine-grained classes, for instance dark-eyed juncos which have multiple distinct color morphs, some more common than others. Since the method explicitly seeks prototypical examples, is this at the expense of recognition of anomalies or rarities within-concept? Have the authors seen any interesting failure modes for their method, where simpler methods succeed?", "limitations": "The authors have not addressed this at all, which I see as a significant weakness. What are the potential benefits or harms when building prototypical definitions of concepts? What might this mean for instances which do not fit the prototypical visual appearance of a given concept on the internet? How might biases in internet data limit our prototypes?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors show that their method performs well for single-label and multi-labe, on WebVision1k and Imagenet1k, and shows some open-set generalizability. In particular, they define 235 categories from these datasets as exhibiting “polysemy concepts”, and show that their method is particularly effective for these concepts. The gains shown are nice, and the more detailed performance breakdown of the 235 class versions of both datasets is interesting (though it would be nice to also show the performance breakdown for the non-polysemy concepts to help build reader intuition of the impact of polysemy). The method does seem to clearly have an advantage in multi-label challenges, and performs better than CoTeach or VSGraph on open-set concepts. The qualitative examples in table 3 are quite compelling, though they seem potentially highly cherry-picked.", "weaknesses": "Notably, their system is quite complex, and feels a bit like a bag of tricks (albeit an effective one), and the method both performs quite similarly to MoPro and often underperforms VSGraph on Top1 (the authors point out that these may not be directly comparable, as they use different backbones, but this makes it difficult to understand or interpret the table. Perhaps they could somehow point out which methods are comparable along which axes in the table? As it is, it’s not very easy to determine the methods value vs other methods). I would also have liked to see more thorough ablations to better understand how impactful their suggested components were vs the additional computational complexity added, to better understand how “worth it” each component would be to use or implement. I find table 4 to be a bit difficult to interpret. I would also like a more thorough analysis of why top-1 suffers, but top-5 benefits (this may be related to over-regularization somehow?). \n\n\nNits:\nIn the abstract: “exacerbates fine-grained visual concept learning.” should perhaps be “exacerbates the challenge of fine-grained visual concept learning.”\n\nLine 26: “The large scale” -> “Large scale”\n\nLine 305: “study” -> “studies”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1.1 We improve Fig. 1 (c) with non-polysemy performance (see PDF).\n\nA1.2 CAPro is indeed a complicated, systematic solution. It is specifically designed to address web noise instead of piling up tricks.\nIt follows a similar paradigm of prototypical learning as MoPro [28] and PCL [27].\nDifferent to them, we take semantic noise into serious consideration.\nOur advantage against MoPro is highlighted in 235 polysemy concepts to demonstrate the benefits of visual-semantic alignment.\n\nA1.3 In Tab. 1, CAPro does not excel VSGraph in top1 but surpasses it in top5 on WebV/ImgN1k. On Ggl/ImgN500, CAPro outperforms it with 7.9%/8.9% in top1. CAPro consistently outperforms VSGraph in Tabs. 2-4 for different benchmarks, tasks, and encoders.\n\nA1.4 First, VSGraph adopts the same R50 but is trained with a batch-size of 1,024. The benefits of a larger batch size are studied in NCR [58]. Its top1 Accs. for 1024 & 256 batch-sizes are 75.7% & 73.9%. We believe batch size is the reason that a baseline [78] surpasses most SOTAs. Due to the limited budget, training with a batch-size of 1024 is currently not affordable, but we will experiment in the future.\n\nSecond, we would like to clarify that:\nMethods in the same parts of Tab. 1 are comparable to each other. Methods in rows 1-5 are not comparable to CAPro due to different backbones. Methods in rows 6-8 are not comparable to CAPro due to optimized training settings. CAPro is fairly compared with all methods in rows 9-18.\n\nA1.5 First, we present the params. & GFLOPs of encoders.\n|Encoders|Params.|GFLOPs|\n|---|---|---|\n|R50|25M|3.8|\n|MiniLM|22M|4.7|\n|XLNet|110M|29|\n|GPT-Neo|1.3B|3400|\n\nSecond, we present the costs vs. gains for text enhancement, where N is the # of nodes, k is # of neighbors per node, and $d_v$ is the dim. of $\\mathbf{v}$.\n|Text Encod.|Text Enhance.|Cost|Ggl500 Top1|ImgN500 Top1|\n|---|---|---|---|---|\n|MiniLM/XLNet/GPT-Neo|VSGraph|$O(N^2d_v)$+$O(N{d_v}^2+Nkd_v)$|72.0/71.6/72.0|66.9/66.8/67.2|\n|MiniLM/XLNet/GPT-Neo|Ours|+$O(3Nkd_v)$+$O(4k)$+$O(4klog(4k))$|+3.5/+3.8/+3.7|+4.6/+4.7/+4.4|\n\nWith k=5 & k=10 for WebV1k & NUS-WIDE, our improvement over VSGraph is worthy at the expense of such a low cost.\n\nThird, we present the costs vs. gains for reference provider, where m is the batch size, d_v is the dim. of v, Q is the size of dictionary, d_p is the dim. of z, and C is the # of classes.\n|Text Encod.|Ref. Provider|Cost|Ggl500 Top1|ImgN500 Top1|\n|---|---|---|---|---|\n|MiniLM|MixUp/BootStrap/LabelSmooth/SCC|--|75.7/75.5/75.4/73.8|71.4/71.3/71.2/70.2|\n|MiniLM|NCR| $O(m^2(d_v+C))$|-0.2/+0/+0.1/+1.7|+0.1/+0.2/+0.3/+1.3|\n|MiniLM|Our CB| $O(mQ(d_p+C))$|+0.3/+0.5/+0.6/+2.2|+0.6/+0.7/+0.8/+1.8|\n\nOperations of our CB are fast to compute since PyTorch supports efficient matrix multiplication on GPUs.\nOur $d_p$=128 is 16x << $d_v$=2048 in NCR & our m=256 is 4x << m=1024 in NCR.\nFor WebV1k, our cost is 1.35x < NCR.\nFor NUS-WIDE, our cost is 20.37x << NCR.\nIt is reasonable to conclude that our CB is more efficient & effective than NCR.\n\nFinally, text encod.& enhance. are off-line and executed only once. They do not participate in network optimization. Besides, the pretrained text encoders are only used for inference under 1 V100 GPU. Therefore, the additional cost is acceptable in return for semantically-correct web images.\n\nA1.6 First, for Tab. 4, methods in rows 2-7 outperform the 1st row with better top1&top5, validating the textual knowledge. For different text encoders, we outperform VSGraph in both top1 & top5.\n\nSecond, SCC estimates confidence independently and neglects the relationship between an instance and its prototype. NCR brings no gains. Its effect is limited without a large batch size. Bootstrap & label smooth degrades both top1 & top5 on ImgN500 while mix-up benefits top1 on Ggl500.\n\nMix-up improves CAPro in top1 but lowers top5. It adopts convex combinations for both inputs & targets, enforcing a stronger regularization than our CB where we only recombine targets.\nFor WebV1k, examples with noisy labels still resemble their prototypes and therefore neighbor knowledge brings useful reference. Mix-up does not consider appearance similarity and causes over-regularization.\n\nA1.7 Thank you. We fix typos.\n\nA1.8 First, CAPro can handle fine-grained categories on WebV1k. The introduction of atypicals increases the risk of noise. For generalization on anomalies or rarities, one solution is to choose both top-K and randomly sampled instances.\n\nSecond, for WebV1k, both MoPro&CAPro underperform the vanilla on a total of 387&312 classes. Top5 failures: screen, sunGlasses, bellCote, ballPlayer, popBottle. For ImgN1k, MoPro&CAPro underperform the vanilla on a total of 450&358 classes. Top-5 failures: silkyTerrier, walkerHound, academicGown, standardSchnauzer, bellCote.\n\nFindings:\n\n1) Domain gap exists between web and realistic datasets.\n\n2) The vanilla tends to overfit training set so that it outperforms on highly similar concepts: screen & monitor, sunGlasses & sunGlass.\n\nMistakes on silky & yorkshire Terrier, walker & englishFox hound are ascribed to over-regularization. The inter-class relationship might be used for class-wise adjustment.\n\nA1.9 Two benefits:\n\nNoisy web data can be corrected by measuring the instance-prototype distance.\nInter-class relationship can be statistically studied to shed light on similarities between species.\n\nThree drawbacks:\n\nTolerance to the intra-class minority. Web data follow long-tailed distribution. The more common one instance is, the greater the likelihood that it gets exposed.\n=>to introduce randomness into init. & update of prototypes.\n\nThe domain bias of web data. Their styles (advert. & render.) are different from realistic ones. Specific modalities (infrared & CT) are unavailable.\n=>to prepare realistic images for guided-training.\n\nPrior knowledge about class hierarchy. Coarse-grained or improper descriptions about hierarchical structure would devalue semantic alignment.\n=>to perform a thorough analysis of concepts."}, {"review_id": "K9to6BN5lz", "reviewer": "Reviewer_XBs5", "summary": "To handle the label noise problems especially the semantic noise in webly supervised learning, the authors propose a unified prototypical contrastive learning framework named as Cross-modality Aligned Prototypes (CAPro). It exploits web data across modalities to formulate semantically-correct textual and visual prototypes. Besides, collective bootstrapping is proposed to encourage smoother and wiser label reference from appearance-similar instances in a manner of dictionary look-up. Extensive experiments on WebVision1K and NUS-WIDE(Web) demonstrate that the proposed CAPro could handle realistic noise under different scenarios. It also achieves new state-of-the-art performance on open-set recognition.", "questions": "Please try to address the weaknesses above.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe authors propose a new cross-modality prototypical learning framework named as CAPro, which aims to handle various noises, especially semantic noise.\n2.\tThe proposed CAPro framework is carefully designed and fully explores the cross-modality intervention to filter out noise.\n3.\tExtensive experiments have been conducted to prove the effectiveness of the proposed method. On the other hand, the code is available and enables reproducing this work.\n", "weaknesses": "1.\tThe framework figure, i.e., fig. 2, is too complex to be understood. The arrows are not in the same directions, and the symbols are not introduced in caption or texts. There is also lack of explicit module division in the figure. These aspects increase the difficulty to understanding the whole framework.\n2.\tIn experiments, it seems that different methods obtain different “vanilla” performance even with the same backbone, such as R50. Why is the performance different among different methods? Does it mean unfair comparison?\n3.\tThe whole framework is complex since it introduces many modules to handle noise. The whole training process should be organized as explicit algorithm to facilitate the understanding.\n4.\tThere are a lot of hyper-parameters in the total objective. How to choose them during experiments?\n5.\tThere are some typos in the manuscript.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "More explanations will be added to the manuscript.\n\nA2.1 Thank you for the comments.\n\nA2.2 First, we would like to further explain the function of each module in Fig. 2:\n\nSiamese image encoders: extract features $\\mathbf{v}_i$, $\\mathbf{v}_i'$ from inputs $\\mathbf{x}_i$ and their augmented counterparts $\\mathbf{x}_i'$.\n\nText encoder: generates embeddings $\\mathbf{s}_i$, $\\mathbf{s}^c$ respectively from the instance $\\mathbf{t}_i$ and the category $\\mathbf{t}^c$.\n\nClassifier: maps $\\mathbf{v}_i$ to predictions $\\mathbf{p}_i$ over $C$ classes.\n\nProjector: distills discriminative low-dim.al embeddings $\\mathbf{z}_i$ from $\\mathbf{v}_i$, followed by $\\ell_2$-normalization for unit-sphere constraint on $\\mathbf{z}_i$.\n\nReconstructor: recovers $\\tilde{\\mathbf{v}}_i$ from $\\mathbf{z}_i$ to be close to $\\mathbf{v}_i$.\n\nAuxiliary classifier: outputs predictions $\\mathbf{q}_i$ on $\\mathbf{z}_i$.\n\nDictionary: records keys for both contrastive learning and collective bootstrapping. The latest embeddings $\\mathbf{z}_i'$ are enqueued while the oldest are dequeued.\n\nSecond, we will modify Fig. 2 to make it easier to read and follow (see PDF). The refined captions are as follows:\n\nOverview of CAPro. Images $\\mathbf{x}_i$ and texts $\\mathbf{t}_i$ are respectively fed into the image and text encoders for features $\\mathbf{v}_i$ and $\\mathbf{s}_i$. Then, $\\mathbf{v}_i$ is projected into the embedding space as $\\mathbf{z}_i$, followed by the reconstruction from $\\mathbf{z}_i$ to $\\mathbf{\\tilde{v}}_i$. Visual prototypes $\\mathbf{z}^c$ are initialized with anchor instances that are selected by matching enhanced texts $\\mathbf{\\tilde{s}}_i$ to textual prototypes $\\mathbf{s}^c$ for semantic alignment. They are constantly polished up by clean images and engage in contrastive learning to constrain cluster distribution. Collective bootstrapping exploits visual dictionary for regularization on the auxiliary classifier output $\\mathbf{q}_i$, where each key embedding is matched to the query for the reference $\\mathbf{b}_i$. Web labels $y_i$ are simultaneously refined as $\\tilde{y}_i$ for ``denoised'' supervision on the classifier output $\\mathbf{p}_i$.\n\nA2.3 First, even under the same R50 backbone, different results of \"vanilla\" baselines are reported [26,28,58,78]. We believe the training settings, especially the batch-size, are the reasons that certain vanilla baselines surpass most SOTAs. In the present study, we follow MoPro [28] to use the standard settings of ImageNet training. Please refer to our supplementary (line 21).\n\nSecond, we take VSGraph [78] and NCR [58] as examples to show how the batch-size would affect performance.\nBoth VSGraph [78] and NCR [58] adopt the R50 backbone.\n\nBut VSGraph, NCR, and their vanilla baselines are trained with a batch size of $1,024$. And their vanilla methods surpass most SOTAs that are trained with a batch-size of $256$.\n\nThe benefits of a larger batch-size (1,024 over 256) on WebV1k are studied in NCR [58]. Its top-1 Accs for batch-sizes of $1,024$ and $256$ are respectively $75.7\\\\%$ and $73.9\\\\%$.\n\nWe believe batch-size is the key factor that affects comparability. \n\nDue to the limited GPU budget, training with a batch-size of $1,024$ is currently not affordable but we are willing to experiment in further research.\n\nThird, we would like to clarify how to fairly interpret Tab. 1:\n\nThe comparison between SOTA methods with ours:\n\nMethods in rows 1-5 are not comparable with the proposed CAPro since their backbones are different.\nMethods in rows 6-8 are not comparable due to their optimized training settings.\nMethods in rows 9-18 are all trained with R50 with a batch size of $256$, which are comparable with ours.\n\nThe comparison between SOTA methods with their vanilla baselines:\n\nSCC can be compared with the 4th row.\nVSGraph and CoTeach can be compared with the 6th row.\nMoPro can be compared with the 9th row.\nOur CAPro can be compared with the second last row.\n\nA2.4 Thank you for the suggestion. We prepare an algorithm to clearly explain the entire process (see PDF) and will be added to our paper.\n\nA2.5 First, for the total objective (line 242), we follow MoPro [28] to use $\\lambda_{pro}=1$ and $\\lambda_{ins}=1$. Out of simplicity, we also use $\\lambda_{prj}=1$ as the default.\n\nSecond, we would like to explain the effect of $\\lambda_{pro}$, $\\lambda_{ins}$, and $\\lambda_{prj}$ on regularzation.\n\nA larger $\\lambda_{pro}$ may pull instances too close to their prototypes, which \"shrinks\" class clusters in the embedding space.\n\nA larger $\\lambda_{ins}$ will enforce stronger visual discriminability between two instances. It may cause two examples from the same category to differ greatly and thereafter downgrades the visual prototype update and class cluster regularization.\n\nA larger $\\lambda_{prj}$ improves the reconstruction quality of $\\tilde{\\mathbf{v}}_i$, which encourages $\\mathbf{z}_i$ to retain more information of $\\mathbf{v}_i$ in the embedding space.\n\nThe projection-reconstruction loss is only involved in the pre-training stage (see line 62 in our supplementary), $\\lambda_{prj}$ will not affect the prototypical and instance-wise contrastive learning in the following stage.\n\nThird, for custom web dataset, we suggest that $\\lambda_{pro}$, $\\lambda_{ins}$, and $\\lambda_{prj}$ should be tuned according to the performance results under\n\n1)$\\lambda_{pro}=0$ vs. $\\lambda_{pro}=1$;\n\n2)$\\lambda_{ins}=0$ vs. $\\lambda_{ins}=1$;\n\n3)$\\lambda_{prj}=0$ vs. $\\lambda_{prj}=1$.\n\nAccording to our experiments on both single-label and multi-label datasets, the default settings of $\\lambda_{pro}=1$, $\\lambda_{ins}=1$, and $\\lambda_{prj}=1$ should work well.\n\nFor settings of other hyper-params., please refer to our response A4.4.\n\nA2.6 We will double-check grammar and word spelling.\n\nA2.7 We will modify the manuscript accordingly. Please see the point-by-point response above."}, {"review_id": "OybtbzZKxT", "reviewer": "Reviewer_SPGi", "summary": "This paper dives into the study of webly-supervised learning and aims to utilize the neglected alt-text of web images to enhance the learning process. To this end, the authors propose the approach called Cross-modality Aligned Prototypes (CAPro). CAPro is adopted with two modules, namely, text matching&enhacement, and collective bootstrapping. The former aims to assign text to the corresponding prototypes by resorting to the LLMs and cross-modal nearest neighbor mechanism. The latter provides smoother label reference by extending bootstrapping with collective knowledge. Extensive experiments have been conducted on several benchmarks to verify the effectiveness of the proposed CAPro.", "questions": "My major concern is the  differences between this work and the existing problem and techniques as eloborated on in Weaknesses. Thanks", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Most existing webly-supervised learning works mainly focus on the web images and corresponding (noisy) labels while omitting the potential alt-texts (captions). This paper provides a new perspective that uses the alt-text to complement the webly-supervised learning. From this point, the motivation and idea of this paper are novel and interesting.\n", "weaknesses": "1. My major concern is the differences between this work and the existing problem or techniques including the noisy correspondence, noise-robust learning from NNs, and noise removal. First, this paper claims that the existing webly-supervised learning works mainly address certain types of noise including label-flipping noise and out-of-distribution (OOD), while neglecting the semantic noise, namely, the misalignment between image contents and the associated texts. The claim might be correct for the webly-supervised learning community. However, the so-called semantic noise is very similar to the definition of noisy correspondence [61,62, 65]. The authors should provide more discussion to clarify the differences between the so-called noisy correspondence and semantic noise. If the two problems are somewhat similar, I think it would be better to give more discussion on the related works like 'Noisy Correspondence Learning with Meta Similarity Correction'. Second, the differences between the used KNN-graph mechanism and the works in 'Noise-Robust Learning from Neighbors' are encouraged to be further discussed. Third, the proposed noise removal strategy seems to be similar to the sophisticated WSL method (MoPro), which is an important baseline for WSL learning. \n2. The performance improvement is limited (See Table 1). This paper adopts a relatively complex pipeline and additional resort to the existing LLMs. However, the performance is marginal compared to the sophisticated WSL baseline (MoPro, ICLR 2020). \n2. There are some typos and unclear statements. For example: \ni) the definition of 'concept definition texts (Line 56-57)' is lacking\nii) 'with visual guidance from image neighbors (Line 61)' is unclear.\niii) 'selection We' (Line 62) ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A3.1 Thank you.\n\nA3.2 We would like to explain the differences between our work and previous studies in noisy correspondence learning.\n\nFirst, the reasons behind these two problems are different.\nThe semantic noise is caused by the polysemy retrieval keywords which are used to crawl web images. For example, when we try to retrieve web images of \"drumstick\" for the percussion music instrument, we may end up with a bunch of images of \"drumstick dishes (chicken)\" and \"drumstick trees (moringa oleifera)\". The associated texts of these irrelevant images indeed contain \"drumstick\", but the image contents are irrelevant to the expected concept.\n\nNoisy correspondence [61, 62, 65] emphasizes the mismatch between an image and its associated text itself. It is mainly caused by mistakes in partitioning the interleaved image and text web data. For example, the caption \"a bunch of cows grazing in a dry field together\" is wrongly assigned to an image of \"giraffes\" [65]. Mismatched image-text pairs mainly hinder the performance of cross-modality retrieval.\n\nSecond, the solutions to these two problems are different.\n\nThe semantic noise belongs to the label noise, which means that web images are incorrectly annotated.  We focus on learning visual representation with categorical noise, which falls under the scenarios of unimodal single-label or multi-label classification.\n\nNoisy correspondence tackles the instance-level mismatched image-text pairs. Its task is to identify alignment errors in the paired data and remove false positives of matching. Most noisy correspondence methods [61, 62, 65] learn to align image and text embeddings to facilitate cross-modal retrieval.\n\nFinally, we agree that there exists an intersection between the two problems. In our webly-supervised learning, we resort to both images and their texts to form semantically-correct visual prototypes. These images and texts can be mismatched and therefore we adopt text enhancement by $k$-reciprocal-NN-based smoothing and reranking to alleviate the noisy correspondence problem. From this point of view, it is indispensable to take noisy correspondence into consideration.\n\nWe add another section to the related work:\n\nNoisy Correspondence Rectification\n\nOne paradigm similar to WSL might by noisy correspondence rectification or calibration [60,65,62,66,61,63,68,79]. It tackles the mismatched image and text pairs and aims to simultaneously learn aligned visual and textual embeddings for improved cross-modal retrieval. Huang et al. [65] utilizes the memorization effect of neural networks to partition clean and noisy data and then learns to rectify correspondence. Hu et al. [61] derives a unified framework with contrastive learning to reform cross-modal retrieval as an N-way retrieval. Han et al. [66] proposes a meta-similarity correction network to view the binary classification of correct/noisy correspondence as the meta-process, which facilitates data purification.\n\nAlthough the noisy correspondence removal is closely related to our task, it differs in two aspects: 1) We focus on the label noise where web images are wrongly-annotated by weak keywords or hashtags. Noisy correspondence emphasizes the instance-level mismatch between an image and its associated text. 2) We aim to learn visual representations with correct categorical labels while most methods on noisy correspondence try to align image and text embeddings to improve cross-modal retrieval.\n\nA3.3 More discussions on the differences between previous nearest-neighbor methods and our CAPro will be added to the related work as follows.\n\nIt is noted that nearest neighbors play a vital role throughout the components of our CAPro, from text enhancement to text matching and collective bootstrapping. Compared with previous methods of learning from neighbors, our mechanism differs in that:\n1) We acquire guidance from cross-modality neighbors, where noisy texts are enhanced by image neighbors to alleviate the mismatch problem. In contrast, most previous studies investigate neighbors within one modality.\n\n2) We exploit reciprocal structures to filter and rerank nearest neighbors for pertinent text matching, while most existing works neglect those top-ranked false positive neighbors.\n\n3) We resort to neighbors for on-line collective bootstrapping in a manner of dictionary look-up instead of explicit global graph construction.\n\nA3.4 First, CAPro shares the same prototypical contrastive learning with MoPro [28]. The differences between MoPro and our CAPro include:\n\nCAPro notices the semantic noise problem and takes advantage of both textual and visual prototypes to handle semantic misalignment. Our visual prototypes are polished only by semantically-correct examples.\n\nCAPro \"creatively\" reuses the dictionary for collective bootstrapping, where visually-similar neighbors provide label references by performing dictionary look-up.\n\nCApro, to our best knowledge, is the first to extend prototypical contrastive learning for multi-label classification, where the overwhelming noise and intra-class imbalance pose great challenges. CAPro performs prototypical learning in subspaces of the shared embedding space to stabilize training.\n\nSecond, please see the response A4.2 for the differences between our noise removal strategy (Eq. 7) and MoPro (Eq. 5).\n\nThird, the advantage of CAPro over MoPro is highlighted in Fig. 1(c), where performance on $235$ polysemy categories is validated. CAPro indeed improves MoPro by addressing the semantic noise that prevails in these classes. In this case, CAPro can be seen as an effective \"booster\" to MoPro, which further enables the handling of semantic misalignment with wiser collective bootstrapping. The performance gains are expected to be much more significant in practice usage.\n\nA3.5 We have corrected these typos and statements.\n\nA3.6 Thank you for the insightful comments.\n\nA3.7 Thank you."}, {"review_id": "zcpXtmjuaI", "reviewer": "Reviewer_QoQL", "summary": "This paper proposes a unified prototypical contrastive learning framework, named Cross-modality Aligned Prototypes (CAPro), to learn visual representations with correct semantics. CAPro exploits web data across modalities to formulate semantically-correct textual and visual prototypes. The authors propose text matching to leverage textual prototypes to establish their noise-robust estimation. They also bring in text enhancement with visual guidance from image neighbors to mitigate the effect of noisy texts on clean sample selection. Further, the authors propose collective bootstrapping (CB) to provide smoother label reference by extending bootstrapping with collective knowledge. Experimental results on WebVision1k and NUS-WIDE (Web) are provided to validate the effectiveness of the proposed method.", "questions": "N/A", "limitations": "The authors have briefly discussed the limitations and broader impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of bridging visual and texture prototypes to cope with webly supervised learning is interesting and promising.\n\n- The motivation is clear, and the paper is written well.\n\n- The code is released. The ablation study seems extensive.", "weaknesses": "- Eq.(5), (6), and (7) seem very similar to those in [28]. It is expected to discuss the difference between the proposed one with [28]. Why is the proposed one better?\n\n- The total objective loss function shown in Ln 242 contains 4 loss weight hyper-parameters. However, only \\lambda_{bts} is discussed in the ablation study. What about other hyper-parameters? Moreover, it is also a concern that the proposed method involves too many hyper-parameters, including these four loss weights, \\gamma, update frequency, etc. How to decide the proper value of these hyper-parameters? What about the robustness?\n\n- In Table 1, the result of NCR[58] (i.e., 73.9) seems wrong. The result shown in the NCR paper is 75.7 for NCR and 76.8 for NCR+Mixup+DA, both surpassing the reported result of the proposed method. This raises a concern about the performance of the proposed method.\n\n- Some minor issues:\n\t- In Ln 62, a period is missing between \"selection\" and \"We\".\n\t- In Ln 69, the sentence uses the past tense, while the former and latter sentences basically use the present tense. It is recommended to change this sentence to present tense to make them consistent.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A4.1 Thank you.\n\nA4.2 First, the Eqs. (5)(6) are exactly the same as Eqs. (1)(4) in MoPro [28] because we adopt the same prototypical and instance-wise contrastive learning.\n\nSecond, the control flow in Eq. (7) is inspired by Eq. (5) in MoPro [28], but we differ in that the labels of the top-$K$ matched examples are kept unchanged. Our design guarantees that these top-$K$ samples would provide consistent guidance on noise removal and prototype update, which avoids overfitting in highly noisy classes. Our superior results under both single-label and multi-label scenarios demonstrate that MoPro [28] is prone to overwhelming noise in certain categories, where clean samples would be easily ruled out by the majority of noisy ones. A detailed comparison between our noisy removal policy Eq. (7) and MoPro [28] is in the supplementary (line 119).\n\nThird, our CAPro differs from MoPro in three aspects:\n\nCAPro notices the semantic noise problem and takes advantage of both textual and visual prototypes to handle semantic misalignment. Our visual prototypes are maintained and polished only by semantically-correct examples.\n\nCAPro \"creatively\" reuses the dictionary (originally set for instance-wise contrastive learning) for collective bootstrapping, where visually-similar neighbors provide label reference by performing dictionary look-up.\n\nCApro, to our best knowledge, is the first to extend prototypical contrastive learning for multi-label classification, where the overwhelming noise ratio and intra-class positive-negative imbalance pose great challenges to optimization. To solve that, CAPro performs prototypical learning in subspaces of the shared embedding space to stabilize training.\n\nA4.3 Please refer to our response A2.5 for the instructions on how to tune the hyper-params. of loss weights in line 242. We will add more explanations on hyper-params. of the loss function.\n\nA4.4 First, ablation studies on $\\lambda^{bts}$ and top-$K$ can be found in lines 301-304 of the manuscript. The discussion on threshold $\\gamma$ and update frequency is in our supplementary (lines 88, 103).\n\nFor $\\gamma$ on single-label datasets, its value is related to the percentage of noise in datasets. For WebV1k/Ggl500 ($34\\\\%$ noise [3]), $\\gamma=0.6$ works better than $\\gamma=0.8$. For one's own web dataset, if the noise ratio is larger, $\\gamma$ should be tuned lower so that wrong labels could be corrected at an earlier stage before overfitting.\n\nFor $\\gamma$ on multi-label datasets, its value is related to both the percentage of noise and the number ratio of positive-negative samples. For NUS-WIDE ($50\\\\%$ noise [78] and $0.02$ for avg. ratio of positive-negative examples), $\\gamma=0.9$ works better than $\\gamma=0.6$. For one's own web dataset, if the noise ratio is smaller and the positive/negative ratio is smaller, $\\gamma$ should be tuned higher so that hard positive samples will not be easily discarded to avoid underfitting.\n\nFor the update frequency, its value is related to the dataset scale and noise level. For WebV1k/Ggl500, visual prototypes should be updated per epoch to improve their diversity, which better handles the domain gap between web and realistic datasets. For NUS-WIDE, the update frequency could be reduced to stabilize training, where the prototypes can be prone to overwhelming negative examples in each category.\n\nFor $\\lambda^{bts}$, we suggest $0.1$ would achieve a balance between the individual and collective label references. A much larger value may cause over-smoothing and over-regularization of visual learning.\n\nFor top-$K$, its value is related to the percentage of noise. If the noise ratio is less than $30\\\\%$, $K$ should be set higher than $50$ to include more diverse examples.\n\nThe settings of other hyper-params. can be found in the supplementary (Tab. S2). We find current settings work well. For one's own dataset, we believe these values can be set as starting points and finetuned accordingly.\n\nAmong all hyper-params., our ablation results show that for $\\lambda^{bts}$, $\\gamma$, and top-$K$, their values do affect performance and should be set following the rules mentioned above (such as the dataset scale, the noise ratio, and the positive-negative ratio).\n\nFor other hyper-params. such as the prototype update frequency, we do not observe significant fluctuation. In other words, the model is robust to these hyper-params.\n\nA4.5 First, the result of $73.9\\\\%$ for NCR [58] is correct for the batch-size of $256$ and can be found in Tab. 6 in their supplementary [58].\n\nSecond, as we pointed out in line 257 (manuscript) and in line 24 (our supplementary), we follow most SOTA methods (see Sec.4.2 in MoPro [28]) to use the standard batch-size of $256$ on the WebV1k dataset.\n\nThird, the improvement by large batch size has been studied by NCR [58] as their results for batch-size=$256$ and batch-size=$1,024$ are respectively $73.9\\\\%$ and $75.7\\\\%$. Their vanilla baseline with R50 backbone achieves a top-1 Acc of $74.9\\\\%$ [58].\n\nFourth, NCR [58] does not report their top-5 results on WebV1k and does not provide both top-1 and top-5 results on ImgN1k, where we cannot tell if their method can properly handle the domain gap between web and realistic datasets.\n\nFinally, we agree that comparability between different methods should be carefully studied. To help interpret Tab. 1, please see our response to the Reviewer HQEo on how to compare different methods.\n\nWe will add results of NCR $\\dagger$ [58] ($75.7\\\\%$ for batch-size of $1,024$) to Tab. 1. More explanations on how to interpret and compare methods will also be added.\n\nA4.6 We correct these sentences accordingly.\n\nA4.7 More discussions on limitations will be added. Please see response A1.9 for details."}]

Author rebuttal:
Dear reviewers, area chairs, and senior area chairs,

We sincerely thank that all four reviewers are positive towards our paper, and provide detailed, constructive comments.

According to these comments and suggestions, we will modify our manuscript by:

1) improving all figures to make them easy to understand and follow;

2) adding more explanations on the differences between our CAPro with MoPro;

3) adding another section into the related work (Noisy Correspondence) to introduce the differences between webly supervised learning and noisy correspondence removal;

4) adding more explanations on the differences between our work and previous methods of learning from noisy labels;

5) adding computational complexity to the ablation study to show the additionally introduced cost;

6) adding more explanations on how to interpret the comparability between SOTA methods when different backbones and training settings are adopted;

7) adding more discussions on how to tune hyper-parameters;

8) adding more discussions on our findings of failure cases;

9) adding more discussions on the limitations;

10) polishing English writings.

Please see the point-by-point response below for each reviewer.

Besides, we provide additional **tables, figures, and algorithms** in the uploaded **one-page PDF** for better explanations. These tables, figures, and algorithms will be magnified in the manuscript, but have to be compressed now to fit into one page only for rebuttal.

Finally, we would like to express our gratitude again for all the valuable comments that would significantly help improve the quality of the manuscript.


Meta-review: There are only positive scores, two weak accept and two borderline accept (considering decisions to increase BA->WA in the comments not reflected in the review form). The reviewers praise the motivation, method design and effectiveness, and the scope of the experiments. Concerns are about complexity and results, but three out of four reviewers explicitly comment that they read the rebuttal and maintain their positive rating. Answers to the fourth reviewer (who did not respond to the rebuttal) seem reasonable.
Decision: Accept (poster)