SlowGuess/NeurIPS_2023_OCR huggingface dataset:
According to our experiments on both single-label and multi-label datasets, the default settings of  $\lambda^{\mathrm{pro}} = 1$ ,  $\lambda^{\mathrm{ins}} = 1$ , and  $\lambda^{\mathrm{prj}} = 1$  should work well on most cases.
For  $\lambda^{\mathrm{bts}}$ , we suggest 0.1 would achieve a balance between the individual and collective label references. A much larger value may cause over-smoothing and over-regularization on visual learning.
Threshold  $\gamma$  For  $\gamma$  on single-label datasets, its value is related to the percentage of noise in datasets. For WebVision1k and Google500 (34% noise [3]),  $\gamma = 0.6$  works better than  $\gamma = 0.8$ . For one's own web dataset, if the noise ratio is larger,  $\gamma$  should be tuned lower so that wrong labels could be corrected at an earlier stage before overfitting. For  $\gamma$  on multi-label datasets, its value is related to both the percentage of noise and the number ratio of positive-negative samples. For NUS-WIDE (50% noise [78] and 0.02 avg. ratio of positive-negative examples),  $\gamma = 0.9$  works better than  $\gamma = 0.6$ . For one's own web dataset, if the noise ratio is smaller and the positive over negative ratio is smaller,  $\gamma$  should be tuned higher so that hard positive samples will not be easily discarded to avoid underfitting.
Prototype Update Frequency For the update frequency, its value is related to the dataset scale and noise level. For WebVision1k and Google500, visual prototypes should be updated per epoch to improve their diversity, which better handles the domain gap between web and realistic datasets. For NUS-WIDE, the update frequency could be reduced to stabilize training, where the prototypes can be prone to the overwhelming negative examples in each category.
Top-  $K$  For top-  $K$ , its value is related to the percentage of noise. If the noise ratio is less than  $30\%$ ,  $K$  should be set higher than 50 to include more diverse examples.
Others The current settings of other hyper-parameters (see Table 6) work well. For one's own dataset, we believe these values can be set as starting points and finetuned accordingly. Among all hyper-parameters, our ablation results show that for  $\lambda^{\mathrm{bts}}$ ,  $\gamma$ , and top- $K$ , their values do affect performance and should be set following the rules mentioned above (such as the dataset scale, the noise ratio, and the positive-negative ratio). For the remaining hyper-parameters such as the prototype update frequency, we do not observe significant fluctuation. In other words, the model is robust to these hyper-parameters.
G Failure Cases
We provide failure cases of our CaPro on the WebVision1k dataset (see Fig. 8). Our observations are summarized as the following.
First, CAPro can handle fine-grained categories on WebVision1k. The introduction of atypicals increases the risk of noise. For generalization on anomalies or rarities, one solution is to choose both top-K and randomly sampled instances.
Second, for WebVision1k, both MoPro and CAPro underperform the vanilla baseline (optimized only by the cross-entropy loss) on a total of 387 and 312 classes, respectively. Top5 failures of classes
n04152593  
Synset('screen.n.03')
n04356056 Synset('sunglasses.n.01')
n02097658 Synset(silky_terrier.n.01')
Figure 8: Failure cases of our CAPro on certain classes where the simple vanilla baseline achieves better performance.
n02089867 Synset('walker_hound.n.01')
include screen, sunGlasses, bellCote, ballPlayer, and popBottle. For ImageNet1k, MoPro and CAPro underperform the vanilla on a total of 450 and 358 classes, respectively. Top-5 failures of classes include silkyTerrier, walkerHound, academicGown, standardSchnauzer, and bellCote.
We also provide interesting findings below:
First, the domain gap exists between web and realistic datasets as the top 5 failure cases on the WebVision1k and ImageNet1k testing sets are quite different.
Second, the vanilla method tends to overfit the training set so that it outperforms on highly similar concepts such as screen vs. monitor and sunGlasses vs. sunGlass.
Third, mistakes on silky vs. yorkshire Terrier and walker vs. englishFox hound are ascribed to over-regularization. The inter-class relationship might be used for class-wise adjustment.
H Computational Complexity
Table 10: The parameters and GFLOPs of different encoders.
Encoders	Number of Parameters	GFLOPs
R50 [89]	25M	3.8
MiniLM [30]	22M	4.7
XLNet [29]	110M	29
GPT-Neo [31]	1.3B	3400
First, we present the number of parameters and GFLOPs for the image and text encoders in Table 11.
Table 11: The cost of the text enhancement of our CAPro with respect to its performance gains.
Text Encoding	Text Enhancement	Cost	Google500 Top1	ImageNet500 Top1
MiniLM [30]	VSGraph [78]	O(N2dv)+O(Ndv2+Nkdv)	72.0	66.9
Ours	+O(3Nkdv)+O(4k)+O(4klog(4k))	+3.5	+4.6
XLNet [29]	VSGraph [78]	O(N2dv)+O(Ndv2+Nkdv)	71.6	66.8
Ours	+O(3Nkdv)+O(4k)+O(4klog(4k))	+3.8	+4.7
GPT-Neo [31]	VSGraph [78]	O(N2dv)+O(Ndv2+Nkdv)	72.0	67.2
Ours	+O(3Nkdv)+O(4k)+O(4klog(4k))	+3.7	+4.4
Second, we present the cost of the text enhancement of our CAPro with respect to its performance gains. Here,  $N$  denotes the number of all nodes in the visual graph.  $k$  is the number of neighbors per node and  $d_v$  is the dimension of the feature  $\mathbf{v}$ . With  $k = 5$  and  $k = 10$  respectively for WebVision1k and NUS-WIDE, our improvement over VSGraph is worthy at the expense of such a low cost.
Third, we present the cost of the reference provider of our CAPro with respect to its performance gains. Here,  $m$  is the batch size,  $d_v$  is the dimension of  $\mathbf{v}$ ,  $Q$  is the size of dictionary,  $d_p$  is the
Table 12: The cost of the reference provider of our CAPro with respect to its performance gains.
Text Encoding	Reference Provider	Cost	Google500 Top1	ImageNet500 Top1
MiniLM [30]	Mix-up [99]	-	75.7	71.4
NCR [58]	O(m2(dv+C))	-0.2	+0.1
Our CB	O(mQ(dp+C))	+0.3	+0.6
MiniLM [30]	Bootstrap [33]	-	75.5	71.3
NCR [58]	O(m2(dv+C))	+0	+0.2
Our CB	O(mQ(dp+C))	+0.5	+0.7
MiniLM [30]	LabelSmooth [100]	-	75.4	71.2
NCR [58]	O(m2(dv+C))	+0.1	+0.3
Our CB	O(mQ(dp+C))	+0.6	+0.8
MiniLM [30]	SCC [22]	-	73.8	70.2
NCR [58]	O(m2(dv+C))	+1.7	+1.3
Our CB	O(mQ(dp+C))	+2.2	+1.8
dimension of  $z$ , and  $C$  is the number of classes. The common reference provider techniques such as the Mix-up, Bootstrapping, label smoothing, and SCC do not incur significant overhead. Operations of our CB are fast to compute for moderate  $m = 256$ ,  $Q = 8192$ ,  $d_p = 128$ , and  $C = 1000$  since PyTorch supports efficient matrix multiplication on GPUs. Besides, compared with NCR, our  $d_p = 128$  is 16x smaller than  $d_v = 2048$  in NCR, and our  $m = 256$  is 4x smaller than  $m = 1024$  in NCR. For WebVision1k, our cost is 1.35x smaller than NCR. For NUS-WIDE, our cost is 20.37x smaller than NCR. It is reasonable to conclude that our CB is more efficient and effective than NCR.
Finally, it is noted that the text encoding and text enhancement methods are performed off-line and executed only once. They do not participate in network optimization. Besides, the pretrained text encoders are only used for inference under 1 V100 GPU. Therefore, the additional cost is acceptable in return for semantically-correct web images.