SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3.3 Balancing intra-/inter-class diversity: the optimal class-to-sample ratio
When  $N$  is fixed, by leveraging the fact that  $N = n \times K$ , we can express  $U$  as
$$
U = \frac {1}{N ^ {\frac {1}{4}}} \left(A x ^ {\frac {1}{4}} + B \frac {1}{x ^ {\frac {1}{4}}}\right) + c, \tag {4}
$$
where  $c = \frac{C}{\sqrt{N}} + D$  is a constant and  $x = \frac{K}{n}$  is the class-to-sample ratio. To minimize  $U$ , we have the optimal class-to-sample ratio  $\bar{x} = \frac{B^2}{A^2}$ . Notably, because both  $A$  and  $B$  have no dependency on  $N$ , the optimal class-to-sample ratio for a specific downstream task is invariant to the size of the pre-training dataset. Motivated by this, one could estimate the optimal class-to-sample ratio using a small  $N$  and then use it to predict the optimal number of classes for building a large pre-training dataset. In particular, given the optimal class-to-sample ratio  $\bar{x}$ , the optimal number of classes is  $\bar{K} = \frac{B}{A}\sqrt{N}$ . Based on Equation 4, one only needs three (class-to-sample ratio, performance) tuples to estimate the constants  $(A, B, c)$  with a fixed  $N$  for computing the optimal class-to-sample ratio.
3.4 When no need to balance intra-/inter-class diversity: a contrasting case
So far, we have studied a common practice of collecting the pre-training dataset, i.e., first sample  $K$  classes and then collect  $n$  samples for each class, and showed that when the size of the pre-training dataset  $N$  is fixed, neither too large nor too small  $K$  is optimal. However, there exist other ways of collecting the pre-training dataset. For example, one could collect a fixed set of data samples and then manipulate the value of  $K$  by clustering the samples into any number of clusters. This raises a natural question: is our conclusion still valid for this case? To answer this question and further understand the condition for our theory, in this section, we study a contrasting case of the data generation process corresponding to the aforementioned example, with which the trade-off no longer exists and, instead,  $K$  dominates the downstream performance. Concretely, the data generation process is given as follows:
Data generation process 2. The pre-training dataset is generated by first i.i.d. sampling data  $\{x_{1},x_{2},\dots ,x_{N}\}$  according to some distribution  $\mathcal{P}_{\mathcal{X}}\in \Delta (\mathcal{X})$  . We then obtain the label  $y_{i}\in [K]$  of each  $x_{i}$  by performing clustering. The final pre-training dataset is given as  $S = \{(x_1,y_1),\dots ,(x_N,y_N)\}$  (and  $\mathcal{P}$  is naturally induced).
The new data generation process is different from the one in Section 3.2. In particular, with different  $K$ , the new data generation process would not introduce new data samples as we manipulate the label by clustering the current sampled data, while for the data generation process in Section 3.2, the sampled data change according to the classes we picked.
As an analogy to Assumption 2, we make the following assumption on the correlation between the representation powers of the pre-training and the downstream task and then present the theorem.
Assumption 3. We assume the pre-training representation error can bound the downstream representation error. Specifically, there exists non-negative constants  $\nu_0^{\vec{P}}(\mathcal{P})$  and  $\nu_{1}^{\vec{P}}(\mathcal{P})$ , such that  $\forall h\in \mathcal{H}$
$$
\mathcal {E} _ {d} (h, \mathcal {P}) \leq \nu_ {1} ^ {\tilde {\mathcal {P}}} (\mathcal {P}) \mathcal {E} _ {p} (h) + \nu_ {0} ^ {\tilde {\mathcal {P}}} (\mathcal {P}).
$$
We further assume that  $\nu_0^{\tilde{P}}(\mathcal{P})$  and  $\nu_{1}^{\tilde{P}}(\mathcal{P})$  concentrate to their means, i.e., there exist  $\nu_{0}^{\tilde{P}}(\mathcal{P}_{\mathcal{X}})$ ,  $\nu_{1}^{\tilde{P}}(\mathcal{P}_{\mathcal{X}})$ ,  $C_0^\tilde{P}$  and  $C_1^\tilde{P}$ , such that  $|\nu_0^{\tilde{P}}(\mathcal{P}) - \nu_0^{\tilde{P}}(\mathcal{P}_{\mathcal{X}})| \leq \frac{C_0^{\tilde{P}}}{\sqrt{K}}$  and  $|\nu_{1}^{\tilde{P}}(\mathcal{P}) - \nu_{1}^{\tilde{P}}(\mathcal{P}_{\mathcal{X}})| \leq \frac{C_{1}^{\tilde{P}}}{\sqrt{K}}$ .
Theorem 3.2. Let Assumptions 1 and 3 hold. For data generation process 2, with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathcal {E} _ {d} \left(f _ {S ^ {d}} \circ h _ {S _ {p}}, \tilde {\mathcal {P}}\right) \leq \left(\nu_ {0} ^ {\tilde {\mathcal {P}}} \left(\mathcal {P} _ {\mathcal {X}}\right) + \frac {C _ {0} ^ {\tilde {\mathcal {P}}}}{\sqrt {K}}\right) \left(5 M _ {\ell} \sqrt {\frac {\log \frac {4}{\delta}}{2 N}} + \frac {2 G \sqrt {2}}{\sqrt {N}}\right) + \nu_ {1} ^ {\tilde {\mathcal {P}}} \left(\mathcal {P} _ {\mathcal {X}}\right) + \frac {C _ {1} ^ {\tilde {\mathcal {P}}}}{\sqrt {K}} \\ + 5 M _ {\ell} \sqrt {\frac {\log \frac {4}{\delta}}{2 \tilde {N}}} + 2 \sqrt {2} G \frac {1}{\sqrt {\tilde {N}}}. \\ \end{array}
$$
According to Theorem 3.2, we can see that there is no longer a trade-off between the inter-class diversity  $K$  and the intra-class diversity  $n$ . Instead, the bound gets smaller when  $K$  is larger. To verify this, we conduct experiments under this setting (Figure 4 in the Appendix A.3), and observe a tendency of lower error rate with increasing  $K$ .
Discussion. The classes and data samples correspond to two dimensions of diversity. When the size of the pre-training dataset is fixed, in the case of Theorem 3.1, both diversities would vary in a see-saw-like way with different  $K$ , but for Theorem 3.2, since we fix the data samples used, only one dimension of diversity varies when  $K$  is different and therefore the downstream performance only depends on the varying diversity  $(K)$ . This difference reveals the underlying cause of the downstream performance trade-off rather than relying solely on  $K$  and  $n$  values.
4 Justification and Application
In this section, we first verify our theoretical findings via empirical results. Then, we show the effectiveness of using the optimal class-to-sample ratio, which is estimated based on our theory with a relatively small number of data samples, to build larger pre-training datasets.
4.1 Are the trade-off curves for different  $N$  aligned?
According to our findings in Section 3.3, the test error on a downstream task is a convex function with respect to the class-to-sample ratio (Equation 4) and the optimal class-to-sample ratio  $\bar{x} = \frac{B^2}{A^2}$  is invariant to the size of pre-training dataset  $N$ . To empirically verify this, we visualize the performance on downstream tasks as a function of the class-to-sample ratio with different  $N$  in Figure 2.
(b) FGVCAircraft
(c) Flowers102
(d) MIT67
(e) Stanford40
(f) StanfordDogs
Figure 2: Test error rate across class-to-sample ratio. The vertical bar is the standard deviation.
From the figures, we can see that the curves of different  $N$  for a specific downstream task are aligned, as well as the optimal class-to-ratios, which follows our theoretical findings. This indicates that empirically, one could extrapolate the optimal class-to-ratio estimated with a small  $N$  for building a large-scale pre-training dataset. Note that the rightmost point of each curve corresponds to using all the classes in ImageNet, i.e.,  $K = 1000$ , and we can see that such a standard design choice does not lead to optimal downstream performance, especially with small pre-training datasets. In addition, we conducted experiments with different pre-trained datasets (Appendix A.4), different model backbones (Appendix A.6), and downstream tasks of different domains (Appendix A.5) to demonstrate that our findings are consistent across many scenarios.
4.2 Predicting the optimal number of pre-training classes
As a direct application of our theoretical and empirical findings, one could estimate the optimal class-to-sample ratio with a small  $N$  and use it to decide the optimal number of classes when building a larger pre-training dataset. In particular, denoted by  $\bar{x}$  the optimal class-to-sample ratio, the optimal number of classes for a given  $N$  is  $\bar{K} = \sqrt{\bar{x}N}$ . We refer to this approach as Extrapolation. We empirically compare it against the following methods of deciding the number of classes when building a pre-training dataset: 1) Standard: the number of classes equals 1000 as the standard design choice of ImageNet; 2) Grid Search: the number of classes corresponding to the data point with the lowest error rate for each curve in Figure 2; 3) Fitting: given the target size, we use the corresponding data points in Figure 2 to fit the theoretically-derived performance function (Equation 4), and then analytically calculate the optimal number of classes. We use the calculated number of classes to build a pre-training dataset and measure the performance of a model trained with it. In reality, the latter two baselines require repeatedly training models on pre-training datasets with the target size yet different numbers of classes and are therefore time-consuming and data-intensive.
We set the target size of pre-training dataset as  $\{50\mathrm{K}, 100\mathrm{K}\}$  and round the number of classes to an integer if needed. For our Extrapolation method, we only use three data points with  $N$  being much smaller than the target size to estimate the optimal class-to-sample ratio:  $N = 5000$  and  $K = \{10, 50, 200\}$ . We use the estimated optimal class-to-sample ratio for both target sizes. The results as well as the number of classes selected by the above methods can be found in Table 1. From
Table 1: Test error rate on downstream tasks (the first row of each task, lower is better), and the number of classes in the pre-training dataset (the second row).
N	Method	Target Dataset
CIFAR10	FGVCAircraft	Flowers102	MIT67	Stanford40	StanfordDogs
50K	Standard (K=1000)	29.19±0.14	77.80±0.28	34.08±0.35	57.51±0.18	62.45±0.36	64.96±0.38
Grid Search	26.24±0.44	75.96±0.43	32.70±0.21	54.10±0.4	57.05±0.44	59.12±0.2
(200)	(200)	(500)	(200)	(100)	(200)
Fitting	26.25±0.47	76.00±0.44	32.13±0.10	53.60±0.39	57.10±0.3	59.76 ± 0.26
(169)	(293)	(415)	(161)	(138)	(260)
Extrapolation	26.27±0.21	76.18±0.14	32.60±0.06	53.01±0.23	57.25±0.14	60.15±0.27
(190)	(168)	(296)	(163)	(134)	(158)
100K	Standard (K=1000)	25.04±0.26	75.21±0.48	27.69±0.45	52.79±0.36	54.69±0.39	54.70±0.45
Grid Search	23.13±0.06	73.70±0.50	27.15±0.44	50.30±0.46	51.45±0.33	51.98±0.30
(500)	(500)	(500)	(200)	(200)	(500)
Fitting	22.67±0.33	73.45±0.33	26.67±0.33	50.82±0.33	52.24±0.33	52.24±0.33
(276)	(372)	(655)	(249)	(207)	(392)
Extrapolation	23.12±0.13	73.30±0.17	26.98±0.41	50.42±0.25	52.32±0.06	53.17±0.26
(269)	(238)	(418)	(231)	(190)	(233)
Table 2: Total number of samples used for building the pre-training dataset.
N	Method	Target Dataset
CIFAR10	FGVCAircraft	Flowers102	MIT67	Stanford40	StanfordDogs
50K	Standard (K=1000)	50K
Grid Search	150K (5 trials)
Fitting	158.164K	161.570K	159.403K	158.694K	159.268K	160.538K
Extrapolation	55.418K	55.183K	55.830K	55.117K	54.598K	55.045K
100K	Standard (K=1000)	100K
Grid Search	260K (4 trials)
Fitting	272.336K	271.836K	268.164K	269.878K	261.981K	270.579K
Extrapolation	103.937K	103.608K	104.517K	103.515K	103.049K	103.401K
the results, we can see that although the ImageNet dataset is widely used, its number of classes  $(K = 1000)$  is not optimal for building a pre-training dataset of  $50\mathrm{K} / 100\mathrm{K}$  samples, since the Standard underperforms other methods. Besides, methods except for the Standard all render similar test error rate even though their number of classes are different, which reveals that the performance is not sensitive to the number of classes as long as we pick a reasonable number. Thus, our Extrapolation method is superior to Grid Search and Fitting, since it needs much fewer samples to estimate the number of classes, while both Grid Search and Fitting require building the pre-training dataset of target size multiple times.
4.3 Extra data needed for estimating the optimal class-to-sample ratio
One advantage of the Standard method is that it does not require extra data for estimating the optimal number of classes. In contrast, other methods would introduce extra data unused in the final pretraining dataset. For example, when estimating the optimal class-to-sample ratio, one may sample data from 1000 classes but eventually find the optimal number of classes is 600, then the data of the additional 400 classes would not be used in the final pre-training dataset. We then investigate how much data is needed by different methods to build the final pre-training dataset. We list the total number of samples used by different methods in Table 2. From the table, we can see that Grid Search and Fitting require much more samples than the target size of the pre-training dataset, since they involve building the pre-training dataset of the target size multiple times, while the number of extra data needed by Extrapolation is relatively small, because it estimates the optimal number of classes using a small  $N$  of 5000. In addition, using Extrapolation, one only needs to estimate the optimal class-to-sample ratio once and then use it for building pre-training datasets with different sizes without re-estimation.
As the Extrapolation requires more data than the Standard, a fairer comparison between them needs to ensure the total number of data used is similar, and the Standard would have a slightly larger pre-training dataset since it does not spend any data budget for estimation. In Figure 3, we compare
the Extrapolation to the Standard whose pre-training dataset size is slightly larger than the total number of data used by the Extrapolation. Each bar plot represents a specific downstream task, and the number in parentheses indicates the size  $(N)$  of the corresponding pre-training dataset. We can observe that in most cases, the Extrapolation demonstrates improved performance over the Standard even when the latter uses a larger pre-training dataset, which further justifies the effectiveness of the Extrapolation method.
Figure 3: Each bar plot visualizes the error rates of different methods on a specific downstream task. The number in parentheses is  $N$ , i.e. the size of the corresponding pre-training dataset.
We briefly review recent studies on supervised pre-training from data-centric perspectives. First, on the composition of the pre-training dataset, [9] presents a scaling law that predicts the test loss on downstream tasks under varying source dataset compositions, while [14] studies the performance on downstream tasks when subsets of the pre-training dataset are removed. Second, on the label space of supervised pre-training, [26] offers a statistical analysis explaining pre-training techniques' success in NLP, showing that class diversity in pre-training tasks substantially enhances sample efficiency in downstream tasks, while the study by [12] explores the impact of pre-training label granularity on downstream tasks, emphasizing the importance of selecting an appropriate level of label granularity. Lastly, [7] explores the impact of pre-training data distribution on transfer performance, finding the choice of the pre-training dataset to be crucial. In contrast, we dive into the trade-off of the intra-/inter-class diversity in the supervised pre-training dataset. The study related the most to ours is [13], where the authors empirically examined the importance of pre-training data characteristics on downstream performance. While covering a wide range of pre-training data characteristics, this study only briefly explores the trade-off of intra-/inter-class diversity in the pre-training dataset (Section 5.5 in [13]). Specifically, the authors only considered two different cases of intra-/inter-class diversity, i.e.,  $K = \{500, 1000\}$  for ImageNet. In contrast, we, both empirically and theoretically, show how such a trade-off would impact the downstream performance and our theory uncovers a surprising property of the optimal class-to-sample ratio: it is invariant to the size of the pre-training dataset.
In this study, we explore the trade-off of the intra-/inter-class diversity in supervised pre-training datasets of fixed size. We discovered that the optimal downstream performance is achieved through a balance of intra-/inter-class diversity. Our theory demonstrates that downstream performance depends on both diversities, and the optimal class-to-sample ratio remains constant regardless of the dataset size. We apply this finding to predict the optimal number of classes in pre-training datasets and provide evidence of its effectiveness across many scenarios.
[18] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
[19] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[20] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[21] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[22] Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition, pages 413-420. IEEE, 2009.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[24] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
[25] Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and Li Fei-Fei. Human action recognition by learning bases of action attributes and parts. In 2011 International conference on computer vision, pages 1331–1338. IEEE, 2011.
[26] Yulai Zhao, Jianshu Chen, and Simon S Du. Blessing of class diversity in pre-training. International Conference on Artificial Intelligence and Statistics, 2022.
[27] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1464, 2017.
A.1 Limitation and Potential Negative Social Impact
Some potential negative societal impacts might arise from this research, such as:
Bias Amplification: When the research discusses the optimal class-to-sample ratio in pre-training datasets, it does not discuss how these classes are determined. There's a risk that the choice of classes and the samples within these classes could reflect and perpetuate existing biases in society. For instance, if the classes are determined by stereotypical or biased criteria, models trained on these datasets could amplify these biases in their predictions or recommendations.
Overemphasis on Quantity over Quality: This research might also create an overemphasis on the quantity (size and diversity) of the data at the expense of its quality. Poor data quality could lead to the development of inaccurate or unreliable machine learning models.
A.2.1 Training Details
We build our code on Python and Pytorch. We fix the model to be the ResNet-18 [10]. For pre-training, we set the number of epochs to be 100 and the batch size to be 64. We use the Adam optimizer for training with a learning rate of 0.1, a momentum of 0.9, and a weight decay of 1e-4. We repeat each experiment 3 times with different seeds and report the mean and variance of the results. All experiments ran on a machine with an Intel(R) Xeon(R) CPU E5-2678 v3 with 512G memory and two 48G NVIDIA RTX A6000 GPUs.
A.2.2 Details of Dataset
The Pre-training Dataset:
- ImageNet [4]. It is an image dataset organized according to the WordNet hierarchy. Each meaningful concept in WordNet, possibly described by multiple words or word phrases, is called a "synonym set" or "synset". There are more than 100,000 synsets in WordNet; the majority of them are nouns  $(80,000+)$ .
- Place365 [25]. It has 1,803,460 training images with the image number per class varying from 3,068 to 5,000. The validation set has 50 images per class and the test set has 900 images per class.
The Downstream Tasks Dataset
- Stanford Actions 40 [25]. It contains images of humans performing 40 actions. There are about 180-300 images per class. We do not use bounding boxes and other annotation information for training. There are a total of 9,532 images, making it the smallest dataset in our benchmark experiments.
- Stanford Dogs 120 [15]. It contains images of 120 breeds of dogs worldwide. There are precisely 100 examples per category in the training set. It is used for the task of fine-grained image categorization. We do not use the bounding box annotations. There are a total of 20,580 images.
- MIT Indoors 67 [22]. It is a scene classification dataset containing 67 indoor scene categories, each consisting of 80 images for training and 20 for testing. Indoor scene recognition is challenging because spatial properties, background information, and object characters are expected to be extracted. There are 15,620 images in total.
- CIFAR10 [16]. It is a collection of images commonly used to train machine learning and computer vision algorithms. It contains 60,000 32x32 color images in 10 different classes. The ten classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. There are 6,000 images of each class.
- Flowers102 [20]. It is an image classification dataset consisting of 102 flower categories. The flowers are chosen to be flowers commonly occurring in UK. Each class consists of
between 40 and 258 images. The images have large scale, pose and light variations. In addition, some categories have significant variations within the category and several very similar categories.
- FGVCAircraft [18]. It contains 10,200 images of aircraft, with 100 images for each of 102 different aircraft model variants, most of which are airplanes. Each image's (main) aircraft is annotated with a tight bounding box and a hierarchical airplane model label. Aircraft models are organized in a four-level hierarchy.
- DomainNet-real and DomainNet-painting [21]. DomainNet-real and DomainNet-painting belong to real and painting domains, respectively, and both comprise 345 categories. DomainNet-real contains over 170k images, while DomainNet-painting has more than 70k images.
A.3 Empirical Verification of Theorem 3.2
(f) StanfordDogs
Figure 4: Test error rate across class-to-sample ratio. The vertical bar at each point is the standard deviation.
As shown in Figure 4, we can find that, across all datasets, the results generally showed a decreasing trend in test error rate as the class-to-sample ratio increased. This empirically supports the theoretical assertion that there is no downstream performance trade-off caused by the inter-class diversity  $K$  and the intra-class diversity  $n$ . Instead, the downstream performance improves with increasing  $K$ , as suggested in Theorem 3.2.
A.4 Experiments on Places365 as Pre-training Dataset
We use the Places365[27] for our pre-training and the result is represented in Figure 5. We then performed evaluations on the same batch of downstream tasks as in the main body of the paper to demonstrate the trade-off between intra- and inter-class diversity is consistent with our main findings.
(a) CIFAR10
(d) MIT67
Figure 5: Test error rate across class-to-sample ratio. ResNet-18 pre-trained on the Places365 dataset.
(a) DomainNet painting
Figure 6: Test error rate across class-to-sample ratio. ResNet-18 pre-trained on ImageNet and tested on two datasets of different domains from the DomainNet benchmark.
(b) DomainNet real
A.5 Experiments on Downstream Tasks of Different Domains