SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure A1: Extended performance comparison of different in-domain DP methods for transfer learning across all downstream tasks. Experiment settings are consistent with Fig. 4.
The main results reported in Fig. 4 are stable: analysis with detailed numerical results and standard deviations. In Tab. A2, we provide the exact numerical results used to generate Fig. 4. These numbers give a more granular view of the performance comparisons. The results, calculated over three independent trials, show that the magnitude of the standard deviations is quite small relative to the mean values. This indicates that the trends and conclusions drawn from Fig. 4 are generally valid and not significantly affected by trial variations. To maintain readability and clarity, we choose not to include these minor standard deviations in Fig. 4.
Figure A2: Ablation study on the sensitivity of FM against the choice of cluster numbers. For FM, the original dataset is pre-clustered into  $K$  clusters, with  $K = 2000$  as the default setting in this study. Here, various  $K$  values (500, 1000, and 2000) are studied in this experiment. Other settings are aligned with Tab. 2.
FM performance is robust to the choice of cluster number. The results presented in Fig. A2 provide insights into the sensitivity of FM performance to variations of the cluster number  $(K)$ . This extended study from Tab. 2 shows that FM performance remains robust as the cluster number varies from 500 to 2000, preserving the benefits of DP for transfer learning without much performance degradation. This implies that FM can provide reliable and stable results even when the hyperparameter settings are not optimal.
LM identifies the most influential source classes. To validate that the classes with the highest LM scores are the most influential, we present the pruning trajectory in Fig. A3, where pruning is executed in the reverse order of the class scores, different from the proposed DP implementation. That is, classes with the smallest scores are retained, while the ones with the highest scores are pruned. Remarkably, even a slight pruning ratio (such as  $10\%$ ) leads to a significant degradation in downstream performance. This evidence underscores the benefits of source classes with high LM scores in promoting the downstream performance.
FM can be smoothly applied to MoCov3. We conducted experiments to illustrate the effectiveness of our proposed method (FM) when applied to the more recent SSL framework MoCov3 [117] using
Figure A3: Performance comparison on the downstream performance of LM with the "Ordered" and the "Reversed" pruning order. Here, the "Ordered" strategy retains source classes with the highest scores, while the "Reversed" order prunes these classes first. Other settings are aligned with Fig. 4.
Figure A4: Flatness evaluations on the models pretrained with the pruned source data using different methods. The models are first pretrained and then finetuned using LP. The flatness are evaluated with respect to the downstream training loss and are quantified by the reversed sharpness evaluated through five widely acknowledged sharpness metrics. The results are normalized to  $0\% \sim 100\%$ , with  $100\%$  denoting the highest (best) flatness across all the methods given one specific flatness metric.
the ViT structure. In line with our SSL experiment plan detailed in Tab. 2 of the paper, Tab. A3 tested FM on three downstream datasets, specifically OxfordPets, Flowers102, and SUN397, with a source data class pruning ratio ranging from  $50\%$  to  $80\%$ . The results affirm that the principal conclusions drawn from MoCov2 remain consistent with MoCov3 on ViT. Our method, FM, successfully identifies data subsets within the source dataset that can be pruned at high ratios without compromising downstream performance (termed as "winning subsets"). For instance, in one particular case, the FM-based winning subsets achieve a pruning ratio of up to  $70\%$  on OxfordPets.
Explore LM in the multi-task setting. A limitation of our method is its task-specific nature in data influence analysis. As our methods assess source data influence for specific downstream tasks, developing a universal DP solution for multiple tasks simultaneously is challenging. As a preliminary study, we examine the performance of source dataset pruning for multiple downstream tasks simultaneously in Fig. A5. While LM can still identify winning subsets, the maximum pruning ratio diminishes as more tasks are considered. We will include this limitation in the Conclusion section.
LM helps remove data biases through DP for a downstream task. To investigate the scenarios with data biases, we conducted experiments on CIFAR-10C (the out-of-distribution scenario) [118]. We first pruned ImageNet given CIFAR-10 as a downstream task and evaluated the model on CIFAR-10C with different corruption types. Tab. A4 shows LM results with different pruning ratios for 5 strong perturbation types in CIFAR-10C. Impressively, LM can achieve winning subsets with pruning up to  $80\%$ , even better than on CIFAR-10, confirming our method's effectiveness to filter biased data points to some degree.
Table A3: Experiments with MoCov3 on ViT. Other settings follow Tab. 2 of the original submission. The performance surpassing the unpruned setting (pruning ratio  $0\%$ ) is highlighted in cyan. The best result in each setting is marked in bold. FM consistently outperforms other baselines and can find winning subsets with pruning ratios of more than  $50\%$ .
Dataset	OxfordPets
Pruning Ratio	0%	50%	60%	70%	80%
RANDOM	87.34	82.13	80.27	75.42	68.34
MODERATE	86.17	85.29	84.01	81.32
GRAND	87.21	86.11	83.19	80.78
FM (ours)	87.68	87.51	87.39	84.14
Dataset	SUN397
Pruning Ratio	0%	50%	60%	70%	80%
RANDOM	60.36	59.22	58.45	56.73	54.29
MODERATE	60.13	59.61	58.21	56.44
GRAND	60.39	59.27	58.95	57.15
FM (ours)	60.49	60.55	60.42	59.88
Dataset	Flowers102
Pruning Ratio	0%	50%	60%	70%	80%
RANDOM	93.96	92.41	91.65	90.17	88.41
MODERATE	93.75	92.41	91.42	90.11
GRAND	93.88	93.21	91.77	90.45
FM (ours)	94.11	94.28	93.97	91.42
Figure A5: DP achieved by LM in the multi-task setting given 4 downstream tasks. This expands Fig. 4, i.e., the single-task setting, where source data is pruned based on an individual task.
Experiments on the few-shot benchmark VTAB [119]. We extend our experiments to include the VTAB benchmark [119]. As shown in Tab. A5, even in the few-shot setting, our major conclusions remain valid. LM can effectively pinpoint the most influential source subsets and eliminate the less valuable ones. For most tasks considered, winning subsets with pruning ratios up to  $40\%$  are successfully found.
Subsets obtained by LM improve the flatness of loss landscape on downstream tasks. Fig. A4 evaluates the flatness of the loss landscape of the model pretrained on an  $80\%$  -pruned source dataset and subsequently finetuned on the downstream OxfordPets dataset [56]. Better flatness in the loss landscape is typically associated with better transferability. We quantify this flatness using the measure of reversed sharpness calculated via five widely accepted metrics:  $\epsilon$  -sharpness [120], low pass filter-based measure (LPF) [121], the max eigenvalue of the Hessian  $(\lambda_{\mathrm{max}}(\mathbf{H}))$  [122], the Frobenius norm of the Hessian  $(|\mathbf{H}|_F)$  [122], and Fisher Rao Norm (FRN) [123]. The results suggest that pretraining on the subset selected by LM leads the model towards a flatter region in
Table A4: Experiments on CIFAR-10C. LM-based source dataset pruning on ImageNet (given CIFAR-10 as the downstream task) applies to transfer learning against CIFAR-10C. 5 out of the 19 corruption types are tested.
Dataset	Pruning Ratio
0%	20%	40%	60%	80%
CIFAR10	96.83	96.88	97.03	96.57	95.41
+ Gaussian Noise	82.13	82.67	82.89	82.60	81.19
+ Defocus Blur	84.73	85.22	85.36	84.92	82.75
+ Impulse Noise	84.62	85.21	84.78	85.93	85.11
+ Shot Noise	83.18	83.25	83.49	83.76	83.24
+ Speckle Noise	83.11	83.59	83.29	83.57	82.27
Table A5: Experiments on the few-shot transfer learning benchmark VTAB. Seven tasks in the NATURAL set are studied following the setting of Fig. 4. Each task contains 800 training and 200 testing samples.
Caltech101	CIFAR-100	DTD	Flowers102	DxfordPets	SUN397	SVHN	Mean
No Prune	80.23	46.39	62.48	90.39	88.42	33.32	87.24	69.78
Pruning 20%	80.39	46.65	62.88	90.78	88.57	34.22	87.45	70.13
Pruning 40%	80.34	46.49	62.92	90.88	88.62	33.81	87.21	70.04
Pruning 60%	78.65	45.51	62.57	90.12	87.35	33.57	86.38	69.16
Pruning 80%	73.31	41.33	61.18	89.42	85.11	31.49	84.22	66.58
the downstream loss landscape, potentially contributing to the superior transferability of LM when compared to the baseline methods.
Figure A6: (Left) Interpretation merit of the data pruning strategy by LM. (Right) Feature distribution visualization using t-SNE for the source class selection by LM for OxfordPets with  $90\%$  pruning ratio.
Feature distribution analysis. Fig. A6 provides visual explanations of DP at both deep representation (feature) and data levels. Here we focus on the LM method with a pruning ratio of  $90\%$  given the downstream dataset OxfordPets. The other settings are consistent with Fig. 4. In Fig. A6 (right), we visualize the source ImageNet classes (including pruned ones and retrained ones) and target OxfordPets classes in terms of their class-wise feature centroids in a 2D space achieved by t-SNE[124]. The class feature centroid is obtained by averaging the features of data points within a class, extracted by the pretrained source model on the full ImageNet dataset. As we can see, all retained source classes are grouped together and are close to the target classes. This indicates that the source classes that the pruning process share the most resemblance with the target data. In contrast, the pruned source classes are more dispersed and located further away from the target data classes. Furthermore, Fig. A6 (left) exhibits image examples of target classes as well as pruned and retrained source classes. We observe that image examples in the retained source classes (e.g., relating to
Table A6: Performance on surrogate model size. Experiments follow the setting of Fig. 4: RN-101 is first pretrained on the pruned source dataset (ImageNet) based on the surrogate model, and then finetuned on the downstream task OxfordPets. Under different surrogate models, the source class selection overlapping ratio with the used surrogate model RN-18 in the submission is reported under  $50\%$  pruning ratio.
Surrogate Model Architecture	RN-20s	VGG-2	RN-32s	VGG-4	RN-44s	RN-56s	VGG-8	RN-18 (Default)
Param. # (M)	0.236	0.417	0.516	0.698	0.706	0.896	5.53	11.69
Source Acc. (%)	36.25	22.56	40.77	29.44	43.74	45.72	58.45	68.73
Largest Pruning Ratio of Winning Subsets (%)	60	50	80	70	80	80	80	80
Source Class Selection Overlap (%)	89.3	84.4	90.7	87.2	93.5	94.8	97.7	100
animals) are semantically closer to the target data points (relating to pets) than the pruned ones. This highlights the ability of LM to effectively identify and retain the most relevant classes for the downstream tasks. We provide more examples for FM in Fig. A7.
Examining the top selected classes by FM and their image examples. In Fig. A7, we showcase the top-10 source classes chosen by FM, as determined by the endowed scores. These selected classes closely correspond to the downstream datasets' subjects, demonstrating FM's effectiveness in identifying relevant classes for transfer learning. This finding also aligns with our observations in Fig. A6, showing that FM identifies source classes resembling downstream data.
Figure A7: The source classes with top-10 scores selected by FM for the 8 downstream tasks studied in this work. For each class, the class label (name) as well as an representative image example is presented.