SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the pretrain phase, our approach learns part representations that are disentangled and diverse, but it does not achieve invariance to the incidental correlations of image background. During the fine-tuning stage, we utilize the learned foreground latent code to extract the relevant foreground information from a given image  $x$ :  $x_{f} = x\odot \mathcal{I}(L_{F})$ , where  $\odot$  denotes the
Hadamard product. The teacher network receives the original image, while the student network receives the foreground-only image. By distilling knowledge between the [class] tokens and foreground latent codes  $L_{F}$  of the student and teacher networks, we achieve invariance to the incidental correlations of image background.
$$
\mathcal {L} _ {c l s} ^ {i n v} = \mathcal {L} _ {c e} \left(\mathcal {F} _ {\phi} ^ {t} \left(\mathcal {F} _ {\theta} ^ {t} (x)\right), \mathcal {F} _ {\phi} ^ {s} \left(\mathcal {F} _ {\theta} ^ {s} \left(x _ {f}\right)\right)\right); \mathcal {L} _ {p} ^ {i n v} = \mathcal {L} _ {c e} \left(L _ {F} ^ {t} (x), L _ {F} ^ {s} \left(x _ {f}\right)\right) \tag {8}
$$
The two proposed invariant regularizations serve distinct purposes:  $\mathcal{L}_{cls}^{inv}$  encourages the model to classify images independently of the background, while  $\mathcal{L}_p^{inv}$  ensures that the latent foreground code captures relevant foreground information even when the background is absent, making the learned parts invariant to the incidental correlations.
Invariant Fine-tuning Objective. Finally, our fine-tuning objective is given as:
$$
\mathcal {L} _ {F T} = \lambda_ {c l s} \mathcal {L} _ {c l s} + \lambda_ {c l s} ^ {i n v} \mathcal {L} _ {c l s} ^ {i n v} + \lambda_ {p} ^ {i n v} \mathcal {L} _ {p} ^ {i n v} \tag {9}
$$
where  $\lambda_{cls},\lambda_{cls}^{inv}$ , and  $\lambda_p^{inv}$  are the weights given to each loss term and are tuned on the validation set after pretraining.
We evaluate the proposed approach on four datasets: MiniImageNet [35], TieredImageNet [40], FC100 [37], and ImageNet-9 [53]. The MiniImageNet, TieredImageNet, and FC100 are generally used as benchmark datasets for few-shot learning. For MiniImageNet, we use the data split proposed in [39], where the data samples are split into 64, 16, and 20 for training, validation, and testing, respectively. The TieredImageNet [40] contains 608 classes divided into 351, 97, and 160 for meta-training, meta-validation, and meta-testing. On the other hand, FC100 [37] is a smaller resolution dataset  $(32 \times 32)$  that contains 100 classes with class split as 60, 20, and 20.
To investigate the impact of background signals and data corruption on classifier performance, researchers introduced ImageNet-9 (IN-9L) [53]. IN-9L is a subset of ImageNet comprising nine coarse-grained classes: dog, bird, vehicle, reptile, carnivore, insect, instrument, primate, and fish. Within these super-classes, there are 370 fine-grained classes, with a training set of 183,006 samples. The authors of [53] created different test splits by modifying background signals, resulting in 4050
Input image (x)
ViT
$\Sigma_{H}\mathcal{I}(MSA(x_{f}))$
$\Sigma_{H}\mathcal{I}(MCA(\mathbf{D}))$
ViT +  $\mathcal{L}_{mix}$
$\Sigma_{H}\mathcal{I}(MSA(x_{f}))$
$\mathcal{I}(L_F)$
(a) Visualizing heatmaps of part projections using  $\mathrm{ViT} + \mathcal{L}_{\mathrm{mix}}$
$\mathcal{I}(\mathbf{D}^1)$
$\mathcal{I}(\mathbf{D}^2)$
$\mathcal{I}(\mathbf{D}^3)$
$\mathcal{I}(\mathbf{D}^4)$
$\mathcal{I}(\mathbf{D}^1)$
(b) Visualizing heatmaps of part projections using  $\mathrm{ViT} + \mathcal{L}_{\mathrm{mix}} + \mathcal{L}_Q$
Figure 5: Visualizing the quality of learned parts for input image from Figure 4. In Figure 5(a) and 5(b), we show the foreground mixture  $\mathcal{I}(L_F)$  and four foreground parts selected randomly from  $n_f$ . Meanwhile, Figure 5(c) and 5(d) show the sparse and orthogonal norms as metrics for evaluating the quality of the part matrix  $\mathbf{P}$ .
Figure 4: Visualizing MSA and MCA layers. The joint representation is obtained by averaging all attention heads  $(\sum_{H})$ . We study the effect of  $\mathcal{L}_{mix}$  on the interpretability of part based learners.
(c) Computing  $||\mathbf{P}||_1$
(d) Computing  $||\mathbf{PP}^{\mathrm{T}} - \mathbf{I}||_{1}$
samples per split to evaluate various classifiers. We use three of these test splits for our evaluation: Original (with no changes to the background), M-SAME (altering the background from the same class), and M-RAND (altering the background from a different class). Additionally, [53] introduced a metric called BG-GAP to assess the tendency of classifiers to rely on background signal, which is measured as the difference in performance between M-SAME and M-RAND.
We use a ViT-S backbone for all our experiments and follow the same pipeline in iBOT [60] for pre-training, keeping most hyperparameters unchanged. We use a batch size of 480 with a learning rate of 0.0005 decayed alongside a cosine schedule and pre-train DPViT for 500 epochs for all four datasets. Fine-tuning is carried out on the same train data for 50 epochs utilizing the class labels (similar to [22]). The DPViT architecture has  $K = 64$  and  $n_f = 40$  for all of our experiments. The pre-training and fine-tuning are carried out on 4 A40 GPUs. More details related to the architectures and optimization are provided in Appendix.
We start our experimental study by examining how incidental correlation affects part-learners' interpretability and quality of learned parts. This study is conducted on the MiniImageNet dataset. We then present experimental results on few-shot learning on the MiniImageNet, TieredImageNet, and FC100 datasets. Lastly, we provide a quantitative analysis of the influence of incidental background correlations on the various test splits in the ImageNet-9 dataset, followed by ablation studies and a discussion.
5.1 How do incidental correlations affect the interpretability of part learners?
To examine how incidental correlations impact various components, we solely employ the  $\mathcal{L}_{cls}$  loss to train DPViT on the MiniImageNet dataset. (Note that  $\mathcal{L}_{cls}$  is equivalent to ViT with parts). We then present a visualization of the attention layers (MSA and MCA) in Figure 4. The MSA layers effectively recognize relevant in-context information of some objects, but the CSA layers fail to pinpoint foreground details. This is because incidental correlations in the background dominate the CSA layers. However, incorporating the  $\mathcal{L}_{mix}$  regularization results in improved localization by the MCA layers, which are no longer influenced by incidental correlations. An important point to highlight is that the design of MSA layers causes them to focus on objects specific to a particular class. As a result, their effectiveness is reduced when multiple objects are present (as seen in Figure 4, where MSA misses objects on the left side). Conversely, CSA layers learn class-agnostic features
Method	Backbone	MiniImageNet	TieredImageNet	FC100
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
ProtoNets (2017) [47]	ResNet12	60.39±0.16	78.53±0.25	65.65±0.92	83.40±0.65	37.50±0.60	52.50±0.60
DeepEMD v2 (2020) [57]	ResNet12	68.77±0.29	84.13±0.53	71.16±0.87	86.03±0.58	46.47±0.26	63.22±0.71
COSOC (2021) [32]	ResNet12	69.28±0.49	85.16±0.42	73.57±0.43	87.57±0.10	-	-
MixtFSL (2021) [1]	ResNet12	63.98±0.79	82.04±0.49	70.97±1.03	86.16±0.67	-	-
Match-feat (2022) [2]	ResNet12	68.32±0.62	82.71±0.46	71.22±0.86	85.43±0.55	-	-
Label-Halluc (2022) [24]	ResNet12	67.04±0.70	85.87±0.48	71.97±0.89	86.80±0.58	47.37±0.70	67.92±0.70
FeLMi (2022) [44]	ResNet12	67.47±0.78	86.08±0.44	71.63±0.89	87.07±0.55	49.02±0.70	68.68±0.70
SUN (2022) [10]	VIT	67.80±0.45	83.25±0.30	72.99±0.50	86.74±0.33	-	-
FewTure (2022) [23]	Swin-Tiny	72.40±0.78	86.38±0.49	76.32±0.87	89.96±0.55	47.68±0.78	63.81±0.75
HCTransformer (2022) [22]	3× VIT-S	74.74±0.17	89.19±0.13	79.67±0.20	91.72±0.11	48.27±0.15	66.42±0.16
SMKD (2023) [30]	VIT-S	74.28±0.18	88.82±0.09	78.83±0.20	91.02±0.12	50.38±0.16	68.37±0.16
ConstNet (2021) [54]	ResNet12	64.89±0.23	79.95±0.17	70.15±0.76	86.10±0.70	43.80±0.20	59.70±0.20
TPMN (2021) [52]	ResNet12	67.64±0.63	83.44±0.43	72.24±0.70	86.55±0.63	46.93±0.71	63.26±0.74
CORL (2023) [21]	ResNet12	65.74±0.53	83.03±0.33	73.82±0.58	86.76±0.52	44.82±0.73	61.31±0.54
VIT-with-parts (Lcls)	VIT-S	72.15±0.20	87.61±0.15	78.03±0.19	89.08±0.19	48.92±0.13	67.75±0.15
Ours - DPViT	VIT-S	73.81±0.45	89.85±0.35	79.32±0.48	91.92±0.40	50.75±0.23	68.80±0.45
Table 1: Evaluating the performance of our proposed method on three benchmark datasets for few-shot learning - MiniImageNet, Tiered-ImageNet, and FC100. The top blocks show the non-part methods while the bottom block shows the part-based methods. The best results are bold, and  $\pm$  is the  $95\%$  confidence interval in 600 episodes.
Method	IN-9L ↑	Original ↑	M-SAME ↑	M-RAND ↑	BG-GAP ↓
ResNet-50 [53]	94.6	96.3	89.9	75.6	14.3
WRN-50×2 [53]	95.2	97.2	90.6	78.0	12.6
ConstNet	90.6	92.7	86.1	69.2	17.1
ViT-S pre [11]	82.5	84.9	72.2	50.3	21.9
CT [41]	84.7	85.5	73.1	51.5	21.6
ViT-with-parts	95.1	97.2	91.5	81.7	9.8
Ours - DPvIT	96.9	98.5	93.4	87.5	5.9
Table 2: Performance evaluation on domain shift of varying background and common data corruptions on ImageNet-9. Evaluation metric is Accuracy %.
Method	1-shot ↑	5-shot ↑	||P||1↓	||PPT-I||1↓
SMKD [30]	60.93	80.38	-	-
Lcls	61.24	81.12	8.41	25.82
Lcls+Lmix	62.81	83.25	8.73	24.61
Lcls+Lmix+LQ	62.15	82.95	0.35	0.56
Table 3: Ablation of different loss terms during pretraining of DP-ViT. We show the effect of each loss term on the MiniImageNet dataset.
consistently throughout the training set, enabling them to perform well in the presence of multiple objects. For further investigation of the complementary properties of  $MSA$  and  $MCA$ , please see the Appendix. Additionally, the Appendix includes visualizations of individual attention heads.
5.2 Studying the quality of learned part representations
As previously stated, the interpretability of learned parts is directly influenced by the sparsity and diversity of the part matrix  $\mathbf{P}$ . This is achieved by examining the impact of  $\mathcal{L}_Q$  during pretraining. We present visualizations of the learned foreground parts in Figure 5(a) and 5(b) for the input image  $x$  from Figure 4. While both setups successfully learn the foreground mixture,  $\mathcal{I}(L_F)$ , without  $\mathcal{L}_Q$ , the parts degenerate into a homogeneous solution (Figure 5(a)), lacking sparsity and diversity. With the inclusion of  $\mathcal{L}_Q$ , however, the learned parts become sparse and diverse (Figure 5(b)).
We use the  $L$ -1 and orthogonal norms of the part matrix to assess the sparsity and diversity of the parts. As illustrated in Figure 5(c) and Figure 5(d), the addition of  $\mathcal{L}_Q$  maintains bounded norms, inducing sparsity and diversity among the parts. The results also highlight higher norm values by employing  $\mathcal{L}_{cls} + \mathcal{L}_{mix}$  losses, which show the degeneracy caused by the introduction of mixture loss. Moreover, the higher sparse and orthogonal norms for  $\mathcal{L}_{cls}$  demonstrate that part-based methods generally do not maintain the quality of parts.
5.3 Few-shot Learning
We compare DPViT with recent part-based methods: ConstNet [54], TPMN [52], and CORL [21]; ViT-based methods: SUN [10], FewTure [23], HCTransformer [22], and SMKD [30] (for SMKD we compare with their prototype-based few-shot evaluation on all the datasets); and recent ResNet based few-shot methods: Match-feat [2], Label-halluc [24], and FeLMi [44].
As shown in Table 1, the proposed method outperforms existing part-based methods by clear margins. Moreover, DPViT achieves competitive performance compared to current ViT-based methods, especially on the FC100 dataset with low-resolution images. The FSL results show that the self-supervised ViT methods give an edge over the existing ResNet-based backbones.
Setting	Org ↑	M-S ↑	M-R ↑	||P||1 ↓	||PPT - I||1 ↓
Lcls	98.1	91.5	82.7	1.4	2.3
Lcls + Linv	98.5	93.4	87.5	1.4	2.3
Lcls + Linv + Lpinv	98.2	93.1	87.2	0.3	0.5
Table 4: Ablation of different loss terms during finetuning of DP-ViT. We show the effect of each loss term on the ImageNet-9 test splits.
Setting	1-shot ↑	5-shot ↑	||P||1 ↓	||PPT - I||1 ↓
λs=0.1, λo=0.1	73.9	89.9	0.7	0.9
λs=0.5, λo=0.5	73.8	89.8	0.3	0.5
λs=2.0, λo=2.0	72.4	88.2	0.1	0.2
Table 6: Evaluating the tradeoff between interpretability and few-shot generalization on MiniImageNet.
Method	Backbone	MiniImageNet -> CUB
ProtoNet	ResNet-18	67.1
MixtFSL [2]	ResNet-18	68.7
ConstNet [54]	ResNet-12	68.8
SUN [10]	ViT-S	72.1
Proposed-DPViT	ViT-S	77.1
Table 5: Cross-domain evaluation. We evaluate models trained on MiniImageNet on the CUB dataset.
0%	10%	50%	100%
1-s	5-s	1-s	5-s	1-s	5-s	1-s	5-s
DPViT	72.3	88.1	72.9	88.6	73.7	89.8	73.8	89.8
Table 7: Results on limited supervision of foreground masks on the MiniImageNet dataset. Here, we use 1-s for 1-shot and 5-s for 5-shot accuracy.
5.4 Studying the impact of incidental correlations of background on ImageNet-9
The impact of incidental background correlation on the ImageNet-9 dataset is examined, and the corresponding results are presented in Table 8. DPViT achieves the highest performance across IN-9L, Original, M-SAME, and M-RAND among the different test splits. Remarkably, our proposed DPViT method significantly reduces the BG-GAP value to 5.9, indicating its resilience to the effects of incidental correlation caused by varying backgrounds. In comparison to non-part methods such as ResNet-50 and WRN-  $50 \times 2$  (as shown in Table 8), ConstNet [54], which is a part-based method, suffers more from the negative impact of incidental correlation, underscoring the detrimental effect on the generalization of part-learners. To evaluate ConstNet, we utilize the provided source code by the authors to train their model on the IN-9L training data.
6 Ablation study and Discussion
In Section 5.1 and 5.2, we investigate how the inclusion of  $\mathcal{L}_{mix}$  and  $\mathcal{L}_Q$  impacts the interpretability of DPViT during the pretraining phase. We will now conduct an ablation study to assess the impact of both these loss components during the pre-training phase. As indicated in Table 3, introducing  $\mathcal{L}_{mix}$  results in enhanced 1-shot and 5-shot performance compared to the baseline model represented by  $\mathcal{L}_{cls}$ . It's worth noting that the SMKD [30] model performs worse than the part-ViT baseline  $(\mathcal{L}_{cls})$ . Upon incorporating  $\mathcal{L}_Q$ , the few-shot performance experiences a slight decrease, but the norms remain constrained, indicating improved quality of part representations.
Additionally, in this section, we examine the advantages of incorporating  $\mathcal{L}_{cls}^{inv}$  and  $\mathcal{L}_p^{inv}$  during the fine-tuning process on the ImageNet-9 dataset. This ablation study focuses on the significance of invariance terms in handling background-related incidental correlations. The results presented in Table 4 demonstrate that the introduction of  $\mathcal{L}_{cls}^{inv}$  enhances generalization on M-S (M-SAME) and M-R (M-RAND), thereby illustrating the benefits of inducing invariance through  $\mathcal{L}_{cls}^{inv}$  across varying backgrounds. Moreover, the inclusion of  $\mathcal{L}_{cls}^{inv}$  yields a lower BG-GAP value of 5.9, in contrast to 8.8 obtained without  $\mathcal{L}_{cls}^{inv}$ . Notably, the introduction of  $\mathcal{L}_{cls}^{inv}$  has no impact on the quality of parts, as evidenced by the final norm values of  $\mathbf{P}$ . By employing  $\mathcal{L}_p^{inv}$  in conjunction with other losses, the norms remain bounded, thereby preserving the interpretability of parts. We also observe a minimal decrease in classification performance upon introducing  $\mathcal{L}_p^{inv}$ , highlighting the trade-off between generalization and interpretability that will be explored in the following section.
6.1 Cross-domain evaluation
In order to assess the advantages of eliminating incidental correlations for transfer learning, we perform experiments on a cross-domain dataset, specifically MiniImageNet to CUB. In this setup, we evaluate the performance of models trained on MiniImageNet when applied to the CUB dataset. As demonstrated in Table 5, the DPViT model stands out by achieving the highest classification performance. These experiments highlight the clear benefits of employing part-based methods and eliminating incidental correlations.
6.2 Tradeoff between interpretability and generalization
We observe a tradeoff between the interpretability of parts and their generalization. To explore this, we conduct an ablation study during pre-training on the MiniImageNet dataset. As indicated in Table
Figure 6: Robustness of DPViT towards a weak foreground extractor on MiniImageNet.
(a) Foreground patches
Figure 7: Disentanglement of foreground/background patches extracted by DPViT.
(b) Background patches
6, assigning higher values to  $\lambda_{s}$  and  $\lambda_{o}$  places greater emphasis on  $\mathcal{L}_Q$ , leading to lower norm values and consequently improving the quality of parts. However, this also results in a slight reduction in few-shot accuracy. After careful analysis, we determine that an optimal value of 0.5 for both  $\lambda_{s}$  and  $\lambda_{o}$  strikes a balance, maintaining the quality of parts while preserving few-shot generalization.
6.3 Partial observability of foreground mask
Our training procedure employs foreground masks acquired through a foreground extractor, similar to the one described in [53]. To study the dependence of DPViT on the availability of foreground masks, we examine the weak/limited supervision scenario for foreground masks, where only a small subset of samples possesses the corresponding masks. As depicted in Table 7, we observe that DPViT achieves comparable performance even when only  $10\%$  of the training samples have mask information. The performance difference is less than  $1.5\%$  for 5-shot and less than  $0.8\%$  for 1-shot performance. Furthermore, Figure 7 presents visualizations of image patches surrounding a random foreground and a background part. We also find that in the setup with a  $0\%$  foreground mask, equivalent to  $\lambda_{mix} = 0$ , no disentanglement is observed in the extracted patches. (More visualizations can be found in the Appendix section).
6.4 Working with weak foreground extractor
The features of DPViT  $(\mathcal{F}_{\theta})$  depend entirely on the input sample  $\mathbf{x}$  in order to learn part representations, without explicitly incorporating the mask information. The mask serves as a weak signal to regularize our training objective and separate the foreground parts from the background. Additionally, introducing Gaussian noise in the latent codes enhances DPViT's ability to handle misalignment issues with the foreground masks. Consequently, the learned features remain unaffected by mistakes made by the existing foreground extractor. Moreover, we find that the mixture-of-parts representations can accurately determine the foreground location even when the mask information is missing or incorrect (as illustrated in Figure 6).
6.5 Limitations of DPViT
A constraint within our framework involves relying on a pre-existing foreground extractor. In certain scenarios, such as the classification of tissue lesions for microbiology disease diagnosis, obtaining an existing foreground extractor might not be feasible. Similarly, DPViT focuses on learning components that are connected to the data, yet it doesn't encompass the connections between these components, like their arrangement and hierarchical combination. Introducing compositional relationships among these components could enhance comprehensibility and facilitate the creation of a part-based model capable of learning relationships among the parts.
In this work, we study the impact of incidental correlations of image backgrounds on the interpretability and generalization capabilities of part learners. We introduce DPViT, a method that effectively learns disentangled part representations through a mixture-of-parts approach. Furthermore, we enhance the quality of part representations by incorporating sparse and orthogonal regularization constraints. Through comprehensive experiments, we demonstrate that DPViT achieves competitive performance comparable to state-of-the-art methods, all while preserving both implicit and explicit interpretability.
The computation resources used in preparing this research were provided by Digital Research Alliance of Canada, and The Vector Institute, Canada.
[50] Pavel Tokmakov, Yu-Xiong Wang, and Martial Hebert. Learning compositional representations for few-shot recognition. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 6372-6381, 2019.
[51] Xixi Wang, Xiao Wang, Bo Jiang, and Bin Luo. Few-shot learning meets transformer: Unified query-support transformers for few-shot classification. arXiv preprint arXiv:2208.12398, 2022.
[52] Jiamin Wu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. Task-aware part mining network for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8433-8442, 2021.
[53] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. International Conference on Learning Representations (ICLR), 2020.
[54] Weijian Xu, Yifan xu, Huaijin Wang, and Zhuowen Tu. Attentional constellation nets for few-shot learning. In International Conference on Learning Representations, 2021.
[55] Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Unsupervised discovery of parts, structure, and dynamics. International Conference on Learning Representations (ICLR), 2019.
[56] Fengyuan Yang, Ruiping Wang, and Xilin Chen. Semantic guided latent parts embedding for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5447-5457, 2023.
[57] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12203-12213, 2020.
[58] Hongguang Zhang, Jing Zhang, and Piotr Koniusz. Few-shot learning via saliency-guided hallucination of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2770-2779, 2019.
[59] Jianguo Zhang, Marcin Marszałek, Svetlana Lazebnik, and Cordelia Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study. International journal of computer vision, 73(2):213-238, 2007.
[60] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations, 2022.
[61] Song-Chun Zhu, David Mumford, et al. A stochastic grammar of images. Foundations and Trends® in Computer Graphics and Vision, 2(4):259-362, 2007.
Mitigating the Effect of Incidental Correlations on Part-based Learning (Supplementary Material)
A.1 Why the term "incidental correlations" for image background?
The concept of "incidental correlations" is derived from the notion of incidental endogeneity [12], which describes unintentional but genuine correlations between variables. In the context of our study, image backgrounds are not considered spurious because they offer contextual information that aids in decision-making. Therefore, the relationship between image backgrounds and classification is not anti-causal, as would be true if the backgrounds were spurious. We argue that the imbalance of specific image backgrounds in the training data is the primary factor contributing to the introduction of incidental correlations.
A.2 Training and inference details for pertaining and fine-tuning DPViT