Title: Mitigating the Effect of Incidental Correlations on Part-based Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: part-based learning

interpretability

few-shot learning

vision transformers
Abstract: Intelligent systems possess a crucial characteristic of breaking complicated problems into smaller reusable components or parts and adjusting to new tasks using these part representations. However, current part-learners encounter difficulties in dealing with incidental correlations resulting from the limited observations of objects that may appear only in specific arrangements or with specific backgrounds. These incidental correlations may have a detrimental impact on the generalization and interpretability of learned part representations. This study asserts that part-based representations could be more interpretable and generalize better with limited data, employing two innovative regularization methods. The first regularization separates foreground and background information's generative process via a unique mixture-of-parts formulation. Structural constraints are imposed on the parts using a weakly-supervised loss, guaranteeing that the mixture-of-parts for foreground and background entails soft, object-agnostic masks. The second regularization assumes the form of a distillation loss, ensuring the invariance of the learned parts to the incidental background correlations. Furthermore, we incorporate sparse and orthogonal constraints to facilitate learning high-quality part representations.
By reducing the impact of incidental background correlations on the learned parts, we exhibit state-of-the-art (SoTA) performance on few-shot learning tasks on benchmark datasets, including MiniImagenet, TieredImageNet, and FC100. We also demonstrate that the part-based representations acquired through our approach generalize better than existing techniques, even under domain shifts of the background and common data corruption on the ImageNet-9 dataset.

Mitigating the Effect of Incidental Correlations on Part-based Learning

Abstract

Intelligent systems possess a crucial characteristic of breaking complicated problems into smaller reusable components or parts and adjusting to new tasks using these part representations. However, current part-learners encounter difficulties in dealing with incidental correlations resulting from the limited observations of objects that may appear only in specific arrangements or with specific backgrounds. These incidental correlations may have a detrimental impact on the generalization and interpretability of learned part representations. This study asserts that partbased representations could be more interpretable and generalize better with limited data, employing two innovative regularization methods. The first regularization separates foreground and background information's generative process via a unique mixture-of-parts formulation. Structural constraints are imposed on the parts using a weakly-supervised loss, guaranteeing that the mixture-of-parts for foreground and background entails soft, object-agnostic masks. The second regularization assumes the form of a distillation loss, ensuring the invariance of the learned parts to the incidental background correlations. Furthermore, we incorporate sparse and orthogonal constraints to facilitate learning high-quality part representations. By reducing the impact of incidental background correlations on the learned parts, we exhibit state-of-the-art (SoTA) performance on few-shot learning tasks on benchmark datasets, including MiniImagenet, TieredImageNet, and FC100. We also demonstrate that the part-based representations acquired through our approach generalize better than existing techniques, even under domain shifts of the background and common data corruption on the ImageNet-9 dataset. The implementation is available on GitHub: https://github.com/GauravBh1010tt/DPViT.git

1 Introduction

Many datasets demonstrate a structural similarity by exhibiting "parts" or factors that reflect the underlying properties of the data [15, 18, 21, 28, 31, 43, 54]. Humans are efficient learners who represent objects based on their various traits or parts, such as a bird's morphology, color, and habitat characteristics. Part-based methods learn these explicit features from the data in addition to convolution and attention-based approaches (which only learn the internal representations), making them more expressive [7, 21, 41, 52, 54]. Most existing part-based methods focus on the unsupervised discovery of parts by modeling spatial configurations [14, 21, 52, 54, 61], while others use part localization supervision in terms of attribute vectors [25, 41, 50, 56] or bounding boxes [7]. Partbased methods come with a learnable part dictionary that provides a direct means of data abstraction and is effective in limited data scenarios, such as few-shot learning. Furthermore, the parts can be combined into hierarchical representations to form more significant components of the object

description [50, 55]. Part-based learning methods offer advantages in terms of interpretability and generalization, particularly in safety-critical domains like healthcare, aviation and aerospace industry, transportation systems (e.g., railways, highways), emergency response, and disaster management. These fields often face challenges in collecting large training samples, making part-based learning methods valuable. Furthermore, the ability to explain decisions becomes crucial in these contexts.

Various studies have indicated that correlations between image background and labels can introduce biases during machine learning model training [29, 32-34, 45, 46, 53, 59]. These correlations exist because specific background configurations create shortcuts for the model during training [32, 53]. While background information is crucial for decision-making, imbalanced background configurations can create unintended correlations, leading to undesirable outcomes. These correlations negatively impact the interpretability and generalization of part-based learners. For instance, let's consider a scenario where a laptop, a charger, and an iPod are on a table. In one case, let's examine the situation without any context or background information. Without background information, the model may struggle to understand the purpose and significance of components or parts such as the laptop, charger, and iPod on a table. It could fail to differentiate between these objects or grasp their functionalities, resulting in a lack of recognition and understanding. Conversely, suppose the model is predominantly trained with examples of these items on tables. In that case, it may overly focus on the background elements, such as the table itself, disregarding the individual entities. Thus it becomes essential to handle incidental correlations of image background to achieve a balanced understanding. Existing part-based approaches fail to handle incidental correlations that arise due to specific background signals dominating the training data (analogous to Figure 1(b)), thereby hampering their interpretability and generalization to limited data scenarios.

Having high-quality part representations is essential for achieving proficiency in part-based learning. In this context, quality pertains to the sparsity and diversity of the learned parts. Sparsity ensures that only a few parts are responsible for a given image, as images comprise a small subset of parts. Conversely, diversity in part representations prevents the parts from converging into a single representation and facilitates each part's learning of a unique data representation. Although incidental correlations can negatively affect learned parts' quality, the quality of part-based methods is a significant challenge that all part learners face. While previous studies have addressed the issue of part quality [50, 52], their solutions do not assure the sparsity and diversity of the learned parts, thereby failing to guarantee high-quality parts.

To solve the aforementioned challenges, we introduce the Disentangled Part-based Vision Transformer (DPViT), which is trained to be resilient to the incidental correlations of image backgrounds and ensures high-quality part representations. We propose a disentangled pre-training phase, which separates the generative process of the foreground and background information through a unique mixture-of-parts formulation. We impose structural constraints on the parts to ensure that the mixtureof-parts for foreground and background includes soft, object-agnostic masks without requiring direct supervision of part localization. The parts are learned to be invariant to incidental correlations using a self-supervised distillation fine-tune phase. To address the issue of the quality of learned parts, we impose sparse and spectral penalties on the part matrix to guarantee the high quality of learned part representations. We include an assessment of the sparse and spectral norms of the part matrix as a quantitative indicator of the learned parts' quality. Finally, we evaluate the effectiveness of our method on benchmark few-shot datasets, including MiniImagenet [35], TieredImageNet [40],

and FC100 [6]. To demonstrate the robustness of our proposed method to incidental correlations of backgrounds and common data corruptions, we use the benchmark ImageNet-9 dataset [53].

Our key contributions can be summarized as follows:

We propose regularization techniques to disentangle the generative process of foreground and background information through a mixture-of-parts formulation. Additionally, we employ a selfsupervised distillation regularization to ensure that the learned parts remain invariant to incidental correlations of the image background.

We ensure the high quality of learned parts by employing both sparsity and spectral orthogonal constraints over the part matrix. These constraints prevent the parts from degenerating and encourage a diverse range of part representations.

Apart from our evaluation of standard few-shot benchmark datasets, we also analyze the impact of incidental correlations of background and typical data distortions by utilizing the benchmark ImageNet-9 dataset [53].

2 Related Work

Part-based learning . The advantages of learning part-based representations have been extensively researched in image recognition tasks [15, 18, 31, 38, 43, 49, 52, 55, 56]. Earlier methods attempted to learn parts by defining a stochastic generative process [19, 43]. Part-based methods have been broadly classified into unsupervised and supervised categories. Unsupervised methods concentrate on learning the spatial relationship between parts by using part dictionaries without the supervision of part localization [14, 15, 20, 21, 28, 49, 52, 54, 61]. In contrast, supervised part-based methods rely on the supervision of part localization through attribute vectors [16, 27, 41, 50] or part bounding box information [7]. In the literature, parts are also referred to as concepts when supervision about part localization is involved [7, 41].

Discovering parts in an unsupervised way is a more challenging scenario that is applicable to most practical problems. Part dictionaries help data abstraction and are responsible for learning implicit and explicit data representations. For example, [28] clustered DCNN features using part-based dictionaries, while [26] introduced a generative dictionary-based model to learn parts from data. Similarly, [20] uses part-based dictionaries and an attention network to understand part representations. The ConstellationNet [54] and CORL [21] are some of the current methods from the constellation family [14, 61], and use dictionary-based part-prototypes for unsupervised discovery of parts from the data. Our approach also belongs to this category, as we only assume the part structure without requiring any supervision of part localization.

Incidental correlations of image background . Image backgrounds have been shown to affect a machine learning model's predictions, and at times the models learn by utilizing the incidental correlations between an image background and the class labels [4, 32, 42, 45, 46, 53, 58, 59]. To mitigate this issue, background researchers have used augmentation methods by altering background signals and using these samples during the training [32, 53, 58]. [53] performed an empirical study on the effect of image background on in-domain classification. They introduce several variants of background augmentations to reduce a model's reliance on the background. Similarly, [58] uses saliency maps of the image foreground to generate augmented samples to reduce the effect of the image background. Recently, [32] showed the effectiveness of background augmentation techniques for minimizing the effect of incidental correlations on few-shot learning.

Unlike these methods, our approach does not depend on background augmentations but instead learns the process of generating foreground and background parts that are disentangled. Furthermore, our proposed approach is not sensitive to the quality of foreground extraction and can operate with limited supervision of weak foreground masks.

Few-shot learning and Vision Transformers . In recent years, few-shot learning (FSL) has become the standard approach to evaluate machine learning models' generalization ability on limited data [1, 2, 5, 13, 24, 36, 44, 47, 54]. Vision transformers (ViT) [11] have demonstrated superior performance on FSL tasks [10, 22, 23, 30, 48, 51], and self-supervised distillation has emerged as a popular training strategy for these models [8, 17, 22, 30, 51]. A recent trend involves a two-stage procedure where models are pretrained via self-supervision before fine-tuning via supervised learning [8, 17, 22, 30, 60]. For example, [17] leverages self-supervised training with iBOT [60] as a pretext task, followed by inner loop token importance reweighting for supervised fine-tuning. HCTransformer [22] uses attribute surrogates learning and spectral tokens pooling for pre-training vision transformers

and performs fine-tuning using cascaded student-teacher distillation to improve data efficiency hierarchically. SMKD [30] uses iBOT pre-training and masked image modeling during fine-tuning to distill knowledge from the masked image regions.

Our approach employs a two-stage self-supervised training strategy of vision transformers akin to [22, 30, 60]. However, unlike existing methods that focus on generalization in few-shot learning, our primary objective is to learn part representations that are invariant to incidental correlations of the image background. Our training procedure is designed to facilitate learning disentangled and invariant part representations, which is impossible through existing two-stage self-supervised pipelines alone.

3 Problem Formulation and Preliminaries

In few-shot classification, the aim is to take a model trained on a dataset of samples from seen classes D seen with abundant annotated data, and transfer/adopt this model to classify a set of samples from a disjoint set of unseen/novel classes D novel with limited labeled data. Formally, let D seen = { ( x , y ) } , where x ∈ X corresponds to an image and y ∈ Y seen corresponds to the label among the set of seen classes. We also assume that during training, we have limited supervision of class-agnostic foreground-background mask ( M f , M b ) for regularization during training, which can be easily obtained by any weak foreground extractor as a preprocessing step (following [53]). Please note that no mask information is required for D novel at inference.

We follow the work of [9, 60] on self-supervised training of ViTs to design our pretrain phase. During training, we apply random data augmentations to generate multiple views of the a given image x v ∈ D seen . These views are then fed into both the teacher and student networks. Our student network, with parameters θ s , includes a ViT backbone encoder and a projection head ϕ s that outputs a probability distribution over K classes. The ViT backbone generates a [ cls ] token, which is then passed through the projection head. The teacher network, with parameters θ t , is updated using Exponentially Moving Average (EMA) and serves to distill its knowledge to the student by minimizing the cross-entropy loss over the categorical distributions produced by their respective projection heads.

For inference, we use the standard M -way, N -shot classification by forming tasks ( T ), each comprising of support set ( S ) and query set ( Q ), constructed from D novel . Specifically, a support set consists of M × N images; N random images from each of M classes randomly chosen from Y novel . The query set consists of a disjoint set of images, to be classified, from the same M classes. Following the setup of [47], we form the class prototypes ( c m ) using samples from S . The class prototypes and learned feature extractor ( F θ ) are used to infer the class label ˆ y for an unseen sample x q ∈ Q using a distance metric d .

4 Proposed Methodology

Given an input sample x ∈ R H × W × C , and a patch size f , we extract flattened 2D patches x f ∈ R N × ( F 2 · C ) , where N is the number of patches generated and ( F, F ) is the resolution of each image patch. Similar to a standard ViT architecture [11], we prepend a learnable [ class ] token and positional embeddings to retain the positional information. The flattened patches are passed to multi-head self attention layers and MLP blocks to generate a feature vector z p = MSA ( x f ) .

Next, we define the parts as part-based dictionaries P = { p k ∈ R F 2 · C } K k =1 , where p k denotes the part-vector for the part indexed as k . The part-matrix ( P ) is initialized randomly and is considered a trainable parameter of the architecture. Note that the dimension of each part-vector is equal to the dimension of flattened 2D patches, which is F 2 · C . For each part p k , we compute a distance map D k ∈ R N where each element in the distance map is computed by taking dot-product between the part p k and all the N patches: D k = x f · p k .

Using the distance maps D ∈ R N × K , we introduce a multi-head cross-attention mechanism and compute the feature vector: z d = MCA ( F ψ ( D )) , where F ψ is an MLP layer which upsamples D : K → F 2 · C . The cross-attention layer shares a similar design to self-attention layers; the only

TransformerEncoder

1

L ×

Dot product

Parts (P)

8~N(0,1)

-Activations

Mixture-of-Part

Concatenation/

sum

Distance Maps

1

MLP

MLP

MSA

Multi-head self

attention

mic

1

1

1

Norm

Norm

MCA

cross attention

Multi-head

B

1

1

1

Multi-Head

~N(0, 1) Gaussian noise

Attention

1

Multi-Head

MCA

1

Norm

Attention

1

MSA

MLP

Norm

TransformerEncoder

1

1

Cls token

Pos Emb +

Distance

Maps

Patches

Embedded

difference is the dimensions of input distance maps. The cross-attention helps contextualize information across part-dictionary and the spatial image regions and provides complementary properties to MSA layers. (Please refer to Appendix for experiments on complementary properties of MSA and MCA ).

Finally, we add the output feature vectors of MSA and MCA to form the feature extractor F θ defined in Eqn 1:

Disentanglement of foreground-background space using mixture-of-parts . We start by dividing the parts-matrix P ∈ R K × F 2 · C into two disjoint sets: foreground set P F ∈ R n f × C and background set P B ∈ R n b × C , such that K = n f + n b .

Next, we construct latent variables to aggregate the foreground and background information using a mixture-of-parts formulation over the computed distance maps D :

where, α k and β k are the learnable weights given to the k th part-vector in the corresponding mixture, whereas δ f and δ b are Gaussian noises sampled from N (0 , 1) . The Gaussian noise is added to the latent codes to ensure that mixture-of-parts are robust to common data distortions. Please note that the purpose of Gaussian noise is not to induce variability in the latent codes, as foreground information for a given image is deterministic.

Finally, our disentanglement regularization takes the form of an alignment loss between the latent codes and the class-agnostic foreground-background masks:

where, I ( L ) is the bilinear interpolation of a given latent code L to the same size as M .

During the architectural design phase, we employ distinct components ( P ) for each encoder block. Consequently, z p and z d are calculated in an iterative manner and subsequently transmitted to the subsequent encoder block. Regarding the computation of L mix , we utilize the distance maps D from the concluding encoder block. While it's feasible to calculate L mix iteratively for each block and then aggregate them for a final L mix computation, our observations indicate that this approach amplifies computational expenses and results in performance deterioration. As a result, we opt to compute L mix using the ultimate encoder block.

Learning high-quality part representations . A problem with minimizing the mixture objective defined in Eqn 5 is that it may cause the degeneration of parts, thereby making the part representations less diverse. One solution is to enforce orthogonality on the matrix P m × n by minimizing || P T P -I || , similar to [50]. However, the solution will result in a biased estimate as m<n ; that is, the number of

parts ( K ) is always less than the dimensionality of parts ( F 2 · C ). In our experiments, we observed that increasing K beyond a certain threshold degrades the performance as computational complexity increases, and is consistent with the findings in [54]. (Please refer to our Appendix section for experiments on the different values of K ). To minimize the degeneration of parts, we design our quality assurance regularization by minimizing the spectral norm of P T P -I , and by adding L 1 sparse penalty on the part-matrix P . The spectral norm of P T P -I has been shown to work with over-complete ( m<n ) and under-complete matrices ( m ≥ n ) [3].

where I is the identity matrix, λ s and λ o are the regularization coefficients for sparsity and orthogonality constraints. σ ( P ) is the spectral norm of the matrix P which is computed using the scalable power iterative method described in [3].

Disentangled Pretraining Objective . We pretrain DPViT using the following loss function:

where λ cls , λ mix , λ s , and λ o are the weights given to each loss term and are tuned on the validation set.

4.1 Invariant fine-tuning

In the pretrain phase, our approach learns part representations that are disentangled and diverse, but it does not achieve invariance to the incidental correlations of image background. During the fine-tuning stage, we utilize the learned foreground latent code to extract the relevant foreground information from a given image x : x f = x ⊙ I ( L F ) , where ⊙ denotes the

Hadamard product. The teacher network receives the original image, while the student network receives the foreground-only image. By distilling knowledge between the [ class ] tokens and foreground latent codes L F of the student and teacher networks, we achieve invariance to the incidental correlations of image background.

The two proposed invariant regularizations serve distinct purposes: L inv cls encourages the model to classify images independently of the background, while L inv p ensures that the latent foreground code captures relevant foreground information even when the background is absent, making the learned parts invariant to the incidental correlations.

Invariant Fine-tuning Objective . Finally, our fine-tuning objective is given as :

where λ cls , λ inv cls , and λ inv p are the weights given to each loss term and are tuned on the validation set after pretraining.

5 Experiments

We evaluate the proposed approach on four datasets: MiniImageNet [35], TieredImageNet [40], FC100 [37], and ImageNet-9 [53]. The MiniImageNet, TieredImageNet, and FC100 are generally used as benchmark datasets for few-shot learning. For MiniImageNet, we use the data split proposed in [39], where the data samples are split into 64, 16, and 20 for training, validation, and testing, respectively. The TieredImageNet [40] contains 608 classes divided into 351, 97, and 160 for metatraining, meta-validation, and meta-testing. On the other hand, FC100 [37] is a smaller resolution dataset (32 × 32) that contains 100 classes with class split as 60, 20, and 20.

To investigate the impact of background signals and data corruption on classifier performance, researchers introduced ImageNet-9 (IN-9L) [53]. IN-9L is a subset of ImageNet comprising nine coarse-grained classes: dog, bird, vehicle, reptile, carnivore, insect, instrument, primate, and fish. Within these super-classes, there are 370 fine-grained classes, with a training set of 183,006 samples. The authors of [53] created different test splits by modifying background signals, resulting in 4050

(a) Visualizing heatmaps of part projections using ViT

+

L

mix

(b) Visualizing heatmaps of part projections using ViT

+

L

mix

+

L

Q

||

P

||

1

||

PP

T

-

I

||

1

ViT

ViT + Lmix

Input image (x)

EHI(MSA(cf))

EHI(MCA(D))

EHI(MSA(cf))

EHI(MCA(D))

8

Lcls+Lmix +LQ

Valueofnorm

6

Lcls+Lmiz

Lels

4

2

0

0

50

100

150

NumberofEpochs

T(LF)

I(D

I(D²)

T(D²

T(D

4

I(LF)

T(D1)

I(D2)

T(D²

T(D4)

25

Valueofnorm

20

Lels+Lmiz

15

Lels

10

5

0

0

50

100

150

NumberofEpochs

samples per split to evaluate various classifiers. We use three of these test splits for our evaluation: Original (with no changes to the background), M-SAME (altering the background from the same class), and M-RAND (altering the background from a different class). Additionally, [53] introduced a metric called BG-GAP to assess the tendency of classifiers to rely on background signal, which is measured as the difference in performance between M-SAME and M-RAND.

We use a ViT-S backbone for all our experiments and follow the same pipeline in iBOT [60] for pre-training, keeping most hyperparameters unchanged. We use a batch size of 480 with a learning rate of 0.0005 decayed alongside a cosine schedule and pre-train DPViT for 500 epochs for all four datasets. Fine-tuning is carried out on the same train data for 50 epochs utilizing the class labels (similar to [22]). The DPViT architecture has K =64 and n f =40 for all of our experiments. The pre-training and fine-tuning are carried out on 4 A40 GPUs. More details related to the architectures and optimization are provided in Appendix.

We start our experimental study by examining how incidental correlation affects part-learners' interpretability and quality of learned parts. This study is conducted on the MiniImageNet dataset. Wethen present experimental results on few-shot learning on the MiniImageNet, TieredImageNet, and FC100 datasets. Lastly, we provide a quantitative analysis of the influence of incidental background correlations on the various test splits in the ImageNet-9 dataset, followed by ablation studies and a discussion.

5.1 How do incidental correlations affect the interpretability of part learners?

To examine how incidental correlations impact various components, we solely employ the L cls loss to train DPViT on the MiniImageNet dataset. (Note that L cls is equivalent to ViT with parts). We then present a visualization of the attention layers ( MSA and MCA ) in Figure 4. The MSA layers effectively recognize relevant in-context information of some objects, but the CSA layers fail to pinpoint foreground details. This is because incidental correlations in the background dominate the CSA layers. However, incorporating the L mix regularization results in improved localization by the MCA layers, which are no longer influenced by incidental correlations. An important point to highlight is that the design of MSA layers causes them to focus on objects specific to a particular class. As a result, their effectiveness is reduced when multiple objects are present (as seen in Figure 4, where MSA misses objects on the left side). Conversely, CSA layers learn class-agnostic features

consistently throughout the training set, enabling them to perform well in the presence of multiple objects. For further investigation of the complementary properties of MSA and MCA , please see the Appendix. Additionally, the Appendix includes visualizations of individual attention heads.

5.2 Studying the quality of learned part representations

As previously stated, the interpretability of learned parts is directly influenced by the sparsity and diversity of the part matrix P . This is achieved by examining the impact of L Q during pretraining. We present visualizations of the learned foreground parts in Figure 5(a) and 5(b) for the input image x from Figure 4. While both setups successfully learn the foreground mixture, I ( L F ) , without L Q , the parts degenerate into a homogeneous solution (Figure 5(a)), lacking sparsity and diversity. With the inclusion of L Q , however, the learned parts become sparse and diverse (Figure 5(b)).

We use the L -1 and orthogonal norms of the part matrix to assess the sparsity and diversity of the parts. As illustrated in Figure 5(c) and Figure 5(d), the addition of L Q maintains bounded norms, inducing sparsity and diversity among the parts. The results also highlight higher norm values by employing L cls + L mix losses, which show the degeneracy caused by the introduction of mixture loss. Moreover, the higher sparse and orthogonal norms for L cls demonstrate that part-based methods generally do not maintain the quality of parts.

5.3 Few-shot Learning

We compare DPViT with recent part-based methods: ConstNet [54], TPMN [52], and CORL [21]; ViT-based methods: SUN [10], FewTure [23], HCTransformer [22], and SMKD [30] (for SMKD we compare with their prototype-based few-shot evaluation on all the datasets); and recent ResNet based few-shot methods: Match-feat [2], Label-halluc [24], and FeLMi [44].

As shown in Table 1, the proposed method outperforms existing part-based methods by clear margins. Moreover, DPViT achieves competitive performance compared to current ViT-based methods, especially on the FC100 dataset with low-resolution images. The FSL results show that the self-supervised ViT methods give an edge over the existing ResNet-based backbones.

Table 7: Results on limited supervision of foreground masks on the MiniImageNet dataset. Here, we use 1-s

for 1-shot and 5-s for 5-shot accuracy.

5.4 Studying the impact of incidental correlations of background on ImageNet-9

The impact of incidental background correlation on the ImageNet-9 dataset is examined, and the corresponding results are presented in Table 8. DPViT achieves the highest performance across IN-9L, Original, M-SAME, and M-RAND among the different test splits. Remarkably, our proposed DPViT method significantly reduces the BG-GAP value to 5 . 9 , indicating its resilience to the effects of incidental correlation caused by varying backgrounds. In comparison to non-part methods such as ResNet-50 and WRN-50 × 2 (as shown in Table 8), ConstNet [54], which is a part-based method, suffers more from the negative impact of incidental correlation, underscoring the detrimental effect on the generalization of part-learners. To evaluate ConstNet, we utilize the provided source code by the authors to train their model on the IN-9L training data.

6 Ablation study and Discussion

In Section 5.1 and 5.2, we investigate how the inclusion of L mix and L Q impacts the interpretability of DPViT during the pretraining phase. We will now conduct an ablation study to assess the impact of both these loss components during the pre-training phase. As indicated in Table 3, introducing L mix results in enhanced 1-shot and 5-shot performance compared to the baseline model represented by L cls . It's worth noting that the SMKD [30] model performs worse than the part-ViT baseline ( L cls ). Upon incorporating L Q , the few-shot performance experiences a slight decrease, but the norms remain constrained, indicating improved quality of part representations.

Additionally, in this section, we examine the advantages of incorporating L inv cls and L inv p during the fine-tuning process on the ImageNet-9 dataset. This ablation study focuses on the significance of invariance terms in handling background-related incidental correlations. The results presented in Table 4 demonstrate that the introduction of L inv cls enhances generalization on M-S (M-SAME) and M-R (M-RAND), thereby illustrating the benefits of inducing invariance through L inv cls across varying backgrounds. Moreover, the inclusion of L inv cls yields a lower BG-GAP value of 5 . 9 , in contrast to 8 . 8 obtained without L inv cls . Notably, the introduction of L inv cls has no impact on the quality of parts, as evidenced by the final norm values of P . By employing L inv p in conjunction with other losses, the norms remain bounded, thereby preserving the interpretability of parts. We also observe a minimal decrease in classification performance upon introducing L inv p , highlighting the trade-off between generalization and interpretability that will be explored in the following section.

6.1 Cross-domain evaluation

In order to assess the advantages of eliminating incidental correlations for transfer learning, we perform experiments on a cross-domain dataset, specifically MiniImageNet to CUB. In this setup, we evaluate the performance of models trained on MiniImageNet when applied to the CUB dataset. As demonstrated in Table 5, the DPViT model stands out by achieving the highest classification performance. These experiments highlight the clear benefits of employing part-based methods and eliminating incidental correlations.

6.2 Tradeoff between interpretability and generalization

We observe a tradeoff between the interpretability of parts and their generalization. To explore this, we conduct an ablation study during pre-training on the MiniImageNet dataset. As indicated in Table

Input image (x)

Fore mask (M§)

I(LF)

I(LB)

6, assigning higher values to λ s and λ o places greater emphasis on L Q , leading to lower norm values and consequently improving the quality of parts. However, this also results in a slight reduction in few-shot accuracy. After careful analysis, we determine that an optimal value of 0 . 5 for both λ s and λ o strikes a balance, maintaining the quality of parts while preserving few-shot generalization.

6.3 Partial observability of foreground mask

Our training procedure employs foreground masks acquired through a foreground extractor, similar to the one described in [53]. To study the dependence of DPViT on the availability of foreground masks, we examine the weak/limited supervision scenario for foreground masks, where only a small subset of samples possesses the corresponding masks. As depicted in Table 7, we observe that DPViT achieves comparable performance even when only 10% of the training samples have mask information. The performance difference is less than 1.5% for 5-shot and less than 0.8% for 1-shot performance. Furthermore, Figure 7 presents visualizations of image patches surrounding a random foreground and a background part. We also find that in the setup with a 0% foreground mask, equivalent to λ mix = 0 , no disentanglement is observed in the extracted patches. (More visualizations can be found in the Appendix section).

6.4 Working with weak foreground extractor

The features of DPViT ( F θ ) depend entirely on the input sample x in order to learn part representations, without explicitly incorporating the mask information. The mask serves as a weak signal to regularize our training objective and separate the foreground parts from the background. Additionally, introducing Gaussian noise in the latent codes enhances DPViT's ability to handle misalignment issues with the foreground masks. Consequently, the learned features remain unaffected by mistakes made by the existing foreground extractor. Moreover, we find that the mixture-of-parts representations can accurately determine the foreground location even when the mask information is missing or incorrect (as illustrated in Figure 6).

6.5 Limitations of DPViT

A constraint within our framework involves relying on a pre-existing foreground extractor. In certain scenarios, such as the classification of tissue lesions for microbiology disease diagnosis, obtaining an existing foreground extractor might not be feasible. Similarly, DPViT focuses on learning components that are connected to the data, yet it doesn't encompass the connections between these components, like their arrangement and hierarchical combination. Introducing compositional relationships among these components could enhance comprehensibility and facilitate the creation of a part-based model capable of learning relationships among the parts.

7 Conclusion

In this work, we study the impact of incidental correlations of image backgrounds on the interpretability and generalization capabilities of part learners. We introduce DPViT, a method that effectively learns disentangled part representations through a mixture-of-parts approach. Furthermore, we enhance the quality of part representations by incorporating sparse and orthogonal regularization constraints. Through comprehensive experiments, we demonstrate that DPViT achieves competitive performance comparable to state-of-the-art methods, all while preserving both implicit and explicit interpretability.

Acknowledgments and Disclosure of Funding

The computation resources used in preparing this research were provided by Digital Research Alliance of Canada, and The Vector Institute, Canada.

References

Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné. Mixture-based feature space learning for few-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9041-9051, 2021.

Arman Afrasiyabi, Hugo Larochelle, Jean-François Lalonde, and Christian Gagné. Matching feature sets for few-shot image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9014-9024, 2022.

Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems , 31, 2018.

Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Bastien Pasdeloup, Stéphane Pateux, and Vincent Gripon. Easy-ensemble augmented-shot-y-shaped learning: State-of-the-art few-shot classification with simple components. Journal of Imaging , 8(7):179, 2022.

Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. International Conference on Learning Representations (ICLR) , 2018.

Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. In International Conference on Learning Representations (ICLR) , 2021.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo. Self-promoted supervision for few-shot transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX , pages 329-347. Springer, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR) , 2021.

Jianqing Fan, Fang Han, and Han Liu. Challenges of big data analysis. National science review , 1(2):293-314, 2014.

Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang. Melr: Meta-learning via modeling episode-level relationships for few-shot learning. In International Conference on Learning Representations (ICLR) , 2021.

Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence , 28(4):594-611, 2006.

Sanja Fidler and Ales Leonardis. Towards scalable representations of object categories: Learning a hierarchy of parts. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1-8, 2007.

Sanja Fidler and Ales Leonardis. Towards scalable representations of object categories: Learning a hierarchy of parts. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1-8, 2007.

Hanan Gani, Muzammal Naseer, and Mohammad Yaqub. How to train vision transformer on small-scale datasets? arXiv preprint arXiv:2210.07240 , 2022.

Qingzhe Gao, Bin Wang, Libin Liu, and Baoquan Chen. Unsupervised co-part segmentation through assembly. In International Conference on Machine Learning , pages 3576-3586. PMLR, 2021.

Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Advances in Neural Information Processing Systems (NeurIPS) , 29, 2016.

Ju He, Adam Kortylewski, and Alan Yuille. Compas: Representation learning with compositional part sharing for few-shot classification. arXiv preprint arXiv:2101.11878 , 2021.

Ju He, Adam Kortylewski, and Alan Yuille. Corl: Compositional representation learning for few-shot classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3890-3899, 2023.

Yangji He, Weihan Liang, Dongyang Zhao, Hong-Yu Zhou, Weifeng Ge, Yizhou Yu, and Wenqiang Zhang. Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9119-9129, 2022.

Markus Hiller, Rongkai Ma, Mehrtash Harandi, and Tom Drummond. Rethinking generalization in few-shot classification. arXiv preprint arXiv:2206.07267 , 2022.

Yiren Jian and Lorenzo Torresani. Label hallucination for few-shot classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7005-7014, 2022.

Yuhe Jin, Weiwei Sun, Jan Hosang, Eduard Trulls, and Kwang Moo Yi. Tusk: Task-agnostic unsupervised keypoints. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Adam Kortylewski, Qing Liu, Huiyu Wang, Zhishuai Zhang, and Alan Yuille. Combining compositional models and deep networks for robust object classification under occlusion. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 1333-1341, 2020.

Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences , 40, 2017.

Renjie Liao, Alexander Schwing, Richard S Zemel, and Raquel Urtasun. Learning deep parsimonious representations. In International Conference on Neural Information Processing Systems (NeurIPS) , pages 5083-5091, 2016.

Chia-Ching Lin, Hsin-Li Chu, Yu-Chiang Frank Wang, and Chin-Laung Lei. Joint feature disentanglement and hallucination for few-shot image classification. IEEE Transactions on Image Processing , 30:9245-9258, 2021.

Han Lin, Guangxing Han, Jiawei Ma, Shiyuan Huang, Xudong Lin, and Shih-Fu Chang. Supervised masked knowledge distillation for few-shot transformers. arXiv preprint arXiv:2303.15466 , 2023.

Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjorn Ommer. Unsupervised part-based disentangling of object shape and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10955-10964, 2019.

Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang, Lingxi Xie, Zenglin Xu, and Qi Tian. Rectifying the shortcut learning of background for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS) , 34:13073-13085, 2021.

Xu Luo, Jing Xu, and Zenglin Xu. Channel importance matters in few-shot image classification. In International conference on machine learning , pages 14542-14559. PMLR, 2022.

Chong Ma, Lin Zhao, Yuzhong Chen, David Weizhong Liu, Xi Jiang, Tuo Zhang, Xintao Hu, Dinggang Shen, Dajiang Zhu, and Tianming Liu. Rectify vit shortcut learning by visual saliency. arXiv preprint arXiv:2206.08567 , 2022.

Oriol metas, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems (NeurIPS) , 29:36303638, 2016.

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. International Conference on Learning Representations (ICLR) , 2018.

Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018.

Patrick Ott and Mark Everingham. Shared parts for deformable part-based models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1513-1520, 2011.

Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. International Conference on Learning Representations (ICLR) , 2017.

Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. International Conference on Learning Representations (ICLR) , 2018.

Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. Attention-based interpretability with concept transformers. In International Conference on Learning Representations (ICLR) , 2022.

Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint arXiv:1808.03305 , 2018.

David A Ross and Richard S Zemel. Learning parts-based representations of data. Journal of Machine Learning Research , 7(11), 2006.

Aniket Roy, Anshul Shah, Ketul Shah, Prithviraj Dhar, Anoop Cherian, and Rama Chellappa. Felmi: Few shot learning with hard mixup. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. Unsupervised discovery of parts, structure, and dynamics , 2020.

Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalkquantifying and controlling the effects of context in classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8218-8226, 2019.

Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS) , 2017.

Mingze Sun, Weizhi Ma, and Yang Liu. Global and local feature interaction with vision transformer for few-shot image classification. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 4530-4534, 2022.

Yihong Sun, Adam Kortylewski, and Alan Yuille. Weakly-supervised amodal instance segmentation with compositional priors. arXiv preprint arXiv:2010.13175 , 2020.

Pavel Tokmakov, Yu-Xiong Wang, and Martial Hebert. Learning compositional representations for few-shot recognition. In IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6372-6381, 2019.

Xixi Wang, Xiao Wang, Bo Jiang, and Bin Luo. Few-shot learning meets transformer: Unified query-support transformers for few-shot classification. arXiv preprint arXiv:2208.12398 , 2022.

Jiamin Wu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. Task-aware part mining network for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8433-8442, 2021.

Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. International Conference on Learning Representations (ICLR) , 2020.

Weijian Xu, Yifan xu, Huaijin Wang, and Zhuowen Tu. Attentional constellation nets for few-shot learning. In International Conference on Learning Representations , 2021.

Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Unsupervised discovery of parts, structure, and dynamics. International Conference on Learning Representations (ICLR) , 2019.

Fengyuan Yang, Ruiping Wang, and Xilin Chen. Semantic guided latent parts embedding for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5447-5457, 2023.

Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12203-12213, 2020.

Hongguang Zhang, Jing Zhang, and Piotr Koniusz. Few-shot learning via saliency-guided hallucination of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2770-2779, 2019.

Jianguo Zhang, Marcin Marszałek, Svetlana Lazebnik, and Cordelia Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study. International journal of computer vision , 73(2):213-238, 2007.

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations , 2022.

Song-Chun Zhu, David Mumford, et al. A stochastic grammar of images. Foundations and Trends® in Computer Graphics and Vision , 2(4):259-362, 2007.

Mitigating the Effect of Incidental Correlations on Part-based Learning (Supplementary Material)

A Appendix

A.1 Why the term "incidental correlations" for image background?

The concept of "incidental correlations" is derived from the notion of incidental endogeneity [12], which describes unintentional but genuine correlations between variables. In the context of our study, image backgrounds are not considered spurious because they offer contextual information that aids in decision-making. Therefore, the relationship between image backgrounds and classification is not anti-causal, as would be true if the backgrounds were spurious. We argue that the imbalance of specific image backgrounds in the training data is the primary factor contributing to the introduction of incidental correlations.

A.2 Training and inference details for pertaining and fine-tuning DPViT

Training Details . Our approach involves pre-training the Vision Transformer backbone and projection head using the same method described in the iBOT paper [60]. We mostly keep the hyper-parameter settings unchanged without tuning. By default, we use the Vit-Small architecture, which consists of 21 million parameters. The patch size is set to 16 as our default configuration. The student and teacher networks have a shared projection head for the [cls] token output. The projection heads for both networks have an output dimension of 8192. We adopt a linear warm-up strategy for the learning rate over 10 epochs, starting from a base value of 5e-4, and then decaying it to 1e-5 using a cosine schedule. Similarly, the weight decay is decayed using a cosine schedule from 0.04 to 0.4. We employ a multi-crop strategy to improve performance with 2 global crops (224×224) and 10 local crops (96×96). The scale ranges for global and local crops are (0.4, 1.0) and (0.05, 0.4), respectively. Following [60], we use only the local crops for self-distillation with global crops from the same image. Additionally, we apply blockwise masking to the global crops inputted into the student network. The masking ratio is uniformly sampled from [0, 1, 0.5] with a probability of 0.5, and with a probability of 0.5, it is set to 0. Our batch size is 480, with a batch size per GPU of 120. DPViT is pre-trained for 500 epochs for the given training set for all the datasets.

We use the value of λ cls = 1 , λ s = 0 . 5 , λ o = 0 . 5 for all the datasets. In the case of ImageNet-9, we incorporate the class labels by incorporating a logit head onto the projection heads. This allows us to calculate the cross-entropy loss based on the provided class labels. The explicit utilization of class labels is necessary for the ImageNet-9 dataset because the evaluation involves straightforward classification rather than few-shot learning.

Fine-tuning Details . Once the pretraining stage is completed, we proceed to train the model using the supervised contrastive loss, which involves distilling knowledge from [cls] tokens across different views of images (referred to as L cls in Equation 9 of the main draft). The fine-tuning process is conducted for 50 epochs using the same training data in the given dataset. We maintain the same set of hyperparameters used in the initial pretraining stage without additional tuning.

We use the value of λ inv cls = 1 , and λ inv p = 0 . 5 for all the datasets.

Inference Details . For inference purposes, we utilized a feature representation obtained by the [cls] token of the teacher network. We also found concatenating the weighted average pooling of the generated patches with the [cls] token useful in a few-shot evaluation. The weights for the weighted average pooling are determined by taking the average of the attention values of the [cls] token across all heads of the final attention layer.

In the case of ImageNet-9, the logit head is used to infer the class label for the given sample in the test set.

A.3 Details regarding the multi-head attention modules

The design of our attention layers draws inspiration from the standard self-attention mechanism, commonly known as qkv self-attention (SA) [11]. In our implementation, we calculate a weighted sum over all values v in the input sequence z , where z has dimensions of R N × D . The attention

weights A ij are determined based on the pairwise similarity between two elements of the sequence and their corresponding query q i and key k j representations.

Multihead self-attention (MSA) is an expansion of the self-attention mechanism, where we perform k parallel self-attention operations, known as "heads," and then combine their outputs through concatenation. In order to maintain consistent computation and the number of parameters when adjusting the value of k , the dimension D h (as defined in Equation 10) is typically set to D/k .

A.4 Details regarding the power iterative method to compute spectral norm

We follow the power iterative method described in [3] to compute the spectral norm for ( P T P -I ) . Starting with a randomly initialized v ∈ R n , we iteratively perform the following procedure a small number of times (2 times by default) :

The power iterative method reduces computational cost from O ( n 3 ) to O ( mn 2 ) , which is practically much faster when used with our training procedure.

A.5 Comparing ViT-S [11] and Concept Transformer (CT) [41] on ImageNet-9

In addition to the findings presented in Section 5.4 (Table 2 in the main draft), we conducted a comparison with vanilla ViT-S pretrained on Imagenet and ConceptTransformers (CT) as well. CT, as described in the study by [41], has a notable limitation in that it relies on attribute supervision for part localization information. This restriction restricts the applicability of CT in scenarios where attribute information is absent, such as in the case of ImageNet-9. To train CT without attributes, we utilized the code provided by the authors and deactivated the attribute loss, allowing CT to be trained without relying on the attribute information 2 . This adjustment significantly decreases the performance of CT but enables a fair comparison with other methods on ImageNet-9. It is worth noting that CT employs the ViT-S backbone pretrained on ImageNet as its default architecture. Moreover, we train ConstNet [54] using the source code provided by the authors 3 .

As indicated in Table 8, DPViT demonstrates superior performance compared to both ViT-S pretrained on ImageNet and CT, exhibiting a clear advantage. CT can be seen as a pretrained ViT-S model with the inclusion of part dictionaries, but it experiences a noticeable drop in performance when confronted with the presence of incidental correlations in the image backgrounds (as observed in the low M-SAME and M-RAND performance in Table 8). This demonstrates that the part learners in general cannot effectively deal with the incidental correlations of backgrounds and are susceptible to varying backgrounds.

A.6 Ablation study with different values of K and n f on MiniImageNet

In this analysis, we investigate the impact of varying the number of parts, denoted as K , on the MiniImageNet dataset. Specifically, we explore the effects of altering the number of foreground parts, represented by n f , as well as the number of background vectors, which can be calculated as K -n f . Table 9 presents the obtained results, demonstrating the influence of different values of K , n f , and n b on parts, foreground parts, and background parts, respectively.

Our findings indicate that maintaining K = 64 and selecting n f = 2 K/ 3 yields the highest performance. When employing a significantly lower number of part vectors, the model's capacity becomes insufficient, leading to performance degradation. Conversely, employing a larger value of K results in increased computational complexity associated with distance maps, subsequently leading to lower performance.

A.7 Computational complexity of DPViT

Adding part-dictionaries to MCA layers slightly increases the trainable parameters from 21 M (ViT-S) to 25 M (DPViT). It is also possible to share the attention layers, analogous to the Siamese networks, for MSA and MCA, which keeps the number of trainable parameters to 21 M . DPViT results in a similar performance in terms of few-shot accuracy when the attention layers are shared, as shown in Table 10.

A.8 Stage-1 pertaining comparison with SMKD [30]

Table 11 showcases the few-shot evaluation results of DPViT on the MiniImageNet dataset. In addition, we compare the performance of DPViT with the first-stage performance of SMKD [30]. It is worth noting that both DPViT and SMKD utilize the iBOT [60] pretraining strategy. However, incorporating part-dictionaries, MSA, and MCA layers in DPViT's pretraining phase contributes to its superior performance compared to SMKD.

A.9 Studying complementary properties of MSA and MCA

Based on Section 5.1 in the main draft, our study focuses on examining the complementary characteristics of MSA and MCA. MSA is designed to be effective for images containing a small number of objects, but it struggles to capture the spatial relationships among multiple objects. In contrast, MCA layers utilize distance maps to learn spatial relationships and prioritize objects without considering

(b) Attention heads of MSA

(c) Attention heads of MCA

their specific classes. In simpler terms, MSA may overlook certain objects that are not crucial for classification, while MCA emphasizes learning spatially similar objects.

Additionally, we present the visualization of attention heads in Figure 9, 10, and 12. The MSA heads excel at identifying objects for classification but may overlook relevant objects with significant spatial context, such as the "charger" in Figure 9 and the "garbage box" in Figure 12. On the other hand, the MCA layers perform well in scenarios involving multiple objects (Figure 9 and 12), but struggle when spatially similar objects are present, as seen with the confusion between the "red grass" and the "fish" in Figure 10.

A.10 Optimization functions for L mix

The segmentation masks M f and M b are composed of binary values. Our selection of the L 2 -norm for L mix is informed by empirical observations. Additionally, we conducted experiments with alternative loss functions like L1 and cosine distance during the pretraining phase. As shown in Table, while the L1-norm results in lower performance, the cosine distance results in unstable training with training loss reaching +infinity, eventually NAN . We achieve the best performance with the L2-norm.

A.11 Noise functions for δ

We conduct an ablation study on the choice of noise functions used for δ . As shown in Table 13, we ascertain that Gaussian noise is more effective in promoting resilience within the latent codes, while simultaneously preserving the interpretability of parts when compared with salt-and-pepper and speckle noise during the pretraining phase.

A.12 Visualization foreground parts

We present additional part visualizations for Figure 10(a) and 11(a). These are shown in Figure 12(a) and 12(b).

(b) Attention heads of MSA

A.13 Qualitative comparison of extracted patches with ConstNet [54]

In order to showcase the acquired parts of DPViT and ConstNet, we provide visualizations in Figure 13 and 14. This is achieved by selecting the nearest patches to the parts. Figure 13 illustrates the separation of foreground and background concepts accomplished by our model, whereas Figure 14 exhibits the patches surrounding the learned parts from the ConstNet model.

While DPViT learns to disentangle the foreground patches from the backgrounds, the patches extracted by ConstNet suffer from the entanglement caused due to incidental correlations of backgrounds.

(a) Parts visualizations for Figure 10(a)

X

I(LF)

I(D1)

I(D2)

I(D3

3

T(D

1

1

1

I(LF)

I(D')

I(D2)

I(D3)

I(D4)

X

(b) Parts visualizations for Figure 11(a)

1ES

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds up a part-based learning algorithm for few-shot classification with components preventing incidental correlations during training. ViT is adopted as the architecture, which facilitates part interpretation by recognizing each image patch as a specific part/foreground/background. Experiments on several few-shot learning benchmarks show the effectiveness of the approach, and an additional experiment on ImageNet-9 verifies that the approach can indeed reduce incidental correlations during training.\n\nSTRENGTHS:\n+ The proposed algorithm achieves interpretable representations to some degree, which is important for few-shot learning. \n+ Image background has been shown harmful for few-shot learning at both training and test time, and this paper tackles the background problem at training time in a novel way. The experiments on ImageNet-9 clearly verify that the method can remove incidental correlations.\n+ The proposed algorithm achieves SOTA performance on several few-shot learning benchmarks.\n\nWEAKNESSES:\n- Missing important ablation studies. (1) To verify the effectiveness of part-based learning, the authors need to show the performance of a vanilla ViT with the loss $L_{cls}$. (2) No ablation studies on the use of $L_{mix}$ and $L_{Q}$. Importantly, the authors do not show the baseline performance using part-based learning with $L_{cls}$ alone during pretraining (especially on few-shot learning benchmarks). Thus the usefulness of removing incidental correlations has not been verified.\n\n- There are confusing parts in the method description. For example, it is not clear if $z_p$ and $z_d$ are iteratively computed for every block of the network or just computed for the feature of the last block. The same confusion goes for distance maps and embedded patches. The architecture plot in Figure 2 does not answer this question (e.g., what's the input to the second block?). Some other confusions are listed in the questions part.\n\nminor: \n- A typo in line 188,  it should be $n_f \\times F^2 \\times C$.\n\nQUESTIONS:\n- Why can Gaussian noise make mixture-of-parts robust to common data distortions? it seems its use is to relax the target $L_{mix}$ to be softer instead.\n- What's the exact form of $M_f$ and $M_b$? Are they just binary masks? Why $l_2$-norm is used in $L_{mix}$? Since the target is a mask, it seems cosine distance is more appropriate.\n- Please clarify the confusing parts listed in the weaknesses part.\n\nLIMITATIONS:\nNo limitations are listed in the paper. I suggest the authors conduct experiments on cross-domain datasets like tieredImageNet->MSCOCO (in Meta-Dataset). Such experiments can show clearer advantages of part-based methods and the removal of incidental correlations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a strong part-based learning method for few-shot learning. Noticing the incidental correlations between part signals, the method proposed in the paper, named DP-ViT, learned to disentangle foreground and background parts. This is done by introducing a mixture-of-experts formulation of parts. To further encourage high-quality and diverse parts, the DP-ViT further employs a regularization for orthogonality.The method proposed by the authors achieves SOTA on multiple few-shot learning datasets, including MiniImagenet, TieredImageNet, and FC100. With the DP-ViT framework, the authors also illustrate the incidental correlation in imagery data and how DP-ViT improves the performance of models.\n\nSTRENGTHS:\nThe authors have made sound improvements to two regularizations on part-based models. For me, these ideas make sense and the authors have demonstrated interpretation of these modules in the experimental sections. Additionally, it is nice to see that the proposed solutions improve the performance of few-shot learning.\n\nWEAKNESSES:\nI don't directly work on this topic, and I think the authors have provided adequate analysis and explanation on the few-shot learning part. However, I think this paper could have a more general impact. Please check the questions below.\n\nQUESTIONS:\n1. It looks to me that DP-ViT is a quite general improvement to the ViT architecture. However, to support its full utility, I think an experiment on ImageNet is needed, where DP-ViT can outperform a vanilla ViT or a part-based ViT baseline.\n\n2. I personally would suggest adding an ablation study for the number of parts, K, especially what will happen when K is increased. It is interesting to see that K=64 parts is sufficient for a dataset.\n\nLIMITATIONS:\nI haven't found such a section from the authors. But I tend to think that this work does not have a negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the impact of incidental correlations on part-learning and proposes several regularization methods to mitigate this. The first regularization separates foreground and background to guide the part-based learning towards relevant input regions. To this end, the paper proposes a mixture of parts formulation to collect latent codes into masks and supervise them (weakly) with the pretrained/unsupervised foreground extractor. Then, the second regularization is employed to impose invariance to background variation. Additionally, sparsity and orthogonality regularization is used to refine part representations.\n\nSTRENGTHS:\n1. The paper tackles an important problem. Part-based methods are becoming more and more valuable nowadays due to increased interpretability\n2. The paper points out the failure mode of part-based learning\n3. The text is nicely written and easy to follow. Figures nicely highlight most relevant findings.\n\nWEAKNESSES:\n1. The choice of sparse and spectral norms is not well explained in my opinion. I understand the need for sparsity in the image space when training part-based models, but in the paper, sparsity is enforced on the matrix of part representations instead. What is the physical meaning of this? Also, why is the spectral norm chosen to optimize for orthogonality? It is not evident straight away why would it be a better alternative to just L2 / Frobenius.\n\n2. Why the background model is needed as a separate entity? Is modeling just the foreground (the background is then all the rest of the image naturally) not sufficient? When the background is modeled explicitly as a mixture of parts, then the part-model should learn to encode all background patches as well, right? This looks like an unnecessary complication.\n\n3. The whole framework consists of 3 regularizers, each with its weighting. This introduces more hyper-parameters to the model and increases resources needed to optimize the performance. With this, it is not clear if the (rather limited) performance improvement provided by the method is worth the increased demand for hyper-parameter optimization.\n\n4. Most of the methods in experimental evaluation are built on top of different backbones. It makes it harder to disentangle improvement from regularization and improvement from the backbone. It would make sense to use the model trained with L_{cls} only as a baseline for comparison in Table 1 and Table 2.\n\nUPD: Authors addressed most of my concerns in the rebuttal.\n\nQUESTIONS:\nI suggest authors address weaknesses for the rebuttal.\n\nLIMITATIONS:\nI suggest authors elaborate on hyper-parameter optimization for the proposed framework.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to produce a more robust and interpretable part-based learning method for image classification problems (though, probably extensible to other tasks). The main contributions include 1) the Disentanglement of foreground/background regions via a weakly supervised loss and 2) the Use of sparse and orthogonality constraints to discourage degenerate solutions 3) Finetuning using invariance constraint to ensure predictions are invariant for the learned background parts. The experiments show competitive results compared to recent methods and provide (arguably) more interpretable results.\n\nSTRENGTHS:\n[S1] Clearly motivated and technically sound: The paper addresses a well-known problem of spurious correlations in neural networks. While there are multiple types of spurious correlations, the paper only focuses on spurious correlations stemming from foreground and background regions. While that makes it more limited, it also provides clearer motivation, and the solution proposed can also address the specific problem better.  Indeed, the proposed solutions appear technically sound and well-designed to discourage correlations which are also well corroborated by the results.\n\n[S2] Adequate details/ablations: There are many parts to the proposed method, but the paper has done a reasonably good job with ablations and discussion of the effects of different components.\n\nWEAKNESSES:\n[W1] Some details are slightly unclear: The pretraining portion is mostly quite clear but I have some clarity issues with the finetuning (distillation) portion of the proposed method. I am not sure I fully grasp how Eqn 8 part 2, ensures the fg code captures details in the absence of background. Part 1 is clearer but still needs a bit more details (i.e., if the goal is to make predictions invariant of background, why distill from a teacher model that has access to both? I assume that the choice is made based on empirical results; e.g., fg only model might not converge well), but this needs to be discussed in more detail.\n\n[W2] Effects of error propagation from foreground/background \"weak supervision\": The existing solution requires access to a model that can segment/divide images into fg and bg during training. While I agree that such models are easy to use and are widely available, the paper is unclear about what the effects of the quality of the underlying model are on the proposed method. The quality of fg/bg themselves might have issues with correlations. This could perhaps be simulated by using corruption techniques to see how the quality of foreground/background segmentation affects proposed methods and also get an idea of how this method might fare with newer/future methods for fg/bg segmentation. L333 studies the availability of masks but not whether a noisy/incorrect mask might affect the model.\n\nQUESTIONS:\n- Why is the orthogonality constraint applied to the whole matrix rather than foreground/background separately? Is this done for practical reasons (i.e., it would be difficult to implement) or is there a motivation behind this choice?\n\n- Please see W1.\n\nLIMITATIONS:\nThe paper aims to reduce spurious correlations that can help mitigate some biases/lack of generalization in neural networks. I do not see an explicit potential for negative societal impact. The limitation section is missing and could include technical limitations/assumptions behind the work (Does not address intra-fg spurious correlations, requires access to masks, etc..), but is not a deal-breaker.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}