Mitigating the Effect of Incidental Correlations on Part-based Learning
Authors: Gaurav Bhatt
Deepayan Das
Leonid Sigal
Vineeth N. Balasubramanian
Venue: NeurIPS 2023 poster (2023)
Keywords: part-based learning
interpretability
few-shot learning
vision transformers

Abstract: Intelligent systems possess a crucial characteristic of breaking complicated problems into smaller reusable components or parts and adjusting to new tasks using these part representations. However, current part-learners encounter difficulties in dealing with incidental correlations resulting from the limited observations of objects that may appear only in specific arrangements or with specific backgrounds. These incidental correlations may have a detrimental impact on the generalization and interpretability of learned part representations. This study asserts that part-based representations could be more interpretable and generalize better with limited data, employing two innovative regularization methods. The first regularization separates foreground and background information's generative process via a unique mixture-of-parts formulation. Structural constraints are imposed on the parts using a weakly-supervised loss, guaranteeing that the mixture-of-parts for foreground and background entails soft, object-agnostic masks. The second regularization assumes the form of a distillation loss, ensuring the invariance of the learned parts to the incidental background correlations. Furthermore, we incorporate sparse and orthogonal constraints to facilitate learning high-quality part representations.
By reducing the impact of incidental background correlations on the learned parts, we exhibit state-of-the-art (SoTA) performance on few-shot learning tasks on benchmark datasets, including MiniImagenet, TieredImageNet, and FC100. We also demonstrate that the part-based representations acquired through our approach generalize better than existing techniques, even under domain shifts of the background and common data corruption on the ImageNet-9 dataset.

Mitigating the Effect of Incidental Correlations on Part-based Learning
Gaurav Bhatt=l3, Deepayan Das
Leonid Sigal 13 Vineeth N Balasubramanian' 2
The University of British Columbia, Indian Institute of Technology Hyderabad The Vector Institute, Canada
Abstract
Intelligent systems possess a crucial characteristic of breaking complicated prob- lems into smaller reusable components or parts and adjusting to new tasks using these part representations. However, current part-learners encounter difficulties in dealing with incidental correlations resulting from the limited observations of objects that may appear only in specific arrangements or with specific backgrounds. These incidental correlations may have detrimental impact the generalization and interpretability of learned part representations. This study asserts that part- based representations could be more interpretable and generalize better with limited data, employing two innovative regularization methods. The first regularization separates foreground and background information's 's generative process via unique mixture-of oottprrs formulation. Structural constraints are imposed on the parts using weakly-supervisee loss, guaranteeing that the mixture-of-parts for foreground and background entails soft, object- masks. The second regularization assumes form of distillation loss, ensuring the invariance of the learned parts to the incidental background correlations. Furthermore, we incorporate sparse and orthogonal constraints to facilitate learning high high-quality part representations. By reducing the impact of incidental background correlations on the learned parts, we exhibit state-of-the-art (SoTA) performance on few-shot learning tasks on bench- mark datasets, including Minilmagenet, TieredImageNet, and FC100. We also demonstrate that the part-based representations acquired through our approach gen- eralize better than existing techniques, even under domain shifts of the background and common data corruption on the ImageNet-9 9 dataset. The implementation is available on GitHub: https htaps:///gtthute com http://///aaaaaa11100/////..1...
Introduction
Many datasets demonstrate a structural similarity by exhibiting "parts" or factors that reflect the underlying properties of the data [15] [8] represent objects based on their various traits or parts, such as bird's morphology, color, and habitat characteristics. Part-based methods learn these explicit features from the data addition convolution and attention-based approaches (which only learn the internal representations) making them more expressive 7 2 52 54). Most existing part-base based methods focus on the unsupervised discovery of parts by modeling spatial configurations [14] 21
21
28
43 [4]. Humans efficient learners who
52 54 61]. while others use part localization supervision in terms of attribute vectors 225 50 56 bounding boxes 7 Part- based methods come with a learnable part dictionary that provides direct means of data abstraction and is effective in limited data scenarios, such as few-shot learning. Furthermore, the parts can be combined into hierarchical representations to form more significant components of the object
"First author; Email: gauravbhatt.co istr gmail.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Input image
(b) VIT with parts
(c) Proposed - DPViT
Figure 1: Impact of incidental correlations on the interpretability of part learners. We visualize the attention maps projected by the learned part dictionaries. Figure [(b)iilusttrate the ViT-S backhone featuring Icarnable part dictionary. However, it encounters difficulties in correctly identifying significant elements like the laptop. giving more attention to the background instead. In contrast, the proposed DPVIT method successfully detects the most crucial parts of the image even in the presence of incidental correlations.
Part-based learning methods offer advantages in terms of interpretability and generalization, particularly in safety-critical domains like healthcare, aviation and aerospace industry, transportation systems (e.g., railways, highways), emergency response, and disaster management. These fields often face challenges in collecting large training samples, making part-based learning methods valuable. Furthermore, the ability to explain decisions becomes crucial in these contexts.
description [50 55).
Various studies have indicated that correlations between image background and labels can introduce [3] [59] These correlations exist because specific background configurations create shortcuts for the model during training 32 53
biases during machine learning model training [29 32 45 46
While background information is crucial for decision-making. imbalanced background configurations can create unintended correlations, leading to undesirable outcomes. These correlations negatively impact the interpretability and generalization of part-based learners. For instance, let's consider scenario where a laptop, charger, and an iPod are on a table. In one case, let's examine the situation without any context or background information. Without background information, the model may struggle understand the purpose and significance of components or parts such as the laptop, charger, and iPod on a table. It could fail to differentiate between these objects or grasp their functionalities, resulting in a lack of recognition and understanding. Conversely, suppose the model is predominantly trained with examples of these items on tables. In that case, it may overly focus on the background elements, such as the table itself, disregarding the individual entities. Thus it becomes essential to handle incidental correlations of image background to achieve a balanced understanding. Existing part-based approaches fail to handle incidental correlations that arise due to specific background signals dominating the training data (analogous to Figure ((b),, thereby hampering their interpretability and generalization to limited data scenarios.
Having high-quality part representations is essential for achieving proficiency in part-based learning. In this context, quality pertains the sparsity and diversity of the learned parts. Sparsity ensures that only few parts are responsible for a given image, as images comprise a small subset of parts. Conversely, diversity in part representations prevents the parts from converging into a single representation and facilitates each part's learning of a unique data representation. Although incidental correlations can negatively affect learned parts' quality, the quality of part-based methods a significant challenge that all part learners face. While previous studies have addressed the issue of part quality [50] (2), their solutions do not assure the sparsity and diversity of the learned parts, thereby failing to guarantee high-quality parts.
To solve the aforementioned challenges, we introduce the Disentangled Part-based Vision Transformer (DPVIT), which is trained to be resilient to the incidental correlations of image backgrounds and ensures high-quality part representations. We propose a disentangled pre-training phase, which separates the generative process of the foreground and background information through a unique mixture-of-partss formulation. We impose structural constraints on the parts to ensure that the mixture- of of-parts for foreground and background includes soft, object-agnostic masks without requiring direct supervision of part localization. The parts are learned to be invariant to incidental correlations using a self-supervised distillation fine-tune phase. To address the issue of the quality of learned parts, we impose sparse and spectral penalties on the part matrix to guarantee the high quality of learned part representations. We include an assessment of the sparse and spectral norms of the part matrix as a quantitative indicator of the learned parts' quality. Finally, we evaluate the effectiveness of our method on benchmark sho-shot datasets, including Minilmagenet [35], TicredImageNet [40].
2

---

and FC100 [6]. To demonstrate the robustness of our proposed method to incidental correlations of backgrounds and common data corruptions, we use the benchmark ImageNet-9 dataset [53].
Our key contributions can be summarized as follows:
We propose regularization techniques to disentangle the generative process of foreground and background information through a mixture-of of of-pprts formulation. Additionally, we employ self- supervised distillation regularization to ensure that the learned parts remain invariant to incidental correlations of the image background.
We ensure the high quality of learned parts by employing both sparsity and spectral orthogonal constraints over the part matrix. These constraints prevent the parts from degenerating and encourage a diverse range of part representations.
Apart from our evaluation of standard few-shot benchmark datasets, we also analyze the impact of incidental correlations of background and typical data distortions by utilizing the benchmark ImageNet-9 dataset [53]]
2 Related Work
Part-based learning. The advantages of learning part- based representations have been extensively researched in image recognition tasks [15] 18] 31
38 43
49
52_
55
56). Earlier methods attempted
to learn parts by defining stochastic generative process 19
43 Part-based methods have been
broadly classified into unsupervised and supervised categories. Unsupervised methods concentrate on learning the spatial relationship between parts by using part dictionaries without the supervision of part localization [5] 20, [1 [28 49 52 54 [6]. In contrast, supervised part- based methods rely on the supervision of part localization through attribute vectors [16] 27 41 [0] or part bounding
box information 77. In the literature, parts are also referred to as concepts when supervision about
part localization is involved 7 41).
Discovering parts an unsupervised way a more challenging scenario that is applicable to most practical problems. Part dictionaries help data abstraction and are responsible for learning implicit and explicit data representations. For example, (28 clustered DCNN features using part-based dictionaries, while (26) introduced generative fictionary-based model to learn parts from data. Similarly. (20) uses past- dictionaries and an attention network to understand part representations. The ConstellationNet (54) and CORL [21] are some of the current methods from the constellation family [14 [I], and use dictionary-basse based part-pre protottoss for unsupervised discovery of parts from only assume the part structure without the data. Our approach also belongs to this category,
requiring any supervision of part localization.
Incidental correlations of image background. Image backgrounds have been shown to affect a machine learning model's predictions, and at times the models learn by utilizing the incidental correlations between an image background and the class labels 4
32 42
45
46
53
58
59 To
mitigate this issue, background researchers have used augmentation methods by altering background
signals and using these samples during the training 332 on the effect of image background on in-domain classification. They introduce several variants of background augmentations to reduce a model's reliance on the background. Similarly, [58) uses saliency maps of the image foreground to generate augmented samples to reduce the effect of the image background. Recently, [32) showed the effectiveness of background augmentation techniques for minimizing the effect of incidental correlations on few-shot learning.
53 58) (53) performed an empirical study
Unlike these methods, our approach does not depend on background augmentations but instead learns the process of generating foreground and background parts that are disentangled. Furthermore, our proposed approach is not sensitive to the quality of foreground extraction and can operate with limited supervision of weak foreground masks.
Few-shot learning and Vision Transformers. In recent years, few-shot learning (FSL) has become the standard approach evaluate machine learning models' generalization ability on limited data ( Vision transformers (VIT) [1] have demonstrated superior performance
2
13, 24
36
44
47
54]
on FSL tasks 10,
51, and self supervised distillation has emerged as popular 51I. A recent trend involves a two-stage procedure training strategy for these models 8 17 22 30
22
23
48
where models are pretrained via self-supervision before fine-tuning via supervised learning [8] 17
22 30 60]. For example, [17] leverages self-super training with iBOT [60] as pretext task, followed by inner loop token importance reweighting for supervised fine-tuning. HCTTransformer
22 uses attribute surrogates learning and spectral tokens pooling for pre- trainini vision transformers
3

---

and performs fine-tuning using cascaded Student-teacher distillation to improve data efficiency hierarchically. SMKD [30] uses iBOT pre-training and masked image modeling during fine-tuning to distill knowledge from the masked image regions.
Our approach employs two-stage self-supervised training strategy of vision transformers akin to [30 60]. However, unlike existing methods that focus on generalization in few-shot learning, our primary objective learn part representations that invariant to incidental correlations of the image background. Our training procedure is designed to facilitate learning disentangled and invariant part representations, which is impossible through existing two-stage self-supervised pipelines alone.
3 Problem Formulation and Preliminaries
In few-shot classification, the aim is to take a model trained on a dataset of samples from seen classes Daeen with abundant annotated data, and transfer/adopt this model to classify a set of samples from disjoint set of unseen/novel classes Drovel with limited labeled data. Formally, let Daeen {(x, y) where x X corresponds to an image and y Yseen corresponds seen classes. We also assume that during training, we have limited supervision of class-agnostic gregrooud-baccgrouee
the label among the set of
mask (M) for regularization during training, which can be easily obtained by any weak foreground extractor as preprocessing step (following [53] Please note that no mask information is required for Dmnoel at inference.
We follow the work of (9. 660 on self-supervised training of VITs to design our pretrain phase. During training, we apply random data augmentations to generate multiple views of the given image x" E Daeenn These views are then fed into both the teacher and student networks. Our student network, with parameters O., includes VIT backbone encoder and projection head ø. that outputs probability distribution over K classes. The ViT backbone generates [cls] token, which then passed through the projection head. The teacher network, with parameters 6g is updated using Exponentially Moving Average (EMA) and serves to distill its knowledge to the student by minimizing the cross-entropy loss over the categorical distributions produced by their respective projection heads.
Cots = (xxx)) Loo LFo
For inference, we use the standard M-way, --shot classification by forming tasks (7), each compris- ing of support set (S) and query set (Q), constructed from Duovel Specifically, support set consists of M x N images; N random images from each of M classes randomly chosen from Wnove! The query set consists of disjoint set of images, to be classified, from the same M classes. Following the setup of 477, we form the class prototypes (cm)) using samples from S. The class prototypes and learned feature extractor (Fa) are used to infer the class label á for an unseen sample x" E using a distance metric d.
1 = arg max d(FFoxx)), cm): Cm = N
1
Σ Fo(x). (x.8m)ES
(2)
m
4 Proposed Methodology
Given an input sample x E Hx and patch size f. we extract flattened 2D patches Xy E R--x(FF-C), where N is the number of patches generated and (F, F) is the resolution of each image patch. Similar to a standard ViT architecture [TI. we prepend a learnable [class) token and positional embeddings to retain the positional information. The flattened patches are passed to multi-head self attention layers and MLP blocks to generate a feature vector 2p = MSA(x,).
Next, we define the parts as past- dictionaries P = (P& E FRFCC Jk=1- where P& denotes the part-vector for the part indexed as k. The part-matrix (P) is initialized randomly and is considered trainable parameter of the architecture. Note that the dimension of each part-vector is equal to the dimension of flattened 2D patches, which is F22.. C. For each part Ps. we compute a distance map D E I where each element in the distance map is computed by taking dot-product between the part P& and all the N patches: D*
Xx Pk-
Using the distance maps D E R. N we introduce a multi-head cross-attention mechanism and compute the feature vector: 2 MCA(FF where Fg. is MLP layer which upsamples D K F2. F2.C. C. The cross-attention layer shares a similar design to self-attention layers; the only

---

prod
Parts (P)
Ditunce Mays
MLP
Muni
Norm
sileetion
Murlshwed 0100 ottentinn
Null-bead Adention
Alth Duossion nelie
Altertion
Imput Bample
NDA
Fo
Norn
MSA
Transforer Encoder
Desence
Enitedded
Figure 2: Overview of proposed architecture DPVIT. We employ learnable part dictionary generate formulation incorporating foreground and background information. The spatial distance maps, computed by the part dictionary, are utilized to determine the mixture of latent codes for foreground and background. Our transformer encoder comprises multi-heas heat- self-attention (MSA) and multi-head rross-attention (MCA) layers. The MSA layer takes embedded patches as input, while MCA layers utilize distance maps as input.
difference is the dimensions of imput distance maps. The atosntattnnioon helps contextualize informa- tion across part-dictionary and the spatial image regions and provides complementary properties to MS A layers. (Please refer to Appendix for experiments on complementary properties of MSA and MCA).
Finally, add the output feature vectors of MSA and MCA to form the feature extractor Fg defined in Eqn[]
Fø
E @ za]
Disentanglement of foreground-backgroune space using mixture-of-parts. We start by dividing
the parts-matrix P E R into disjoint sets: foreground set Pp R and background set PB E IR xx such that K = n + Tib-
Next, construct latent variables to aggregate the foreground and background information using mixture-of-partt. formulation over the computed distance maps D:
LF
Σ D kEnF
T5
Σ D 5
where, ag and 3g are the learnable weights given to the Ath part-vector in the corresponding mixture, whereas Sy and ó are Gaussian noises sampled from (o 1). The Gaussian noise is added to the latent codes to ensure that mixture-of-pp -frttatt are robust to common data distortions. Please note that the purpose of Gaussian noise is not to induce variability in the latent codes, as foreground information for given image is deterministic.
Finally, our disentanglement regularization takes the form of an alignment loss between the latent codes and the class-agnostic foreground backgrooun masks:
Lmiz [Z(LF) -Myll + [[[(LB) Mallz
(5)
where, I(L) is the bilinear interpolation of given latent code to the same size as M.
During the architectural design phase, we employ distinct components (P) for each encoder block. Consequently, Zp and 2 are calculated an itcrative manner and subsequently transmitted to the subsequent encoder block. Regarding the computation of Lmiz, utilize the distance maps D from the concluding encoder block. While it's feasible to calculate Lyni iteratively for each block and then aggregate them for final Lyni computation, our observations indicate that this approach amplifies computational expenses and results in performance deterioration. As result, we opt to compute Lmiz using the ultimate encoder block.
Learning high-quality part representations. A problem with minimizing mixture objective defined in Eqn55is that it may cause the degeneration of parts, thereby making the part representations less diverse. One solution enforce orthogonality on the matrix pmxn by minimizing |PP'P-11 similar to [50]. However, the solution will result in a biased estimate < that is, the number of

---

parts (K) is always less than the dimensionality of parts (F² C). In our experiments, we observed that increasing K beyond certain threshold degrades the performance as computational complexity increases, and is consistent with the findings in [54] (Please refer to our Appendix section for experiments on the different values of K). To minimize the degeneration of parts, we design our quality assurance regularization by minimizing the spectral norm of P I, and by adding I.1 sparse penalty on the part-matrix P. The spectral norm of p has been shown to work with over-complete (m < n) and under-complete matrices (m 2 n) 3
Lo(As, PIIl + do (o PF" -) I) + +o o PB Pn"-1)] -n]
(6)
where I is the identity matrix, A, and X are the regularization coefficients for sparsity and orthogo- nality constraints. o (P) the spectral norm of the matrix P which computed using the scalable power iterative method described in [3]
Disentangled Pretraining Objective. We pretrain DPVIT using the following loss function:
PT dotehots + Amizhmir + Lo Xo)
where Acta: Amita Ag and Xo are the weights given to each loss term and are tuned on the validation set.
Student
14
4.1
Invariant fine-tuning
In the pretrain phase, our approach learns part representations that are disentangled and diverse, but it does not achieve in- variance to the incidental correlations of image background. During the fine-tuning stage, we utilize the learned foreground latent code to extract the relevant foreground information from a given image æ: If x + I(LF), where O denotes the Hadamard product. The teacher network receives the original image, while the student network receives the foreground-only image. By distilling knowledge between the [class] tokens and fore- ground latent codes LF of the student and teacher networks, achieve invariance to the incidental correlations of image background.
EMA Update
Teacher
2
Figure 3: Invariant fine-tuning DPVIT via distillation framework.
ciny Lee (F
The two proposed invariant regularizations serve distinct purposes: L encourages the model to classify images independently of the background, while Line ensures that the latent foreground code captures relevant foreground information even when the background is absent, making the learned parts invariant to the incidental correlations.
Invariant Fine-tuning Objective. Finally, our fine-tu fine-tuning objective is given as
LFT = Actafets + X Gine
inv + Ap
CP
where Aeta, Ais and sine are the weights given to each loss term and are tuned the validation set after pretraining.
Experiments
We evaluate the proposed approach on four datasets: MinilmageNet (35], TieredImageNet [4)] FC100 (37), and ImageNet-9 [53]. The MinilmageNet, migeed and FC100 are generally used as benchmark datasets for fho-shot learning. For MinilmageNet, we use the data split proposed in [39]. where the data samples split into 64, 16, and 20 for training, validation, and testing, respectively. The TieredImageNet 40) contains 608 classes divided into 351, 97, and 160 for meta- training, vata-validation, and meta-testing, On the other hand, FC100 [37] is smaller resolution dataset (32 x 32) that contains 100 classes with class split as 60, 20, and 20.
To investigate the impact of background signals and data corruption on classifier performance, researchers introduced ImageNet-9 (IN-9L) (53]. IN-9L a subset of ImageNet comprising nine coarse-grained classes: dog, bird, vehicle, reptile, carnivore, insect, instrument, primate, and fish. Within these super-classes, there are 370 fine-grained classes, with a training set of 183,006 samples. The authors of (53) created different test splits by modifying background signals, resulting in 4050

---

ViT
ViT + Cia
Input image (X)
(MSA(())
EnI(MCA(D)
(MS(()())))
(MC(MCA(D))
Figure 4: Visualizing MSA and MCA layers. The joint representation is obtained by averaging all attention heads
En) We study the effect of Liir on the interpretability of part based learners.
I(Le)
I(D")
I(D*)
I(D")
Z(D")
La
Value
50 100
150
Number Epochs
(a) Visualizing heatmaps of part projections using ViT+
(c) Computing ||]|
I(Lp)
I(D")
z(D")
Z(D
Eda + Cost + Cq Cai feus
Valieaf
15
Zu
10
150
Number Epochs
(b) Visualizing heatmaps of part projections using ViT+Cmiis La
(d) Computing |PPP"
Figure 5: Visualizing the quality of learned parts for input image from Figure In Figure 5(a) and 5bb
show the foreground mixture I(Irr) and four foreground parts selected randomly from ny-- Meanwhile, Figure and 5(d) show the sparse and orthogonal norms as metrics evaluating the quality of part matrix P.
5(c)
samples per split to evaluate various classifiers. We use three of these test splits for our evaluation:
Original (with no changes to the background), M-SAME (altering the background from the same class), and M-RAND (altering the background from different class). Additionally, (53] introduced a metric called BG-GAP to assess the tendency of classifiers to rely on background signal, which measured as the difference in performance between M-SAME and M-RAND.
We use VIT-S backbone for all our experiments and follow the same pipeline iBOT [60] for pre-training, keeping most hyperparameters unchanged. We use batch size of 480 with a learning rate of 0.0005 decayed alongside cosine schedule and pre-train DPVIT for 500 epochs for all four datasets. Fine-tuning is carried out on the same train data for 50 epochs utilizing the class labels (similar to (22)) The DPVIT architecture has K=64 and n 4- for all of our experiments. The pre- tre-training and fine-tun fine-tuning are carried out on 4 A40 GPUs. More details related to the architectures and optimization are provided in Appendix.
We start our experimental study by examining how incidental correlation affects part-learners interpretability and quality learned parts. This study is conducted on the MinilmageNeet dataset. We then present experimental results on few-shot learning on the MinilmageNet, TieredImageNet, and FC100 datasets. Lastly, we provide a quantitative analysis of the influence of incidental background correlations on the various test splits in the ImageNet-9 dataset, followed by ablation studies and discussion.
5.1 How do incidental correlations affect the interpretability part learners?
To examine how incidental correlations impact various components, we solely employ the Cets loss to train DPVIT on the MinilmageNee dataset. (Note that Cots is equivalent to ViT with parts). We then present visualization the attention layers (MSA and MC.A) Figure The MSA layers effectively recognize relevant in-context information of some objects, but the CSA layers fail to pinpoint foreground details. This is because incidental correlations in the background dominate the CSA layers. However, incorporating the Lmir regularization results in improved localization by MC'A layers, which are no longer influenced by incidental correlations. An important point to highlight is that the design of MSA A layers causes them to focus on objects specific to particular class. As result, their effectiveness is reduced when multiple objects are present (as seen in Figure
A where MSA misses objects the left side). Conversely, CSA layers learn class-agnostic features
7

---

MiniImageNet
TieredImageNet
FC100
Method
Backhone
I-shot
5-shot
1-sbot
5-shot
1-shot
5-shot
ProtoNets (2017)
+7
ResNet12
60.39 16
78.53
65.65
83,40
52.50 20.50
DeepEMD v2 (2021)
57
ResNet12
68. 20
84. 84.13+0.5a
71. 26.167
86.03320.5
63.2220.71 26.47
COSOC (2021)
ResNet12
69.28 4
85.16
23.57
87.57 20.10
MixtFSL (2021) [
ResNett2
63.98
82.04.0.0 19
1..97
56. 56.1620.67
Match-feat (2022)
Non-Part
ResNet12
68. 68.32.0...
82.71
71. 71.222
85, 25.43
Labci-Halluc (2022) [4]
ResNet11
67.04.0.10
85.87
71.977
86, 86.80
47 47.37
67. 67.9220.70
(202)) 44
ResNet12
67. 67.47
86.08
71.663+0.99
87.07 +0.55
49.0220.70
68. 68.68
SUN (2022) [0]
VIT
67. 0..44
83, 0..25
72.99 +0.50
86.74 2...s
FewTure (2022)
25
Swin-Tiny
72.40.0.7
86.38 20.49
76.32
89.9640.55
63.81 27.68
HCTrrasffrmer (2022)
3x VIT-S
74.74
29.11
79.67
91.72
48 27
66.42 20001
SMKD (2022) 30
VIT:S
24.28
88.82g0..0
78.83g0.20
91.0220.11
50.38
68. 20.37
ConstNet (202 54
ResNet12
20
79.95
70.15
86, 16.10
43 20.80
59. 20.0
TPMN (2021)
52
ResNet12
67.6420.3
67.
83. 83.44
72.2440.70
86.5540..5
46 20.77
63. 23.26
CORL (2023)
ResNet12
65. 65.74g0.03
83.03
73.82
86.76
44.82 20.7
61.31g0.4
VTT-with- (1.ot))
VITSS
72.15-0.20
87.61 20.15
28.03
89.08
48 20.13
67. 27.75
Ours DPVIT
VIT-S
73.81=0.45
89,85
79.32+0.48
91.92:0.40
50.75=0.28
68.80
Table 1; Evaluating the performance of our proposed method on three benchmark datasets for few-shot learning- MinilmageNet, Tiered- Nett and FC100. The top blocks show the non-part methods while the bottom block shows the part-base bast-based methods. The best results are bold, and is the 95% confidence interval in 600 episodes.
Method
IN-91.1
Original
M-SAME
M-RAND
BG-GAP
Mothod
1-sbot 1
S-sbot 1
||P|/ L.
||PP"
1
ReoNas50 [3]
wRY-50=2 55
96.6
95.2
96.1
97.2
89:9
40.5
78.00
75.6
12.6
14.3
SMKD Bu
60.93
80.38
CanstNer
90.6
02.7
69.2
Cote
61.24
81.12
8:41
25.82
VITSS
1T]
$2.5
$4.9
50.3
21.9
Lisa Lins
62.81
83.25
8.73
24.61
88.5
51.5
Eui Emie Co
62.15
82.95
0.35
0.56
WT-wiih-parts
Ours DPVIT
95.1
96M
97.2
98.5
934
01.5
81.7
87.5
988
59
Table 3: Ablation of different loss terms during pre-
Table 2: Performance evaluation on domain shift of
training of DP-VIT. We show the effect of cach loss term the MinilmageNet dataset.
varying background and common data corruptions
ImageNet-9, Evaluation metric is Accuracy %.
consistently throughout the training set, enabling them to perform well in the presence of multiple objects. For further investigation of the complementary properties of MSA and MCA, please see the Appendix. Additionally, the Appendix includes visualizations of individual attention heads.
5.2 Studying the quality of learned part representations
As previously stated, the interpretability of learned parts is directly influenced by the sparsity and diversity of the part matrix P. This is achieved by examining the impact of Lg during pretraining. We present visualizations of the learned foreground parts in Figure 5(a) and (5() for the input image x from Figure 4 While both setups successfully learn the foreground mixture, I(LF), without Lo. the parts degenerate into homogeneous solution (Figure 5(a)). lacking sparsity and diversity. With the inclusion of Lq. however, the learned parts become sparse and diverse (Figure 5()).
We use the L-1 and orthogonal norms of the part matrix to assess the sparsity and diversity of the parts. As illustrated in Figure and Figure (5(a), the addition of La maintains bounded norms, inducing sparsity and diversity among the parts. The results also highlight higher norm values by employing Lats + Lymix losses, which show the degeneracy caused by the introduction of mixture loss. Moreover, the higher sparse and orthogonal norms for Lats demonstrate that part-based methods generally do not maintain the quality of parts.
5.3 Few-shot Learning
We compare DPVIT with recent part-based methods: ConstNet [54] TPMN [52], and CORL 221]: ViT-based methods: SUN [10), FewTure (23), HCTransformer 222 and SMKD [30 (for SMKD we compare with their prototype- few-shot evaluation on all the datasets); and recent ResNet based few-shot methods: Match-feat (2) Label-halluc (24), and FeLMi 44
As shown in Table I the proposed method outperforms existing part-based methods by clear margins. Moreover, DPVIT achieves competitive performance compared to current ViT-based methods, espe- cially on the FC100 dataset with low-resolution images. The FSI. results show that the self-supervised ViT methods give edge over the existing ResNet-based backbones.
8

---

Setting
Org"
M-S
M-R1
[
[PP
-1
Mothod
Backbone
MinilmageNet >> CUB
98.1
91.5
82.7
14
2.5
ProtoNet
ResNet-18
67.1
Lis. Dire Cu
Lots dal
98.2
98.5
93.4
87.2
87.5
1.4
0.5
2.3
MixtFSL
ResNet-18
68.7
Table 4: Ablation of different loss terms during fine-
ConstNet
S4
ResNet-12
68.8
tuning of DP-ViT. We show the effect of each loss term on the ImageNet-9 test splits.
SUN [0]
VIT-S
72.1
Proposed-DPVVT
VIT-S
77.1
Table 5: Cross-domain evaluation. We evaluate mod- els trained on MinilmageNet on the CUB dataset.
Setting
I-shor 1
5-shot 1
|P]|a
|PP"
T//1
0%
10%
50%
100%
1-s
5-s
5-s
1-s
5-s
1,-0.11
A,1
73.9
89.9
0.7
0.9
A.50.5 0,5
0.5
DPVI 72.3
88.1
72.9
88.6
73.7
89.8
73.8
89.8
0, 2,=20
72.4
88.2
0.1
0.2
Table 7: Results on limited supervision of foreground masks on the MinilmageNet dataset. Here, we use 1-s for I-shot and 5-s for 5-shot accuracy.
Table Evaluating the tradeoff between interpretabil- ity and few-shot generalization on MinilmageNet.
5,4 Studying the impact of incidental correlations of background on ImageNet-99
The impact of incidental background correlation on the ImageNet-9 dataset is examined, and the corresponding results are presented in Table g DPVIT achieves the highest performance across IN-9I., Original, M-SAME, and M-RAND among the different test splits. Remarkably, our proposed DPVIT method significantly reduces the BG-GAP value to 5.9, indicating its resilience the effects of incidental correlation caused by varying backgrounds. In comparison to non-part methods such as ResNet-50 and WRN-50 x 2 (as shown Table 8). ConstNet [54), which a part-based method, suffers more from the negative impact of incidental correlation, underscoring the detrimental effect on the generalization of part-learners. To evaluate ConstNet, we utilize the provided source code by the authors to train their model on the IN-9L training data.
6 Ablation study and Discussion
In Section 5.1 and 5.2 we investigate how the inclusion of Cyii and Lo impacts the interpretabillity of DPVIT during the pretraining phase. We will now conduct ablation study to assess the impact of both these loss components during the pre-training phase. As indicated in Table 3 introducing Lumi results in enhanced 1-shot and 5-shot performance compared the baseline model represented by Lets- It's worth noting that the SMKD [30) model performs worse than the pirt-ViT baseline (Cols). Upon incorporating Co. the few-shot performance experiences a slight decrease, but the norms remain constrained, indicating improved quality of part representations.
Additionally, in this section, we examine the advantages of incorporating Live and Liav during the fine-tuning process on the ImageNet-9 dataset. This ablation study focuses on the significance of invariance terms in handling Buckground-relater incidental correlations. The results presented in Table 4 demonstrate that the introduction of Livs" enhances generalization on M-S (M-SAME) and M-R (M-RAND), thereby illustrating the benefits of inducing invariance through Cas across varying backgrounds. Moreover, the inclusion of Line yields lower BG-GAP value of 5.9, in contrast to 8.8 obtained without Cit.". Notably, the introduction of C has no impact on the quality of parts, as evidenced by the final norm values of P. By employing Line in conjunction with other losses, the norms remain bounded, thereby preserving the interpretability of parts. We also observe a minimal decrease in classification performance upon introducing Live highlighting the trade-off' between generalization and interpretability that will be explored in the following section.
6.1 Cross-domain evaluation
In order assess the advantages of eliminating incidental correlations for transfer learning, we perform experiments on cross- cross-domain dataset, specifically MinilmageNett to CUB. In this setup, we evaluate the performance of models trained on MinilmageNet when applied to the CUB dataset. As demonstrated in Table [5, the DPVIT model stands out by achieving the highest classification performance. These experiments highlight the clear benefits of employing part-based methods and eliminating incidental correlations.
6.2 Tradeoff between interpretability and generalization
We observe a tradeoff between the interpretability of parts and their generalization. To explore this, we conduct an ablation study during pre-training the MinilmageNet dataset. As indicated in Table

---

(a) Foreground patches (h) Background patches
Figure 6: Robustness of DPVIT towards weak foreground extractor MinilmageNet.
Figure 7: Disentanglement foreground/background
patches extracted by DPViT.
assigning higher values to Åa and X, places greater emphasis on Lo. leading to lower norm values and consequently improving the quality of parts. However, this also results in slight reduction in few-shot accuracy. After careful analysis, we determine that an optimal value of 0.5 for both X and Ao strikes balance, maintaining the quality of parts while preserving few-shot generalization.
6.3 Partial observability of foreground mask
Our training procedure employs foreground masks acquired through a foreground extractor, similar to the one described in [53]. To study the dependence of DPVIT on the availability of foreground masks, we examine the weak/limited supervision scenario for foreground masks, where only a small subset of samples possesses the corresponding masks. As depicted in Table 7 we observe that DPVIT achieves comparable performance even when only 10% of the training samples have mask information. The performance difference is less than 1.5% for 5-shot and less than 0.8% for 1-shot performance. Furthermore, Figure [ presents visualizations of image patches surrounding a random foreground and a background part. We also find that in the setup with a 0% foreground mask, equivalent Amiz no disentanglement is observed in the extracted patches. (More visualizations can be found in the Appendix section).
6.4 Working with weak foreground extractor
The features of DPVIT (Fe) depend entirely on the input sample x in order to learn part representa- tions, without explicitly incorporating the mask information. The mask serves as a weak signal to regularize our training objective and separate the foreground parts from the background. Additionally, introducing Gaussian noise in the latent codes enhances DPVIT's ability handle misalignment issues with the foreground masks. Consequently, the learned features remain unaffected by mistakes made by the existing foreground extractor. Moreover, we find that the mixture-of-part representations can accurately determine the foreground location even when the mask information is missing or incorrect (as illustrated in Figure 6).
6.5 Limitations of DPVIT
A constraint within our framework involves relying on pre-existing foreground extractor. In certain scenarios, such as the classification of tissue lesions for microbiology disease diagnosis, obtaining an existing foreground extractor might not be feasible. Similarly, DPVIT focuses on learning components that are connected to the data, yet doesn't I encompass the connections between these components, like their arrangement and hierarchical combination. Introducing compositional relationships among these components could enhance comprehensibility and facilitate the creation of a part-based model capable of learning relationships among the parts.
7 Conclusion
In this work, we study the impact of incidental correlations of image backgrounds on the interpretabil- ity and generalization capabilities of part learners. We introduce DPViT, method that effectively learns disentangled part representations through mixture-of of parts approach. Furthermore, we enhance the quality of part representations by incorporating sparse and orthogonal regularization constraints. Through comprehensive experiments, we demonstrate that DPVIT achieves competitive performance comparable to sftee-ofthh-an methods, all while preserving both implicit and explicit interpretability.
of-parts
10

---

Acknowledgments and Disclosure of Funding
The computation resources used in preparing this research were provided by Digital Research Alliance of Canada, and The Vector Institute, Canada.
References
[1] Arman Afrasiyabi, lean-François Lalonde, and Christian Gagné. Mixture-based feature space learning for few-shot image classification. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9041-9051, 2021.
[2] Arman Afrasiyabi, Hugo Larochelle, Yean-François Lalonde, and Christian Gagné. Matching feature sets for few-shot image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9014-9024, 2022.
[3] Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
[4] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[5] Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Bastien Pasdeloup. Stéphane Pateux, and Vincent Gripon. Gasy-eeseembl segmented-shot-y-saappe learning: Statc-of-the-att few-shot classification with simple components. Journal Imaging, 8(7):179, 2022.
[6] Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. International Conference Learning Representations (ICLR), 2018.
[7] Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. In International Conference on Learning Representations (ICLR), 2021.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[10] Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo. Self-promoted supervision for few-shot transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX, pages 329-347 Springer, 2022.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR), 2021.
[12] Jianqing Fan, Fang Han, and Han Liu. Challenges of big data analysis. National science review, 1(2):293-314, 2014.
[13] Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang. Melr: Meta-learning via modeling episode-level relationships for few-shot learning. International Conference on Learning Representations (ICIR), 2021.
[14] Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IFEE transactions pattern analysis and machine intelligence, 28 2006.
11

---

[15] Sanja Fidler and Ales Leonardis. Towards scalable representations of object categories: Learning a hierarchy of parts. In IEEE Conference on Computer Vision and Patter Recognition (CVPR), pages 1-8, 2007.
[16] Sanja Fidler and Ales Leonardis. Towards scalable representations of object categories: Learning a hierarchy of parts. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1-8, 2007.
[17] Hanan Gani, Muzammal Naseer, and Mohammad Yaqub. How train vision transformer small-scale datasets? arXiv preprint arXiv:2210.07244 2022.
[18) Qingzhe Gao, Bin Wang, Libin Liu, and Baoquan Chen. Unsupervised co-part segmentation through assembly. In International Conference Machine Learning, pages 3576-3586. PMLR, 2021.
[19] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Advances in Neural Information Processing Systems (NeurlPS), 29, 2016.
[20] Ju He, Adam Kortylewski, and Alan Yuille. Compas: Representation learning with composi- tional part sharing for few-shot classification. arXiv preprint wrXiv:2101.11876 2021.
[21] Ju He, Adam Kortylewski, and Alan Yuille. Corl: Compositional representation learning for few-shot classification. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 3890-3899, 2023.
[22] Yangji He, Weihan Liang, Dongyang Zhao, Hong-Yt Yu Zhou, Weifeng Ge, Yizhou Yu, and Wenqiang Zhang. Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9119-9129, 2022.
[23] Markus Hiller, Rongkai Ma, Mehrtash Harandi, and Tom Drummond. Rethinking generalization in few-shot classification. arXiv preprint (rXiv:2206007266 2022.
[24] Yiren Jian and Lorenzo Torresani. Label hallucination for few-shot classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7005-7014, 2022.
[25] Yuhe Jin, Weiwei Sun, Jan Hosang, Eduard Trulls, and Kwang Moo Yi. Tusk: Task agnostti unsupervised keypoints. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[26] Adam Kortylewski, Qing Liu, Huiyu Wang, Zhishuai Zhang, and Alan Yuille. Combining compositional models and deep networks for robust object classification under occlusion. In IEEF/CVE Winter Conference on Applications of Computer Vision ( (WACV), pages 1333-1341, 2020.
[27] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[28) Renjie Liao, Alexander Schwing, Richard S Zemel, and Raquel Urtasun. Learning deep parsimonious representations. In International Conference Neural Information Processing Systems (NeurIPS), pages 5083-5091, 2016.
[29] Chia-Ching Lin, Hsin-Li Chu, Yu-Chiang Frank Wang, and Chin-Laung Lei. Joint feature disentanglement and hallucination for few-shot image classification. IEEE Transactions on Image Processing, 30:9245-9255 2021.
[30] Han Lin, Guangxing Han, Jiawei Ma, Shiyuan Huang, Xudong Lin, and Shih-Fu Chang. Supervised masked knowledge distillation for few-shot transformers. arXiv preprint arXiv:2303. /5466, 2023.
[31] Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjorn Ommer. Unsupervised part-based disentangling of object shape and appearance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10955-10964, 2019.
12

---

[32] Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang, Lingxi Xie, Zenglin Xu, and Qi Tian. Rectifying the shorteut learning of background for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 34: 13773333-,, 2021.
[33] Xu Luo, Jing Xu, and Zenglin Xu. Channel importance matters in few-shot image classification. In International conference on machine learning, pages 14542-14559 PMLR, 2022.
[34] Chong Ma, Lin Zhao, Yuzhong Chen, David Weizhong Liu, Xi Jiang. Tuo Zhang, Xintao Hu, Dinggang Shen, Dajiang Zhu, and Tianming Liu. Rectify vit shortcut learning by visual saliency. arXiv preprini arXiv:2206.08560 2022.
[35] Oriol metas, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances Neural Information Processing Systems (NeurIPS), 29:3630- 3638, 2016.
[36] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. International Conference on Learning Representations (ICLR), 2018.
[37] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
[38] Patrick Ott and Mark Everingham. Shared parts for deformable part-based models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1513-1520, 2011.
[39] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. International Conference Leurning Representations (ICLR), 2017.
[40] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenen- baum, Hugo Larochelle, and Richard S Zemel. Meta-learning for superviee few-shot classification. International Conference on Learning Representations (ICLR), 2018.
[41] Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. Attention-based interpretability with concept transformers. In International Conference on Learning Representationss (ICLR), 2022.
[42] Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint 18Xiv:180..03.000 2018.
[43] David A Ross and Richard S Zemel. Learning parts-based representations of data. Journal of Machine Learning Research, 7(11), 2006.
[44] Aniket Roy, Anshul Shah, Ketul Shah, Prithviraj Dhar, Anoop Cherian, and Rama Chellappa. Felmi: Few shot learning with hard mixup. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[45] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. Unsupervised discovery of parts, structure, and dynamics, 2020.
[46] Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk- quantifying and controlling the effects of context in classification and segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8218-8222 8226, 2019.
[47] Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems (NeurlPS), 2017.
[48] Mingze Sun, Weizhi Ma, and Yang Liu. Global and local feature interaction with vision transformer for few-shot image classification. In Proceedings of the 31 ACM International Conference on Information & Knowledge Management, pages 4530-4534 2022.
[49] Yihong Sun, Adam Kortylewski, and Alan Yuille. Weakly-supervised amodal instance segmen- tation with compositional priors. arXiv preprint arXiv:2010.13175,
13

---

[50] Pavel Tokmakov, Yu-Xiong Wang. and Martial Hebert. Learning compositional representations for few-shot recognition. In /EEE/CVF International Conference on Computer Vision (ICCV), pages 6372-6381, 2019. 2019.
[51] Xixi Wang, Xiao Wang, Bo Jiang, and Bin Luo. Few-shot learning meets transformer: Unified qupro-s transformers for few-shot classification. arXiv preprint arXiv:2208. 12398, 2022.
[52] Jiamin Wu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. Task-a part mining network for few-shot learning. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 8433-8442, 2021.
[53] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry, Noise or signal: The role of image backgrounds object recognition. International Conference on Learning Representations (ICLR), 2020.
[54] Weijian Xu, Yifan xu, Huaijin Wang, and Zhuowen Tu. Attentional constellation nets for few-shot learning. In International Conference on Learning Representations, 2021.
[55] Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Unsupervised discovery of parts, structure, and dynamics. International Conference on Learning Representations (ICLR),
[56] Fengyuan Yang, Ruiping Wang, and Xilin Chen. Semantic guided latent parts embedding for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 5447-5457, 2023.
[57] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's distance and structured classifiers In Proceedings of the IEEE/CVF conference compuler vision and paltern recognition, pages 12203-12213, 2020.
[58] Hongguang Zhang, Jing Zhang, and Piotr Koniusz. Few-shot learning via aaliency-guided hallucination of samples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 2770-2777 2019.
[59] Jianguo Zhang, Marein Marszalek, Svetlana Lazebnik, and Cordelia Schmid. Local features and kernels for classification of texture and object categories: comprehensive study. International journal of computer vision, 73(2):213-238, 2007.
[60] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Repre- sentations, 2022.
[61] Song-Chun Zhu, David Mumford, al. stochastic grammar of images. Foundations and Trends@ in Computer Graphics and Vision, 2(4))259-36) -62, 2007.
14

---

Mitigating the Effect of Incidental Correlations on Part-based
Learning (Supplementary Material)
A Appendix
A.1
Why the term "incidental correlations" for image background?
The concept of "incidental correlations" is derived from the notion of incidental endogeneity [2]. which describes unintentional but genuine correlations between variables. In the context of our study, image backgrounds are not considered spurious because they offer contextual information that aids in decision-makee
Therefore, the relationship between image backgrounds and classification is not anti-causal, as would be true if the backgrounds were spurious. We argue that the imbalance of specific image backgrounds in the training data the primary factor contributing to the introduction of incidental correlations.
A.2 Training and inference details for pertaining and fine-tuning DPVIT
Training Details. Our approach involves pre- pre-traininn training the Vision Transformer backbone and projection head using the same method described in the iBOT paper [60]. We mostly keep the parammte settings unchanged without tuning. By default,
the Vit-Small architecture, which consists of 21 million parameters. The patch size is set to 16 as our default configuration. The student and teacher networks have shared projection head for the [cls] token output. The projection heads for both networks have an output dimension of 8192. We adopt a linear warm-up strategy for the learning rate over 10 epochs, starting from base value of 5c-4, and then decaying le-5 using a cosine schedule. Similarly, the weight decay is decayed using a cosine schedule from 0.04 to 0.4. We employ a multi-crop strategy improve performance with 2 global crops (224x224) and 10 local crops (96x96). The scale ranges for global and local crops are (0.4, 1.0) and (0.05, 0.4), respectively. Following [60]. we use only the local crops for self-distillation with global crops from the same image. Additionally, we apply blockwise masking to the global crops inputted into the student network. The masking ratio uniformly sampled from [0, 1, 0.5] with a probability of 0.5, and with probability of 0.5, it is set to O. Our batch size is 480, with batch size per GPU of 120. DPVIT's pre-trained for 500 epochs for the given training set for all the datasets.
We use the value of Acta = 1, A = 0.5, we incorporate the class labels by incorporating logit head onto the projection heads. This allows us to calculate the cross-entropy loss based on the provided class labels. The explicit utilization of class labels is necessary for the ImageNet-9 dataset because the evaluation involves straightforward classification rather than few-shot learning.
0.5 for all the datasets. In the case of ImageNet-9,
Fine-tuning Details. Once the pretraining stage is completed, we proceed to train the model using the supervised contrastive loss, which involves distilling knowledge from [cls] tokens across different views of images (referred conducted for 50 epochs using the same training data the given dataset. We maintain the same set of hyperparameters used in the initial pretraining stage without additional tuning.
Lat in Equation 9 of the main draft). The fine-tuning process is
We use the value of iine 1, and Ximu 0.5 for all the datasets.
Inference Details. For inference purposes, we utilized a feature representation obtained by the [cls] token of the teacher network. We also found concatenating the weighted average pooling of the generated patches with the [cls] token useful in few-shot evaluation. The weights for the weighted average pooling are determined by taking the average of the attention values of the [cls] token across all heads of the final attention layer.
In the case of ImageNet-9, the logit head used infer the class label for the given sample in the test set.
A.3 Details regarding the multi-head attention modules
The design of our attention layers draws inspiration from the standard self-attention mechanism, commonly known as gkv self-attention (SA) [I]. In our implementation, we calculate a weighted sum over all values v in the input sequence where has dimensions of NxD. The attention
15

---

weights As are determined based on the pairwise similarity between two elements of the sequence and their corresponding query 9' and key k' representations.
q k, v] = zUkkv
Ugkeo E DRD33DD
(10)
A = softmar (qk' (W/D/VD))))
A A
(11)
SA(z) Av.
(12)
Multihead self-attention (MSA) is expansion of the self-attention mechanism, where we perform k parallel self-attention operations, known as "heads," and then combine their outputs through concatenation, order to maintain consistent computation and the number of parameters when adjusting the value of k, the dimension Dr (as defined in Equation 10 is typically set to D/k.
M (SA(z) = (AA ... (SA()))))s))))::sss)))))))))))) ( mssa
U Unsa E DD
(13)
A.4 Details regarding the power iterative method to compute spectral norm
We follow the power iterative method described in 3 to compute the spectral norm for (P"P 1). Starting with randomly initialized IR", we iteratively perform the following procedure a small number times (2 times by default):
||8|| ||]
u u+ I (P'P 1
(14)
The power iterative method reduces computational cost from O to O(m²)) which is practically much faster when used with our training procedure.
A.5 Comparing VIT-S 11 and Concept Transformer (CT) 41 on ImageNet-9
In addition the findings presented in Section 5.4 (Table 2 in the main draft), we conducted comparison with vanilla ViT-S pretrained on Imagenet and ConceptTrrnnformers (CT) as well. CT, as described the study by [4], has a notable limitation in that relies on attribute supervision for part localization information. This restriction restricts the applicability of CT in scenarios where attribute information absent, such as the case of ImageNet-9. To train CT without attributes, we utilized the code provided by the authors and deactivated the attribute loss, allowing CT to be trained without relying on the attribute information This adjustment significantly decreases the performance of CT but enables fair comparison with other methods on ImageNet-9. It is worth noting that CT employs the ViT-S backbone pretrained on ImageNet as its default architecture. Moreover, we train ConstNet [54] using the source code provided by the authors
As indicated in Table DPVIT demonstrates superior performance compared to both VIT-S pretrained on ImageNet and CT, exhibiting a clear advantage. CT can be seen pretrained ViT-S model with the inclusion of part dictionaries, but it experiences a noticeable drop performance when confronted with the presence of incidental correlations in the image backgrounds (as observed in the low M-SAME and M-RAND performance in Table (). This demonstrates that the part learners in general cannot effectively deal with the incidental correlations of backgrounds and are susceptible to varying backgrounds.
A.6 Ablation study with different values of K and ny on MinilmageNet
In this analysis, we investigate the impact of varying the number parts, denoted as K, on the MinilmageNet dataset. Specifically, explore the effects of altering the number of foreground parts, represented by ny,, as well as the number of background vectors, which can be calculated as K Rz- Table 9 presents the obtained results, demonstrating the influence of different values of K, n,y and ny on parts, foreground parts, and background parts, respectively.
TransopTrr 41 http::://gtttabbbc/////oo/eeeeee htte://eeaaaatte///////////////e
*ConstNet
16

---

Method
IN-9L 1
Original
M-SAME 1 M-RAND
BG-GAP
ResNet-S0 53
94.6
96.3
89.9
73.6
14.3
WRN-50
95.2
97.2
90.6
78.0
12.6
ConstNet
90.6
92.7
86.1
VII-S pretrained
[11
82.5
84.9
72.2
50.3
21.9
Concepriransformer
(41
84.7
85.5
73.1
51.5
21.6
Ours DPVIT
96.9
98.5
93.4
87.5
5.9
Table 8:
Performance evaluation on domain shift of varying background and common data corruptions on ImageNet-9. Evaluation metric Accuracy %.
(a) Original
(b) MIXED-SANE
(c) MIXED-RAND
Figure 8: Visualizing the test splits from ImageNet-9 dataset.
K=32
K=64
K=96
K=128
Foreground parts
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
K/2
72.2.0.2
87.8
72.9
88.1
72.1
88. 48.1
72.1
87.1
2K/3
72.2+0.2
88, 88.1
73.8
89. 89.344.4
73.1
88. 48.4
72.240.5
87.4
ny 4K/3
72.3
88.420.4
73.410.5
88,510.4
73.2
87.910.
72.5.0.5
87,8
Table 9: Ablation of varying the number of foreground -ackkududdb vectors, along with part-vectors
used. We show the results on the minilmageNNe dataset.
Our findings indicate that maintaining K 64 and selecting ns 2K/3 yields the highest performance. When employing significantly lower number of part vectors, the model's capacity becomes insufficient, leading to performance degradation. Conversely, employing a larger value of K results in increased computational complexity associated with distance maps, subsequently leading to lower performance.
A.7 Computational complexity of DPVIT
Adding part-dictionaries to MCA layers slightly increases the trainable parameters from 21M (ViT-S) to 25M (DPViT). It is also possible to share the attention layers, analogous to the Siamese networks, for MSA and MCA, which keeps the number of trainable parameters to 21 M. DPVIT results in similar performance in terms of few-shot accuracy when the attention layers are shared, as shown in Table I0
A.8 Stage-1 pertaining comparison with SMKD 30
Table II showcases the few-shot evaluation results of DPVIT on the MinilmageNet dataset. In addition, we compare the performance of DPVIT with the first-stage performance of SMKD [30]. worth noting that both DPViT and SMKD utilize iBOT [60] pretraining strategy. However, incorporating part-dictionaries, MSA, and MCA layers in DPVIT's pretraining phase contributes to its superior performance compared SMKD.
A.9 Studying complementary properties of MSA and MCA
Based on Section 5.1 in the main draft, our study focuses on examining the complementary character- istics of MSA and MCA. MSA is designed to be effective for images containing small number of objects, but struggles to capture the spatial relationships among multiple objects. contrast, MCA layers utilize distance maps to learn spatial relationships and prioritize objects without considering
17

---

Setting
1-shot 7
S-shot 1
[|]]]
|PP'
I||
Method
1-shot f
5-shol
Shared
73.6
89.6
0.4
0.5
SMKD (30
60.93
80.38
Unshared
73.8
0.3
DPVIT
62.81
83.25
Table 10: Siamese DPVIT. Sharing MSA and
Table 11: Few-shot performance after Ist stage pretrain phase on MinilmageNet.
MCA layers and evaluation on MinilmageNet.
Setting
1-shot f
5-shot f
Setting
I-shot
5-shot f
L2-norm
65.12
82.95
Gaussian
65.12
82.95
L.Inorm
58.73
79.51
Salt-and-pepper
62.21
82.75
Cosine
non
Speckle
61.60
81,50
Table 12: Different optimizers for Lmiz. Ex- perimenting with different loss terms during pre- training.
Table 13: Different noise functions for ő intro-
duced in Equation 4.
(a) Original
Attention heads of MSA
(c) Attention heads of MCA
Figure 9: Visualizing the attention heads for MSA and MCA.
their specific classes. In simpler terms, MSA may overlook certain objects that not crucial for eaeresizess
classification, while MCA emphasizes learning spatially similar objects.
10 and 12 The MSA heads excel at identifying objects for classification but may overlook relevant objects with significant spatial context, such as the "charger" in Figure g and the "garbage box" in Figure 12 On the other hand, the MCA layers perform well in scenarios involving multiple objects (Figure 9 and 12]. but struggle when spatially similar objects present, "fish" in Figure TO
Additionally, present the visualization of attention heads in Figure 9
with the confusion between the "red grass" and the
A.10 Optimization functions for Lmiz
The segmentation masks My and M,, are composed of binary values. Our selection of the L2-norm for Lmiz is informed by empirical observations. Additionally, we conducted experiments with alternative loss functions like LI and cosine distance during the pretraining phase. As shown in Table, while the L1-norm results in lower performance, the cosine distance results in unstable training with training loss reaching +infinity, eventually N AN. We achieve the best performance with the 1.2-norm.
A.11 Noise f'unctions for
We conduct ablation study the choice of noise functions used for 8. As shown in Table 13 we ascertain that Gaussian noise is more effective in promoting resilience within the latent codes, while simultaneously preserving the interpretability of parts when compared with salt-and-pepper and speckle noise during the pretraining phase.
A.12 Visualization foreground parts
We present additional part visualizations Figure 10(a) and [T(a) These are shown in Figure T2(a)
and 12(b)
18

---

(a) Original
Attention heads of MSA
(c) Attention heads of MCA
Figure 10: Visualizing the attention heads for MSA and MCA.
(a) Original
(b) Attention heads of MSA
(c) Attention heads of MCA
Figure Visualizing the attention heads for MSA and MCA.
A.13
Qualitative comparison extracted patches with ConstNet [54]
In order to showcase the acquired parts of DPVIT and ConstNet, provide visualizations in Figure 13 and [4] This is achieved by selecting the nearest patches the parts. Figure [33 illustrates the separation of foreground and background concepts accomplished by our model, whereas Figure [4] exhibits the patches surrounding the learned parts from the ConstNet model.
While DPVIT learns to disentangle the foreground patches from the backgrounds, the patches extracted by ConstNet suffer from the entanglement caused due to incidental correlations of backgrounds.
X
I(Lp)
I(D
I(D²)
I(D*)
I(D*)
(a) Parts visualizations for Figure 1O(a)
X
I(Lp)
I(D+)
I(D²)
I(D")
I (D*)
(b) Parts visualizations Figure [1(a)
Figure 12: Visualizing foreground parts learned by DPVIT.
19

---

(a) Foreground patches extracted by DPVIT
(b) Background patches extracted by DPVIT
Figure 13: Visualizing foreground and background patches extracted by DPVIT around a random foreground and background part for images from the validation of MinilmageNee.
(a) Patches extracted by ConstNet 54
(b) Patches extracted by ConstNet [54]
Figure 14: Visualizing patches extracted by ConstNet 54 around a random parts for images from the validation set of MinilmageNet.
20


Reviews:
[{"review_id": "ZHW5BFWjAk", "reviewer": "Reviewer_DtGy", "summary": "The paper proposes to produce a more robust and interpretable part-based learning method for image classification problems (though, probably extensible to other tasks). The main contributions include 1) the Disentanglement of foreground/background regions via a weakly supervised loss and 2) the Use of sparse and orthogonality constraints to discourage degenerate solutions 3) Finetuning using invariance constraint to ensure predictions are invariant for the learned background parts. The experiments show competitive results compared to recent methods and provide (arguably) more interpretable results.", "questions": "- Why is the orthogonality constraint applied to the whole matrix rather than foreground/background separately? Is this done for practical reasons (i.e., it would be difficult to implement) or is there a motivation behind this choice?\n\n- Please see W1. ", "limitations": "The paper aims to reduce spurious correlations that can help mitigate some biases/lack of generalization in neural networks. I do not see an explicit potential for negative societal impact. The limitation section is missing and could include technical limitations/assumptions behind the work (Does not address intra-fg spurious correlations, requires access to masks, etc..), but is not a deal-breaker. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "[S1] Clearly motivated and technically sound: The paper addresses a well-known problem of spurious correlations in neural networks. While there are multiple types of spurious correlations, the paper only focuses on spurious correlations stemming from foreground and background regions. While that makes it more limited, it also provides clearer motivation, and the solution proposed can also address the specific problem better.  Indeed, the proposed solutions appear technically sound and well-designed to discourage correlations which are also well corroborated by the results.\n\n[S2] Adequate details/ablations: There are many parts to the proposed method, but the paper has done a reasonably good job with ablations and discussion of the effects of different components. ", "weaknesses": "[W1] Some details are slightly unclear: The pretraining portion is mostly quite clear but I have some clarity issues with the finetuning (distillation) portion of the proposed method. I am not sure I fully grasp how Eqn 8 part 2, ensures the fg code captures details in the absence of background. Part 1 is clearer but still needs a bit more details (i.e., if the goal is to make predictions invariant of background, why distill from a teacher model that has access to both? I assume that the choice is made based on empirical results; e.g., fg only model might not converge well), but this needs to be discussed in more detail.\n\n[W2] Effects of error propagation from foreground/background \"weak supervision\": The existing solution requires access to a model that can segment/divide images into fg and bg during training. While I agree that such models are easy to use and are widely available, the paper is unclear about what the effects of the quality of the underlying model are on the proposed method. The quality of fg/bg themselves might have issues with correlations. This could perhaps be simulated by using corruption techniques to see how the quality of foreground/background segmentation affects proposed methods and also get an idea of how this method might fare with newer/future methods for fg/bg segmentation. L333 studies the availability of masks but not whether a noisy/incorrect mask might affect the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and suggestions. We respond below to each of the concerns/suggestions.\n\n> [W1] Some details are slightly unclear: The pretraining portion is mostly quite clear but I have some clarity issues with the finetuning (distillation) portion of the proposed method. I am not sure I fully grasp how Eqn 8 part 2, ensures the fg code captures details in the absence of background. Part 1 is clearer but still needs a bit more details (i.e., if the goal is to make predictions invariant of background, why distill from a teacher model that has access to both? I assume that the choice is made based on empirical results; e.g., fg only model might not converge well), but this needs to be discussed in more detail.\n\nIn the process of fine-tuning, the initial component of Equation 8 (referred to as $L_{cls}^{inv}$) guarantees that the student model acquires pertinent foreground insights from a teacher model equipped with access to both foreground and background information. This design decision further guarantees the incorporation of pertinent background insights during the learning of representations. As recommended, we carry out an empirical investigation to examine the impact of knowledge distillation from a teacher model possessing solely foreground information:\n\n|Architecture\t|1-shot %\t| 5-shot %|\n|--------| -------- | -------- |\n|Distill only foreground\t|73.69$_{\\pm{0.23}}$\t| 88.51$_{\\pm{0.17}}$\t|\n|Distill foreground + background\t|73.81$_{\\pm{0.21}}$\t|89.85$_{\\pm{0.20}}$|\n\nAs demonstrated in the table above, the teacher model equipped with the ability to distill knowledge from both foreground and background information exhibits slightly superior performance compared to the model that solely accesses foreground-related information.\n\nDuring the finetuning process, we abstain from employing the mixture loss due to the utilization of learned latent codes for generating foreground masks. The incorporation of $L_p^{inv}$ essentially confines the foreground elements within the foreground space. Alternatively, another approach would involve utilizing the foreground masks to calculate the mixture loss (akin to the pretraining stage). If this route is taken, the inclusion of the $L_p^{inv}$ loss becomes unnecessary when fine-tuning. Through empirical testing, we have confirmed that both of these design options yield comparable performance outcomes.\n\n> [W2] Effects of error propagation from foreground/background \"weak supervision\": The existing solution requires access to a model that can segment/divide images into fg and bg during training. While I agree that such models are easy to use and are widely available, the paper is unclear about what the effects of the quality of the underlying model are on the proposed method. The quality of fg/bg themselves might have issues with correlations. This could perhaps be simulated by using corruption techniques to see how the quality of foreground/background segmentation affects proposed methods and also get an idea of how this method might fare with newer/future methods for fg/bg segmentation. L333 studies the availability of masks but not whether a noisy/incorrect mask might affect the model.\n\nWe agree with the reviewer that studying the quality of foreground/background segmentation could be simulated by adding corruptions to the ground truth masks. We conduct an ablations studying the effect of corrupting the ground truth masks using the Gaussian noise with a manual corruption strength ($\\lambda_c$) : \n\n$M_f = \\lambda_c \\delta + M_f; M_b = \\lambda_c \\delta + M_b; \\delta \\sim N(0,1)$.\n\n| | $\\lambda_c$=0.1 | $\\lambda_c$=0.5  | $\\lambda_c$=1.0  |\n| -------- | -------- | -------- | -------- |\n| Method |1-shot/5-shot | 1-shot/5-shot | 1-shot/5-shot| \n| DPViT | $73.05_{\\pm{0.15}}$  / 88.56$_{\\pm{0.18}}$ | $72.83_{\\pm{0.20}}$/ 87.92$_{\\pm{0.19}}$ | $72.12_{\\pm{0.17}}$/ 87.51$_{\\pm{0.15}}$ |\n\nAs depicted in the Table above, DPViT displays a degree of robustness against minor to moderate corruptions in segmentation masks (<=0.5). However, once this threshold is surpassed, the interpretability of DPViT experiences a marked decline. We substantiated this observation through qualitative analysis. Please refer to Figure 2 and 3 in the uploaded rebuttal pdf for qualitative analysis.\n\n> Why is the orthogonality constraint applied to the whole matrix rather than foreground/background separately? Is this done for practical reasons (i.e., it would be difficult to implement) or is there a motivation behind this choice?\n\nWe would like to clarify that the orthogonality is applied to foreground and background parts separately rather than the whole matrix (as indicated by Equation 6):\n\n$\\mathcal{L}_{Q} (\\lambda_s, \\lambda_o) = \\lambda_s ||\\mathbf{P} ||_1 + \\lambda_o \\Big[\\sigma \\big(\\mathbf{P_F} \\cdot \\mathbf{P_F}^{T} - \\mathbf{I} \\big)+ \\sigma \\big( \\mathbf{P_B} \\cdot \\mathbf{P_B}^{T} - \\mathbf{I}\\big) \\Big] $\n\n> Limitations:\nThe paper aims to reduce spurious correlations that can help mitigate some biases/lack of generalization in neural networks. I do not see an explicit potential for negative societal impact. The limitation section is missing and could include technical limitations/assumptions behind the work, but is not a deal-breaker.\n\nLimitations of our work: \n- A constraint within our framework involves relying on a pre-existing foreground extractor. In certain scenarios, such as the classification of tissue lesions for microbiology disease diagnosis, obtaining an existing foreground extractor might not be feasible. \n- At present, DPViT focuses on learning components that are connected to the data, yet it doesn't encompass the connections between these components, like their arrangement and hierarchical combination. Introducing compositional relationships among these components could enhance comprehensibility and facilitate the creation of a part-based model capable of learning relationships among the parts."}, {"review_id": "7Al2sd7ON9", "reviewer": "Reviewer_gyRR", "summary": "The paper addresses the impact of incidental correlations on part-learning and proposes several regularization methods to mitigate this. The first regularization separates foreground and background to guide the part-based learning towards relevant input regions. To this end, the paper proposes a mixture of parts formulation to collect latent codes into masks and supervise them (weakly) with the pretrained/unsupervised foreground extractor. Then, the second regularization is employed to impose invariance to background variation. Additionally, sparsity and orthogonality regularization is used to refine part representations.", "questions": "I suggest authors address weaknesses for the rebuttal.", "limitations": "I suggest authors elaborate on hyper-parameter optimization for the proposed framework.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an important problem. Part-based methods are becoming more and more valuable nowadays due to increased interpretability\n2. The paper points out the failure mode of part-based learning\n3. The text is nicely written and easy to follow. Figures nicely highlight most relevant findings. ", "weaknesses": "1. The choice of sparse and spectral norms is not well explained in my opinion. I understand the need for sparsity in the image space when training part-based models, but in the paper, sparsity is enforced on the matrix of part representations instead. What is the physical meaning of this? Also, why is the spectral norm chosen to optimize for orthogonality? It is not evident straight away why would it be a better alternative to just L2 / Frobenius.\n\n2. Why the background model is needed as a separate entity? Is modeling just the foreground (the background is then all the rest of the image naturally) not sufficient? When the background is modeled explicitly as a mixture of parts, then the part-model should learn to encode all background patches as well, right? This looks like an unnecessary complication.\n\n3. The whole framework consists of 3 regularizers, each with its weighting. This introduces more hyper-parameters to the model and increases resources needed to optimize the performance. With this, it is not clear if the (rather limited) performance improvement provided by the method is worth the increased demand for hyper-parameter optimization.\n\n4. Most of the methods in experimental evaluation are built on top of different backbones. It makes it harder to disentangle improvement from regularization and improvement from the backbone. It would make sense to use the model trained with L_{cls} only as a baseline for comparison in Table 1 and Table 2.\n\nUPD: Authors addressed most of my concerns in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and suggestions. We respond below to each of the concerns/suggestions.\n\n> The choice of sparse and spectral norms is not well explained in my opinion. I understand the need for sparsity in the image space when training part-based models, but in the paper, sparsity is enforced on the matrix of part representations instead. What is the physical meaning of this? ...\n\nWe have explained the need of sparse and orthogonal norm in the context of our work in L57-L65 : “*Sparsity ensures that only a few parts are responsible for a given image, as images comprise a small subset of parts. Conversely, diversity in part representations prevents the parts from converging into a single representation and facilitates each part’s learning of a unique data representation*”. The need of sparsity is to select a subset of parts from the part-matrix for a given sample. Similarly, the need of orthogonality is to impose diversity in the foreground/background parts.\n\nMoreover, the choice of spectral norm is explained in L200-L213 : “ *One solution is to enforce orthogonality on the matrix $\\mathbf{P}^{m\\times n}$ by minimizing $||\\mathbf{P}^T\\mathbf{P} - \\mathbf{I}||$. However, the solution will result in a biased estimate as $m<n$; that is, the number of parts ($K$) is always less than the dimensionality of parts ($F^2\\cdot C$). In our experiments, we observed that increasing $K$ beyond a certain threshold degrades the performance as computational complexity increases. (Please refer to our Appendix section 1.6 (Table 2) for experiments on the different values of $K$). To minimize the degeneration of parts, we design our quality assurance regularization by minimizing the spectral norm of $\\mathbf{P}^T\\mathbf{P} - \\mathbf{I}$, and by adding $L_1$ sparse penalty on the part-matrix $\\mathbf{P}$. The spectral norm of $\\mathbf{P}^T\\mathbf{P} - \\mathbf{I}$ has been shown to work with over-complete ($m<n$) and under-complete matrices ($m\\geq n$)* [1].”\nWe structure our power iterative spectral norm methodology based on the framework presented in [1]. It's important to emphasize that the study conducted in [1] concentrates on ensuring orthogonality across all weights within the neural network. Conversely, in our approach, we employ spectral norm to introduce diversity among the matrices associated with foreground and background components. For a comprehensive verification of the spectral norm concept, please refer to [1]\"Can we gain more from orthogonality regularizations in training deep networks?\" (NeurIPS 2018).\n\n\n>Why the background model is needed as a separate entity? Is modeling just the foreground (the background is then all the rest of the image naturally) not sufficient? ...\n\nThe background parts provides essential contextual information during training, which is cruical for learning interpretable part representations. To verify this, we conduct an ablation study where n_f = K, and n_b = 0. Furthermore, we switch off the background influence on $L_{mix}$ by modifying the Equation 5 and 6 as:\n\n$\\mathcal{L}_{mix} = || \\mathcal{I}(L_F) - \\mathcal{M}_f||_2$\n\n$\\mathcal{L}_{Q} (\\lambda_s, \\lambda_o) = \\lambda_s ||\\mathbf{P} ||_1 + \\lambda_o \\Big[\\sigma \\big(\\mathbf{P_F} \\cdot \\mathbf{P_F}^{T} - \\mathbf{I} \\big) \\Big] $\n\n| Design |1-shot %| 5-shot %  |  $\\|P\\|_1$ | $\\|PP^T - I\\|_1$ |\n| -------- | -------- | -------- | -- | -- |\n| Foreground |72.81$_{\\pm{0.15}}$ |88.21$_{\\pm{0.18}}$ | 0.35$_{\\pm{0.21}}$| 0.99$_{\\pm{0.11}}$ | \n| Foreground+Background |73.05$_{\\pm{0.15}}$ | 88.56$_{\\pm{0.18}}$ | 0.32$_{\\pm{0.21}}$| 0.32$_{\\pm{0.17}}$ | \n\nThe foreground-only model exhibit comparable few-shot performance to foreground+background model, but the acquired components lack diversity, resulting in reduced interpretability. This is evident from the higher orthogonal-norm value observed in the case of the foreground-only model.\n\n>The whole framework consists of 3 regularizers, each with its weighting. This introduces more hyper-parameters to the model and increases resources needed to optimize the performance. ...\n\nAlthough our framework incorporates multiple hyperparameters, extensive hyperparameter tuning is not required. As outlined in Appendix Section 1.2, we maintain consistency in most hyperparameters across all employed datasets. Specifically, we set $\\lambda_{cls}$ = 1, $\\lambda_{s}$ = 0.1, $\\lambda_o$ = 0.1, $\\lambda_{cls}^{inv}$ = 1, and $\\lambda_p^{inv}$ = 0.5 for all experiments conducted on MiniImagenet, TieredImageNet, FC100, Imagenet-9, and Imagenet-1K datasets. For a more comprehensive understanding of our approach to hyperparameter optimization, kindly refer to Appendix Section 1.2.\n\n>Most of the methods in experimental evaluation are built on top of different backbones. It makes it harder to disentangle improvement from regularization and improvement from the backbone. It would make sense to use the model trained with L_{cls} only as a baseline for comparison in Table 1 and Table 2.\n\nFor a better evaluation of our work, we conduct experiments with the $L_{cls}$ baseline. As suggested by the reviewer, we compare $L_{cls}$ model with DPViT in Table 1 and Table 2 as follows:\n\nTable1:\n| | MiniImageNet | TieredImageNet  | FC100  |\n| -------- | -------- | -------- | -------- |\n| Model |1-shot/5-shot | 1-shot/5-shot | 1-shot/5-shot| \n| $L_{cls}$ |$72.15_{\\pm{0.20}}$/87.61$_{\\pm{0.15}}$ | $78.03_{\\pm{0.19}}$/89.08$_{\\pm{0.19}}$ | $48.92_{\\pm{0.13}}$/67.75$_{\\pm{0.15}}$|\n| DPViT |$73.81_{\\pm{0.17}}$/89.85$_{\\pm{0.18}}$ | $79.32_{\\pm{0.19}}$/91.92$_{\\pm{0.20}}$ | $50.75_{\\pm{0.20}}$/68.80$_{\\pm{0.15}}$|\n\nTable 2:\n|\tMethod | IN-9L | Org\t|M-SAME\t|M-RAND |BG-GAP|\n|--------| -------- | -------- | -------- | -------- | -------- |\n|$L_{cls}$ | 95.1$_{\\pm{0.21}}$ | 97.2$_{\\pm{0.25}}$ | 91.5$_{\\pm{0.19}}$ | 81.7$_{\\pm{0.15}}$ | 9.2$_{\\pm 0.2}$ |\n|DPViT | 96.9$_{\\pm{0.15}}$ | 98.5$_{\\pm{0.20}}$ | 93.4$_{\\pm{0.19}}$ | 87.5$_{\\pm{0.23}}$ | 5.9$_{\\pm 0.2}$ |"}, {"review_id": "jOPYujOmmb", "reviewer": "Reviewer_QGVs", "summary": "The authors propose a strong part-based learning method for few-shot learning. Noticing the incidental correlations between part signals, the method proposed in the paper, named DP-ViT, learned to disentangle foreground and background parts. This is done by introducing a mixture-of-experts formulation of parts. To further encourage high-quality and diverse parts, the DP-ViT further employs a regularization for orthogonality.The method proposed by the authors achieves SOTA on multiple few-shot learning datasets, including MiniImagenet, TieredImageNet, and FC100. With the DP-ViT framework, the authors also illustrate the incidental correlation in imagery data and how DP-ViT improves the performance of models.\n\n", "questions": "1. It looks to me that DP-ViT is a quite general improvement to the ViT architecture. However, to support its full utility, I think an experiment on ImageNet is needed, where DP-ViT can outperform a vanilla ViT or a part-based ViT baseline.\n\n2. I personally would suggest adding an ablation study for the number of parts, K, especially what will happen when K is increased. It is interesting to see that K=64 parts is sufficient for a dataset.", "limitations": "I haven't found such a section from the authors. But I tend to think that this work does not have a negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors have made sound improvements to two regularizations on part-based models. For me, these ideas make sense and the authors have demonstrated interpretation of these modules in the experimental sections. Additionally, it is nice to see that the proposed solutions improve the performance of few-shot learning.", "weaknesses": "I don't directly work on this topic, and I think the authors have provided adequate analysis and explanation on the few-shot learning part. However, I think this paper could have a more general impact. Please check the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and suggestions. We respond below to each of the concerns/suggestions.\n\n> It looks to me that DP-ViT is a quite general improvement to the ViT architecture. However, to support its full utility, I think an experiment on ImageNet is needed, where DP-ViT can outperform a vanilla ViT or a part-based ViT baseline.\n\nAs suggested by the reviewer, we conduct an experiment on ImageNet-1K, and compare it with the vanilla ViT-S model. \n\n| Model | Val Acc %|\n|------|------|\n| ViT-S | 54.5 %|\n|**DPViT** | **72.1** %|\n\nPlease note that we train both models over 100 epochs using approximately 1 million training samples, employing a batch size of 512. For DPViT, we use the same set of hyperparameters that we used for ImageNet-9 dataset (please refer to Appendix Section 1.2). For ViT-S, we use Adam optmizer with learning rate 0.001 along with cosine scheduler (the code is adopted from **timm** library). Subsequently, we evaluate the models using the provided validation set consisting of 50,000 samples. Ideally, DPViT could undergo pretraining on an expansive dataset like ImageNet-21K or JFT300M, followed by fine-tuning on ImageNet-1K. Regrettably, due to limitations in time and resources, we were unable to carry out such extensive pretraining for both models.\n\n> I personally would suggest adding an ablation study for the number of parts, K, especially what will happen when K is increased. It is interesting to see that K=64 parts is sufficient for a dataset.\n\nWe have reported an ablation with a varying number of parts K on MiniImageNet dataset in Appendix Section 1.6 (Table 2):\n\n| | K=32 | K=64 | K=96 | K=128 |\n|--|--|--|--|--|\n|$n_f$ | 1-s/5-s | 1-s/5-s | 1-s/5-s |1-s/5-s|\n| $n_f=K/2$ | $72.2_{\\pm 0.2}$ / $87.8_{\\pm 0.4}$ | $72.9_{\\pm 0.5}$ / $88.1_{\\pm 0.4}$| $72.1_{\\pm 0.2}$ / $88.1_{\\pm 0.4}$ | $72.1_{\\pm 0.5}$/ $87.1_{\\pm 0.5}$ \n| $n_f=2K/3$ | $72.2_{\\pm 0.2}$/ $88.1_{\\pm 0.4}$ |$73.8_{\\pm 0.5}$ / $89.3_{\\pm 0.4}$ | $73.1_{\\pm 0.2}$ / $88.1_{\\pm 0.4}$ | $72.2_{\\pm 0.5}$ / $87.4_{\\pm 0.5}$\n|$n_f = 4K/3$ | $72.3_{\\pm 0.2}$/ $88.4_{\\pm 0.4}$ | $73.4_{\\pm 0.5}$/ $88.5_{\\pm 0.4}$ |$73.2_{\\pm 0.2}$/ $87.9_{\\pm 0.4}$ | $72.5_{\\pm 0.5}$/ $87.8_{\\pm 0.5}$ \n\nWe observed that increasing $K$ beyond a certain threshold degrades the performance as computational complexity increases. \n\n\n> Limitations: I haven't found such a section from the authors. But I tend to think that this work does not have a negative societal impact.\n\nLimitations of our work: \n- A constraint within our framework involves relying on a pre-existing foreground extractor. In certain scenarios, such as the classification of tissue lesions for microbiology disease diagnosis, obtaining an existing foreground extractor might not be feasible. \n- At present, DPViT focuses on learning components that are connected to the data, yet it doesn't encompass the connections between these components, like their arrangement and hierarchical combination. Introducing compositional relationships among these components could enhance comprehensibility and facilitate the creation of a part-based model capable of learning relationships among the parts."}, {"review_id": "fN1OWcMIgK", "reviewer": "Reviewer_Tpfu", "summary": "This paper builds up a part-based learning algorithm for few-shot classification with components preventing incidental correlations during training. ViT is adopted as the architecture, which facilitates part interpretation by recognizing each image patch as a specific part/foreground/background. Experiments on several few-shot learning benchmarks show the effectiveness of the approach, and an additional experiment on ImageNet-9 verifies that the approach can indeed reduce incidental correlations during training.", "questions": "- Why can Gaussian noise make mixture-of-parts robust to common data distortions? it seems its use is to relax the target $L_{mix}$ to be softer instead.\n- What's the exact form of $M_f$ and $M_b$? Are they just binary masks? Why $l_2$-norm is used in $L_{mix}$? Since the target is a mask, it seems cosine distance is more appropriate.\n- Please clarify the confusing parts listed in the weaknesses part.", "limitations": "No limitations are listed in the paper. I suggest the authors conduct experiments on cross-domain datasets like tieredImageNet->MSCOCO (in Meta-Dataset). Such experiments can show clearer advantages of part-based methods and the removal of incidental correlations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The proposed algorithm achieves interpretable representations to some degree, which is important for few-shot learning. \n+ Image background has been shown harmful for few-shot learning at both training and test time, and this paper tackles the background problem at training time in a novel way. The experiments on ImageNet-9 clearly verify that the method can remove incidental correlations.\n+ The proposed algorithm achieves SOTA performance on several few-shot learning benchmarks.", "weaknesses": "- Missing important ablation studies. (1) To verify the effectiveness of part-based learning, the authors need to show the performance of a vanilla ViT with the loss $L_{cls}$. (2) No ablation studies on the use of $L_{mix}$ and $L_{Q}$. Importantly, the authors do not show the baseline performance using part-based learning with $L_{cls}$ alone during pretraining (especially on few-shot learning benchmarks). Thus the usefulness of removing incidental correlations has not been verified.\n\n- There are confusing parts in the method description. For example, it is not clear if $z_p$ and $z_d$ are iteratively computed for every block of the network or just computed for the feature of the last block. The same confusion goes for distance maps and embedded patches. The architecture plot in Figure 2 does not answer this question (e.g., what's the input to the second block?). Some other confusions are listed in the questions part.\n\nminor: \n- A typo in line 188,  it should be $n_f \\times F^2 \\times C$.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and suggestions. We respond below to each of the concerns/suggestions.\n\n> Missing important ablation studies....\n\nWe perform an ablation analysis to investigate the impact of $L_{mix}$ and $L_Q$ during the pretraining phase on the MiniImageNet dataset. Additionally, we provide the baseline results achieved solely through $L_{cls}$, along with the pretraining performance of SMKD [1]:\n\n| Model | 1-shot % | 5-shot % |  $\\|P\\|_1$ | $\\|PP^T - I\\|_1$ |\n|--|--|--|--|--|\nSMKD [1] | 60.93 | 80.38 | - |   -\n$L_{cls}$ | 61.24$_{\\pm 0.2}$ | 81.12$_{\\pm 0.2}$ | 8.41$_{\\pm 0.1}$ | 25.82$_{\\pm 0.2}$\n$L_{cls}+L_{mix}$ | 62.81$_{\\pm 0.1}$ | 83.25$_{\\pm 0.2}$ | 8.73$_{\\pm 0.2}$ | 24.61$_{\\pm 0.1}$\n$L_{cls}+L_{mix}+L_Q$ | 62.15$_{\\pm 0.2}$ | 82.95$_{\\pm 0.2}$ | 0.35$_{\\pm 0.1}$ | 0.56$_{\\pm 0.2}$\n\nThe above table highlights the relevance of each component during the pretraining.\n\n> There are confusing parts in the method description....\n\nDuring the architectural design phase, we employ distinct components ($\\mathbf{P}$) for each encoder block. Consequently, $z_p$ and $z_d$ are calculated in an iterative manner and subsequently transmitted to the subsequent encoder block. Regarding the computation of $L_{mix}$, we utilize the distance maps (**D**) from the concluding encoder block. While it's feasible to calculate $L_{mix}$ iteratively for each block and then aggregate them for a final $L_{mix}$ computation, our observations indicate that this approach amplifies computational expenses and results in performance deterioration. As a result, we opt to compute $L_{mix}$ using the ultimate encoder block.\nWe agree with the reviewer's assessment that the current draft lacks clarity in explaining the method. We intend to rectify this shortcoming in the forthcoming final version by revising the description accordingly.\n\n> Why can Gaussian noise make mixture-of-parts robust to common data distortions?....\n\nThrough empirical investigation, we ascertain that Gaussian noise is more effective in promoting resilience within the latent codes, while simultaneously preserving the interpratabilty of parts. In the subsequent table, we delve into the impact of salt-and-pepper and speckle noise during the pretraining phase:\n\n\n| Noise Type | 1-shot % | 5-shot %\n|--|--|--|\nGaussian | 62.15$_{\\pm 0.3}$ | 82.95$_{\\pm 0.1}$\nSalt-and-pepper | 62.21$_{\\pm 0.2}$ | 82.75$_{\\pm 0.3}$\nSpeckle | 61.60$_{\\pm 0.2}$ | 81.52$_{\\pm 0.1}$\n\nAlthough there isn't a substantial distinction in the few-shot performance during the pretraining stage, we note that the utilization of salt-and-pepper and speckle noise fails to uphold the quality of parts. This, in turn, leads to a decline in the interpretability of part representations, making it advisable to abstain from their use. For qualitative analysis, please refer to figure 1 in the uploaded rebuttal pdf.\n\n\n> What's the exact form of $M_f$ and $M_b$? Are they just binary masks?...\n\nThe segmentation masks $M_f$ and $M_b$ are composed of binary values. Our selection of the $L_2$-norm for $L_{mix}$ is informed by empirical observations. Additionally, we conducted experiments with alternative loss functions like L1 and cosine distance during the pretraining phase:\n\n\n| Mix Loss Type | 1-shot % | 5-shot %\n|--|--|--|\nL2-norm | 62.15$_{\\pm 0.2}$ | 82.95$_{\\pm 0.2}$\nL1-norm | 58.73$_{\\pm 0.3}$ | 79.51$_{\\pm 0.2}$\nCosine | (nan loss) | (nan loss)\n\nWhile L1-norm results in a lower performance, the cosine distance results in unstable training with training loss reaching +infinity, eventually nan. We achieve the best performance with L2-norm. \n\n> No limitations are listed in the paper. I suggest the authors conduct experiments on cross-domain datasets like tieredImageNet-MSCOCO (in Meta-Dataset). Such experiments can show clearer advantages of part-based methods and the removal of incidental correlations.\n\nAs suggested by the reviewer, we conduct a study on the cross-domain task. We use the setup from [2] and use MiniImageNet -> CUB evaluation (5-way-5-shot). For this analysis, we refer to the baseline performances reported in Table 9 by [2]. Moreover, we use the official code for ConstNet [3] and SUN (ViT-S) [4] to evaluate cross-domain tasks:\n\n\n| Method | Backbone | MiniImageNet -> CUB\n|--|--|--|\nNeg-Margin* | ResNet-18 | 67.3\nProtoNet* | ResNet-18 | 67.1\nRelationNet* | ResNet-18 | 57.7\nBaseline* | ResNet-18 | 65.5\nBaseline++* | ResNet-18 | 64.3\nPos-Margin* | ResNet-18 | 64.9\nMixtFSL [2] | ResNet-18 | 68.7\nSum-min [2] | ResNet-12 | 67.3\nConstNet [3] | ResNet-12 | 68.8\nSUN (ViT-S) [4] | ViT-S | 72.1\n**DPViT** | **ViT-S** | **77.1**\n\\* results are reported from [2], Table 9. \n\n\n- [1] SMKD: Supervised masked knowledge distillation for few-shot transformers. (CVPR'23)\n- [2] Matching Feature Sets for Few-Shot Image Classification. (CVPR'22)\n- [3] Attentional constellation nets for few-shot learning. (ICLR'21)\n- [4] Self-promoted supervision for few-shot transformer. (ECCV'22)\n\nMoreover, we also include limitations of our work: \n- A constraint within our framework involves relying on a pre-existing foreground extractor. In certain scenarios, such as the classification of tissue lesions for microbiology disease diagnosis, obtaining an existing foreground extractor might not be feasible. \n- At present, DPViT focuses on learning components that are connected to the data, yet it doesn't encompass the connections between these components, like their arrangement and hierarchical combination. Introducing compositional relationships among these components could enhance comprehensibility and facilitate the creation of a part-based model capable of learning relationships among the parts.\n\n> A typo in line 188, it should be $n_f \\times F^2\\cdot C$\n\nWe appreciate your observation of this typographical error. We will make the necessary corrections."}]

Author rebuttal:
We thank all reviewers for their positive feedback: proposed method tackles an important problem in a novel way with increased interpretability in representations [gyRR, Tpfu]; is clearly motivated and is technically sound [DtGy, QGVs]; achieves state-of-the-art performance with improvements over existing methods [QGVs, Tpfu]; has adepquate details/ablations [DtGy]; is nicely written and easy to follow [gyRR]. We respond to the comments of each reviewer individually below.

Meta-review: All reviewers found the proposed method to be sound and effective and the results promising. The rebuttal successfully addressed reviewer comments and all reviewers recommend acceptance. The authors are encouraged to improve the final paper version by following reviewer recommendations.
Decision: Accept (poster)