SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training Details. Our approach involves pre-training the Vision Transformer backbone and projection head using the same method described in the iBOT paper [60]. We mostly keep the hyper-parameter settings unchanged without tuning. By default, we use the Vit-Small architecture, which consists of 21 million parameters. The patch size is set to 16 as our default configuration. The student and teacher networks have a shared projection head for the [cls] token output. The projection heads for both networks have an output dimension of 8192. We adopt a linear warm-up strategy for the learning rate over 10 epochs, starting from a base value of  $5\mathrm{e} - 4$  , and then decaying it to 1e-5 using a cosine schedule. Similarly, the weight decay is decayed using a cosine schedule from 0.04 to 0.4. We employ a multi-crop strategy to improve performance with 2 global crops  $(224\times 224)$  and 10 local crops  $(96\times 96)$ . The scale ranges for global and local crops are (0.4, 1.0) and (0.05, 0.4), respectively. Following [60], we use only the local crops for self-distillation with global crops from the same image. Additionally, we apply blockwise masking to the global crops inputted into the student network. The masking ratio is uniformly sampled from [0, 1, 0.5] with a probability of 0.5, and with a probability of 0.5, it is set to 0. Our batch size is 480, with a batch size per GPU of 120. DPViT is pre-trained for 500 epochs for the given training set for all the datasets.
We use the value of  $\lambda_{cls} = 1, \lambda_s = 0.5, \lambda_o = 0.5$  for all the datasets. In the case of ImageNet-9, we incorporate the class labels by incorporating a logit head onto the projection heads. This allows us to calculate the cross-entropy loss based on the provided class labels. The explicit utilization of class labels is necessary for the ImageNet-9 dataset because the evaluation involves straightforward classification rather than few-shot learning.
Fine-tuning Details. Once the pretraining stage is completed, we proceed to train the model using the supervised contrastive loss, which involves distilling knowledge from [cls] tokens across different views of images (referred to as  $\mathcal{L}_{cls}$  in Equation 9 of the main draft). The fine-tuning process is conducted for 50 epochs using the same training data in the given dataset. We maintain the same set of hyperparameters used in the initial pretraining stage without additional tuning.
We use the value of  $\lambda_{cls}^{inv} = 1$ , and  $\lambda_p^{inv} = 0.5$  for all the datasets.
Inference Details. For inference purposes, we utilized a feature representation obtained by the [cls] token of the teacher network. We also found concatenating the weighted average pooling of the generated patches with the [cls] token useful in a few-shot evaluation. The weights for the weighted average pooling are determined by taking the average of the attention values of the [cls] token across all heads of the final attention layer.
In the case of ImageNet-9, the logit head is used to infer the class label for the given sample in the test set.
A.3 Details regarding the multi-head attention modules
The design of our attention layers draws inspiration from the standard self-attention mechanism, commonly known as qkv self-attention (SA) [11]. In our implementation, we calculate a weighted sum over all values  $\mathbf{v}$  in the input sequence  $\mathbf{z}$ , where  $\mathbf{z}$  has dimensions of  $\mathbb{R}^{N\times D}$ . The attention
weights  $A_{ij}$  are determined based on the pairwise similarity between two elements of the sequence and their corresponding query  $\mathbf{q}^i$  and key  $\mathbf{k}^j$  representations.
$$
[ \mathbf {q}, \mathbf {k}, \mathbf {v} ] = \mathbf {z} \mathbf {U} _ {q k v} \quad \mathbf {U} _ {q k v} \in \mathbb {R} ^ {D \times 3 D _ {h}}, \tag {10}
$$
$$
A = \operatorname {s o f t m a x} \left(\mathbf {q k} ^ {\top} / \sqrt {D _ {h}}\right) \quad A \in \mathbb {R} ^ {N \times N}, \tag {11}
$$
$$
S A (\mathbf {z}) = A \mathbf {v}. \tag {12}
$$
Multihead self-attention (MSA) is an expansion of the self-attention mechanism, where we perform  $k$  parallel self-attention operations, known as "heads," and then combine their outputs through concatenation. In order to maintain consistent computation and the number of parameters when adjusting the value of  $k$ , the dimension  $D_h$  (as defined in Equation 10) is typically set to  $D / k$ .
$$
M S A (\mathbf {z}) = \left[ S A _ {1} (z); S A _ {2} (z); \dots ; S A _ {k} (z) \right] \mathbf {U} _ {m s a} \quad \mathbf {U} _ {m s a} \in \mathbb {R} ^ {k \cdot D _ {h} \times D} \tag {13}
$$
A.4 Details regarding the power iterative method to compute spectral norm
We follow the power iterative method described in [3] to compute the spectral norm for  $(\mathbf{P}^T\mathbf{P} - \mathbf{I})$ . Starting with a randomly initialized  $v \in \mathbb{R}^n$ , we iteratively perform the following procedure a small number of times (2 times by default):
$$
u \leftarrow (\mathbf {P} ^ {T} \mathbf {P} - \mathbf {I}) v, v \leftarrow (\mathbf {P} ^ {T} \mathbf {P} - \mathbf {I}) u, \sigma (\mathbf {P} ^ {T} \mathbf {P} - \mathbf {I}) \leftarrow \frac {| | v | |}{| | u | |}. \tag {14}
$$
The power iterative method reduces computational cost from  $\mathcal{O}(n^3)$  to  $\mathcal{O}(mn^2)$ , which is practically much faster when used with our training procedure.
A.5 Comparing ViT-S [11] and Concept Transformer (CT) [41] on ImageNet-9
In addition to the findings presented in Section 5.4 (Table 2 in the main draft), we conducted a comparison with vanilla ViT-S pretrained onImagenet and ConceptTransformers (CT) as well. CT, as described in the study by [41], has a notable limitation in that it relies on attribute supervision for part localization information. This restriction restricts the applicability of CT in scenarios where attribute information is absent, such as in the case of ImageNet-9. To train CT without attributes, we utilized the code provided by the authors and deactivated the attribute loss, allowing CT to be trained without relying on the attribute information<sup>2</sup>. This adjustment significantly decreases the performance of CT but enables a fair comparison with other methods on ImageNet-9. It is worth noting that CT employs the ViT-S backbone pretrained on ImageNet as its default architecture. Moreover, we train ConstNet [54] using the source code provided by the authors<sup>3</sup>.
As indicated in Table 8, DPViT demonstrates superior performance compared to both ViT-S pretrained on ImageNet and CT, exhibiting a clear advantage. CT can be seen as a pretrained ViT-S model with the inclusion of part dictionaries, but it experiences a noticeable drop in performance when confronted with the presence of incidental correlations in the image backgrounds (as observed in the low M-SAME and M-RAND performance in Table 8). This demonstrates that the part learners in general cannot effectively deal with the incidental correlations of backgrounds and are susceptible to varying backgrounds.
A.6 Ablation study with different values of  $K$  and  $n_f$  on MiniImageNet
In this analysis, we investigate the impact of varying the number of parts, denoted as  $K$ , on the MiniImageNet dataset. Specifically, we explore the effects of altering the number of foreground parts, represented by  $n_f$ , as well as the number of background vectors, which can be calculated as  $K - n_f$ . Table 9 presents the obtained results, demonstrating the influence of different values of  $K$ ,  $n_f$ , and  $n_b$  on parts, foreground parts, and background parts, respectively.
2ConceptTransformer [41] - https://github.com/IBM/concept_transformer
${}^{3}$  ConstNet [54] - https://github.com/mlpc-ucsd/ConstellationNet
Method	IN-9L ↑	Original ↑	M-SAME ↑	M-RAND ↑	BG-GAP ↓
ResNet-50 [53]	94.6	96.3	89.9	75.6	14.3
WRN-50×2 [53]	95.2	97.2	90.6	78.0	12.6
ConstNet [54]	90.6	92.7	86.1	69.2	17.1
ViT-S pretrained [11]	82.5	84.9	72.2	50.3	21.9
ConceptTransformer [41]	84.7	85.5	73.1	51.5	21.6
Ours - DPViT	96.9	98.5	93.4	87.5	5.9
(b) MIXED-SAME
(c) MIXED-RAND
Figure 8: Visualizing the test splits from ImageNet-9 dataset.
Table 8: Performance evaluation on domain shift of varying background and common data corruptions on ImageNet-9. Evaluation metric is Accuracy %.
Foreground parts	K=32	K=64	K=96	K=128
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
nf = K/2	72.2±0.2	87.8±0.4	72.9±0.5	88.1±0.4	72.1±0.2	88.1±0.4	72.1±0.5	87.1±0.5
nf = 2K/3	72.2±0.2	88.1±0.4	73.8±0.5	89.3±0.4	73.1±0.2	88.1±0.4	72.2±0.5	87.4±0.5
nf = 4K/3	72.3±0.2	88.4±0.4	73.4±0.5	88.5±0.4	73.2±0.2	87.9±0.4	72.5±0.5	87.8±0.5
Table 9: Ablation of varying the number of foreground-background vectors, along with part-vectors used. We show the results on the miniImageNet dataset.
Our findings indicate that maintaining  $K = 64$  and selecting  $n_f = 2K / 3$  yields the highest performance. When employing a significantly lower number of part vectors, the model's capacity becomes insufficient, leading to performance degradation. Conversely, employing a larger value of  $K$  results in increased computational complexity associated with distance maps, subsequently leading to lower performance.
A.7 Computational complexity of DPViT
Adding part-dictionaries to MCA layers slightly increases the trainable parameters from  $21M$  (ViT-S) to  $25M$  (DPViT). It is also possible to share the attention layers, analogous to the Siamese networks, for MSA and MCA, which keeps the number of trainable parameters to  $21M$ . DPViT results in a similar performance in terms of few-shot accuracy when the attention layers are shared, as shown in Table 10.
A.8 Stage-1 pertaining comparison with SMKD [30]
Table 11 showcases the few-shot evaluation results of DPViT on the MiniImageNet dataset. In addition, we compare the performance of DPViT with the first-stage performance of SMKD [30]. It is worth noting that both DPViT and SMKD utilize the iBOT [60] pretraining strategy. However, incorporating part-dictionaries, MSA, and MCA layers in DPViT's pretraining phase contributes to its superior performance compared to SMKD.
A.9 Studying complementary properties of MSA and MCA
Based on Section 5.1 in the main draft, our study focuses on examining the complementary characteristics of MSA and MCA. MSA is designed to be effective for images containing a small number of objects, but it struggles to capture the spatial relationships among multiple objects. In contrast, MCA layers utilize distance maps to learn spatial relationships and prioritize objects without considering
Setting	1-shot ↑	5-shot ↑	||P||1 ↓	||PPT - I||1 ↓
Shared	73.6	89.6	0.4	0.5
Unshared	73.8	89.8	0.3	0.5
Table 10: Siamese DPViT. Sharing MSA and MCA layers and evaluation on MiniImageNet.
Setting	1-shot ↑	5-shot ↑
L2-norm	65.12	82.95
L1-norm	58.73	79.51
Cosine	nan	nan
Table 12: Different optimizers for  $\mathcal{L}_{\text {mix }}$ . Experimentering with different loss terms during pretraining.
Method	1-shot ↑	5-shot ↑
SMKD [30]	60.93	80.38
DPViT	62.81	83.25
Table 11: Few-shot performance after  $1^{st}$  stage pretrain phase on MiniImageNet.
Setting	1-shot ↑	5-shot ↑
Gaussian	65.12	82.95
Salt-and-pepper	62.21	82.75
Speckle	61.60	81.50
Table 13: Different noise functions for  $\delta$  introduced in Equation 4.
(b) Attention heads of MSA
Figure 9: Visualizing the attention heads for MSA and MCA.
(c) Attention heads of MCA
their specific classes. In simpler terms, MSA may overlook certain objects that are not crucial for classification, while MCA emphasizes learning spatially similar objects.
Additionally, we present the visualization of attention heads in Figure 9, 10, and 12. The MSA heads excel at identifying objects for classification but may overlook relevant objects with significant spatial context, such as the "charger" in Figure 9 and the "garbage box" in Figure 12. On the other hand, the MCA layers perform well in scenarios involving multiple objects (Figure 9 and 12), but struggle when spatially similar objects are present, as seen with the confusion between the "red grass" and the "fish" in Figure 10.
A.10 Optimization functions for  $\mathcal{L}_{\text{mix}}$
The segmentation masks  $M_{f}$  and  $M_{b}$  are composed of binary values. Our selection of the  $L_{2}$ -norm for  $L_{mix}$  is informed by empirical observations. Additionally, we conducted experiments with alternative loss functions like L1 and cosine distance during the pretraining phase. As shown in Table, while the L1-norm results in lower performance, the cosine distance results in unstable training with training loss reaching +infinity, eventually NAN. We achieve the best performance with the L2-norm.
A.11 Noise functions for  $\delta$
We conduct an ablation study on the choice of noise functions used for  $\delta$ . As shown in Table 13, we ascertain that Gaussian noise is more effective in promoting resilience within the latent codes, while simultaneously preserving the interpretability of parts when compared with salt-and-pepper and speckle noise during the pretraining phase.
A.12 Visualization foreground parts
We present additional part visualizations for Figure 10(a) and 11(a). These are shown in Figure 12(a) and 12(b).
Figure 10: Visualizing the attention heads for MSA and MCA.
(b) Attention heads of MSA
(c) Attention heads of MCA
Figure 11: Visualizing the attention heads for MSA and MCA.
A.13 Qualitative comparison of extracted patches with ConstNet [54]
In order to showcase the acquired parts of DPViT and ConstNet, we provide visualizations in Figure 13 and 14. This is achieved by selecting the nearest patches to the parts. Figure 13 illustrates the separation of foreground and background concepts accomplished by our model, whereas Figure 14 exhibits the patches surrounding the learned parts from the ConstNet model.
While DPViT learns to disentangle the foreground patches from the backgrounds, the patches extracted by ConstNet suffer from the entanglement caused due to incidental correlations of backgrounds.
(b) Parts visualizations for Figure 11(a)
Figure 12: Visualizing foreground parts learned by DPViT.
(a) Foreground patches extracted by DPViT
(b) Background patches extracted by DPViT
Figure 13: Visualizing foreground and background patches extracted by DPViT around a random foreground and background part for images from the validation set of MiniImageNet.
(a) Patches extracted by ConstNet [54]
(b) Patches extracted by ConstNet [54]
Figure 14: Visualizing patches extracted by ConstNet [54] around a random parts for images from the validation set of MiniImageNet.