SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The key patches contain the most class-related information of the input images, and the model should focus its attention on them to obtain discriminative representation in the current task, not only for the support samples, but also for the query ones. Since the position prompts (or key patches) of the query samples are unknown without label information, unlike the existing prompts [35], our position prompts can not be used in the input or middle phase of forward inference. To this end, we use the position prompts as the prediction target for attention, and optimize the cross-entropy loss between their many-hot presentation $^2$  and attention logits during fine-tuning. As shown in Equation 2 and 3, given the input tokens (e.g., [CLS] token and patch tokens) of the top layer  $\mathbf{Z}_{in}^{L} = \{\mathbf{z}_{n}\}_{n=1}^{N} \in \mathbb{R}^{N \times d}$ , we obtain the Query  $\mathbf{Q} = \{\mathbf{q}_{n}\}_{n=1}^{N} = \mathbf{Z}_{in}^{L} \mathbf{W}_{q}$  and Key  $\mathbf{K} = \{\mathbf{k}_{n}\}_{n=1}^{N} = \mathbf{Z}_{in}^{L} \mathbf{W}_{k}$  for calculating the attention logits  $\{\mathbf{q}_{i} \mathbf{k}_{j}^{T} / \sqrt{d_{k}}\}$ ,  $i, j = 1, \ldots, N$ , where  $\mathbf{W}_{q} = \text{Concat}([\mathbf{W}_{q}^{h}]_{h=1})$  and  $\mathbf{W}_{k} = \text{Concat}([\mathbf{W}_{k}^{h}]_{h=1})$ . The fine-tuning objective is
$$
\min  _ {\theta} \sum_ {\left(x ^ {s}, y ^ {s}\right) \in \mathcal {T} _ {s}} \left[ \operatorname {c e} \left(f _ {\theta} \left(x ^ {s}\right), y ^ {s}\right) - \alpha \cdot \mathcal {R} \right], \quad \mathcal {R} = \frac {1}{N \cdot | \Omega |} \sum_ {n = 1} ^ {N} \sum_ {t \in \Omega} \ln \frac {\exp \left(\mathbf {q} _ {n} \mathbf {k} _ {t} ^ {T} / \tau\right)}{\sum_ {m = 1} ^ {N} \exp \left(\mathbf {q} _ {n} \mathbf {k} _ {m} ^ {T} / \tau\right)} \tag {8}
$$
where  $\theta$  denotes tunable full or partial parameters of the model,  $\mathrm{ce}(\cdot ,\cdot)$  denotes cross-entropy classification loss,  $\alpha$  is a loss coefficient,  $\Omega$  is the index set of the key patches in  $x^{s}$ ,  $\tau$  is a temperature coefficient. We calculate the objective term  $\mathcal{R}$  only at the top layer to operate the features with more refine semantic. Specially, for the pyramidal architectures,  $\mathcal{R}$  is applied to the top layer of each stage. This loss enables the model to automatically identify the key patches and focus on them during fine-tuning, and this ability then can generalize to the query samples. The position prompts are fixed during fine-tuning since updating them only obtains trivial improvement but introduces obvious cost.
Theoretical analysis. The key patches contain the most class-related information of the input and we want to increase their information in the output feature. We first prove that the objective term  $\mathcal{R}$  can increase the information from the key patches in the input tokens  $\mathbf{Z}_{in}^{L} = \{\mathbf{z}_{n}\}_{n=1}^{N}$ . Concretely, we consider the mutual information  $I(z, z_{\Omega})$ , where the random variable  $z$  presents the input token with  $\{\mathbf{z}_n\}_{n=1}^N$  as its samples, the random variable  $z_{\Omega}$  presents the key patch with  $\{\mathbf{z}_t | t \in \Omega\}$  as its samples. The InfoNCE lower bound estimate [42, 44] of  $I(z, z_{\Omega})$  thus is
$$
\hat {I} _ {\text {I n f o N C E}} (z, z _ {\Omega}) = \frac {1}{N \cdot | \Omega |} \sum_ {n = 1} ^ {N} \sum_ {t \in \Omega} \left[ \ln \frac {f \left(\mathbf {z} _ {n} , \mathbf {z} _ {t}\right)}{\frac {1}{N} \sum_ {m = 1} ^ {N} f \left(\mathbf {z} _ {n} , \mathbf {z} _ {m}\right)} \right] \leq I (z, z _ {\Omega}) \tag {9}
$$
where the critic  $f(\cdot, \cdot)$  is used to model the density ratio and can be any positive real score function. When we use a log-bilinear function as critic, i.e.,
$$
f (\mathbf {z} _ {n}, \mathbf {z} _ {m}) = \exp \left(\mathbf {z} _ {n} \mathbf {W} _ {q} \mathbf {W} _ {k} ^ {T} \mathbf {z} _ {m} ^ {T} / \tau\right) = \exp \left(\mathbf {q} _ {n} \mathbf {k} _ {m} ^ {T} / \tau\right), \quad \forall n, m \in \{1, \dots , N \} \tag {10}
$$
2For index set  $\Omega$  , it is a vector of length  $|\Omega |$  , with the values of the positions in  $\Omega$  be 1 and other ones be 0.
Table 2: Few-shot classification accuracy (%) on 20-way 1-shot / 5-shot tasks. We use the Swin-T/7 pre-trained on ImageNet-1k with iBOT for backbone initialization.
Model	CUB	Cars	Places	Plantae
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
MetaOptNet [32]	49.1	81.1	21.0	51.1	48.9	70.4	36.0	63.1
FT	52.7	87.1	20.9	59.7	48.1	69.3	35.9	65.9
LoRA [24]	53.6	87.4	20.4	57.9	48.9	69.5	37.1	65.8
SSF [34]	54.9	87.6	21.4	57.2	48.7	70.8	35.9	64.7
FT + FORT	59.1 (6.4)	88.3 (1.2)	22.3 (1.4)	61.6 (1.9)	49.3 (1.2)	70.4 (1.1)	37.2 (1.3)	66.9 (1.0)
LoRA + FORT	59.9 (6.3)	88.5 (1.1)	21.9 (1.5)	59.4 (1.5)	49.7 (0.8)	70.4 (0.9)	38.7 (1.6)	66.9 (1.1)
SSF + FORT	60.1 (5.2)	88.9 (1.3)	23.4 (2.0)	58.8 (1.6)	49.8 (1.1)	72.0 (1.2)	37.6 (1.7)	66.0 (1.3)
Table 3: Few-shot classification accuracy (%) on 20-way 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained on WIT [45] with CLIP for backbone initialization.
Model	CUB	Cars	Places	Plantae
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
MetaOptNet [32]	68.0	88.7	67.6	90.7	51.6	73.8	46.2	73.0
zero-shot [45]	84.1	84.1	88.0	88.0	76.6	76.6	61.2	61.2
CoOp [78]	84.4	90.4	91.3	94.6	77.3	81.1	63.8	76.2
Tip-Adapter-F [73]	86.9	92.0	92.2	95.3	79.8	82.0	68.3	79.3
PLOT++ [7]	87.4	92.0	92.2	95.5	79.9	82.7	67.7	78.8
LoRA [24]	86.3	92.6	92.3	95.8	79.8	84.1	67.4	80.0
LoRA + FORT	87.8 (1.5)	93.8 (1.2)	93.6 (1.3)	97.0 (1.2)	80.6 (0.8)	84.9 (0.8)	68.5 (1.1)	81.0 (1.0)
the objective term  $\mathcal{R}$  satisfies
$$
\mathcal {R} = \hat {I} _ {I n f o N C E} (z, z _ {\Omega}) - \ln N \tag {11}
$$
Therefore, increasing the objective term  $\mathcal{R}$  can increase the mutual information between the input tokens and key patches. On the other hand, the objective term  $\mathcal{R}$  can also be derived as
$$
\mathcal {R} = \mathcal {R} ^ {U} + \mathcal {R} ^ {A}, \quad \mathcal {R} ^ {U} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \ln \sum_ {m = 1} ^ {N} \exp \left(\mathbf {q} _ {n} \mathbf {k} _ {m} ^ {T} / \tau\right), \mathcal {R} ^ {A} = \frac {1}{N \cdot | \Omega |} \sum_ {n = 1} ^ {N} \sum_ {t \in \Omega} \mathbf {q} _ {n} \mathbf {k} _ {t} ^ {T} / \tau \tag {12}
$$
Increasing  $\mathcal{R}^U$  reduces the similarity between input token pairs and obtains a more uniform distribution of input tokens, which thus increases the entropy  $H(z)$ . Increasing  $\mathcal{R}^A$  aligns the input tokens with the key patches and makes them more similar, which thus decreases the conditional entropy  $H(z|z_{\Omega})$ . Since  $I(z,z_{\Omega}) = H(z) - H(z|z_{\Omega})$ , the objective term  $\mathcal{R}$  can increase the mutual information  $I(z,z_{\Omega})$ . In fact, if  $p(\mathbf{z}_n|\mathbf{z}_m)\propto \exp (\mathbf{z}_n\mathbf{W}_q\mathbf{W}_k^T\mathbf{z}_m^T /\tau),n,m\in \{1,\ldots ,N\}$ ,  $\mathcal{R}^U$  and  $\mathcal{R}^A$  have constant bias with the Monte-Carlo estimate of  $H(z)$  and  $-H(z|z_{\Omega})$  respectively, see derivation in appendix. Finally, the objective term  $\mathcal{R}$  increases the information from the key patches in the output feature (e.g., [CLS] token or average patch tokens) from two aspects: 1) it increases the attention scores of the output feature to the key patches, which directly increases their contribution and also reduces the noise information from other patches, as shown in Equation 4; 2) it increases the information from the key patches in the input tokens, which further transports to the output feature.
4.1 Few-shot classification
Settings. The experiments are conducted on the few-shot benchmark from [58] (i.e., CUB [61], Cars [30], Places [75] and Plantae [59] datasets), and two other fine-gained datasets: Aircraft [39] and Pets [43]. We use the pre-trained ViT-B/16 released by DINO [6] and CLIP [45] and Swin-T/7 released by iBOT [76] for examining different architectures and pre-training ways. Here we only consider the contrastive pre-trained models [6] since their representations have better linear separability than the generative pre-trained models [18, 68] and are more suitable for few-shot classification. We evaluate both the backbone-frozen methods, e.g., machine learning classifiers (Nearest Neighbor classifier, Ridge Regression and Support Vector Machine), meta-solvers (ProtoNet [52], R2D2 [4]
DINO
CLIP
Figure 3: The attention map of [CLS] token in the pre-trained ViT-B/16 from DINO and CLIP, and the position prompts (red patches) obtained using the attention w/ or w/o gradient information.
$\therefore m - 1 \neq  0$  ;
$\therefore m = \frac{3}{11}$  ;
Attention+Grad Prompts
$\therefore m = \frac{3}{11}$
and MetaOptNet [32]) and linear probing, and full or parameter-efficient fine-tuning, e.g., VPT [26], LoRA [24] and SSF [34]. For the vision-language pre-trained model from CLIP, we also evaluate the specially-designed models: CoOp [78], Tip-Adapter-F [73] and PLOT++ [7], and use 'a photo of a [CLASS]' as the prompts for classifier initialization in the fine-tuning methods. The language prompts are unusable for DINO and iBOT, so we use the classifier initialization based on MetaOptNet. The number of the key patches  $P$  is set to 14 for ViT-B/16 and 7 for Swin-T/7. We verify the effectiveness of our model, FORT, on the fine-tuning methods. All models are evaluated in the 20-way 1-shot / 5-shot settings, instead of overly-easy 5-way settings, using 2,000 randomly sampled tasks with 15 query samples per class, and we report the average accuracy. All experiments can be conducted on single Tesla V100 with 32GB memory. More fine-tuning details are provided in Appendix B, where we also provide the results on 5-way settings for comparison with traditional setting.
Table 4: Few-shot classification accuracy (%) on 20-way 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained on ImageNet-1K with DINO for backbone initialization.
Model	Aircraft	Pets
1-shot	5-shot	1-shot	5-shot
MetaOptNet [32]	25.5	55.5	71.9	89.6
FT	27.1	64.6	71.6	89.0
LoRA [24]	26.0	62.7	72.7	89.8
SSF [34]	26.5	61.6	72.7	90.0
FT + FORT	28.7	66.0	72.6	89.9
LoRA + FORT	31.3	63.9	74.8	90.9
SSF + FORT	30.9	63.1	74.4	91.2
Table 5: Attention-weighted summation of importance scores  $(\times 10^{-2})$  on 20-way 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained on ImageNet-1K with DINO for initialization.
Model	1-shot	5-shot
support	query	support	query
Orig Attn	0.72	0.72	0.72	0.72
FT	0.74	0.73	0.74	0.75
LoRA [24]	0.71	0.71	0.75	0.76
SSF [34]	0.74	0.73	0.79	0.79
FT + FORT	0.80	0.77	0.90	0.91
LoRA + FORT	1.00	0.97	0.96	0.97
SSF + FORT	1.06	1.03	1.04	1.04
Results. The results are provided in Table 1, 2, 3 and 4, where the  $95\%$  confidence interval is  $\pm 0.3$  (1-shot) or  $\pm 0.2$  (5-shot) and the number in parentheses is the increment of accuracy beyond the base fine-tuning methods. As shown in Table 1, 2 and 4, with classifier initialization, the fine-tuning methods typically can achieve better performance than the model-frozen methods, since the backbone is further adapted to task distribution. However, fine-tuning sometimes struggles to achieve non-trivial improvement beyond the classifier initialization (i.e., MetaOptNet) or even performs worse, especially on the 1-shot tasks. Our FORT can help the fine-tuning methods to obtain significant improvement on both ViT and Swin. Further, FORT typically achieves more improvement on 1-shot tasks than that on 5-shot tasks. This is intuitive, since with more support samples the model can attend to the frequently-occurring entities more in each class, which are typically the key entities. FORT is more suitable for solving the hardest 1-shot tasks. As shown in Table 3, on the vision-language pre-trained model, fine-tuning with simple classifier head (i.e., LoRA) achieves comparable or better performance than the specifically-designed methods: CoOp, Tip-Adapter-F and PLOT++. In fact, Tip-Adapter-F uses the ensemble of a linear classifier head and the similarity-based label propagation module. PLOT++ introduces multiple learnable language prompts and visual prompts for obtaining multiple classifier weights and image embeddings, and then calculates their optimal-transport distance for classification, which has huge computational burden. Our FORT can further improve the performance beyond the fine-tuning methods. Finally, FORT is compatible with the parameter-efficient fine-tuning even better than with full fine-tuning, which is convenient for industrial deployment.
Figure 4: Visualization of the patches (white parts) with top highest attention scores and covering about  $95\%$  attention of [CLS] token in ViT-B/16. We use DINO pre-trained model for initialization.
Position prompts. The different pre-training ways could lead to different attention behavior in the vision transformers. Figure 3 'Attention' shows the attention map of [CLS] token in the top layer of pre-trained ViT-B/16 from both the single-modal (DINO) and multi-modal (CLIP) methods. DINO can attend to the foreground objects with the highest attention scores, while CLIP typically attends to more parts besides them. This is intuitive, since the supervision signals for the image encoder in CLIP are the sentence descriptions, which typically involve both the foreground objects and scene, e.g., 'a bird flies over the sea'. Importantly, the key entities typically can not get the highest attention scores in CLIP pre-trained model, which makes it difficult to accurately locate them only based on the attention information. Figure 3 'Attention Prompts' shows the position prompts obtained using only the attention scores, i.e.,  $\lambda = 0$  in Equation 6. For DINO, the obtained position prompts can indicate the key entities, while for CLIP, the obtained position prompts could point to other class-independent parts besides the key entities. To this end, we introduce the gradient information as an auxiliary. Figure 3 'Attention+Grad Prompts' shows the position prompts obtained using both the attention and gradient information, where  $\lambda = 1$  for DINO and  $\lambda = 50$  for CLIP. We use MetaOptNet to calculate the classifier weight for DINO, and use language prompts for CLIP. The gradient information helps to accurately locate the key entities in CLIP, and also improve that in DINO.
Attention enhancement. We conduct qualitative and quantitative experiments to verify that our method can enhance the attention of the model to the key patches. Qualitatively, Figure 4 visualizes the patches with top highest attention scores of [CLS] token in ViT-B/16. Concretely, when the attention scores of all patches are  $a_1, \ldots, a_N$  in descending order, we choose the patches  $\{1, \ldots, n\}$ , which satisfy  $S_{n-1} < 0.95 \leq S_n$  with  $S_n = \sum_{i=1}^n a_i$  and  $S_N = 1$ . We use the pre-trained model from DINO for backbone initialization and fine-tune on random 1-shot tasks. We provide the results of original attention ('Orig Attn') and the attention after fine-tuning w/ ('Our Attn') or w/o ('FT Attn') our method. The original model can not focus its attention on the key patches, nor can the full or parameter-efficient fine-tuning achieve this property. Our method can prompt the model the positions of key patches and help the model to focus most attention (95%) on them, and this ability can generalize from the support set to query samples. This phenomenon is not obvious on full fine-tuning, and the reason may be it uses smaller learning rate and less epochs than LoRA and SSF to avoid over-fitting. The full attention maps are provided in appendix and we can observe the same phenomenon. Further, we examine the attention-weighted summation of importance scores, i.e.,  $\sum_{i=1}^{N} \mathbf{s}_i * \mathbf{a}_i^0$ , where s is defined in Equation 7 and  $\mathbf{a}^0$  is its position-aligned attention scores. The larger the value, the more focused on the key patches the attention. Table 5 provides the results averaged on CUB, Cars, Places and Plantae datasets with 2000 tasks each setting. The fine-tuning methods obtain the similar scores with original attention, while our FORT achieves non-trivial increase. Consistently, the increment on full fine-tuning is smaller than that on LoRA and SSF. For standard fine-tuning, the increment on 5-shot tasks is larger than that on 1-shot tasks. In fact, with more support samples, the model can attend to the frequently-occurring key entities in each class.
Importance calculation. In Section 3.4, we combine both gradient and attention information for calculating the importance score of each patch. Here we ablate the different information item and compare their performance. We follow the experimental settings in Section 4.1 and use the per-trained models from DINO or CLIP for initialization. The results are provided in Table 6, where 'FORT (attn)' denotes using only attention information (i.e., setting  $\lambda = 0$  in Equation 6), 'FORT (grad)' denotes using only gradient term G, 'FORT (attn+grad)' denotes using both attention and gradient information, that is our final method. Here we pick the best hyper-parameters for each method, even though they may differ from each other. For DINO pre-trained models, the attention information
Table 6: Few-shot classification accuracy (%) on 20-way 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained by DINO or CLIP for backbone initialization.
Model	DINO	CLIP
CUB	Cars	CUB	Cars
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
LoRA	57.9	88.2	23.3	64.3	86.3	92.6	92.3	95.8
+ FORT (attn)	62.1	89.0	26.3	65.0	86.6	92.6	92.5	95.8
+ FORT (grad)	60.0	88.8	25.2	64.7	87.8	93.8	93.6	97.0
+ FORT (attn+grad)	62.5	89.5	26.8	65.7	87.8	93.8	93.6	97.0
(b)  $\tau$
(c)  $\alpha$
Figure 5: Ablation studies on hyper-parameter  $\lambda$ ,  $\tau$  and  $\alpha$ . We examine the absolute increment of accuracy beyond the base fine-tuning methods averaged on CUB, Cars, Places and Plantae datasets.
can effectively locate the key patches and contributes more to the performance than the gradient information. For CLIP pre-trained models, the attention information can not indicate the positions of the key patches and thus the gradient information plays a major role. Combining both attention and gradient information works for different pre-trained models and can obtain the best performance.
Hyper-parameters. We conduct ablation studies on the hyper-parameters from Equation 6 and 8:  $\lambda$ ,  $\tau$  and  $\alpha$ , and examine the absolute increment of accuracy beyond the base fine-tuning methods averaged on CUB, Cars, Places and Plantae datasets. For each hyper-parameter, we set the other ones to its corresponding best values. For  $\lambda$ , we use LoRA as the base fine-tuning method and evaluate different pre-trained models, i.e., ViT-B/16 from DINO and CLIP. For  $\tau$  and  $\alpha$ , we use the ViT-B/16 from DINO and evaluate different base fine-tuning methods, i.e., full fine-tuning, LoRA and SSF. Figure 5 (a) shows that using DINO pre-trained model is suitable for small  $\lambda$  since its attention is enough for locating key patches, while using CLIP pre-trained model needs large  $\lambda$  to utilize gradient information. Figure 5 (b) shows that our method is generally robust to the temperature coefficient  $\tau$ , and the performance on 1-shot tasks is relatively more sensitive than that on 5-shot tasks. Figure 5 (c) shows that the loss coefficient  $\alpha$  is an important hyper-parameter and needs to be carefully chosen, and the performance on 5-shot tasks is more sensitive than that on 1-shot tasks. In fact, if the coefficient  $\alpha$  is too large, the attention enhancement loss could relatively inhibit the optimization of the classification loss (e.g., skew gradient direction), which is harmful to fitting the input-label distribution. The more training samples, the more important it is to fit input-label distribution, so the 5-shot tasks prefer smaller coefficient  $\alpha$  than 1-shot tasks.
In this work, we aim to adapt the pre-trained vision transformers to few-shot image classification tasks. It is nontrivial, since even with well classifier initialization, the standard fine-tuning methods sometimes can not obtain significant improvement beyond the initialization, especially on 1-shot tasks. To this end, we propose a new prompt for vision data, position prompts, to explicitly prompt the model where are the key entities and focus on them during fine-tuning. Our method can effectively improve the performance of full or parameter-efficient fine-tuning on few-shot tasks. The limitations of our methods are: 1) our method is suitable for multi-head attention module, and unusable for other vision backbones without attention, e.g., ResNet [20] or MLP-Mixer [57]; 2) for some professional fields, e.g., medical imaging, we may still need experts to manually label the position prompts.
Table 7: Candidate values of hyper-parameters for fine-tuning.
hyper-parameter	value
learning rate	1e-5, 3e-5, 5e-5, 8e-5, 1e-4, 3e-4, 5e-4, 8e-4, 1e-3, 3e-3, 5e-3, 8e-3, 1e-2
#epoch	5, 10, 15, 20
temperature τ	1, 5, 10, 15, 20
coefficient α	5e-4, 8e-4, 1e-3, 3e-3, 5e-3, 8e-3, 1e-2, 3e-2, 5e-2, 8e-2, 1e-1, 3e-1, 5e-1, 8e-1, 1
A Derivation of Monte-Carlo estimate
In Section 3.5, we claim that if  $p(\mathbf{z}_n|\mathbf{z}_m) \propto \exp (\mathbf{z}_n\mathbf{W}_q\mathbf{W}_k^T\mathbf{z}_m^T /\tau) = \exp (\mathbf{q}_n\mathbf{k}_m^T /\tau), n,m \in \{1,\dots ,N\}$ ,  $\mathcal{R}^U$  and  $\mathcal{R}^A$  have constant bias with the Monte-Carlo estimate of  $H(z)$  and  $-H(z|z_{\Omega})$  respectively, where
$$
\mathcal {R} ^ {U} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \ln \sum_ {m = 1} ^ {N} \exp \left(\mathbf {q} _ {n} \mathbf {k} _ {m} ^ {T} / \tau\right), \quad \mathcal {R} ^ {A} = \frac {1}{N \cdot | \Omega |} \sum_ {n = 1} ^ {N} \sum_ {t \in \Omega} \mathbf {q} _ {n} \mathbf {k} _ {t} ^ {T} / \tau \tag {13}
$$
$$
p (z | \hat {z}) = \frac {\exp \left(z \mathbf {W} _ {q} \mathbf {W} _ {k} ^ {T} \hat {z} ^ {T} / \tau\right)}{Z (\hat {z})}, \quad p (z | z _ {\Omega}) = \frac {\exp \left(z \mathbf {W} _ {q} \mathbf {W} _ {k} ^ {T} z _ {\Omega} ^ {T} / \tau\right)}{Z (z _ {\Omega})} \tag {14}
$$
where the random variable  $\hat{z}$  follows the same distribution as  $z$  and also has  $\{\mathbf{z}_m\}_{m=1}^N$  as its some samples, and  $Z(\cdot)$  is the partition function for normalization. Here we assume that the energy state remains the same for different  $\hat{z}$  or  $z_{\Omega}$ , i.e.,  $Z(\cdot)$  keeps a constant  $Z$ . The detailed derivations are
$$
\begin{array}{l} H (z) = - \mathbb {E} _ {p (z)} \ln p (z) = - \mathbb {E} _ {p (z)} \ln \mathbb {E} _ {p (\hat {z})} p (z | \hat {z}) \\ \approx - \frac {1}{N} \sum_ {n = 1} ^ {N} \ln \mathbb {E} _ {p (\hat {z})} p (\mathbf {z} _ {n} | \hat {z}) \\ \approx - \frac {1}{N} \sum_ {n = 1} ^ {N} \ln \frac {1}{N} \sum_ {m = 1} ^ {N} p (\mathbf {z} _ {n} | \mathbf {z} _ {m}) \\ \approx - \frac {1}{N} \sum_ {n = 1} ^ {N} \ln \sum_ {m = 1} ^ {N} \exp \left(\mathbf {q} _ {n} \mathbf {k} _ {m} ^ {T} / \tau\right) + \ln N Z \\ \approx \mathcal {R} ^ {U} + \ln N Z \tag {15} \\ \end{array}
$$
$$
\begin{array}{l} - H (z | z _ {\Omega}) = \mathbb {E} _ {p (z, z _ {\Omega})} \ln p (z | z _ {\Omega}) \\ \approx \frac {1}{N \cdot | \Omega |} \sum_ {n = 1} ^ {N} \sum_ {t \in \Omega} \ln p (\mathbf {z} _ {n} | \mathbf {z} _ {t}) \\ \approx \frac {1}{N \cdot | \Omega |} \sum_ {n = 1} ^ {N} \sum_ {t \in \Omega} \mathbf {q} _ {n} \mathbf {k} _ {t} ^ {T} / \tau - \ln Z \\ \approx \mathcal {R} ^ {A} - \ln Z \tag {16} \\ \end{array}
$$
From these results, the above claim is proved. Besides, we have  $I(z,z_{\Omega}) = H(z) - H(z|z_{\Omega}) \approx \mathcal{R}^{U} + \mathcal{R}^{A} + \ln N = \mathcal{R} + \ln N$ , i.e., we obtain the Equation 11 again.
B More experiments
Fine-tuning details. The resolution of input images is  $224 \times 224$ , and we do not use data augmentation following the standard setting. In fact, we find the base data augmentation (e.g., random resized cropping, flip or random erasing) can not obtain improvement on the chosen datasets of our