Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning
Authors: Shuai Yi
Yixiong Zou
Yuhua Li
Ruixuan Li
Venue: ICML 2025 spotlightposter (2025)
Keywords: Cross-Domain Few-Shot Learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We find a phenomenon that disrupting image patches' continuity (e.g., shuffle patches) affects differently on source and target domains. We delve into it for an interpretation and propose a method based on it for CDFSL.

Abstract: Vision Transformer (ViT) has achieved remarkable success due to its large-scale pretraining on general domains, but it still faces challenges when applying it to downstream distant domains that have only scarce training data, which gives rise to the Cross-Domain Few-Shot Learning (CDFSL) task. Inspired by Self-Attention's insensitivity to token orders, we find an interesting phenomenon neglected in current works: disrupting the continuity of image tokens (i.e., making pixels not smoothly transited across patches) in ViT leads to a noticeable performance decline in the general (source) domain but only a marginal decrease in downstream target domains. This questions the role of image tokens' continuity in ViT's generalization under large domain gaps. In this paper, we delve into this phenomenon for an interpretation. We find continuity aids ViT in learning larger spatial patterns, which are harder to transfer than smaller ones, enlarging domain distances. Meanwhile, it implies that only smaller patterns within each patch could be transferred under extreme domain gaps. Based on this interpretation, we further propose a simple yet effective method for CDFSL that better disrupts the continuity of image tokens, encouraging the model to rely less on large patterns and more on smaller ones. Extensive experiments show the effectiveness of our method in reducing domain gaps and outperforming state-of-the-art works. Codes and models are available at https://github.com/shuaiyi308/ReCIT.

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
Shuai Y
Yixiong Zou Yuhua Li
Ruixuan Li
Abstract
Vision Transformer (ViT) has achieved remark- able success due its large-scale pretraining on general domains, but it still faces challenges when applying it to downstream distant domains that have only scarce training data, which gives rise to Few-Shot Learning (CDFSL) task. Inspired by Selff-Attentoon's 's insensitivity token orders, find interesting phenomenon neglected in current works: disrupting the con- tinuity of image tokens (i.e., making pixels not smoothly transited across patches) in VIT leads to a noticeable performance decline in the general (source) domain but only a marginal decrease downstream target domains. This questions the role of image tokens' continuity in VIT's gener- alization under large domain gaps. In this paper, we delve into this phenomenon for an interpre- tation. We find continuity aids VIT in learning larger spatial patterns, which are harder to trans- fer than smaller ones, enlarging domain distances. Meanwhile, implies that only smaller patterns within each patch could be transferred under ex- treme domain gaps. Based on this interpretation. further propose simple yet effective method for CDFSL that better disrupts the continuity of image tokens, encouraging the model to rely less on large patterns and more on smaller ones. Ex- tensive experiments show the effectiveness of our method reducing domain gaps and outperform- ing state-of-the-art works. Codes and models are available at
Original
Remeve
Shuffle Patches
Shottl
Sauffie
Patch Amplitude
Patch Phase
(a) Four strateggie for breaking continuity of ol'inage pulches
Original
the
0.75
Remove Posttion Embedding
Shaffle Patches
Shuttle Patch Amplitude
0.25
Shuffle Patch Phase
Nourco-Domsin
Tange-Demssin
(b) Impact on source and target domains
Figure 1. (a) Four approaches are utilized disrupt continuity of image tokens, i.e. making the pixels smoothly transited across patches. (b) We find an interesting phenomenon: although disrupting the continuity of image tokens in the source domain has a substantial impact on the performance of ViT-based models, the model's performance in the target domain, which undergoes an equivalent level of continuity disruption, is only marginally affected. In this paper, we will delve into this phenomenon for an interpretation, explore the role of image tokens' continuity in model generalization, and propose methods based on better cross doss-n few-shot learning.
1. Introduction
Vision Transformer (ViT) has achieved great success across numerous tasks (Yuan et al., 2021; Wu et al., 2022) because of its ability to learn from large-scale datasets (Naseer et al., 2021), which makes it prevailing option for downstream applications by generalizing upstream- pretrainde VIT to downstream expert tasks. However, in real-world scenar- ios, downstream tasks can be in domains distant from up- stream large-scale datasets, and it may not be easy for down- stream tasks to collect sufficient training samples, which makes challenging for the generalization and finetuning of VIT (Zou al., 2022; 2024b). To mitigate this issue, Cross- Domain Few-Shot Learning (CDFSL) has been proposed, aiming to transfer general knowledge from source domains, such as ImageNet (Deng et al., 2009). to target domains, like medical datasets (Mohanty et al., 2016), that possess only a scarcity of training samples (Oh et al., 2022).
'School Computer Science and Technology, Huazhong Uni- versity of Science and Technology. Wuhan, China School of Artifi- cial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. Corresoondence to: Yixiong Zou <yixiongz @ hust. edu.en>> Yuhua <ideliyuhua @ weshhuuuuuu edu.cn.. Ruixuan Li <rxli @ www.t.ddu.c.>...........
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
However, the generalization of ViT under large domain

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
gaps is still under-explored. Different from Convolutional Neural Networks (CNN), ViT partitions the image into non- overlapping patches for input and takes Multi-head Self- Attention (MSA) (Chen et al., 2023) to model the coherence of patches. However, MSA itself is not sensitive to the order of input tokens, i.c. two subsequent tokens do not have to be continuous in pixels. The only assurance of continu- ity is the positional embeddings (Dosovitskiy et al., 2021), which is relatively weak. Inspired by this characteristic, we find an intriguing phenomenon that is ignored by current works: when the continuity of images is disrupted e.g., by removing positional embeddings, shuffling image patches, or shuffling the amplitude or phase of image patches in the frequency domain (Fig. Ia), the performance of ViT experiences noticeable decline in the source domain but decreases only marginally on target domains (Fig. lb). This phenomenon questions the role of image tokens' continuity in ViT's generalization under large domain gaps.
decline when disrupting image tokens' continuity on target domains than on general (source) domains.
We delve into this phenomenon for an interpretation: continuity aids VIT in learning large spatial patterns; however, under large domain gaps, large patterns across patches are hard to transfer, making the disrup- tion of large patterns less effective on target domains.
Based on this interpretation. we further propose novel method to better disrupt the continuity of image tokens for CDFSL, thereby enabling the model to reduce its reliance on large patterns and enhance its learning of smaller ones, thereby enhancing its transferability.
Extensive experiments on four benchmark datasets val- idate our rationale and state-of ofte-of-thhe-rr performance.
2. Delve into Continuity of Image Tokens in Cross-Domain Few-Shot Learning
In this paper, we delve into this phenomenon for an interpre- tation. We discover that disrupting image tokens' continuity can paradoxically be beneficial in reducing domain gaps. Then, we find that the more disrupted image tokens' con- tinuity (by binding fewer patches not disrupted), the more generally increased domain similarity between source and target domains. Based on these experiments, we interpret the continuity as an aid of ViT in learning larger spatial patterns. However, since large spatial patterns, c.g., a whole dog, are harder to transfer to target domains than smaller pat- terns, e.g., the head of a dog, under extremely large domain gaps, this phenomenon implies that only smaller patterns within each patch could be transferred to target domains, therefore interrupting image tokens' continuity has only marginal effect on target-domain performance.
2.1. Preliminaries
Cross-Domain Few-Shot Learning (CDFSL) entails a model to acquire knowledge from a source-domain dataset abun- dant with training samples (c.g., minilmageNet (Vinyals et al., 2016)) then transfer it to downstream tasks, enabling learning of target-domain datasets using merely a handful of training instances. Finally, the model is evaluated on target datasets.
Specifically, we denote the source dataset as D {w. yt
) Ji=1 with xf and yf symbolizing the ith training sample and its corresponding label, respectively. Analo- gously, DT = {>T vu ) represents the target dataset. During the learning and evaluation phases on D" to ensure a fair comparison, current research (Fu et al., 2022; Zou et al., b) employs a k-way n-shot paradigm. This involves sampling from D" construct limited datasets, known as episodes, each comprising k classes with training sam- ples per class. Based on these episodes, the model learns from the A * n samples, collectively termed the support set {z] Miy Tik,n === and its performance is assessed using testing samples from the same k classes, referred to as the query set {x
Drawing upon this interpretation, we further propose a sim- ple but effective method tailored for the CDFSL task that better disrupts the continuity of image tokens, encourag- ing the model strengthen the learning of smaller spatial patterns and reduce its reliance on large ones, thereby en- hancing the model's generalization to downstream tasks. Specifically, we integrate the continuity disruption of image tokens in both spatial and frequency domains, and construct a balanced disruption among different style distributions, ensuring the diversity of the disruption. Extensive experi- ments four CDFSL benchmarks with large domain gaps show that we can outperform ofth-affthhe-rt performance and effectively reduce domain gaps.
The Vision Transformer (ViT) has recently gained signif- icant popularity in vision-related tasks. It operates by di- viding an image E R&# xC into fixed-sized patches Ip Mx Where (HI W) denotes the resolution of the original image, C represents the number of chan- nels, (P, P) signifies the resolution of each image patch, and M = HWWP22 is the resulting count of patches. This count, M, also functions as the number of input tokens for the Transformer. Then each image patch is flattened and projected into a D-dimensional space through a trainable
In summary, our contributions can be listed as follows.
To the best of our knowledge, we are the first to con- sider image tokens' continuity in the CDFSL task.
We find a phenomenon that is neglected by others: pretrained VIT undergoes a much less performance
2

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
linear projection E E ((PP)))x termed patch embed- dings. Additionally, learnable embedding called class token (denoted as Telass) is prepended to the sequence of patch embeddings. To maintain positional information, the patch embeddings are added with position emheddings Epss E I((++1)xD, which can be represented as
Original
Remove Posiion Embbolding
0.4
Shutfle Potches
0.2
Shufi Purch Amplitude
Shuffll Punch Phose
20
delassi T Ei
:2p MMM E}
+ Epos.
(L)
ChopDiszesss
EumSAT
ISIC2018
ChestX
Ave.
Domain Similarity measured by the CKA
The resultant sequence of embedding vectors subsequently serves as the input an encoder architecture that consists of L stacked blocks, each encompassing a multiheaded self- attention (MSA) network, Multi-Layer Perceptron (MLP) network, Layernorm (LN) and residual connections. The overall process can be depicted as follows
Figure 2. Disrupting the continuity of input images significantly increases domain similarity, although the performance decreases on all datasets in Fig. 1. Intriguingly, the more decrease Fig. I, the more increase in domain similarity, which shows disrupting continuity can surprisingly mitigate domain discrepancies.
class token, and then adding positional embeddings before inputting them into the encoder:
z| MSA (LN (2-1)) + 21-1:
(2)
21 MLP (LN (24)) + 2/1
(3)
f(xf) = LN (EL).
(4)
20 = Telass Fp E; x E
:Tp
E
+ E
pos+
(8)
In this paper, we focus on exploring ViT's downstream generalization on the CDFSL task. We follow (Fu et al., 2023; Zou et al., a) to employ the DINO (Zhang et al., 2022) pretraining on ImageNet (Deng et al., 2009) as the initialization. Then, we train the ViT on D by minimizing the cross-entropy loss relevant the source-domain label space | with fully connected (FC) layer as
Shuffle Patch Amplitude (SPA): The two subsequent meth- ods involve transitioning image patches from the spatial domain the frequency domain, starting with obtaining the Fourier transformations of the input patches Tp
1''1W'-
F(sp)|m,n]
Σ Σ h- w-
2p w/ exexpp 2n
h H
W
(9)
N Lote (o((()(()))))
Σ L= N
(5) where ,2 = and m. denote spatial frequencies. When Re(F -J) and [m(F(x)[:--) represent the real and imaginary components of the Fourier spectrum, respectively, the corresponding amplitude spectrum A(x) and phase spectrum P. can be expressed as follows
where <(-) represents the FC layer and f(+) denotes VIT. Finally, employ prototype-based classification (Zhou et al., 2023) for target-domain recognition with a distance function .(.,.) as
A(x,)mm, n| v [Ra(F(ap) nl)² T'm (())]m)))) (10)
P(x,)]mm = arctan [mm(F(en)(mm m!)
(1))
vo = arg min d(- 1 E((TT)
[e((F(z)[(mm))
(6)
Our third method disrupts the amplitudes of the patches within an image, merges them with the original phases of these patches, and then applies an inverse Fourier transform to revert the combined data back to the spatial domain as
2.2. Breaking the continuity of image tokens
As illustrated Fig. la, we first contemplate four of the simplest approaches to disrupt the continuity of images, applying each separately the training set within the source domain to facilitate the model's training that domain.
2 = iDFT (Aay @
(12)
where & denote kth patch in image, k'' denote after be shulffed, the kith patch's amplitude.
Remove Position Embedding (RPE): The first method con- sists of inputting the image patches directly into the encoder of ViT, without incorporating positional embeddings as
Shuffle Patch Phase (SPP): The fourth method, in contrast to the third one, disrupts continuity by shuffling the phases while retaining the amplitudes as
20- Felnssi 1r
M
iP -+8)) #). AAgy 2
x = iDFT
(13)
Shuffle Patches (SP): The second approach involves shuf- fling the image patches directly, concatenating them with
3

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
1.00
remove posiiion ambosdiing
0.88
common feature
0.75
within patches: e.g. fish
stuffle patcites
0.63
Sourco-Domain
Target-Domain
ofter stuftle putch ampiindde
0.50
|*1
2*2
4+4
7*7
8*8
14*14
(a) Degrec of disruption 5-shot performance
eñei FTT shuffl paich plase
0.3
Figure 3. Take a fish as an example, although disrupting continuity distorts its overall shape, it is still feasible recognize the fish's patterns in individual patches, such as fins and eyes. This indicates that the continuity between patches primarily assists the model in learning larger spatial patterns; however, even after disrupting the continuity, the model can only recognize the patterns maintained within each patch, which is smaller but easier to transfer.
0.1
CropDiseases
EuroSAT
ISIC2018
ChestX
Ave. 14 14
II*1 2*2 m 4 7**
8*8
(b)Domain Similarity measured by the CKA
2.3. What role does ViT's weak continuity play under large domain gaps?
Figure 4. We sequentially divide the images into where the pseudo-patch size decreases from left right, and shuf- fle them. The smaller the patterns preserved within these pseudo- patches, (a) the more the model's performance decreases, and (b) the more the domain similarity increases. This suggests that breaking the continuity essentially influences the spatial size maintained patterns, where larger ones are discriminative on the source domain harder to transfer, and vice versa.
In Fig. 1, we observe that pretrained ViT is highly sensitive to disruptions in continuity the source domain that is similar to its pretraining data, yet exhibits a much smaller impact on target domains that are distant from the source domain. Therefore, we are inspired to question the role that image tokens' continuity plays under large domain gaps.
Firstly, we quantitatively assess the domain distance be- tween the source and target domains using the Centered Kernel Alignment (CKA (Kornblith et al., 2019)) similarity following (Oh et al., 2022; Davari et al., 2022). Specifically, utilizing ViT as the backbone network, we extract features from images belonging to different domains and then com- pute the CKA similarity between different domains' features by aligning the channel dimension. A higher CKA similar- ity indicates smaller domain distance, implying that the model encompasses less domain-specific information.
an image of fish, although the patches are shuffled, the internal details of each patch still allow us to discern fea- tures such as fish scales, eyes, fins, tail, and other distinctive attributes. Based on such disrupted image, the model extracts features majorly based the patterns maintained within each patch, while the patterns across patches are lost.
Intuitively, patterns within each patch are spatially smaller, such as fish eyes, while patches are interconnected with their neighbors to build spatially larger patterns, such as a whole fish. Simultaneously, larger patterns are always harder to transfer than smaller ones (Zou et al., 2024b). For example, capturing a pattern similar dog is difficult, but capturing a pattern similar to a fish eye is relatively easier, e.g., the dog eyes. Therefore, we hy- pothesize that breaking continuity essentially breaks the large patterns across patches into small patterns within each patch. Under large domain gaps, large patterns are inherently harder to transfer. Therefore, the major effec- tive patterns on target domains are small patterns within each patch. Consequently. disrupting token continuity has much smaller effect on target domains, as the recognition of target domains is only marginally based on large patterns across patches. Similarly, since larger patterns are effective for the source domain, the performance of source-domain recognition drops drastically. On the other hand, breaking
The results are in Fig. 2, from which we observe the domain similarity between the source and target datasets increased significantly, although in Fig. the model's performance declines on all datasets. Intriguingly, the greater the decline in model performance in Fig. 1, the more the increase in domain similarities. This indicates the surprising benefit of disrupting continuity in reducing domain discrepancies, thereby enhancing the model's transferability.
whole fish in
2.4. Why does breaking continuity reduce domain discrepancy?
To account for breaking continuity reduces domain distance, we look back on the disrupted images. Although the connec- tion of each patch is disrupted, patterns within each patch are not disrupted. For example, as shown in Fig. 3, given

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
continuity forces the model to extract features majorly based on smaller patterns, aligning with the patterns used target domains. Therefore, the domain similarity increases.
3.1. Warm-Up Spatial-Domain Disruption
Since the model's pretraining data is significantly different from the disrupted data, the training on the disrupted ones may be difficult. Therefore, gradually increasing the diffi- culty will be beneficial for model training. As illustrated in Fig. 4b, the magnitude of Shuffle Patches' performance drop and Nomainn-similate amplitude-bal baaed disruption. Therefore, we first use Shuffle Patches as a apatial-domal disruption for warming up.
To validate our hypothesis. we disrupt images to varying degrees of shuffling, in order to verify how the maintained patterns' spatial size influences the performance and domain similarity. Specifically, each image is divided into pseudo- patches of sizes 1# 1, 2 2, 4 4. 7 7, 8 * 8, and 14 * 14. For instance, 2 2 signifies dividing the whole image into four equal-sized sections along both its length and width'. Then, these pseudo-patches are randomly shuffled and then reassembled to form the shuffled images, which are then input into VIT. Since patches within each pseudo-patch are not disrupted, the size of spatial patterns is larger.
increase is smaller than that of the
Specifically, our method involves randomly dividing images into varying numbers of patches and subsequently shuffling these patches. This approach encourages the model to learn the internal information within patches of different sizes, gradually enabling it to adapt and learn smaller patterns. As in the Fig. 50, we divide set of images into a random num- ber of equally sized patches (e.g., patches). Afterward, we shuffle the patches belonging to the same image, append the CLS token and positional encoding to them, and then feed them into the ViT as
As in Fig. 4a, the performance consistently decreases with the growing number of ppeude-patches, i.e., smaller patterns within each pseudo-patch. Meanwhile, domain similarities between source and target domains consistently increase as the pseudo-pated -ptcc size reduces, verifying it is the maintained pattern's 's spatial size that influences domain similaritics.
1' = Telassi F E; a B
E.... pxp L
E + Epos:
(14)
2.5. Conclusion and Discussion
Based on it, we interpret as follows. The continuity between image tokens essentially assists models in learning larger spatial patterns, which are beneficial for source-domain classification, therefore breaking the continuity significantly harms source-domain performance. However, excessively large patterns often struggle to transfer to target domains. Conversely, smaller patterns are easier to transfer. Under large domain gaps, most patterns transferred to target do- mains are those small ones that are within each patch, there- fore disrupting the continuity has smaller influences on the target-domain performance. By aligning patterns used in feature extraction for source and target domains small patterns, the domain similarity also increases.
3.2. Balanced Prequeney-Domanne Disruption
Considering the comparison of the four simple continuity disruption methods presented in Fig. 2, it is observed that shuffling the amplitudes in the frequency domain maximizes domain similarity between the source and target domains. This consideration is primarily based on the following two reasons. Firstly, transferred patterns may also encompass regions in adjacent patches, e.g., fish eye split into two ad- jacent patches, suggesting that retaining certain degree of continuity is beneficial. Secondly, studies (Chen et al., 2021) have demonstrated that some information in the frequency domain encompasses domain information. By shuffling the amplitudes, the model can eliminate style biases. There- fore, we aim to take Shuffle Patch Amplitude as a strong frequency-doman disruption to enhance the learning of small and Nomain-agnoster patterns.
3. Method
Building upon the above analysis and interpretation, we conclude that under large domain gaps, maybe only small patterns within each patch can be transferred. Therefore, aim to prioritize learning within patches and reduce models' reliance on large patterns across patches. Specif- ically, during the source-domain training, we propose to disrupt the continuity of image tokens in two steps, includ- ing a warm-up spatial-domain disruption and a balanced frequency-domain disruption (Fig. 5).
Considering the distribution of patches may vary across images, simply shuffling the amplitudes of image patches directly may not diversely disrupt the continuity. Take the classification of jellyfish in the ocean as an example: most patches may be dark water. When we directly shuffle the amplitudes of image patches, the overall image remains largely unchanged, with most areas retaining the continuity. Therefore, we consider balancing the distribution of patches for better disruption of the continuity.
We resize the image to 256 * 256 if the number of tokens is not evenly divided, preserving the number of pixels in cach patch.
Therefore, we first cluster image patches based on their vi- sual appearances, randomly sample styles from each cluster, and finally re-assign the sampled styles to each patch. This approach balances different sizes of image clusters, e.g..
5

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
Warm-Up Spatial-Domal Doomann
Disruption
Mixed Rescaling:
ViT Blocks
Classifier
Shuffling:
Ronndam seleect cow hypa 1..0.99
Balanced Frequency-
Domain Disruption
Decompose Amplitude
Balance & Reasemble Patches
Clustering:
Sampling:
Type
Type 2:
Type
Modeling:
FTT
Amplitude
Type 1:
N:
IPFT
two steps to disrupt the continuity images during source-domaie training. Step ① involves dividing the image into random number of patches and then shuffling them. This approach poses relatively lower difficulty for the model and is utilized during the warming up training. Step decomposes reassembles the amplitudes of clustered image patches to ensure the diversity disruptions, which presents greater challenge for the model and is thus employed during the middle to later stages of training.
Figure 5.
if dark water takes large in the jellyfish image, its sampling probability is similar to those in small areas.
where Ai. is the amplitude corresponding the patch oy Mj is the total numbers of patches in this cluster.
Specifically, as in Fig. 50, an input image E R R is split into the patches Pp E MDD where æi and ry. are the average of pixels in the ith and jth patches (i,j E M). Then, we calculate the similarity distance among these patches by employing the cosine similarity metric
Then, we model the amplitude distribution of each patch within each cluster as multivariate Gaussian distribution. This Gaussian distribution centers around the mean ampli- tude of all patches within the corresponding cluster and its variance can be computed from the differences between the amplitudes of the patches within the cluster and their mean value, thereby obtaining Gaussian distribution that represents that particular amplitude of cluster.
cos(a) oq.)
(15)
| |
Subsequently, we categorize these patches into different types based on their similarity exceeding predefined thresh- old, thereby dividing all patches within an image into mul- tiple clusters. A cluster is defined as comprising a patch =i and its corresponding patches, all of which fall within predefined similarity threshold sim
M
((Cluster
Σ
A1. E Cluster
(18)
M, f=1
M;
(CClusterAg)
M, A #CCluster (alustter.s)) A,)²
Σ
(19)
Cluster, = { cos(æ) xz/)) 2 sim, x E xxp. (16)
After extracting the distributions of multiple amplitudes from an image, we random sample on each cluster distribu- tion, e.g. ECluster AB N (Nu(Cluste A3 o (Cluster
Next, as shown in the Eq. 10, we extract the amplitudes of each patch within each cluster
Cluster As. = (Ag co, x2,) 2 sim, x/ e E xpp, (17)
Finally, we sum these samples with random proportions p to
6

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
Table Comparison with state-of-the-ane works based on VIT-S on target domains.
Method
Shot
FT
Mark
ChestX
ISIC2018
EuroSAT
CropDiseases
Average
MEM-FS (Walsh et al., 2023)
X
TIP-23
22.76
32.97
68,11
81.11
51.24
StyleAdv (Fu al., 2023)
X
CVPR-23
22.92
33.05
72.15
$2.34
FLoR (Zou et al., 2024a)
I
X
CVPR-22 24
22.78
34,20
72.39
81.81
52.80
DAMIM (Ma al., 2024)
X
AAAI-25
22.97
34.66
72.87
82.34
53.21
CD-CLS (Zou et al., b)
X
NeurlPS-22 24
22.93
34.21
74.08
83.51
53.68
AttTTemp (Zou et al., a)
X
NeurlPS-24
23.19
34,92
74.35
84.02
54.12
ReCTT
X
Ours
23.27
35.13
74.56
84.76
54.43
PMF (Shell Xu, 2022)
CVPR-22
21.73
30.36
70.74
80.79
50,91
FLoR (Zou et al., 2024a)
CVPR-24
23.26
35,49
73.09
83.55
$3.85
StyleAdv (Fu et al., 2023)
CVPR-22
22.92
33.99
74,93
84.11
53.99
DAMIM (Ma al., 2024)
AAAI-25
23.38
36.35
73.61
83.90
$4.31
CD-CLS (Zou et al., b)
NeurlPS-24
23.39
35.56
74.97
84.54
54.62
AttTTemp (Zou et al., a)
NeurlPS-24
23.63
38,05
75.09
84.78
55,39
ReCTT
Ours
23.84
38.48
75.23
85.92
55.87
MEM-FS RDA" (Walsh et al., 2023)
TIP-23
23,85
37.07
75.91
83.74
55.14
DAMIM (Ma et al., 2024)
AAAI-25
23.91
38.07
77.23
86.74
$6.49
CD-CLS (Zou et al., b)
NeurIPS-22 24
23.88
37,20
78,41
87.39
56.72
AttnTemp (Zou et al., a)
NeurIPS-24
23.96
40.13
77.40
87.58
$7.23
ReCIT'
Ours
24.42
39.39
78.84
88.45
57.78
MEM-FS (Walsh et al., 2023)
X
TIP-23
26.67
47.38
86.49
93.74
63.57
StyleAdv (Fu et al., 2023)
5
x
CVPR-23
26.97
47.73
88.57
94.85
64.53
FLoR (Zou al., 2024a)
5
CVPR-22 -4
26.71
49.52
90.41
95.28
65.48
DAMIM (Ma et al., 2024)
X
AAAI-25
27.28
50.76
89.50
95.52
65.77
CD-CLS (Zou et al., b)
5
X
NeurlPS-22 24
27.23
50.46
91.04
95.68
66.10
AttnTemp (Zou et al., a)
5
NeurIPS-24
27.72
53.09
90.13
95.53
66.62
ReCIT
5
X
Ours
28.23
52.36
90.42
96.02
66.76
PMF (Shell Xu, 2022)
CVPR
27.27
50.12
85.98
92.96
64.08
StyleAdv (Fu et al., 2023)
5
CVPR-23
26.97
51.23
90.12
95.99
66.08
FLoR (Zou al., 2024a)
5
CVPR -24
27.02
53.06
90.75
96.47
66.83
DAMIM (Ma et al., 2024)
AAAI-25
27.82
54.86
91.18
96.34
67.55
CD-CLS (Zou et al., b)
5
NeurIPS-24
27.66
$4.69
91.53
96.27
67.54
AttnTemp (Zou et al., a)
5
V
NeurIPS-24
28,03
54,91
90.82
96.66
67.61
ReCIT
Ours
28.88
54.91
91.58
68.06
MEM-FS + RDA' (Walsh et al., 2023)
TIP-23
27.98
$1.02
88.77
95.04
65.70
DAMIM" (Ma et al., 2024)
5
AAAI-25
28.10
55.44
91.08
96.49
67.78
CD-CLS' (Zou et al., b)
5
/
NeurIPS-24
28.25
55.66
91.68
96.62
68.05
AttTTemp' (Zou et a)
NeurIPS-2 24
28.41
55.22
91.34
96.74
67.93
ReCIT"
5
Ours
28.97
55.60
91.72
96.97
68.32
assign to patches, N is the number of clusters in an image
4. Experiments
4.1. Dataset and Implementation Details
N
A
PAS
Elusser
(20)
Following current works (Oh et al., 2022), employ the minilmageNet dataset (Vinyals et al., 2016) our source domain, and target domains involve four datasets: CropDis- ease (Mohanty et al., 2016), EuroSAT (Helber et al., 2019), ISIC (Codella et al., 2019), and ChestX (Wang et al., 2017). These datasets cover agriculture, remote sensing, and medi- cal data, with substantial domain discrepancies.
i=1
Apm
P4g
Epro
a)),a)
(21)
1==1 Cpr
This process reconfigures the amplitudes of each patch, ad- dressing the issue of uneven patch distribution within the image and ensuring continuity in amplitudes among patches.
In implementation, set the similarity threshold to 0.3. We adopt ViT-S as our backbone network and initialize it with DINO pretraining on ImageNet following (Caron et al.,
During the target-domain stage, we conduct ceatotyee-based classification (Eq. 6) or finetune-base based classification follow- ing (Zhou al., 2023).
2021: Fu et al., 2023; Zhang et al., 2022). Additionally, our

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
Table 2. Ablation study by S-shot.
41305
Method
CropDisease
EuroSAT
ISIC2018
ChestX
Aue.
66.4%%
47%
Rasdine
04.20ge in
80-4
45 am
63.91
Warm up discaptiin
95. ee
48.83
65.19
62.07%
Balanced disruption
96.02...
90.42au.7
52.36.015
28.23
04 90%
(0) Ramove Position Endbedding
95,02
80.96
47.40
26885am
(b) Stuftted Pusches
94.03
89.07
48.04 -e
27.08-am
64.70
44.0%
65
(c) Shuffle Putch Amp
94.75
58 18.78
29.67
27 04
65.06
10
0.4
(d) Sluffle Patch Phuse
94.94
3
26.
64.98
01 Block v5. F-dcct shetage performaacce
(8) Lacaboll Sobot overage perfommace
(e) Shuffle Image Amp.
96.73
38.96
47,95
26
Figure 6. (a) Applying our approach to any layer results in perfor- mance enhancements, but the greatest improvement is achieved when it is applied to the input layer. (b) A relatively small similar- ity threshold can more effectively balance the style.
(1) Shuftle Cluster Amp. wi Bolancing
95.28 i4.
19.1
49.51 -4:3
am
65.44
(g) Shuttle Patch Amp. w/ Balancing
96.05 14.1
89.25 mur
30.85
28. 12
05.94
model leverages the Adam optimizer(Kingma & Ba, 2017) for 50 epochs, with a learning rate of 10 assigned to the backbone network and 10-3 to the classifier, respectively. Experiments are conducted on NVIDIA GeForce RTX 3090 GPUs.
the importance of disrupting the continuity.
4.3.3. BALANCED DISRUPTION AND WARMING UP
4.2. Comparison with State-of-the-Art Works
Comparing the default shuffling of amplitude (c) and the balanced shuffling (g), we can see the balancing operation consistently increases the performance. Moreover, com- paring the disruption without warming up (g) and our final performance, we can see the warming up step can also con- sistently improve the performance.
We report our comparison with state-of-the-ee works for the 1-shot and 5-shot configurations in Tab. respectively. To ensure fair assessment, we separately compare works that have undergone finetuning (FT) and those that have not. The asterisk (*) indicates a transductive setting. Notably, our results demonstrate superior average performance across all configurations and consistently surpass existing works.
4.4. Sensitivity Study of Hyper-parameters
We conduct an analysis of the hyper-parameters presented in Fig.6 and 7, and see that:
4.3. Ablation Study
The ablation study for each module is presented in Tab.2. It is evident that all types of our proposed methods aimed at disrupting the continuity of image patches contribute positively to the performance in the target domains. Addi- tionally, we conduct comparative analysis between our approaches and other similar works to substantiate the ratio- nale underlying our design choices.
(I) The input layer demonstrates effecttivnness As illus- trated in Fig. 6a, merely applying our approach within the first block (i.e., the input layer) results in a substantial im- provement in the model performance in target domains. Moreover, the application of our method to any layer yields notable improvements compared to not using it, further con- firming the validity of our approach.
(2) A moderately small similarity distance threshold for di- viding patches into clusters yields optimal results but should not be too minute, whereas a smaller threshold refers to larger clusters. As depicted in Fig. 6b, reducing it initially leads to a steady improvement in performance in the target domains. However, once this threshold is below approxi- mately 30%, performance begins to decline.
1. VERIHICTION OF BREAKING THE CONTINUITY
To study the contribution of our methods, we compare our four types of methods for disrupting the continuity of image patches with the baseline model in Tab. 2a. It is evident that all four methods we propose outperform the baseline model significantly on the target domain, thereby demonstrating the effectiveness of our approaches.
(3) A larger standard deviation of sampling the proportions in Eq. 21 yields optimal results. As illustrated in Fig. 7a, increasing the standard deviation leads to a steady enhance- ment in performance until it reaches a plateau. This suggests that a larger standard deviation results in greater variability among patches and a higher degree of disrupted continuity, which is more beneficial for generalization in target domains.
4.3.2. SHUFFLING AMPLITUDE IN PATCH DIMENSION
We study the impact of shuffling amplitude in patch dimen- sion and directly contrast with shuffling amplitude across entire images (c), an approach that has been extensively ex- plored in many previous works (as shown in Table 2b). Our findings reveal that the performance of the latter is inferior to ours, suggesting that manipulating image patches offers a more effective means of enhancing the transfcrability of ViT- based models. Moreover, by gradually reducing the size of maintained continuity to clusters (f) or patches (g), we can see the performance consistently increases, verifying
4.5. Verification of model generalization
1. QUANTITATIVE STUDY
As depicted in Fig. 7b, we assess the domain similarity of the features extracted from the trained backbone network
8

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
This field is primarily explored through two approaches: transferring- methods (Zhou et al., 2023; Zou et al., 2024a), which adapt pre-trained models from extensive source datasets to target domains with scant data, and meta- learning approaches(Fu et al., 2022; Hu & Ma, 2022), which emphasize training models to swiftly adapt to novel tasks. However, no studies have explored the impact of continuity of ViT on cross-domain scenarios.
6..0%
01004
04
Nelee Demanr
64.0%
Standard deviation 5-doot sverage perforance Demaiin Similatty mowardd the CKA
Figure 7. (a) A larger standard deviation indicates greater discrep- ancy and discontinuity among image patches, and consequently, the performance on the target domain improves accordingly. This demonstrates that disrupting continuity is effective in reducing domain discrepancies. (b) Employing our method markedly boosts the similarity across domains.
Continuity. VIT was introduced by (Dosovitskiy et al., 2021), which adopts an input approach by dividing images into patches and explores that when the input images are larger, the model is able to capture more detailed informa- tion. Further investigation is conducted by them on the incorporation of positional encoding, revealing that absolute positional encoding causes ViT to lack the translation invari- ance required for image processing. Then, (Liu et al., 2021) proposes a relative positional encoding to preserve invari- ance but is highly demanding of computational resources. Alternatively, (Chu et al., 2021) proposes a conditional positional encoding to address ViT's translation invariance while conserving computational resources. (Li et al., 2019; Wertheimer al., 2021; Rong et al., 2023) find that the low- level local visual features can be more easily transferred to the target domain than those high-level semantic features. However, none of these studies specifically explore the im- pact of continuity on cross-domain performance. We are the first to investigate the influence of continuity on cross- domain performance.
Image
Baseline
Ours
Image
Baseline
Ours
Figure 8. The heatmap for the source domain displayed the first row illustrates that our method takes into account smaller patterns scattered throughout the image, collectively forming broader perceptual area. The heatmaps for the target domains shown in the following two rows demonstrate that our approach effectively enhances VIT's perception of the target domain.
6. Conclusion
between the source and target domains using the CKA simi- larity metric. The results reveal that our model substantially elevates the domain similarity, suggesting that our model acquires domain domain-agnostic information.
In this paper, we find an intriguing phenomenon that break- ing image tokens' continuity affects differently on the per- formance of source and target domains. We delve into this phenomenon for an interpretation, which inspires us to fur- ther propose a method to break the continuity, encouraging the model to rely less on large patterns and more on small patterns for recognition. Extensive experiments four CDFSL benchmarks validate our rationale and effectiveness.
4.5.2. QUALITATIVE STUDY
The visualization of the attention maps in both the source and target domains is presented in Fig. 8. In contrast to the baseline, which exhibits dispersed attention, our model not only focuses the most diseriminative large patterns but also attends to smaller, less discriminative patterns scattered across the image, forming a larger perceptual field. By transferring these patterns to the target domain, our model achieves a better perception of the target domain.
A Acknowledgments
This work is supported by the National Natural Science Foundation of China under grants 62206102; the National Key Research and Development Program of China under grant 2024YFC3307900; the National Natural Science Foun- dation of China under grants 62436003, 62376103 and 62302184; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through Ant-Ant Research Fund. The com- putation is completed in the HPC Platform of Huazhong University of Science and Technology.
5. Related Work
Cross-Domain Few-Shot Learning(CDFSL, was intro- duced in FWT(Tseng et al., 2020), and novel benchmark for this field has been established in BSCD-FSL(Guu et al., 2020). It strives to transfer knowledge from a proficiently trained source domain to a separate target domain char- acterized by limited labeled data and a large domain gap.

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
Impact Statement
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 248-255. leee, 2009.
We propose a CD-FSL method that disrupts the continuity of image tokens within the Vision Transformer (ViT) archi- tecture. This strategy encourages the model reduce its de- pendence on large patterns and instead rely more smaller, highly transferable patterns to the target domain. Addition- ally, our approach holds promise for application in other fields, including domain generalization, domain adaptation, and few-shot class- inssemnncremtttll learning, where enhancing model transferability poses a universal challenge. Although our evaluations concentrate on four distinct target domains, it is important note that these may not exhaustively cover all potential real-world scenarios. Hence, further evaluation across broader spectrum of target domains is crucial to validate our approach in more realistic and diverse settings.
Dosovitskiy, A., Beyer. L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Fu, Y., Fu, Y., and Jiang, Y.GG. Meta-fdmixup: Cross- domain few-shot learning guided by labeled target data. In Proceedings of the 29th ACM international conference on multimedia, pp. 5326-5334, 2021.
Fu, Y. Xie, Y., Fu, Y., Chen, J., and Jiang, Y.-G. Wave- san: Wavelet based style augmentation network for cross- domain few-shot learning, 2022.
References
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Fu, Y., Xie, Y., Fu, Y., and Jiang, Y.GG. Styleadv: Meta style adversarial training for cross-domain few-shot learning, 2023.
Guo, Y., Codella, N. C., Karlinsky, L.,, Codella, J. V., Smith, J. R., Saenko, K., Rosing, T., and Feris, R. A broader study of cross-domain few-shot learning. In Proceedings of the IEEE/CVF European Conference on Computer Vision, pp. 144. Springer, 2020.
Chen, G., Peng, P., Ma, I.., Li, J., Du, L.,, and Tian, Y. Amplitude-phaad recombination: Rethinking robustness of convolutional neural networks in frequency domain. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 458 4633-467,200 202,
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, 2019.
Chen, X., Hu, Q., Li, K., Zhong, C., and Wang. G. Accumu- lated trivial attention matters in vision transformers on small datasets. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1984-39922 January 2023.
Hu, Y. and Ma, A. J. Adversarial feature augmentation for cross-domain few-shot classification, 2022.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017.
Chu. X., Tian, Z., Zhang. B., Wang. X., and Shen, C. Condi- tional positional encodings for vision transformers. arXiv preprint arXiv:2102. 10882, 2021.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similar- ity of neural network representations revisited. In Interna- tional Conference on Machine Learning, pp. 3519-3529, PMLR, 2019.
Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., Kittler, H., and Halpern, A. Skin lesion analysis toward melanoma detection 2018: challenge hosted by the international skin imaging collaboration (isic), 2019.
Li, P., Gong, S., Wang, C., and Fu, Ranking distance calibration for cross-domaii few-shot learning, 2022.
Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., and Luo, J. Revisiting local descriptor based image-to-class measure for few-shot learning. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition. pp. 7260- 2019.
Das, D., Yun, S., and Porikli, F. Confess: A framework for single source cross-domain few-shot learning. In International Conference on Learning Representations, 2022.
Davari, M., Horoi, S., Natik, A., Lajoie, G., Wolf, G., and Belilovsky, E. Reliability of cka as a similarity measure in deep learning, 2022.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the
10

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., and Wierstra, D. Matching networks for one shot learning. In Proceedings of the International Conference Neural Information Processing Systems, pp. 3637-3645, 2016.
Ma, R., Zou, Y., Li, Y., and Li, R. Reconstruction target matters in masked image modeling for cross- dooss- few- shot learning. arXiv preprint 14 19101, 2024.
Walsh, R., Osman, L., and Shehata, M. S. Masked embed- ding modeling with rapid domain adjustment for few-shot image classification. IEEE Transactions on Image Pro- cessing, 32:4907-4920, 2023. doi: 10. 109/TT1:222. 3306916.
Mohanty, S., Hughes, D., and Salathé, M. Using deep learn- ing for image- based plant disease detection. Frontiers in Plant Science, 7(September), September 2016. ISSN 1664-462X. doi: 00.3389ffpss22116.1141 Publisher Copyright: c 2016 Mohanty, Hughes and Salathé.
Wang. H. and Deng. Z.-H. Cross-domain few-shot classifi- cation via adversarial task augmentation, 2021.
Naseer, M. M., Ranasinghe, K., Khan, S. H., Hayat, M., Shahbaz Khan, F., and Yang, M.-H. Intriguing properties of vision transformers. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc., http://prooeedii.....rr.elllee 2021. URL httpee_iiieeee/pprrr////////L//
Wang, X., Peng, Y., Lu, L.,, Lu, Z., Bagheri, M., and Summers, R. M. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax dis- eases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, July 2017. doi: 10. 10.11099/v::2211..366 URL http: http://xxxdo.iiod.o/ooo// doi.org/ 10. 10.1109/ CVPR. 2017 369.
volume 34, pp.
cc/ see/pape__fte c404 (tw.aaaaaaa.0000016681111111111. Paper. pdf.
Wertheimer, D.. Tang, L., and Hariharan, B. Few-shot classification with feature map reconstruction networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8012-8021, 2021.
Oh, J., Kim, S., Ho, N., Kim, J.-H., Song, H., and Yun, S.-Y. Understanding cross-domain few-shot learning based on domain similarity and few-shot difficulty, 2022.
Wu, K., Zhang, J., Peng. H., Liu, M., Xiao, B., Fu, J., and Yuan, L. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision (ECCV), 2022.
Phoo, C. P. and Hariharan, B. Self-training for few-shot transfer across extreme task differences, 2021.
Raghu, A., Raghu, M., Bengio, S., and Vinyals, O. Rapid learning feature reuse? towards understanding the ef- fectiveness of maml. In Proceedinggs of the International Conference on Learning Representations, 2019.
Xu, H., Liu, L., Zhi, S., Fu, S., Su, Z., Cheng, M.-M., and Liu, Y. Enhancing information maximization with distance-aware contrastive learning for source-free cross- domain few-shot learning. IEEE Transactions on Image Prucessing, 2024.
Rong, Y., Lu, X., Sun, Z., Chen, Y., and Xiong, S. Espt: A self-supervised episodic spatial pretext task for improving few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9596-9605, 2023.
Yuan, L.,, Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.- H., Tay, F. E., Feng, J., and Yan, S. Tokens-to-token vit: Training vision transformers from scratch imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 558-558-6 558,,67,66 October 2021.
Shell Xu, Da Li, J. S. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make difference, 2022.
Zhang, H., Li, F,, Liu, S., Zhang, L., Su, H., Zhu, J., Ni,
L. M., and Shum, H.-Y. Dino: Detr with improved denois- ing anchor boxes for end-to-end object detection, 2022.
Sun, J., Lapuschkin, S., Samek, W., Zhao, Y., Cheung, M.-M., and Binder, A. Explanation-guided training for cross-domain few-shot classification. In 2020 25th inter- national conference on pattern recognition (ICPR), pp. 7609-7616. IEFF, 2021.
Zhang, R., Che, T., Ghahramani, Z., Bengio, Y., and Song, Y. Metagan: An adversarial approach to few-shot learning, In Proceedings of the International Conference Neural Information Processing Systems, pp. 2371-23800 2018.
Zhou, F., Wang, P., Zhang, I.., Wei, W., and Zhang, Y. Re- visiting prototypical network for cross domain few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20061-20077 June 2023.
Tseng, H.-Y., Lee, H.-Y., Huang, J.-B., and Yang, M.-H. Cross-domain few-shot classification via learned feature- wise transformation. In Proceedings of the International Conference on Learning Representations, 2020.

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
Zou, Y., Ma, R., Li, Y., and Li, R. Attention temperature matters in vit-based cross-domain few-shot learning. In The Tiirth- Annual Conference Neural Informa- tion Processing Systems,
Zou, Y., Yi, S., Li, Y., and Li, R. A closer look the cls token for cross-domain few-shot learning- In The Thirty-eighth Annual Conference Neural Information Processing Systems, b.
Zou, Y., Zhang, S., Li, Y., and Li, Margin-basee few-shot class-incremental learning with class-level overfitting mit- igation. Advances in neural information processing sys- tems, 35:27277-27279, 2022.
Zou, Y., Liu, Y., Hu, Y., Li. Y., and Li, R. Flatten long- range loss landscapes for cross-domain few-shot learning, 2024a.
Zou, Y., Zhang, S., Zhou, H., Li, Y., and Li, R. Composi- tional few-shot a7X2v:2405.17022, 2024b.
learning. arXiv preprint
12

---

Appendix for Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
CrepDsseses
EuroSAT
ISIC2018
Figure 9. Samples of the miniImageNet dataset.
A. Dataset Description
minilmageNet (Vinyals al., 2016) is a meticulously selected subset derived from the extensive ImageNet dataset (Deng et al., 2009). It encompasses 100 categories, with each category represented by 600 natural images, total- ing 60,000 images. Consistent with recent research (Zou et al., a;b), we utilize the training portion minilmageNet as our source domain dataset. This comprises 64 classes and total of 38,400 images, with representative samples displayed in Fig. 9.
ChestX
Figure 10. Samples of the CropDiscases, EuroSAT, ISIC2018 and ChestX datasets.
dedicated medical imaging, particularly focusing on the classification of skin lesions. This dataset boasts 10,015 images distributed across 7 distinct categories, serving as pivotal resource for diagnosing skin diseases.
Furthermore, as illustrated in Fig. 10, we adopt the method- ology outlined in (Guo et al., 2020) and employ datasets from four distinct domains as our target domains. These domains include plant disease images, surface satellite im- agery, skin disease images, and chest X-ray images, which will be elaborated in the following sections.
ChestX (Wang et al., 2017) is another medical imaging dataset that centers on chest X-rays. comprises 25,847 images, meticulously categorized into unique classes, of- fering invaluable data for tasks related chest disease clas- sification.
CropDiseases (Mohanty et al., 2016) is specifically de- signed for the recognition of agricultural diseases. It con- tains 43,456 images across 38 different classes, featuring a diverse range of crops, both healthy and diseased, each labeled with a specific disease category.
B. Centered Kernel Alignment
Centered Kernel Alignment (CKA) (Kornblith al., 2019) is statistical technique designed to quantify the similarity between representations learned by disparate neural net- works. Originating from kernel methods, CKA excels in comparing high-dimensional representations. To compute CKA between two sets of data representations X E R and Y E Rx we first calculate the Gram matrices K = X and = YY These matrices encapsulate the inner products between all pairs of data points within
EuroSAT (Helber al., 2019) a comprehensive dataset comprising extensive collection of satellite imagery of Earth's ss surface. It encompasses total of 27,000 images, meticulously categorized into 10 diverse classes, spanning a broad spectrum of geographical and topographical at- tributes.
ISIC2018 (Codella al., 2019) is specialized dataset
13

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
Table 3. Comparison with more state-of-the-ee art works based by 5-way 1-shot accuracy.
Method
Backbone
FT
Mark
ChestX
ISIC2018
EuroSAT
CropDiseases
Average
GNN + FT (Tseng et al., 2020)
ResNet10
ICLR-20
22.00
30.22
55.53
60.74
42.12
MN AFA (Hu & Ma, 2022)
ResNet10
ECCV-22
22.11
32.32
61.28
60.71
44.10
GNN ATA (Wang & Deng, 2021)
ResNet10
IICAI-21
22.10
33.21
61.35
67.47
46.53
GNN + AFA (Hu Ma, 2022)
ResNet10
ECCV-22
22.92
33.21
63.12
67.61
46.97
LDP-net (Zhou et al., 2023)
ResNet10
CVPR-23
23.01
33.97
65.11
69.64
47.18
Fl.oR (Zou al., 2024a)
ResNer10
CVPR-24
23.11
38,11
62.90
73.64
49.69
MEM-FS (Walsh al., 2023)
VIT-S
TIP-23
22.76
32.97
68.11
81.11
51.24
StyleAdv (Fu et al., 2023)
VIT-5
CVPR-23
22.92
33.05
72.15
81.22
52.34
Fl.oR (7ou et al,, 2024a)
VIT-S
CVPR-24
22.78
34,20
72.39
81.81
52.80
DAMIM (Ma et al., 2024)
VIT-S
AAAI-25
22.97
34,66
72.87
82.34
53,21
CD-CLS (Zou et al., b)
VITSS
NeurlPS-24
22.93
34.21
74.08
83.51
53.68
AttnTemm (Zou et
VIT-S
NeurlPS-24
23.19
34.92
84.02
54.12
ReCIT
VIT-S
Ours
23.27
35.13
74.56
84.76
54.43
PMF (Shell Xu, 2022)
VIT-S
CVPR-24
21.73
30.36
70.74
80.79
50.91
FLoR (Zou et al., 2024a)
VIT-S
CVPR-24
23.26
35.49
73.09
83.55
53.85
StylcAdv (Fu et al., 2023)
VIT-S
CVPR-23
22.92
33.99
74.93
84.11
53.99
DAMIM (Ma et al., 2024)
VIT-S
AAAI-25
23.38
36.35
73.61
83,90
54.31
CD-CLS (Zou et al.,
VIT-S
NeurIPS-24
23.39
35.56
74.97
84.54
54.62
Attn'Temp (Zou et al., a)
ViT-S
NeurlPS-24
23.63
38.05
75.09
84.78
55.39
ReCIT
VIT-S
Ours
23.84
38.48
75.23
85.92
55.87
LDP-net" (Zhou al., 2023)
ResNet10
CVPR-23
22.21
33.44
73.25
81.24
52.54
TPN + ATA" (Wang& & Deng, 2021)
ResNet10
IICAI-21
22.45
35.55
70.84
82.47
52.83
RDC (Li et al., 2022)
ResNer10
CVPR-22
22.32
36.28
70.51
85.79
53.73
MEM-FS RDA" (Walsh al., 2023)
VIT-S
TIP-23
23.85
37.07
75.91
83.74
55.14
DAMIM"
2024)
VIT-S
AAAI-25
23.91
38.07
77.23
86.74
56.49
CD-CLS (Zou et al., b)
VIT-S
NeurlPS-22 24
23.88
37.20
78.41
87.39
56.72
AttnTemp (Zou et al.,
VIT-S
NeurIPS-24
23.96
40.13
77,40
87,58
57.23
ReCIT
VIT-S
Ours
24.42
39.39
78.84
88.45
57.78
their respective feature spaces. Subsequently, proceed to center the Gram matrices by applying the following process:
C. Comparison with more SOTAs
As depicted in Tab. and we undertake a comprehensive comparison of diverse ViT-based and CNN-based method- ologies in the context of CDFSL tasks. Our proposed meth- ods consistently outperform all other approaches, attaining superior performance. These results underscore the effec- tiveness of our approach.
Ka HKH, La HLH.
(22)
(23)
where H 4n -1.-1,,17 1 is the centering matrix, In is the identity matrix, and 1, is vector of ones. Finally, the centered kernel alignment is computed as:
D. Source Domain Performance
We have provided the performance the source domain (miniilageNNtt) for reference in Tab. 5.
Tr(KaLa) (T((K2) Tr(L3)
CKA(K,
(24)
Although there is slight performance trade-off on the source domain, this aligns with our hypothesis: disrupting token continuity prioritizes learning smaller, transferable patterns over domain-speciff holistic features. Crucially, the significant gains on target domains demonstrate the ef- fectiveness of our approach for cross tross-domain adaptation.
where Tr denotes the trace of a matrix. CKA is an exception- ally valuable tool for assessing similarity, particularly when comparing different datasets or models. High CKA values serve indicators of strong similarity, whereas low values point to notable differences. In this paper, leverage CKA to evaluate the similarity of features across various datasets and models. Specifically, for each dataset, we extract fea- tures from the model and compute the CKA similarity be- tween the source and target domains, aiming to gain insights into the model's generalization capabilities across diverse data distributions.
E. Detailed related work
Cross-Domain Few-Shot Learning (CDFSL) (Zhou et al., 2023) focuses on training a model on the source domain that can generalize well target domain with limited ex-
14

---

Revisiting Continuity of Image Tokens for Cross-Domain Few-shot Learning
Table 4, Comparison with more state-of the-art art works by 5-way 5-shot accuracy.
Method
Backbone
FT
Mark
ChestX
ISIC2018
EuroSAT
CropDiseases
Average
MN + AFA (Hu & Ma, 2022)
ResNet10
ECCV-22
23.18
39.88
69.63
80.07
53.19
GNN + FT (Tseng al., 2020)
ResNet10
ICLR-20
24.28
40.87
78.02
87.07
57.06
GNN + ATA (Wang & Deng, 2021)
ResNet10
IICAI-21
24.32
44.91
83.75
90.59
60.39
LDP-net (Zhou et al., 2023)
ResNet10
CVPR-23
26.67
48.06
82.01
89.40
61.29
GNN + AFA (Hu & Ma, 2022)
ResNet10
ECCV-22
25.02
46.01
85,58
88.06
61.67
Fl.oR (Zou al., 2024a)
ResNer10
CVPR-24
26.70
51.44
80,87
91.25
62.32
MEM-FS (Walsh al., 2023)
VIT-S
TIP-23
26.67
47.38
86.49
93.74
63.57
StyleAdv (Fu et al., 2023)
VIT-5
CVPR-23
26.97
47.73
88.57
94.85
64.53
Fl.oR (7ou et al,, 2024a)
VIT-S
CVPR-24
26.71
49.52
90.41
95.28
65,48
DAMIM (Ma et al., 2024)
VIT-S
AAAI-25
27.28
50.76
89.50
95.52
65.77
CD-CLS (Zou et al., b)
VIT-S
NeurIPS-24
27.23
50.46
91.04
95.68
66.10
AttnTemm (Zou et
VIT-S
NeurlPS-24
27.72
53.09
90.13
95.53
ReCIT
VIT-S
Ours
28.23
52.36
90.42
96.02
66.76
PMF (Shell Xu, 2022)
VIT-S
CVPR-22
27.27
50.12
85.98
92.96
64.08
StyleAdv (Fu et al., 2023)
VIT-S
CVPR-23
26.97
51.23
90.12
95.99
66.08
FLoR (Zou ct al., 2024a)
VIT-S
CVPR-24
27.02
53.06
90.75
96.47
66.83
DAMIM (Ma et al., 2024)
VIT-S
AAAI-25
27.82
54.86
91.18
96.34
67.55
CD-CLS (Zou et al.,
VIT-S
NeurIPS-24
27.66
54.69
91.53
96.27
67.54
Attn'Temp (Zou et al., a)
VIT-S
NeurlPS-24
28.03
54.91
90.82
96.66
67.61
ReCTT
VIT-S
Ours
28.88
54.91
91.58
96.85
68.06
ConFeSS (Das al., 2022)
ResNer10
ICLR-2022
27.09
48.85
84.65
88.88
62.37
LDP-net" (Zhou et al., 2023)
ResNet10
CVPR-23
26.88
48.44
84.05
91.89
62.82
RDC" (Li et al., 2022)
ResNet10
CVPR-22
25.07
49.91
84,29
93.30
63.14
TPN ATA" (Wang & Deng. 2021)
ResNer10
IJCAI-21
24.74
49.83
85.47
93.56
63.40
MEM-FS RDA" (Walsh al., 2023)
VIT-S
TIP-23
$1.02
95.04
65.70
DAMIM (Ma et al., 2024)
VIT-S
AAAI-25
28.10
55.44
91.08
96.49
67.78
CD-CLS* (Zou et al.,
VITSS
NeurlPS-22 24
28.25
55.66
91.68
96.62
68.05
AttnTTmmp (Zou
VIT-S
NeurIPS-24
28.41
55.22
91.34
96.74
67.93
ReCIT
VIT-S
Ours 28.97
$5.60
91.72
96.97
68.32
Table Ablation study of source-domain training by 5-shot.
CD-FSL. by spanning distributions source styles. IM- DCL (Xu et al., 2024) sets the entire feature positive and negative sets learn the query without accessing the source domain. However, studies have explored the impact of the continuity of VIT in cross-domain scenarios.
Method
Source Domain
Target Domain
Baseline 97.78
63.91
Ours
96.33
66.76
amples. Current methods can be grouped into two types: meta-learning-basee approaches and caanffrrlccaring-basse ones. Meta-learning-basee approaches (Zhang et al., 2018) aim learning aank-agnostic knowledge learn new tasks efficiently, differing in their way of learning the parameter of the initial model on the base class data. MAML (Raghu et al., 2019) aims at learning an initial parameter that can quickly adapt to new tasks, while FWT (Tseng et al., 2020) uses feature-wise transformation to learn representations with improved ability to generalization. An alternative way approaches, tackle the problem is seasserrlearning-baaee tackling the problem based on reusing the model trained the base class data in standard supervised learning way. Among these approaches, LRP (Sun et al., 2021) aims to use the explanation results to guide the learning process. STARTUP (Phoo & Hariharan, 2021), and Meta- FDMixup (Fu et al., 2021) mainly aim at defining relaxed settings for CD-FSL.. Wave-SAN (Fu al., 2022) tackles
15


Reviews:
[{"review_id": "DG4w1aiH0S", "reviewer": "Reviewer_3Jjq", "summary": "This paper provides a novel perspective to improve the performance cross-domain few-shot learning (CDFSL). The key insight is that disrupting the continuity of image tokens in ViT will force the model to learn smaller patterns which are more easily transferred under extreme domain gaps. The observation is interesting and they design several experiments to prove the hypothesis. Based on this motivation, a new model is proposed for CDFSL by disrupting the continuity of image tokens with 2 stages, including a warm-up spatial-domain disruption and a balanced frequency-domain disruption. Experiments show good improvements over previous works.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and valuable suggestions. Below are our responses to your comments:\n\n## **1. Source domain performance**\n\nAs suggested, we have provided the performance on the source domain (miniImageNet) for reference. \n\n| Model    | Source Domain | Target Domain |\n| -------- | ------------- | ------------- |\n| Baseline | 97.78         | 63.91         |\n| **Ours** | 96.33         | 66.76         |\n\nAlthough there is a slight performance trade-off on the source domain, this aligns with our hypothesis: disrupting token continuity prioritizes learning smaller, transferable patterns over domain-specific holistic features. Crucially, the significant gains on **target domains** (e.g., +6.5% on ISIC) demonstrate the effectiveness of our approach for cross-domain adaptation.\n\n## **2. Visualization of disrupted images** \n\nAn anonymous GitHub link ([https://anonymous.4open.science/r/More-visualization/visualizations.png]) with 16 visualized examples of disrupted images in three types is provided. These visualizations illustrate how our method breaks large-scale spatial patterns while preserving fine-grained, domain-agnostic features (e.g., textures, edges), which aligns with our motivation.\n\n\n\nWe appreciate your insightful feedback and hope these revisions address your concerns. Thank you again for your time and effort in reviewing our work!", "claims_and_evidence": "The idea is well-motivated and sufficient experiments have been designed to support the hypothesis.", "methods_and_evaluation_criteria": "The methods is cleverly designed, and model evaluation is standard for CDFSL", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "(1) Well, the experiments show good improvement over the few-shot target domain. The author should also provide the performance on source domain for future reference. This is because the performance gains on the target domain come at the expense of performance on the source domain, as pointed by the author in the manuscript.\n\n(2) Can the authors provide more visualization of disrupted images to better understand the impact of the various disruption methods, like Shuffle Patch Amplitude.", "relation_to_broader_scientific_literature": "This work provides a new perspective to tackle cross-domain few-shot learning tasks (CDFSL), and investigates the influence of continuity of image tokens in ViT on cross-domain performance. This will potentially inspire more follow-ups works in this domain", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "================================\n\nafter reading the rebuttal, my concerns have been mostly addressed. I will keep my original score of recommending acceptance.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "D7ZHBci46f", "reviewer": "Reviewer_yKbi", "summary": "This article explores the impact of image token continuity on model performance in the context of cross-domain few-shot learning (CDFSL). The study demonstrates that disrupting image token continuity can reduce the gap between the source and target domains to some extent, thereby improving the model's generalization ability on the target domain. The authors propose a novel method that disrupts image token continuity in both the spatial and frequency domains, encouraging the model to better learn small-scale spatial patterns rather than relying on large-scale ones. The source-domain training includes two core steps, namely Warm-Up Spatial-Domain Disruption and Balanced Frequency-Domain Disruption. Experimental results show that the proposed method significantly improves model performance on four benchmark datasets, validating the effectiveness of the proposed method.\n\n## Update After Rebuttal\nI have checked the authors' rebuttal, and found most of my concerns have been solved, so I choose to maintain my score of Weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback. Below are our detailed responses to your concerns:\n\n## **1. Classification Values in Figure 1**\n\nThe numerical  values in Fig.1 are summarized below:\n\n| Disruption Method       | Source Acc. | Target Acc. |\n| :---------------------- | :---------- | :---------- |\n| Original                | 97.78       | 63.91       |\n| Remove Pos.             | 90.03       | 61.95       |\n| Shuffle Patches         | 87.25       | 62.25       |\n| Shuffle Patch Amplitude | 68.13       | 59.80       |\n| Shuffle Patch Phase     | 67.41       | 59.78       |\n\n## **2. Source-Domain Performance**\n\nOurs achieves 66.76% on target (+2.85%), with 96.33% vs Baseline’s 97.78% on source.\n\n##  **3. Meta-Dataset Results**\n\nDue to time and resource constraints, we first pretrain on our datasets (miniImagenet), and then we evaluate on parts of the Meta-Dataset under 5-way 5-shot settings:\n\n| Dataset       | Baseline | Ours (Δ)          |\n| :------------ | :------- | :---------------- |\n| Birds         | 94.23    | **96.51 (+2.28)** |\n| FGVC-Aircraft | 70.37    | **70.43 (+0.06)** |\n| Fungi         | 61.03    | **62.98 (+1.95)** |\n| VGG Flower    | 89.64    | **89.72 (+0.08)** |\n| Traffic Sign  | 60.46    | **61.93 (+1.47)** |\n| Ave.          | 75.15    | **76.31(+1.16)**  |\n\n## **4. Computation cost**\n\nOur method introduces **no additional trainable parameters**.  \n\nDuring training, the computational overhead (baseline: 126.42s vs. ours: 140.48s per epoch) stems from the clustering-based frequency disruption during training, which is acceptable compared with the gains in cross-domain performance.  Moreover, some engineering trick, such as bipartite matching, can accelerate the clustering process, which we leave for future works.\n\nDuring inference, since our method introduces no additional parameters, inference time matches the baseline exactly with no computational overhead.\n\n## **5. Justification of Clustering**\n\n### Algorithm Overview\n\n### Step 1: Patchify & Standardize\n\n**Image Segmentation**\n\nSplit an input image into non-overlapping patches:\n$$\n\\mathbf{P} \\in \\mathbb{R}^{N \\times p^2 \\times C}\n$$\n**Normalization**\n\nStandardize each patch to zero mean and unit variance:\n$$\n\\mathbf{P}_i = \\frac{\\mathbf{P}_i - \\mu_i}{\\sigma_i + \\epsilon},\n$$\n\n### Step 2: Similarity Computation\n\n**Cosine Similarity Matrix**: Calculate cosine similarity between all neighbouring patch pairs:\n$$\n\\mathbf{S}_{i,j} = \\frac{\\mathbf{P}_i \\cdot \\mathbf{P}_j}{\\|\\mathbf{P}_i\\| \\cdot \\|\\mathbf{P}_j\\|}\n$$\n\n**Threshold Clustering**: Group patches into clusters beyond the threshold\n$$\n\\text{Cluster}(i,j) = \\begin{cases} \n1 & \\text{if } \\mathbf{S}_{i,j} > \\tau \\newline 0 & \\text{otherwise}\n\\end{cases}\n$$\n\n### Step 3: Cluster Merging\n\n\nFor each image:\n\n- Initialize each image as a singleton cluster.\n- Merge neighbouring patches i and j if beyond threshold\n- Repeat until no merges occur.\n\n## **6.  References and Other Revisions**\n\nOur method is consistent with [1-3] in that we hold the small patterns are easier to transfer than larger ones, but we differ in that (1) our method discusses the transferability of small patterns for ViT while [1-3] are originally designed for CNNs; (2) we provide a simple way to enhance the learning of small patterns by perturbing patches with no additional branches or losses.\n\nWe promise we will add discussions to the [1], [2], [3] and increase font sizes for Fig.5 Readability.\n\n## **7. Balance Between Information Loss and Generalization**\n\nTo complement the CKA metric, we also use the MMD distance to measure the domain distance, with larger value indicating larger distance.\n\n| Disruption Granularity | Source Acc. | Target Acc. | CKA  | MMD  |\n| :--------------------- | :---------- | :---------- | :--- | :--- |\n| 224×224 (pixel-level)  | 21.3        | 20.5        | 0.03 | 0.61 |\n| 112×112                | 23.68       | 22.95       | 0.03 | 0.57 |\n| 56×56                  | 36.33       | 31.09       | 0.04 | 0.56 |\n| 28×28                  | 59.45       | 56.97       | 0.05 | 0.54 |\n| 16×16                  | 64.82       | 61.05       | 0.22 | 0.43 |\n| 14×14                  | 87.25       | 62.25       | 0.22 | 0.41 |\n| 8×8                    | 94.39       | 62.26       | 0.15 | 0.47 |\n| 7×7                    | 94.93       | 62.26       | 0.14 | 0.47 |\n| 4×4                    | 96.49       | 62.26       | 0.12 | 0.49 |\n| 2×2                    | 97.38       | 62.99       | 0.08 | 0.50 |\n| No disruption          | 97.78       | 63.91       | 0.07 | 0.50 |\n\nModerate disruptions achieve optimal **MMD-CKA-accuracy balance**. Extreme disruptions harm both domains (low accuracy, high MMD, low CKA). This analysis confirms our method preserves **transferable semantics** while suppressing domain-specific structures, implying the default patch size is enough to achieve the balance.", "claims_and_evidence": "In the submitted paper, the authors present several important claims primarily centered on the impact of image token continuity on cross-domain few-shot learning (CDFSL). These claims are supported by experiments and theoretical analysis in the text, and overall, they can be considered to have clear and convincing evidence.\n\nI have just one small suggestion: The authors state that disrupting image token continuity significantly affects model performance in the source domain but has a relatively minor impact on the target domain. This phenomenon is one of the core findings of the paper. So could the authors provide the obtained classification values in Figure 1 (e.g., above the bar), which will allow the readers to more intuitively compare the results achieved in source and target domain?", "methods_and_evaluation_criteria": "The proposed methods and the used evaluation criteria are reasonable. The proposed method (disrupting token continuity) is remarkably simple, not requiring complex architectural designs, yet it effectively enhances the model's generalization ability in the target domain. In addition, I personally like the **balance operation** in the frequency-domain disruption step, whose motivation is clearly stated.", "theoretical_claims": "There is no proof and theoretical claim included in the paper.", "experimental_designs_or_analyses": "After checking the experimental part, I believe the experimental designs are soundness, and the results are able to indicate the effectiveness of the proposed method in improving model's generalization ability for solving cross-domain few-shot learning process.\n\nHowever, I have the following concerns:\n\n1. The authors only present the results on the target-domain (out-of-domain results), but in my own opinion, cross-domain learning should not significantly degrade the source-domain (in-domain) performance. So could the authors provide the classification accuracy on the pre-trained miniImageNet dataset?\n\n2. Many recent cross-domain few-shot learning methods [1],[2],[3] have evaluated their performance on a larger and more complex dataset, namely Meta-dataset [4]. So could the authors provide more results on this dataset to show the generalization ability of their method on more diverse target domains?\n\n3. Besides the classification accuracy, the model effeciency is also an important metric. Considering the proposed method involves a clustering process and a balanced sampling in the source-domain training phase, which may introduce additional computational overhead. So could the authors provide some discussions on the model effeciency?\n\n4. The details of the clustering process is missing, the authors only state that \"we set a similarity threshold of 0.3 for clustering image patches, where patches exceeding this threshold are fused into a single cluster.\" Which clustering algorithm is used？Could the authors provide the detailed steps of their clustering process or make appropriate citations of related works？\n\n[1] Li W H, Liu X, Bilen H. Cross-domain few-shot learning with task-specific adapters[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 7161-7170.\n\n[2] Guo Y, Du R, Dong Y, et al. Task-aware adaptive learning for cross-domain few-shot learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1590-1599.\n\n[3] Perera R, Halgamuge S. Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 23794-23804.\n\n[4] Triantafillou E, Zhu T, Dumoulin V, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples[J]. arXiv preprint arXiv:1903.03096, 2019.", "relation_to_broader_scientific_literature": "One of the key findings of this paper is \"disrupting image token continuity encourages the model to better learn small-scale spatial patterns rather than relying on large-scale ones, which can reduce the gap between the source and target domains\". The similar concept is related to many previous works [1],[2],[3] in designing local descriptors for improving the model's generalization ability. So I believe this finding is validated by existing studies and is technically reasonable.\n\n[1] Li W, Wang L, Xu J, et al. Revisiting local descriptor based image-to-class measure for few-shot learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 7260-7268.\n\n[2] Wertheimer D, Tang L, Hariharan B. Few-shot classification with feature map reconstruction networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 8012-8021.\n\n[3] Rong Y, Lu X, Sun Z, et al. Espt: A self-supervised episodic spatial pretext task for improving few-shot learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(8): 9596-9605.", "essential_references_not_discussed": "The statement in the paper \"larger patterns are always harder to transfer than smaller ones\" is a similar concept to the pervious findings that \"The low-level local visual features can be more easily transferred to the target domain than those high-level semantic features\", so I believe some related works [1], [2], [3] should be appropriately cited and discussed.\n\n[1] Li W, Wang L, Xu J, et al. Revisiting local descriptor based image-to-class measure for few-shot learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 7260-7268.\n\n[2] Wertheimer D, Tang L, Hariharan B. Few-shot classification with feature map reconstruction networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 8012-8021.\n\n[3] Rong Y, Lu X, Sun Z, et al. Espt: A self-supervised episodic spatial pretext task for improving few-shot learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(8): 9596-9605.", "other_strengths_and_weaknesses": "Strengths: \n\n1. This paper is well written and easy to follow, the experiments and ablation studies are sufficient to demonstrate the effectiveness of the proposed method.\n\n2. The motivation of designing the model is clearly stated, and some empirical analysis are preformed to well-support their motivation.\n\nWeaknesses:\n\nPlease refer to the questions in \"Methods And Evaluation Criteria\" and \"Experimental Designs Or Analyses\"", "other_comments_or_suggestions": "The texts in some figures (e.g., Figure 5) are too small to read.", "questions_for_authors": "In my own opinion, disrupting the image token continuity can reduce the gap between the source and target domains is a nature observation, since such operation will destory the semantic information in the training data. In an extreme case, we can shullfe every pixel in input image, resulting in nearly random noise, leading to quite similar domain distributions of source and target, but losing almost all the useful information for model training. So solely measuring the CKA similarity between the two domains may not correctly reflect the domain generalization ability (for classification tasks). Could the authors provide more discussions on this? And also explain how to achieve a balance between the information loss and the domain generalization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gs00er6o2P", "reviewer": "Reviewer_unnh", "summary": "This paper investigates the role of image tokens' continuity in Vision Transformers for Cross-Domain Few-Shot Learning. The authors identify an interesting phenomenon: disrupting the continuity of image tokens significantly affects performance in source domains but has only a marginal impact on target domains with large domain gaps. Based on this observation, the authors propose a simple yet effective method to disrupt image token continuity in spatial and frequency domains, encouraging the model to focus on smaller, more transferable patterns. The approach achieves state-of-the-art results on four CDFSL benchmark datasets. The paper also includes detailed analyses and ablation studies to support its claims.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback. Below are our detailed responses to your concerns:\n\n## **1. Justification of Clustering Hyperparameters**\n\nThe clustering threshold in Eq. 16 controls the **granularity of patch grouping** in the frequency domain. Higher threshold values (e.g., >40%) enforce stricter similarity criteria, resulting in smaller clusters (more fragmented groups). At the extreme (threshold → 100%), each patch forms its own cluster, which degenerates to non-cluster random sampling. Since some image regions occupy larger areas (e.g., background), this will lead patches in the dominant areas to change frequency with other patches in the same area, i.e., their frequencies are not changed, maintaining the original continuity. In contrast, lower thresholds (e.g., <40%) allow looser grouping, creating larger clusters. As shown in **Fig. 6b**, the optimal threshold (**30%**) achieves a critical balance.\n\nThe amplitude sampling standard deviation (ϵ) in Eq. 21 **governs the diversity of style proportions assigned to patch clusters** during frequency-domain disruption. Specifically, **smaller ϵ** results in near-uniform style proportions across clusters. This over-regularizes the disruption, limiting diversity and failing to suppress large patterns effectively.  As shown in **Fig. 7**, **Larger ϵ** introduces high variability in style mixing ratios, creating *heterogeneous disruptions* that break global continuity.\n\n\n\n## **2. Comparison with Diversity-Promoting Methods**\n\nWe add comparisons to state-of-the-art diversity-enhancing techniques:\n\n| Method       | CropDisease | EuroSAT    | ISIC       | ChestX     | **Average** |\n| :----------- | :---------- | :--------- | :--------- | :--------- | :---------- |\n| Random-Drop  | 91.23%      | 84.42%     | 43.89%     | 24.06%     | 60.90%      |\n| Wave-SAN  | 94.84%      | 88.79%     | 48.71%     | 26.98%     | 64.82%      |\n| **Ours**     | **96.02%**  | **90.42%** | **52.36%** | **28.23%** | **66.76%**  |\n\nOur method outperforms both approaches because:\n\nRandom-Drop indiscriminately discards patches, losing critical local patterns. Wave-SAN focuses on global frequency bands, overlooking localized high-frequency components critical for fine-grained transfer.\n\n\n\n## **3. Limitations and Non-ViT Applicability**\n\nOur method is **inherently ViT-dependent** due to:\n\n- Reliance on **patch-based tokenization** for spatial and frequency disruptions.\n- **Self-attention mechanisms** that propagate disrupted token relationships.\n\nExperiments with CNNs (DINO-ResNet50) show limited gains (**+0.7% average** vs. **+2.8% for ViT**), as CNNs’ overlapped sliding-window convolutions and local inductive biases hinder explicit token continuity control.  Notably, we also apply our methods to **MLP-Mixers** (see answer 5), and we observe **2.8%** improvements in target-domain accuracy, which verifies our adaptability to token-based structures.\n\n## **4. Sensitivity to Clustering Threshold**\n\nAs shown in Fig. 6b, performance remains stable for thresholds between 10%-35%, with optimal results at 30%.  This \"sweet spot\" balances local pattern retention and global disruption, indicating our method is not sensitive to the specific choice of clustering threshold.\n\n## **5. Generalization to Other Structures**\n\nWhile our method is designed for ViTs, we test an **MLP-Mixer backbone** (patch-based architecture):\n\n| Backbone | CropDisease | EuroSAT    | ISIC       | ChestX     | **Average** |\n| :------- | :---------- | :--------- | ---------- | ---------- | :---------- |\n| Baseline | 85.12%      | 78.34%     | 36.45%     | 22.31%     | 55.56%      |\n| **Ours** | **87.45%**  | **80.21%** | **40.12%** | **25.67%** | **58.36%**  |\n\nThis demonstrates applicability to **other patch-based architectures**. However, traditional CNNs (**+0.7% average**) remain incompatible due to their lack of explicit tokenization and overlapped sliding-window convolutions.\n\n------\n\nWe thank you for highlighting these critical points. Revised sections in the manuscript (marked in blue) address all concerns. Your feedback has significantly strengthened our work!", "claims_and_evidence": "The claim that disrupting the continuity of image tokens reduces domain gaps and improves transferability is supported by experiments on four benchmarks.", "methods_and_evaluation_criteria": "1. The proposed methods are reasonable and align well with the CDFSL task.\n2. The rationale for the warm-up phase and clustering in the frequency domain is intuitively explained, but the clustering threshold could use more justification.", "theoretical_claims": "The paper does not make strong theoretical claims or provide formal proofs. The focus is largely on empirical observations and practical methods.", "experimental_designs_or_analyses": "1. The experimental design is sound and well-structured, with comprehensive ablation studies and comparisons to state-of-the-art methods.\n2. The choice of specific hyperparameters (e.g., clustering threshold, amplitude sampling standard deviation) could be better justified.", "relation_to_broader_scientific_literature": "The paper builds on recent work in CDFSL and ViT architectures. It extends the understanding of positional embeddings and token continuity in ViT.", "essential_references_not_discussed": "No essential references are missing.", "other_strengths_and_weaknesses": "Strengths:\n1. The identified phenomenon about token continuity is novel and provides new insights into ViT's behavior under domain shifts.\n2. The visualizations and domain similarity analysis enrich the paper's interpretability.\n\nWeakness:\n1. The paper lacks a more detailed comparison of the proposed balanced disruption against other diversity-promoting methods.\n2. Some design decisions, such as the choice of clustering threshold, require stronger justification.", "other_comments_or_suggestions": "Include a discussion on the potential limitations of the method, such as its applicability to non-ViT models.", "questions_for_authors": "How sensitive is the model's performance to the choice of the clustering threshold in the balanced frequency-domain disruption?\nCan the proposed method be generalized to CNN-based models? If not, what are the limitations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oGIDrXFAJA", "reviewer": "Reviewer_MSD8", "summary": "This paper investigates the role of image token continuity in Vision Transformers (ViTs) for Cross-Domain Few-Shot Learning (CDFSL). The authors observe that disrupting token continuity (e.g., shuffling patches or perturbing frequency components) significantly degrades source-domain performance but only marginally affects target domains. They hypothesize that continuity aids ViTs in learning large spatial patterns, which are less transferable across domains, while smaller patterns within patches are more domain-invariant. Based on this insight, they propose a method combining spatial and frequency-domain disruptions to encourage reliance on smaller patterns, achieving state-of-the-art performance on CDFSL benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and constructive suggestions. Below are our responses to your concerns:\n\n## **1. More Proofs for Continuity**\n\nOur method is consistent with the proofs of the previous works [1-3] in that the small patterns are easier to transfer than larger ones. But we differ in that (1) our method discusses the transferability of small patterns for ViT while [1-3] are originally designed for CNNs; (2) we provide a simple way to enhance the learning of small patterns by perturbing patches with no additional branches or losses.\n\n[1] Cross-domain few-shot learning with task-specific adapters\n\n[2] Task-aware adaptive learning for cross-domain few-shot learning\n\n[3] Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning\n\n## **2. Frequency Disruption Breaks the Continuity**\n\nQualitatively, images from an anonymous GitHub link ([https://anonymous.4open.science/r/More-visualization/visualizations.png]) illustrate how our method breaks large-scale spatial patterns while preserving fine-grained, domain-agnostic features (e.g., textures, edges), which aligns with our motivation.\n\nQuantitatively, experiments in Fig. 2 and Fig. 4 validate the feasibility of our methods.\n\n## **3. Source domain performance trade-off**\n\nAs suggested, we have provided the performance on the source domain (miniImageNet) for reference. \n\n| Model    | Source Domain | Target Domain |\n| -------- | ------------- | ------------- |\n| Baseline | 97.78         | 63.91         |\n| **Ours** | 96.33         | 66.76         |\n\n While there is a slight performance trade-off on the source domain, this aligns with our hypothesis: disrupting token continuity prioritizes learning smaller, transferable patterns over domain-specific holistic features. Crucially, the significant gains on **target domains** (e.g., +6.5% on ISIC) demonstrate the effectiveness of our approach for cross-domain adaptation.\n\n## **4. Experiments with other pretraining and backbones**\n\n(1) Other pretraining \n\nWe have conducted experiments with CLIP pretraining. Results confirm our method’s generalizability.\n\n| Backbone | CropDisease | EuroSAT    | ISIC       | ChestX     | **Average** |\n| :------- | :---------- | :--------- | :--------- | :--------- | :---------- |\n| Baseline | 93.02%      | 74.37%     | 40.92%     | 23.99%     | 58.08%      |\n| **Ours** | **94.03%**  | **80.90%** | **43.31%** | **24.45%** | **60.67%**  |\n\n(2) Other backbone\n\nWhile our method is designed for ViTs, we test an **MLP-Mixer backbone** (patch-based architecture):\n\n| Backbone | CropDisease | EuroSAT    | ISIC       | ChestX     | **Average** |\n| :------- | :---------- | :--------- | ---------- | ---------- | :---------- |\n| Baseline | 85.12%      | 78.34%     | 36.45%     | 22.31%     | 55.56%      |\n| **Ours** | **87.45%**  | **80.21%** | **40.12%** | **25.67%** | **58.36%**  |\n\nThis demonstrates applicability to **other patch-based architectures**. \n\n## **5. Role of CDFSL vs. large pretrained models**\n\nWhile large vision-language models (VLMs) have demonstrated remarkable generalization capabilities, their effectiveness heavily relies on the assumption that downstream tasks share *similar data domains* with their pre-training corpora (e.g., natural images and generic text). However, in **vertically specialized scenarios** (e.g., medical imaging, remote sensing), where **domain gaps are extreme** and task-specific patterns diverge significantly from generic priors, **directly fine-tuning VLMs often yields suboptimal performance** [4], even worse than training domain-specific models (e.g., UNet) from scratch [5]. Pre-trained models like DINO struggle with medical X-rays (22% accuracy on ChestX) due to fundamentally different texture and structural semantics compared to natural images. However, training large vision-language models from scratch is impractical due to limited data in target domains. This highlights the necessity of CDFSL-specific designs to address domain shifts that challenge even powerful pre-trained models.\n\n[4] Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models\n\n[5] Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentation\n\n## **6. Computation cost**\n\nOur method introduces **no additional trainable parameters**.  \n\nDuring training, the computational overhead (baseline: 126.42s vs. ours: 140.48s per epoch) stems from the clustering-based frequency disruption during training, which is acceptable compared with the gains in cross-domain performance.  Moreover, some engineering tricks, such as bipartite matching, can accelerate the clustering process, which we leave for future works.\n\nDuring inference, since our method introduces no additional parameters, inference time matches the baseline exactly with no computational overhead.", "claims_and_evidence": "Yes, the authors have conducted several experiments to show the importance of image token continuity for CSFSL. However, more theoretical proof would be better for understanding the claim.", "methods_and_evaluation_criteria": "To some extent the proposed method is reasonable. Given the validated claim about image token continuity, the authors propose a method that breaks the continuity during training. As mentioned in L192-L195, such a method can keep local patterns undisrupted. However, it is not clear why the following frequency-domain disruption as in Sec.3.2 can hold such a property.", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "The authors have conducted comprehensive experiments to show the effectiveness of the proposed method. The ablation study can also validate the effectiveness of each design.", "relation_to_broader_scientific_literature": "The insight on the relationship between token continuity and pattern size can help future researches on the design of more powerful ViT backbones. The authors adopt amplitude shuffling but innovate by balancing disruptions across patch clusters. This addresses a limitation of naive frequency augmentation and improves transferability, bridging frequency-domain insights with CDFSL, and potentially other cross-domain tasks.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n1. The method is clearly explained.\n2. This paper brings novel insight about the ViT architecture.\n\n\nWeaknesses:\n1. It would be better if the continuity problem can be analyzed theoretically.\n2. I wonder if breaking continuity could harm the source domain performance severely.\n3. The authors can include experiments with more backbones.\n4. Despite the impact statement, I still have concerns about the role of CDFSL given so many large models with great generalization ability.\n5. Computation cost can be included in the paper.", "other_comments_or_suggestions": "Please refer to weaknesses.", "questions_for_authors": "Please refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper identifies an interesting impact of continuity of ViT tokens on generalization of ViT in few shot learning and propose a solution for it. Both the identified phenomenon and the experiments are well established and are of interest to the research community and therefore I recommend accepting the paper.
Decision: Accept (spotlight poster)