Mind the Gap Between Prototypes and Images in Cross-domain Finetuning

Authors: hongduan  tian
feng  liu
zhanke  zhou
tongliang  liu
chengqi  zhang
bo  han
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 10:37:25

Abstract: In _cross-domain few-shot classification_ (CFC), recent works mainly focus on adapting a simple transformation head on top of a frozen pre-trained backbone with few labeled data to project embeddings into a task-specific metric space where classification can be performed by measuring similarities between image instance and prototype representations. Technically, an _assumption_ implicitly adopted in such a framework is that the prototype and image instance embeddings share the same representation transformation. However, in this paper, we find that there naturally exists a gap, which resembles the modality gap, between the prototype and image instance embeddings extracted from the frozen pre-trained backbone, and simply applying the same transformation during the adaptation phase constrains exploring the optimal representation distributions and shrinks the gap between prototype and image representations. To solve this problem, we propose a simple yet effective method, _contrastive prototype-image adaptation_ (CoPA), to adapt different transformations for prototypes and images similarly to CLIP by treating prototypes as text prompts. 
  Extensive experiments on Meta-Dataset demonstrate that CoPA achieves the _state-of-the-art_ performance more efficiently. Meanwhile, further analyses also indicate that CoPA can learn better representation clusters, enlarge the gap, and achieve the minimum validation loss at the enlarged gap.

Main paper: 
 introduction: 
 Cross-domain few-shot classification [49] (a.k.a. CFC) aims at learning to perform classification on tasks sampled from previously unseen domains with only a few labeled data. Recent works [12, 34, 29, 30] , which aims at fast adapting a light model on top of the frozen pre-trained backbones for feature selection or transformation, has shown impressive generalization capability on CFC tasks. Typically, URL [29] , a representative work of this genre, proposes to fast fine-tune a linear transformation head on top of a frozen pre-trained backbone with the nearest centroid classifier loss [47] . The ultimate goal of URL is to project embeddings into a space where classification can be performed via measuring the similarity between representations of prototypes and image instances. Technically, an assumption implicitly adopted in URL is that the prototype and image instance embeddings share the same representation transformation (see the upper subfigure of Fig. 2 ) during the adaptation phase. Intuitively, however, prototype and image instance embeddings depict different levels of information. Specifically, the embeddings of an image instance encode the instance-level information from the given image, while the embeddings of a prototype contain some abstract and higher-level information that is commonly shared among image instances within the corresponding class yet discriminative from other classes. Thus, applying the same representation transformation as URL might constrain models from exploring the optimal representations respectively for prototypes and image instances. shows that there naturally exists a gap, which resembles the "modality" gap in visual language models, between prototype and image instance embeddings extracted from a frozen pre-trained backbone. However, Fig. (b) shows that the gap between the representations of prototypes and image instances is shrunk after applying the same representation transformation to both the image instance and prototype embeddings.In this paper, we first conduct an analysis to validate the intuition that prototypes and image instances describe different levels of information. Specifically, following Liang et al. [32] , we visualize the distributions of prototype and instance embeddings extracted from a frozen pre-trained backbone and measure the Euclidean distance between the centroids of the normalized prototype and image instance embeddings. According to Fig. 1 (a), the UMAP [37] visualization indicates that there naturally exists a gap, which resembles the modality gap demonstrated by Liang et al. [32] in visual language models (VLMs, e.g. CLIP [42] ), between prototype and image instance embeddings. However, after applying the same representation transformation to both prototype and image instance embeddings as done in URL [29] , we observe that the gap is numerically shrunken (see Fig. 1(b) ). Moreover, by visualizing the cluster distribution of data samples (see Figs. 3(b ) and 3(c)), we further find that applying the same representation transformation to both prototype and image instance embeddings may fail to learn compact and clear representation clusters. With all these empirical results taken into consideration, two aspects can be summarized here: (1) there does exist a gap between the prototype and image instance embeddings extracted from the frozen backbone; (2) the shared representation transformation tends to shrink the gap between the learned prototype and image instance representations and may potentially result in the failure of learning clear and compact representation clusters for classes in the task. Our further investigation into the URL framework in Section 3.3 indicates that applying the same transformation, on the one hand, removes the discriminative information in gradients. On the other hand, our empirical results further reveal that applying the shared transformation constrains learning representations where the gap between prototypes and image instances is preserved.Previous work [32] has demonstrated that contrastive loss helps preserve the modality gap and further contributes to improving the generalization performance of downstream tasks. Thus, in this paper, we propose a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformations for prototype and image embeddings similarly to CLIP [42] by treating prototypes as text prompts. In this way, the discriminative information in gradients can be preserved in different parameters, and the optimal transformations, where the gap between prototype and image representations is held, can be explored in the function space constructed by the two transformations.Extensive experiments on the representative Meta-Dataset benchmark [49] under several task settings demonstrate that CoPA can achieve the state-of-the-art performance with less time consumption and learn more clear and compact representation clusters. In addition, further analysis reveals that the global minimum of the validation loss is achieved at the enlarged gap, which implies the enlargement of the gap is a process of exploring representation distributions for better generalization performance.Our Contribution. In this paper, our contributions can be summarized in the following 4 parts:• We validate there naturally exists a gap, which resembles the modality gap, between prototype and image instance embeddings extracted from a frozen backbone in Section 3.2. • Our analyses in Section 3.2 reveal that the shared representation transformation shrinks the gap between prototype and image instance representations. Our investigation into the URL in Section 3.3 indicates that applying the same transformation removes the discriminative information in gradients and constrains learning representations where the gap is preserved. • To solve this problem, in Section 4, we propose a simple yet effective method, contrastive prototype-image adapation (CoPA), to adapt two different transformations for prototypes and image instances as done in CLIP via substituting text prompts with prototypes. • Extensive results reveal that CoPA can achieve the state-of-the-art performance on Meta-Dataset [49] (Section 5.1), enlarge the gap to achieve the global minimum of the validation loss (Section 5. The datasets, in which training sets are used for pre-training, are called seen domains while the remaining datasets, which are only available for evaluation, are called unseen domains. Thus, the meta dataset can be alternatively expressed as S = S seen ∪ S unseen , S seen ∩ S unseen = ∅, where the notations S seen and S unseen respectively denote the seen and the unseen domains.Typically, a cross-domain few-shot classification task with vary-way vary-shot is randomly sampled in an episodic way. To be specific, at the beginning of each learning episode, a new task T = {D T , Q T } is sampled from a specific sub-dataset of the meta dataset S i ∈ S, whereD T = {(x s i , y s i )} |D T |i=1 denotes the support set of the sampled task, which is used for model training and Q T = {(x q i , y q i )}|Q T | i=1 denotes the query set of the task that is used for model evaluation.Problem Formulation. In this paper, we follow the pre-training framework adopted in previous works [12, 34, 29, 30] and build our method upon a frozen pre-trained backbone.Consider a frozen pre-trained backbone f ϕ * : R din → R dout that is parameterized with a set of optimal parameters ϕ * and a trainable fine-tuning module h θ : R dout → R dout that is parameterized with θ. Given a set of support data D T = {(x i , y i )}|D T | i=1 , we then can obtain a set of representations Z = {(z i , y i )} |D T | i=1 , where z i = h θ (f ϕ * (x i )), from the frozen pre-trained backbone and the fine-tuning module. The ultimate goal of the pre-training framework is learning a set of optimal task-specific parameters θ * from the given support data so that better generalization performance can be achieved on the query data in the task. To be specific, following previous works [34, 29] , the learning problem is formulated as minimizing a nearest centroid classifier (NCC) loss [47] :EQUATIONwhere p(y = c|z; θ) = e d(z,cc ) c ′ e d(z,c ′ ) denotes the likelihood of a sample z belonging to its class c, d(•, •) denotes a measure, such as negative Euclidean distance function or cosine similarity function, to describe the similarity between samples and prototypes. In this paper, we follow URL [29] and adopt cosine similarity as the measure. Moreover, c c denotes the prototype of class c, which is generated with all available samples in the class c and formulated as:c c = 1 |Cc| z∈Cc z, C c = {z i |y i = c}. 
revisit the previous adaptation strategy: 
 In this section, we first revisit the adaptation strategy applied in the previous work and uncover an implicitly adopted assumption that the prototype and image instance representations are transformed with the same (linear) transformation. However, prototypes intuitively depict different levels of information from image instances. Then, we empirically demonstrate that there naturally exists a gap between prototype and image instance embeddings. Our further investigation into URL reveals that applying the same transformation to prototypes and images removes the discriminative information in gradients and constrains learning compact representation clusters where the gap is preserved.In previous works [29, 30] that are based on prototypes, cross-domain few-shot classification is typically formulated as a maximum likelihood estimation problem as mentioned in Eq. (1) . The ultimate goal of this problem is to learn a set of representations such that classification can be performed by measuring the similarity between representations of image instances and prototypes.Typically, the prototype of each class is calculated with all instances within the class (see Section 2 for concrete details). Assume that the transformation is defined as a linear transformation parameterized with Θ ∈ R dout×dout , then the calculation of the prototype c of the class set C can be formulated as:EQUATIONSince the representation transformation is usually linear, the average of image instance representations (the second part) is equivalent to the representation transformation of prototype embeddings (the third part). Thus, it is easy to observe that an assumption is implicitly adopted in the URL framework:Instance-level and prototype-level embeddings share the same representation transformation.Intuitively, compared with image instance representations, the prototype representations contain higher-level information which is commonly shared among the instances within the class yet discriminative from that of other classes. Thus, there seems to exist a gap between prototypes and images. Such an intuition motivates us to take a further step to explore whether such a gap exists and what effect the shared representation transformation imposes on such a gap. To this end, we perform a series of analyses on the prototype and image instance embeddings and their corresponding transformed representations. In addition, we also investigate the mechanism of the adaptation strategy.In this section, we conduct an analysis on the prototype and image instance embeddings and their corresponding representations obtained from the same representation transformation. Specifically, the embeddings are extracted from a frozen pre-trained ResNet-18 [18] backbone, and the representations are transformed from the embeddings with a shared linear transformation model as done in URL [29] ."Modality" Gap between Prototype and Image Instance. According to Liang et al. [32] , there exist modality gaps between different modal embeddings, such as text and image embeddings extracted from visual language models (e.g. CLIP), and preserving such modality gaps facilitates improving the downstream performance. We notice that the prototypes, to some extent, play the same role as text prompts. Specifically, both text prompts and prototypes depict the common concepts shared among all images within the corresponding class yet discriminative from other classes. This observation motivates us to validate whether such a "modality" gap exists between prototypes and image instances.In this section, an empirical analysis is performed on the validation set of ImageNet [44] . Following Liang et al. [32] , we define the gap as the difference between the centroids of normalized prototype and image instance representations (embeddings We visualize the distributions of prototype and image instance embeddings with UMAP [37] in Fig. 1(a) . As shown in the figure, it is easy to observe that the prototype and image instance embeddings are located in two completely separate regions of the feature space, and there naturally exists a gap between prototype and image instance embeddings. Moreover, to verify the general existence of such a gap, we conduct an analysis with 600 randomly sampled tasks on each of all 8 seen domain datasets in Meta-Dataset. The results are reported in Table 3 in Appendix B. The results reveal that the gap also generally exists between prototype and instance embeddings in other datasets.) ⃗ ∆ := 1 |D T | |D T | i=1 z i -1 N C N C j=1 c j ,Larger Gap Facilitates Generalization. To further figure out the property of the prototype-image gap, we conduct the same embedding shift experiment as done in previous work [32] . Specifically, we manually shift prototype and image instance embeddings by scaling the gap between two sets of embeddings/representations to narrow or enlarge the gap and observe the change of validation loss.The results are reported in Fig. 3 (a). The X-axis depicts the size of the gap where the negative value means the position of prototype and image embeddings are mutually replaced. From the figure, it is easy to observe that the validation loss fails to achieve its global minimum at the original gap between prototype and image instance embeddings. Interestingly, we can further observe that the global minimum can be achieved by slightly enlarging the gap. This is consistent with the phenomenon observed by Liang et al. [32] . Thus, an intuitive insight is that appropriately enlarging the gap between the prototypes and image instances contributes to achieving better generalization performance.From our perspective, the phenomenon that the larger gap facilitates improving the generalization performance can be attributed to two aspects. On the one hand, each prototype is generated with all instances within the class. Thus, data samples are naturally closer/more similar to the prototypes of their own class. This potentially results in slight overfitting. Thus, when the gap is slightly enlarged, it is equivalent to decreasing the similarity between prototypes and images. Such an enlargement, in turn, helps alleviate the overfitting and further improve the generalization performance on downstream tasks. On the other hand, as images and prototypes describe different levels of information, good generalization performance can only be achieved when the embeddings are well aligned. Thus, enlarging the gap can also be seen as exploring the optimal distributions of embeddings for alignment.Drawbacks of Shared Transformation. In order to determine the effect of the shared representation transformation on representation learning, we further analyze the prototype and image instance representations obtained from the existing SOTA baseline, URL [29] . The visualization results are respectively presented in Figs. 1(b) and 3(c). According to Fig. 1 (b), we can observe that the gap between prototype and image instance representations is numerically shrunken after applying the same transformation to both prototype and image instance embeddings. Meanwhile, we can also observe from Fig. 3(c ) that URL fails to learn well-clustered representations for classes in the task. All these results, to some extent, demonstrate the drawbacks of applying the shared transformation.In this section, we conduct a further investigation to explore the effect of the shared transformation. L(θ) ≥ - 1 n n i=1 z ⊤ i c c + α n n i=1 z ′ ∈Z z ⊤ i z ′ ,where z ′ is an independent copy of samples in Z, C c denotes sets of sample representations C c = {z i |y i = c}, and α is a constant that satisfies 0 ≤ α < 1/(N C |C j |) for ∀j.The complete proof is available in Appendix G.1. Theorem 3.1 provides a lower bound of Eq. ( 1).According to the bound, if we adopt the right side as the surrogate loss, solving Problem (1) is then equivalent to simultaneously maximizing the similarities between instances and their corresponding prototypes and minimizing the similarities among all samples. Moreover, since z ⊤ i z j < z ⊤ i z i for ∀j ̸ = i, the second term can then be approximated as z∈Z z ⊤ z by omitting the similarity term between different image instance representations. Then, we can reformulate Problem (1) as:L(Θ P , Θ I ) = - 1 |D T | Tr f ϕ * (X)Θ I (Y Y ⊤ f ϕ * (X)Θ P ) ⊤ + α |D T | Tr f ϕ * (X)Θ I Θ ⊤ I f ϕ * (X) ⊤ ,where X ∈ R |D T |×dout and Y ∈ R |D T |×N C respectively denote the support image instances and the corresponding one-hot labels, Θ P ∈ R dout×dout and Θ I ∈ R dout×dout denote the model parameters of linear transformation heads respectively for prototype and image instance embeddings, Tr(•) denotes the matrix trace operation. Y Y ⊤ f ϕ * (X) ∈ R |D T |×dout denotes the prototypes which are expanded to the same size of instance embeddings. In this way, the gradients w.r.t. Θ P and Θ I are:∇ ΘP L(Θ P , Θ I ) = - 1 |D T | Θ ⊤ I f ϕ * (X) ⊤ Y Y ⊤ f ϕ * (X), ∇ ΘI L(Θ P , Θ I ) = - 1 |D T | Θ ⊤ P f ϕ * (X) ⊤ Y Y ⊤ f ϕ * (X) + 2α |D T | Θ ⊤ I f ϕ * (X) ⊤ f ϕ * (X).According to the gradients above, we can observe that the gradients of Θ P depict the similarity between instance representations and prototype embeddings while the gradients of Θ I depict the similarity between prototype representations and instance embeddings. However, in the case of the shared transformation (i.e., Θ P = Θ I = Θ), such differences are removed. Thus, the shared transformation may potentially drop the discriminative information in gradients during the adaptation phase. Meanwhile, the shared transformation also constrains the complexity of the function space, and in turn, prevents exploring the optimal representation distributions for both prototypes and images.Theorem 3.2 (The shared transformation). Consider a support data setD T = {(x i , y i )} |D T | i=1composed of N C classes and a frozen pretrained backbone f ϕ * : R din → R d parameterized with the optimal parameters ϕ * . Let Θ ∈ R d×d be a shared linear transformation across the prototype and image instance embeddings. Then, we can obtain the image instance representationsZ = {z i } |D T | i=1 = {f ϕ * (x i )Θ} |D T |i=1 , and the prototype representationsC = {c i } N C i=1 , where c i = 1 |Ci| z ′ ∈Ci z ′ = 1 |Ci| x ′ ∈Ci f ϕ * (x ′ )Θ.Then we can obtain the bounds of the representation gap:m ∥Θ∥ 2 F ⃗ ∆ emb 2 2 ≤ 1 |D T | z∈Z z - 1 N C c∈C c 2 2 ≤ M ∥Θ∥ 2 F ⃗ ∆ emb 2 2 ,where⃗ ∆ emb = 1 |D T | x∈D T f ϕ * (x) -1 N C N C b=1 1 |C b | x ′ ∈C b f ϕ * (x ′) denotes the gap between prototype and image embeddings, m = min 1≤i≤d cos 2 ( ⃗ ∆ emb , Θ i ) denotes the minimum value of cos 2 ( ⃗ ∆ emb , Θ i ), and M = max 1≤j≤d cos 2 ( ⃗ ∆ emb , Θ j ) denotes the maximum of cos 2 ( ⃗ ∆ emb , Θ j ). The proof is provided in Appendix G.2.1. In Theorem 3.2, we derive the bounds of the gap between prototype and image representations learned from a shared linear transformation. For simplicity, we only consider the unnormalized representations. However, in practice, the gap is calculated with normalized representations. According to the theorem, we find that the bounds are closely related to the Frobenius norm of the transformation matrix and the angle between the embedding gap vector and the column vectors of the transformation matrix.Theoretically, it is intractable to directly analyze the upper bound of the gap between prototype and image representations. Thus, to determine the reason for the gap shrinkage, we empirically track the change of the scale √ M ∥Θ∥ F on 600 randomly sampled tasks. The insight here is that the representation gap will not be larger than the embedding gap if the scale is smaller than 1.0. According to curves in Fig. 4 , we observe that the scale is consistently smaller than 1.0 in all cases. Thus, a shared transformation cannot preserve the original gap between prototype and image instance embeddings.Table 1 : Results on Meta-Dataset under the "train on all datasets" setting. Under the "train on all datasets" setting, the first 8 datasets are treated as "seen domians" while the last 5 are treated as "unseen domains". Mean accuracy and 95% confidence interval are reported. In this section, to address the problem mentioned above, we follow CLIP [42] and propose a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformation modules respectively for the prototypes and image instances in a similar way of contrastive learning.Backbone Pre-training. We pre-train a ResNet-18 [18] in the same way as URL [29] as the backbone. Specifically, 8 domain-specific backbones respectively for all seen domains are firstly pre-trained. Then, a universal encoder is distilled from these backbones and frozen during the meta-test phase.Contrastive Prototype-Image Adaptation. Briefly, CoPA aims to adapt two different transformation models respectively for prototype and image instances by optimizing the symmetric cross-entropy loss. The entire pipeline of CoPA resembles that of CLIP [42] . CoPA replaces the text prompts with prototypes with the simple insight that both prototypes and text prompts depict some higher-level concepts that are common among samples within a class yet discriminative from other classes.Specifically, consider two representation transformation heads, h θP and h θI , respectively, for prototypes and image instances. Given a support set D T = {X, Y }, where X ∈ R |D T |×dout de- notes the support data samples, while Y ∈ R |D T |×N C denotes the corresponding one-hot labels, where N C denotes the number of classes, we can respectively obtain the prototype representationsZ P = h θP (Y Y ⊤ f ϕ * (X)) and image instance representations Z I = h θI (f ϕ * (X)).Then, CoPA learns the representations by minimizing the following symmetry cross entropy objective:EQUATIONwhere τ is a temperature coefficient,Y pseudo = [0, 1, ..., |D T | -1]denotes the pseudo labels, and L CE denotes the cross-entropy loss. The complete CoPA algorithm is presented in Algorithm 1.Discussion. Two aspects are worth noticing in our proposed CoPA. On the one hand, two different transformations are adopted in our proposed CoPA method. In this way, the discriminative information in gradients is preserved in different sets of parameters. Besides, introducing different transformations expands the function space, which further facilitates learning better prototype and image instance representations. On the other hand, the contrastive learning objective, which has been demonstrated to facilitate preserving the modality gap, is adopted as the learning objective. In particular, to match this objective, we modify the prototype embeddings from Y ⊤ f ϕ * (X) to Y Y ⊤ f ϕ * (X) so that the size of the prototype embeddings is consistent with that of image instance embeddings. Such a modification further indicates the cluster structure of the given support data set. Thus, by minimizing the symmetric cross entropy loss, the two transformations are encouraged to align the prototype and image instance representations and learn more class-specific and discriminative representations. 
experiments: 
 In this section, we evaluate CoPA on Meta-Dataset [49] under both "train on all datasets" and "train on ImageNet only" settings with a series of tasks to answer the following questions: (1) Does CoPA achieve better generalization performance? ( 2) Does CoPA benefit from extra learning modules? (3) Why does CoPA perform better? Detailed experimental settings are available in Appendix E.In this section, we evaluate the generalization performance of our proposed CoPA method on Meta-Dataset under both "train on all datasets" and "train on ImageNet only" settings. In order to demonstrate the effectiveness of CoPA, we compare CoPA against several currently state-of-the-art baselines, including fo-Proto-MAML (FP-MAML) [49] , ALFA+fo-Proto-MAML (AFP-MAML) [2] , CNAPs [43] , Simple-CNAPs [3] , SUR [12] , URT [34] , Tri-M [35] , FLUTE [50] , URL [29] .Train on All Datasets. We report the evaluation results under the "train on all datasets" setting in Table 1 . As shown in the table, it is easy to observe that CoPA achieves the best performance on 9 out of 13 datasets among all approaches and outperforms URL in all cases. Specifically, compared with URL, our proposed CoPA method achieves 0.7%, 1.5%, and 1.0% improvements on average respectively on seen, unseen, and all domains. It is also worthwhile to notice that CoPA performs better on unseen domains. Specifically, our proposed CoPA method achieves 3.3%, 2.1%, and 1.1% improvements respectively on Traffic Sign, MSCOCO, and CIFAR 10 datasets. Meanwhile, we also notice that CoPA achieves comparable or even better results on seen domains compared with TSA [30] which plugs extra learning modules into the frozen backbone for more task-specific parameters.Train on ImageNet Only. The results under the "train on ImageNet only" setting are reported in Table 2 . Briefly, CoPA achieves the best performance on 11 out of 13 datasets and consistently outperforms URL in all cases. On average, CoPA achieves 0.4%, 2.1%, and 1.8% improvements respectively on seen, unseen, and all domains. Moreover, the results also reveal that CoPA consistently performs better on unseen domains. Specifically, CoPA achieve impressive improvements on Omniglot (1.5%), Aircraft (4%), Birds (1.3%), Textures (1.8%), Quick Draw (3.4%), Fungi (1.8%), VGG_Flower (2%), Traffic Sign (2.6%), MSCOCO (3.1%), MNIST (1.1%), CIFAR (2.9% & 0.7%).Extra Learning Modules. We follow TSA [30] to plug extra trainable modules into the frozen backbone.The results are reported respectively in Table 1 and 2 . According to the table, we can observe that CoPA benefits from these extra modules and achieves better performance than previous state-of-the-art approaches (e.g., TSA and TA 2 -Net). CoPA + TSA outperforms 11 out of 13 datasets under the "train on all datasets" setting. Similarly, CoPA + TSA consistently achieves better generalization performance on unseen domains.Specifically, CoPA + TSA achieves 5%, 2.6%, and 0.8% improvements respectively on Traffic Sign, MSCOCO, and MNIST datasets. Under the "train on ImageNet only" setting, CoPA + TSA achieves the best performance on 7 out of 13 datasets.Compared with TSA, CoPA + TSA achieves better performance on Birds (0.7%), Textures (0.8%), Fungi (1.5%), VGG_Flower (0.8%), Traffic Sign (4.5%), MSCOCO (1.6%) and MNIST (1.5%).In previous experiments, we have demonstrated that CoPA can achieve better generalization performance. In this section, we aim to validate whether CoPA preserves the gap and learns better representations. Further, we would like to explore why CoPA performs better than previous works.More Results. To verify the effectiveness of CoPA, we conduct more analyses. Firstly, we investigate the representation shift experiment to observe the gap between prototype and image representations.As shown in 5(a), in contrast to URL, the gap between prototype and image representations learned via CoPA is enlarged. Such a phenomenon demonstrates that CoPA can hold the gap between prototypes and image instances. In addition, we further check the image instance representations learned with CoPA. According to Fig. 5 (b), the image representations are more clearly and compactly clustered than Fig. 3(c ). This case reveals that CoPA can facilitate learning more class-specific representations.The results reported above indicate that CoPA can enlarge the representation gap and learn wellclustered image representations simultaneously. In order to further figure out why CoPA is able to improve performance, in this section, we propose to conduct ablation studies on the two important components of CoPA: the different transformations and SCE loss. The results are reported in Table 10 . Ablation: Different Transformation Modules. As we have discussed in Section 3.3, the shared transformation may drop the discriminative information in gradients. Thus, we propose to substitute the shared transformation with the different transformations adopted in CoPA. As shown in the table, although URL+2Heads achieves slightly better results in some cases, its performance drops in the remaining cases. By plotting the test loss curves (cf. Fig. 6 ), we can observe that overfitting takes place.Ablation: SCE Loss. From the table, we can observe that URL with SCE loss achieves comparable performance to the URL baseline in most datasets. Meanwhile, URL with SCE achieves significantly better results on some datasets, such as Fungi and MSCOCO, in which URL baseline tends to overfit (cf. Figs. 27(g) and 27(j)). This indicates that SCE loss can improve generalization performance. In addition, we also replace the SCE loss with the NCC loss on CoPA. According to the results of "CoPA + NCC" shown in the table, it is easy to find that the performance in all cases drops drastically. For example, the performance on Fungi and MSCOCO respectively decreases by 11.6% and 18.7%. All these empirical results demonstrate that SCE loss facilitates improving generalization performance.Effectiveness of Different Transformations. As aforementioned, an advantage of CoPA is applying two different transformations respectively to prototype and image instance embeddings. The goal of applying two different transformations in CoPA is to preserve the discriminative information of gradients in different sets of parameters. Meanwhile, applying two different transformations also increases the complexity of the hypothesis space, which contributes to reducing the approximation error and learning representations in a more flexible way. However, simply increasing hypothesis space complexity may also result in overfitting (cf. Fig. 6 ). According to Fig. 27 , CoPA, which is equipped with SCE loss, benefits from the increased complexity and achieves better performance.Essence of Gap Enlargement. In Fig. 3 (a), we observe that the global minimum of the validation loss can be achieved by slightly enlarging the gap between prototype and images. Meanwhile, according to Fig. 5 (c), the validation loss achieves its global minimum at the enlarged gap learned with CoPA. As aforementioned, we conjecture that the reasons for the phenomenon that gap enlargement facilitates generalization mainly include two aspects: representation alignment and overfitting alleviation.We first start with representation alignment. Typically, representation alignment is performed with SCE loss. Specifically, we omit the temperature coefficient τ and simply rewrite Eq. ( 3) as:EQUATION)Theorem 5.1. Given a set of normalized finite support data representation Z = {(z i , y i )} n i=1 and a set of normalized prototype representations C = {c i } n i=1 , where ||z|| 2 = 1 for ∀z ∈ Z and ||c|| 2 = 1 for ∀c ∈ C, then we are able to obtain a lower bound of SCE loss in Eq. ( 4):L SCE ≥ - 2 n n i=1 z ⊤ i c i + 2 n n i=1 N C k=1 |C k | n z ⊤ i c k ,where C k denotes the set of support data of the class k and N C denotes the number of classes.The proof is in Appendix G.3. According to Theorem 5.1, we find that the lower bound of SCE loss plays a similar role to that of NCC loss. In detail, the lower bound aims at maximizing the similarity between each sample and its corresponding prototype while minimizing the similarities between the sample and all prototypes. When minimizing the lower bound as the surrogate loss, the second term, which measures the similarities between image and prototype representations, is minimized. Thus, the Euclidean distances between representations of prototypes and images (i.e., the gap) are enlarged.An interesting point is that the similarities between images and prototypes are minimized with the weights calculated based on the size of C k . Differently, in NCC loss, the weights are fixed to 1 N C for all cases. Thus, the similarities between samples and the prototypes involving more samples will be significantly reduced. Consequently, the gap between images and prototypes tends to be enlarged.For the overfitting, on the one hand, the potential overfitting discussed in Section 3.2 is mitigated by applying different transformations. On the other hand, the high similarities between samples and prototypes may also result in overfitting. For example, in Fig. 3(a) , the validation loss increases when the gap is narrowed. However, as we have discussed above, minimizing SCE loss tends to reduce such similarities. Thus, the gap enlargement also functions as a regularization to alleviate overfitting.Therefore, with all aspects taken into consideration, the essence of CoPA is a representation alignment of prototypes and images, in which a balance between learning discriminative representations and achieving better generalization performance, is explored in a more flexible hypothesis space. 
conclusion: 
 In this paper, we validate that there naturally exists a gap, which resembles the modality gap in visual language models, between prototype and image instance embeddings, and uncover that the shared transformation shrinks such a gap and constrains the learning of well-clustered representations. In order to solve these problems, we propose CoPA to adapt different transformations respectively for prototype and image instances via optimizing SCE loss. Empirical results on Meta-Dataset reveal that CoPA can achieve state-of-the-art performance. Our further analyses indicate that the essence of CoPA is a representation alignment, where a balance between learning discriminative representations and achieving better generalization performance is explored in a more flexible hypothesis space. 
 
 Appendix: In this paper, in order to solve the problem that the shared transformation adopted in previous works potentially hurts the discriminative information in gradients and constrains learning representations where the gap is preserved, we propose to fine-tune two different transformations respectively for prototype and image instances in the same way of CLIP by treating the prototypes as text prompts.One potential limitation of our proposed CoPA method is the symmetric cross-entropy. Symmetric cross entropy loss is commonly applied in contrastive learning. In contrastive learning, the batch size of the data samples is an important issue for better downstream tasks. Larger batch size usually means better downstream performance. We also notice this phenomenon in this paper. Although CoPA can still achieve relatively better performance, the performance tends to degrade when the size of the data batch becomes small. Thus, this constrains the application of CoPA to some extent.In this paper, we find that there exists a gap, which resembles the modality gap in visual language models, between prototype and instance embeddings. An intuition of such a phenomenon is that prototypes are generated with all data samples within a specific class and consequently can be treated as an abstract concept of all data samples in the class. From this perspective, the prototype information plays the same role as text prompts in visual language models to some extent. Thus, our work provides a direction that we can extract some abstract concept information from the available single modal data and then train/adapt models with such information in some appropriate way, such as contrastive learning, to learn more discriminative and useful representation for downstream tasks. Since visual information is more diverse than texts, we think it will be a promising and feasible way of developing general models for various kinds of visual tasks. Based on this, there are many potential societal consequences of our work, none of which we feel must be specifically highlighted.Cross-domain few-shot classification. Cross-domain few-shot classification is a learning paradigm that aims at learning to perform classification on previously unseen data and domains by merely fast training a model on several labeled training data. Generally, the cross-domain few-shot classification problem can be seen as an extension of conventional few-shot classification problem [52, 14, 47, 6] . The main difference between conventional and cross-domain few-shot classification lies in the domains for evaluation. In the conventional few-shot classification setting, a model is trained and evaluated on the same dataset (domain), where the distribution is invariant. However, in the crossdomain few-shot classification setting, a model is trained on the training sets of several datasets with different distributions and then evaluated on the test sets of datasets that are used for training as well as the datasets that have never been observed during the training/adaptation phase. Thus, compared with the conventional few-shot classification task, the cross-domain few-shot classification task is much more challenging since there exist distribution gaps between the training and previously unseen test datasets. Currently, all existing works regarding cross-domain few-shot classification are mainly based on conventional meta-learning methods, such as Prototypical Networks [47] and MAML [14] . The former is famous for its strong ability to learn effective inductive bias while the latter is impressive for its flexibility of adaptation to new tasks. Generally, the existing works can be divided into two different types according to their ways of training the backbones and classifiers. Some of the existing works train the model from scratch with a sequence of few-shot tasks in the same way as conventional few-shot learning frameworks. A typical case of this kind of learning paradigm is Proto-MAML [49] . Specifically, Proto-MAML proposes to train a model with the modified Prototypical loss in the same way as MAML to take advantage of the strengths of the two frameworks. We would like to note that Prototypical loss plays an important role in current cross-domain few-shot classification research. The reason behind this is that the tasks adopted in cross-domain few-shot classification tasks are usually sampled with varied numbers of classes and shots [49] . In this case, it is infeasible to train a classifier in the same way as the conventional meta-learning framework, where a parameterized classifier is initialized with a fixed number of classes. In addition to Proto-MAML, more works are proposed to solve cross-domain few-shot classification tasks. CNAPs [43] proposes to generate parameters with feature-wise linear modulation (a.k.a., FiLM [41] ) for task-specific modules in their model. However, a potential problem of CNAPs is that the Euclidean distance adopted in the loss implicitly assumes that each cluster is distributed according to a unit normal. This may further result in misclassification of data points. Based on this observation, SimpleCNAPs [3] further proposes a non-parametric classifier from the perspective of class-covariance-based distance metric to improve the classification performance of the original CNAPs method. Moreover, Doersch et al. [10] notice that the representations learned via the Prototypical Network framework may only represent the observed data yet discard the information that might contribute to the recognization of out-ofdistribution classes. Thus, in order to solve this problem, they further propose CrossTransformer to leverage query image information in pixel-level during the adaptation steps via an attention module.Different from the learning frameworks that train the entire pipeline from scratch, the other kinds of methods usually respectively pre-train one or several domain-specific backbones on all available datasets during the training phase in a conventional supervised learning paradigm. Then, during the evaluation phase, the prior knowledge in these pre-trained backbones is transferred to the new tasks through representation fusion or representation adaptation. In cross-domain few-shot classification community, SUR [12] firstly proposes to select relevant features from the 8 pre-trained backbones to represent the data of the given new task via a vector. Later, URT [34] further proposes to quickly train a multi-head transformer on top of 8 pre-trained backbones to learn to select features for unseen data. Besides, Triantafillou et al. [50] proposes to treat the convolutional layers in a model as universal templates while treating the batch normalization layers as the domain-specific modules. Thus, FLUTE is proposed to pre-train a set of universal weights and several domain-specific BN modules and then train a small model to learn to combine these BN layers and transfer the prior knowledge to the new tasks. Although leveraging pre-trained backbones is able to take advantage of high-quality representations learned from previous domains, it is quite time-consuming to perform several forward passes during both training and evaluation phases. In order to solve this concern, Li et al. [29] proposes URL to distill the prior knowledge of all 8 pre-trained backbones in one single backbone. Then, during the test phase, a simple linear layer is fast fine-tuned on top of the distilled multi-domain backbone to transfer the priori to the unseen domains. Later, TSA [30] is proposed to enhance URL via plugging more learnable residual modules into the frozen pre-trained backbone to learn more task-specific features for unseen data. Meanwhile, recent work TA 2 -Net [16] notices that different tasks prefer different learnable modules and proposes to train an Action Generation Network in the way of reinforcement learning to learn to select the optimal module for each task.Contrastive loss and contrastive learning. The ultimate goal of contrastive learning is to learn discriminative and robust representations at the instance level by comparing two augmented counterparts of an image. In practice, the goal is realized through contrastive loss which focuses on maximizing the similarity between positive pairs while minimizing the similarity between negative pairs. Compared with early contrastive loss [8, 17, 46] , the current contrastive loss is constructed upon a batch of data, where the number of negative samples is larger than that of positive samples. Such contrastive loss was first proposed by Sohn [48] as multi-class N-pair loss and then improved and further explored by further works [5, 19, 7, 15, 55, 4] . Based on the InfoNCE, Li et al. [31] further explores an alternative loss from the perspective of the Hilbert-Schmidt Independence Criterion measure. Different from conventional unsupervised contrastive learning, Khosla et al. [23] proposes a fully supervised contrastive learning paradigm, SupCon, via leveraging label information. To be specific, in conventional unsupervised contrastive learning, only the augmented counterparts of an image are treated as positive images. However, in supervised contrastive learning, all images that share the same "general" concept are treated as positive samples. The ultimate goal of SupCon is to pull normalized embeddings from the same class closer than embeddings from different classes. Then, with the emergence of pre-trained vision language models, such as CLIP [42] , embeddings that are generated from other modalities are treated as supervised information and have shown impressive power in multi-modality learning fields. From our perspective, SupCon and CLIP share the same core. The key difference is that SupCon learns representations by comparing each sample with all other samples within the same class while CLIP learns to align the representations of each image sample with the representations of text information that are more "general" than a set of images.Multi-modal contrastive representation learning. Multi-modality learning [21, 28, 54, 56] is a learning paradigm that aims at aligning representations extracted from different data modalities, such as text and images. Since the emergence of CLIP [42] has shown impressive performance on downstream tasks, more and more attention has been attracted to this research topic. Recently, Liang information of a category. Thus, we conjecture that the prior information respectively contained in prototypes and image instances is at different levels. To be specific, image instance representations are encoded with instance-level information closely related to the contents in the given image while prototype representations are a set of class-specific representations. According to the visualizations in [32] , the image and text prompt embeddings extracted from visual language models are located in completely separated areas of the feature space. In that work, the reason for such a phenomenon is attributed to the different kinds of modalities of the input data. Thus, the gap between the clusters of the extracted text and image embeddings is correspondingly called the modality gap.As aforementioned, the prototype resembles the text prompt to some extent since both of them describe some high-level information commonly shared among images within a class. Thus, similar to the modality gap, there might also exist some type of gap between prototype and image representations.To further determine our intuition, in this section, we propose to conduct an analysis to figure out whether the gap between the prototype and image instance embeddings/representations exists.Analysis Settings. To examine the gap between prototypes and image instances, we follow Liang et al. [32] to measure the Euclidean distance between the prototypes and image instances embeddings/representations. To be specific, in the analysis, tasks are first randomly sampled from the validation set of a specific dataset, such as ImageNet [44] , and the corresponding embeddings are obtained by feeding support data of the tasks into a frozen pre-trained ResNet-18 backbone. Then, the prototype for each class in the task is calculated as a mean of all support data within the class following previous works [47, 49, 43, 3, 12, 34, 29, 30] . By respectively applying URL [29] and our proposed CoPA methods on the extracted embeddings, the corresponding representations are obtained. Finally, we reduce the dimension of the prototype and image representations via some dimension reduction tools, such as UMAP [37] , and visualize the results. The results are visualized in Figs. 1, 5(a), 7, and 19. According to the visualization results, there are several interesting phenomena:• According to Figs. 1(a), 7(a), 7(d), 7(g), 7(j), and the results of other datasets (Fig. 19 ), it is easy to observe that the prototype and image instance embeddings extracted from a frozen pre-trained backbone tend to be located in different regions of the feature space. This phenomenon is consistent with what happens in visual language models [32] . It also demonstrates the aforementioned intuition that prototype and image instance data describe the information at different levels and should be treated differently during the adaptation. • Then, according to Figs. 1(b), 7(b), 7(e) and 7(k), we can observe that the "modality" gap is shrunken after applying URL, which imposes an assumption that the same representation transformation is applied to both prototype and image instance embeddings. This phenomenon demonstrates that sharing the same transformation between prototypes and image instances potentially damages the distribution differences between the prototypes and image instances and in turn removes the differences of semantic information between them. • Finally, according to Figs. 5(a), 7(c), 7(f) and 7(l), we can observe that the representations learned from our proposed CoPA method preserve the "modality" gap and the gap is further enlarged compared with the gap between the embeddings. As claimed in Liang et al. [32] , the enlargement of the gap contributes to the improvements in generalization performance. According to our empirical results in Tables 1 and 2 , CoPA achieves the SOTA performance and significantly outperforms URL and URL + TSA, which is consistent with the conclusion. Thus, we can further say that it is the gap between prototype and image instance representations that facilitates CoPA to achieve better empirical performance.In addition to these visualization results, we take a further step to validate whether this phenomenon is common among all tasks of all seen domain datasets. To this end, we collect the results of the gaps of the extracted embeddings and the representations respectively learned from URL and CoPA among 600 randomly sampled tasks under the "train on all datasets" task setting with the The distribution of both positive and negative similarities with the "Average" prototype.random seed 42 on the validation sets of all seen domain datasets. The results are reported in Table 3 . The gaps are calculated following Liang et al. [32] . Given a set of normalized support data embeddings/representations {z i }|D T |i=1 and a set of normalized prototypes generated from the support data {c j } N C j=1 , the gap is calculated as the difference between the averaged prototypes and images:⃗ ∆ := 1 |D T | |D T | i=1 z i - 1 N C N C j=1 c j .Thus, in this way, with the normalized prototype and image instance embeddings/representations, the "modality" gap is further defined as the Euclidean distance between the two sets of embeddings/representations || ⃗ ∆||, where || • || denotes the Euclidean distance.According to the results reported in Table 3 , it is easy to observe that URL consistently shrinks the gaps between prototypes and image instances over all datasets while CoPA consistently enlarges the gaps. These results indicate that URL undermining the "modality" gaps while CoPA preserving and enlarging the gaps is not an occasional case. It commonly happens across all domains.In this paper, in order to select the most appropriate prototypes for CoPA, we conduct an analysis to perform prototype selection. Generally, the prototype for each class ought to contain the most general and discriminative information regarding the class. To this end, three kinds of prototypes are selected as candidates. We will provide detailed descriptions and analyses of them in the following parts.Max Prototype. The "Max" prototype is defined as the sample that is composed of the largest features in each dimension among all support data embeddings extracted from a frozen backbone. The pseudo codes are provided in the following: Average Prototype. The "Average" prototype is the typical prototype adopted in previous few-shot classification works and calculated as the average vector of all support data embeddings extracted from a frozen pre-trained backbone within the given class. The pseudo codes are as the following: Settings. The analysis regarding prototype selection is performed on the support data embeddings (of tasks sampled from the validation set of seen domain datasets) extracted from the frozen pre-trained backbone. The results are obtained from 600 randomly sampled with the random seed 42. Specifically, for each sampled task, the support data are fed into the backbone for the corresponding embeddings. Then, the prototype embeddings for each class are calculated respectively. With the support data and prototype embeddings, we measure two kinds of similarity. The first one is the similarity between each support data sample and the prototype that it belongs to. We call such kind of similarity "positive similarity". The other one is the similarity between each sample and the prototypes of other classes. We call such kind of similarity "negative similarity". The density of both kinds of similarity of the three prototypes (over all seen domain datasets) are visualized in Fig. 8 and Figs. 12 -18. Results. As aforementioned, the prototype should contain information that is commonly shared among samples within the class while distinctive from other classes. In other words, such a prototype has high similarity with the samples within the class while low similarity with the samples from other classes. Thus, for prototypes that depict the representative features of the classes, the positive similarity should be as large as possible while the negative similarity should be as small as possible.According to the results on ImageNet dataset shown in Fig. 8 , the comparison of the three kinds of prototypes mainly includes two parts. On the one hand, the numerical results of the few-shot classification task are evaluated respectively with the three prototypes. On the other hand, the distributions of "positive similarity" and "negative similarity" are respectively measured. From the perspective of numerical results, it is easy to observe that the "Average" prototype achieves the best empirical generalization performance on ImageNet and outperforms the "Max" and the "Max Sample" prototypes. In addition, the "Average" prototype also achieves the best results in positive similarity distribution with an average positive similarity of 0.79 (see Fig. 8(d) ). Interestingly, the "Max Sample" prototype, which owns the highest similarity over all other samples, obtains the lowest positive similarity, even smaller than the "Max" prototype. We conjecture the reason for this phenomenon is that the "Max Sample" prototype does not contain the common features shared in its class though it owns the highest similarity among all other samples within its class. Thus, with all aspects taken into consideration, we select the "Average" prototype for our proposed CoPA.In this section, we intend to provide comprehensive and concrete details about the task settings adopted in the experiment section of our paper. Specifically, we mainly introduce the dataset (i.e. Meta-Dataset [49] ), and the vary-way vary-shot classification task setting in this section.Meta-Dataset is a dataset composed of several datasets that are easy and free to access. It spans a variety of visual concepts with different degrees of fine grain (e.g. from hand-writing to natural scenarios). It was first proposed by Triantafillou et al. [49] for cross-domain few-shot classification tasks and has been a popular and mainstream benchmark in this field. Originally, Meta-Dataset only contained 10 datasets, which are ILSVRC_2012 (a.k.a. ImageNet) [44] , Omniglot [26] , FGVC_Aircraft (Aircraft) [36] , CUB_200-2011 (CU_Birds) [53] , Describable Textures (DTD/Textures) [9] , Qucik Draw [22] , FGVCx Fungi (Fungi) [45] , VGG_Flower [39] , Traffic Sign [20] , MSCOCO [33] . Later, MNIST [27] , CIFAR-10 [25] and CIFAR-100 [25] are further contained by Bateni et al. [3] .In practice, there are two mainstream settings widely applied in the existing works [49, 43, 3, 10, 12, 34, 29, 30] . The first setting is the "train on all datasets" setting. In this setting, 8 datasets, including ImageNet, Omniglot, Aircraft, Birds, Textures, Quick Draw, Fungi, and VGG_Flower, are selected as seen domains while the remaining datasets are treated as unseen domains. In contrast, the other setting, the "train on ImageNet only" setting, only takes ImageNet as the seen domain while all other datasets are treated as unseen domains. In this paper, we report results under both task settings. In particular, if there is no further declaration, the default task setting is the "train on all datasets" setting.Vary-way vary-shot classification task setting is first proposed in Meta-Dataset [49] to mimic the most common situations, where the number of classes and the number of samples in each class are randomly determined, in the real world. Thus, the samples in these tasks are imbalancedly distributed. Such task setting is widely adopted in many previous works [43, 3, 10, 50, 12, 34, 29, 30, 16] . Different from the conventional few-shot classification task setting, where the numbers of classes and shots are invariant in each task, vary-way vary-shot task setting adopted in cross-domain few-shot classification is much more challenging due to the imbalanced data and the uncertain task structure.Roughly, the generation process of vary-way vary-shot task mainly includes two steps: class sampling and data point sampling. To be specific, the first step is sampling classes from a specific dataset. Given a dataset, the number of classes is randomly determined with some probability distribution in the interval [5, N max ], where N max denotes the maximum number of classes of the dataset. In the context of Meta-Dataset, the N max is usually set either 50 or as many classes as there are available.After the number of classes is determined, a set of classes is uniformly sampled from the given dataset. Then, the number of data points for each class is assigned in a random way. Concretely, the Z P = h θP (Y Y ⊤ f ϕ * (X)); Z I = h θI (f ϕ * (X)); Compute SCE loss L(θ P , θ I ) in Eq. (3); Update parameters: θ P ← θ P -η∇ θP L(θ P , θ I ); θ I ← θ I -η∇ θI L(θ P , θ I ); end for first step in data point sampling is determining the number of query data. In the vary-way vary-shot task setting proposed in Meta-Dataset [49] , the number of query data for each class is fixed to the same number (e.g. 10). The number of query data ought to be no more than half of all available data in the given class so that at least 50% of the data can be preserved as support data. Then, based on the number of query data, the support data are sampled from the remaining data with the constraint that the total number of the entire support data (of all classes) should be no more than 500. The number of support data for each class is determined by the size of support data and a set of random scalars sampled from a given interval. In this way, a task with random numbers of classes and shots is sampled from a given dataset. More concrete details are available in the Meta-Dataset article [49] .In this section, we provide detailed implementation settings, including backbone pre-training, model structures, adaptation module initialization, hardware information, and other hyperparameter settings (e.g. learning rate & weight decay), for the convenience of reproducing our proposed CoPA method.In this paper, our proposed CoPA method is built upon a ResNet-18 backbone [18] as done in previous works [12, 34, 29] . For the pre-trained backbone used in the "train on ImageNet only" setting, the backbone is pre-trained in the same way as proposed in SUR [12] . Specifically, the backbone is trained with stochastic gradient descent optimizer and cosine annealing with a learning rate 3 × 10 -2 . The momentum is set to 0.9 and the weight decay is set to 7 × 10 -4 . In addition, the batch size is 64 and the total number of learning iterations is 480,000. All images are reshaped to the size of 84×84.Moreover, for the universal backbone that is used in the "train on all datasets" setting, we follow URL [29] to distill a ResNet-18 backbone from models that are respectively pre-trained on 8 single datasets. The distilled model is also pre-trained with SGD optimizer (learning rate is set to 3 × 10 -2 and weight decay is set to 7 × 10 -4 ) and cosine annealing as well with 240,000 learning iterations. The details of single domain backbone pre-training are available in appendices of URL [29] .In addition to the linear transformation head, we also evaluate our method on visual Transformer (i.e. ViT) [11] to figure out whether the proposed CoPA method benefits from complex and powerful learning modules. In our experimental setting, the visual Transformer we adopted is composed of a single self-attention block. Specifically, each visual Transformer model only contains one attention block followed by a LayerNorm layer [1] and a linear layer. All layers in the ViT, including query, key, value heads, and the linear transformation head, are formulated as 512 × 512 matrices.During the meta-test phase, the adaptation modules (i.e. trainable learning modules) are re-initialized for the adaptation of new tasks at the beginning of each episode. Specifically, all linear layers will be re-initialized to an identity matrix with the size of 512 × 512. Besides, in the visual Transformer case, the weight and bias of the LayerNorm layer are respectively set to 1.0 and 0.0, all class tokens are re-initialized to 0.0, and all position embeddings are re-initialized with a normal distribution N (0, 0.02). In particular, in CoPA + TSA case, the trainable learning modules plugged into the pre-trained backbone are re-initialized to all-ones matrices with a scale coefficient 1e -4.In our method, the optimizer adopted during adaptation is Adam optimizer [24] . For each adaptation episode, the total number of iterations is 50 for linear transformation and 40 for visual Transformer. For linear head, the learning rate is set to 5e-3 for Traffic Sign and MNIST datasets while 1e-3 for the remaining datasets. The weight decay is set to 0.1 for all datasets except for Traffic Sign and MNIST. For the visual Transformer, the learning rate is set to 1e-3 for Traffic Sign and MNIST and 1e-4 for other datatsets. The weight decay is set to 0.1 for all datasets except Traffic Sign and MNIST.All experiments in this paper are conducted with an NVIDIA GeForce RTX 3090 GPU. To make sure that all experimental results are reproducible and the comparisons are fair, all results of our proposed CoPA method and other reproduced methods are the average of 5 reproductions with the random seeds 41, 42, 43, 44, 45. The GPU memory that is required for running our method is about 10 GB.Among all existing methods, as far as we know, the best results on cross-domain few-shot classification tasks with respect to Meta-Dataset are reported by TSA [30] . Besides, recent work, TA 2 -Net [16] , can be seen as a further extension of TSA by adaptively selecting learning modules plugged into the pre-trained backbone. Due to the extra trainable modules plugged into the frozen pre-trained backbone, more task-specific knowledge can be learned to achieve better generalization performance.In this paper, although our proposed CoPA method is able to achieve better generalization performance than TSA on seen domains without having to add any extra learning modules on the pre-trained backbone, there still exist obvious performance gaps on unseen domains. We conjecture that the improvements over unseen domains significantly rely on the extra learning modules. Thus, to verify this conjecture, we followed TSA to plug some extra learning modules into the pre-trained backbone to combine our proposed CoPA with TSA strategy. However, since TSA consumes large amounts of time compared with URL in practice (about 30s/iter), concerning efficiency, we only plugged learning modules on the second and third residual blocks of the pre-trained ResNet-18 backbone.In our implementation of CoPA + TSA case, we optimize the learning modules and transformation heads with Adam optimizer as well. To be specific, the learning rate of the two transformation heads is set to 5e -3 for Traffic Sign, MNIST, CIFAR-10, and CIFAR-100 datasets while 1e -3 for the remaining datasets. For learning modules plugged into the pre-trained backbones, the learning rate is set to 2.5e -3 for Traffic Sign, MNIST, CIFAR-10 and CIFAR-100 datasets while 1e -4 for the remaining datasets. Except for Traffic Sign and MNIST, the weight decay is set to 0.1.In this paper, for fair comparisons between our proposed CoPA method and current state-of-the-art methods, such as URL, TSA, and TA 2 -Net, we ran both CoPA and those existing methods with the same 5 random seeds and reported the average of 5 reproduction experiments as the final results. All results of these methods reported in our paper can be reproduced by setting the seed in the bash file.Similar to CLIP [42] , which performs contrastive learning via two Transformers [51, 11] with text and image data pairs, the proposed CoPA method in this paper fine-tunes two different modules respectively for prototype and image instance embeddings by substituting the text prompts with prototypes. However, the modules adopted in the main results are two linear transformations. It is not clear whether applying more complex modules contributes to the performance. Thus, in this section, we similarly evaluate the performance of CoPA with two visual Transformer (ViT) heads [11] .The implementation details of ViT are available in Appendix E.2. The comparison of performance between CoPA + ViT and CoPA + Linear is reported in Table 4 . According to the results in the table, we can observe that the performance changes between CoPA + Linear and CoPA + ViT are slight under both "train on all datasets" and "train on ImageNet only" settings. Such a phenomenon reveals that the proposed CoPA method is not sensitive to the complexity of model architectures.In this section, we follow previous works [49] to evaluate our proposed CoPA method on fewershot classification tasks. Different from the vary-way vary-shot task setting mentioned above, the fewer-shot classification task contains fewer data samples in each task and thus is more challenging.In addition to the vary-way vary-shot task setting adopted in the main results section, some other task settings, such as 5-way 1-shot and vary-way 5-shot, are also widely adopted in previous works [14, 47, 49, 29, 30] . Compared with the vary-way vary-shot task setting, 5-way 1-shot and vary-way 5-shot tasks are more challenging since fewer data samples are included in the support set of each task. For example, in the vary-way 5-shot task setting, although the number of classes is still randomly determined at the beginning of each episode, the number of data samples (i.e. shot) is fixed to 5. Similarly, in the 5-way 1-shot task setting, both the numbers of classes and data samples in each class are respectively fixed to 5 and 1. In this section, in order to further explore the capability of CoPA, we validate the performance of CoPA on both task settings aforementioned under the "train on all datasets" setting for all datasets in Meta-Dataset. All results are reported in Tables 5 and 6 .Vary-way 5-shot. As shown in Table 5 , compared with previous works, such as SimpleCNAPs [3] , SUR [12] , URT [29] and URL [29] , CoPA achieves the best performance on 7 out of 13 datasets and ranks 3.2 among all approaches. In particular, CoPA outperforms URL, which CoPA is based on, on 9 out of 13 datasets, and achieves 0.2%, 0.5%, and 0.3% improvements on average respectively on seen, unseen, and all domain cases. To be specific, CoPA respectively achieves better performance on Omniglot (0.2%), Aircraft (0.4%), CU_Birds (0.2%), VGG_Flower (0.5%) and Traffic Sign (2.1%) datasets. Meanwhile, we also notice that CoPA achieves better results than TSA, where extra trainable 1 Both the results on URL, TSA and CoPA are the average of 5 random seeds (41 -45) .Table 6 : Results under 5-way 1-shot task settings (under "Trained on all datasets" setting).Mean accuracy, 95% confidence interval reported. 65.8±0.9 67.1±1.4 77.9±0.9 78.6±0.9 78.4±0.9 78.6 ± 0.9 78.9 ± 0.9 Birds 67.9±0.9 59.2±1.0 70.9±0.9 76.2±0.9 76.2±0.9 75.8 ± 0.9 75.8 ± 0.9 Textures 42.2±0.8 42.5±0.8 49.4±0.9 52.0±0.9 52.2±0.9 51.9 ± 0.9 52.0 ± 0.9 Quick Draw 70.5±0.9 79.8±0.9 79.6±0.9 79.1±0.9 79.1±0.9 78.9 ± 0.9 79.0 ± 0. 1 Both the results on URL, TSA and CoPA are the average of 5 random seeds (41 -45) .learning modules are plugged into the frozen pre-trained backbones. Concretely, CoPA outperforms TSA respectively on ImageNet (0.6%), Textures (0.5%), Quick Draw (0.4%), and Fungi (0.3%).Moreover, we also evaluate the performance of CoPA + TSA on vary-way 5-shot tasks. According to the results in the table, CoPA + TSA achieves the best performance on 8 out of 13 datasets and ranks 2.2 among all approaches. In general, CoPA + TSA obtains the similar performance on seen domains to CoPA and TSA while significantly outperforms all other methods on unseen domains. Specifically, compared with the state-of-the-art method TSA, CoPA + TSA achieves 8.5%, 0.4%, 2.6%, and 0.9% improvements respectively on Traffic Sign, MSCOCO, MNIST, and CIFAR-100 datasets. CoPA and CoPA + TSA under the 5-way 1-shot setting during the adaptation process. According to these figures, it is easy to observe that overfitting tends to take place in all cases in URL and several cases in CoPA during the adaptation process. The main reason is that the prototype of each class under the 5-way 1-shot setting is calculated with only one data sample. Thus, such prototypes are too biased to contain the comprehensive and useful information regarding the classes they belong to.All these results demonstrate that our proposed CoPA is robust and able to achieve better generalization performance compared with the existing approaches on CFC tasks even with fewer data samples.5-way 1-shot. As to 5-way 1-shot task, it is a special case in the few-shot classification task since the prototype of each class in this task setting is calculated with only one data sample. Thus, the obtained prototypes may be more biased compared with those obtained in the vary-way vary-shot and vary-way 5-shot task settings. Consequently, the 5-way 1-shot task is much more challenging.According to the results reported in Table 6 , it is easy to observe that our proposed CoPA achieves comparable results compared with URL and ranks 3.2 among all approaches. To be concrete, CoPA achieves the similar performance to URL on the seen domains while only outperforms URL on Traffic Sign dataset among the unseen domains. As for TSA, we find that plugging more extra trainable learning modules into the pre-trained backbone fails to improve the generalization performance on the seen domains. However, the performance on the unseen domains benefits from these extra learning modules. Specifically, although TSA fails to outperform our proposed CoPA on Traffic Sign One conjecture for the phenomenon that CoPA fails to outperform other approaches evidently in most cases is that the prototype of each class generated under the 5-way 1-shot setting fails to contain comprehensive and useful information that is representative enough to depict the general features of the class since it is calculated with only one data sample. In this case, contrastive learning is performed between two identical data samples and overfitting may take place. In order to validate our conjecture, we plot the accuracy curves of URL, CoPA, and CoPA + TSA. The visualization results are presented in Fig. 9 . According to these figures, we can see that it is quite hard to perform adaptation on tasks with extremely few labeled data samples and overfitting tends to take place. For example, For datasets like Omniglot (Fig. 9 (b)) and Aircraft (Fig. 9 (c)), overfitting is obvious on both URL and CoPA. Meanwhile, the application of CoPA even deteriorates the performance.In this paper, due to the consideration of preserving the gap between prototypes and image instances, we propose to adopt symmetric cross-entropy loss as the learning objective. Typically, the symmetric cross-entropy loss is commonly applied in the contrastive learning framework.An important issue in the conventional contrastive learning framework is the size of the data. As demonstrated in previous works [5] , the large batch size is the key to learning useful representations for downstream tasks. Thus, we are also interested in the effect of the batch size of the data. However, due to the task settings, such as the vary-way vary-shot setting, it is intractable to manipulate the batch size via task sampling directly. Thus, we can approximately study the problem by comparing the performance of CoPA respectively on vary-way vary-shot tasks and fewer-shot tasks since the latter contains much fewer data samples in the support set.First of all, we count the number of samples in 600 tasks randomly sampled from the validation set of seen domain datasets (with the random seed 42). The results are reported in Table 7 . According to the table, we can observe that the difference between the minimum and the maximum of the support set size is large. On average, in most cases, the average size of the support set is more than 200. These results indicate that the number of data samples under vary-way vary-shot settings is not too small.Then, we compared the performance of CoPA respectively on vary-way vary-shot tasks, vary-way 5-shot tasks, and 5-way 1-shot tasks. In vary-way vary-shot setting, which contains the most samples, CoPA evidently achieves the best results. However, in vary-way 5-shot tasks, CoPA can still achieve the best results on average, though it fails to outperform on some datasets. On 5-way 1-shot tasks, we can obviously observe that CoPA achieves similar results to URL without absolute advantages.Thus, we can see that the rule is not changed and the large batch size is still preferred in our proposed method. Thus, a potential limitation of our proposed method is that CoPA may fail to achieve good generalization performance with extremely few data samples.In this paper, the original CoPA aims at fast fine-tuning two linear heads respectively for prototype and image instance embeddings on top of the frozen pre-trained backbones in the same way as contrastive learning applied in CLIP [42] . Compared with previous works, such as URL [29] on which CoPA is based, CoPA utilizes more model parameters. To be specific, the number of parameters contained in the trainable modules of CoPA is twice those contained in the trainable module in the URL. Thus, in this case, a concern that may be raised is whether the improvements in generalization performance on CoPA result from more trainable parameters contained in the two linear transformation heads.In order to figure out this question, we conduct an analysis to explore the effect of the number of model parameters. To be specific, we compare CoPA with a variant of URL that includes the same number of trainable model parameters under the "train on all datasets" setting. In the URL, only a simple linear layer with the size of 512 × 512 is trained on support data for further few-shot classification on query data. In order to increase the number of the number of parameters, we propose to substitute the original linear transformation head in the URL with a simple two-layer MLP. The MLP is designed with the structure of Linear-Batch Normalization-Linear. All linear layers in the MLP model are with the size of 512 × 512. Then, the number of parameters is equal to CoPA.The experimental results are reported in Table 9 . According to these results, we can observe that the combination of URL and MLP fails to outperform CoPA and even achieve worse results than URL where fewer parameters are contained. These results demonstrate that simply adding more parameters in the linear transformation head does not positively affect the generalization performance since the implicit assumption about the shared transformation is not removed. The success of CoPA is mainly based on the mechanism of CoPA, where the prototype and image instance representations are explored respectively with different transformations by optimizing symmetric cross entropy loss. Due to the two linear transformation heads, the number of parameters in CoPA is twice of that in URL. Assume that there are N data samples with the dimension of N d , the number of classes is N C , then the size of a linear head isN d × N d .For instance, in our experimental settings, N d = 512 and the size of the linear head is 512 × 512. Based on this, we list the number of parameters and the computational complexity of a forward pass (including representation transformations with linear heads and inner products between samples and prototypes) in Table 8 . Notice that we did not take the computation of feature extraction from the frozen backbone into consideration. However, as shown in Fig. 10 , CoPA is more efficient than URL. The main reason for this phenomenon is that CoPA calculates the prototypes at the beginning of each episode and does not have to calculate prototypes repeatedly in each iteration as done in URL.Table 8 : Comparison of the number of parameters, the computational complexity of URL and CoPA.URL N 2 d N (NC + N d )(N 2 d -N d ) CoPA 2N 2 d (N N d + NC N d + 2N 2 )(N 2 d -N d )In this section, we perform a series of ablation studies to evaluate the abilities of our proposed CoPA method and determine the functions of the components, such as the SCE loss, adopted in the CoPA.Symmetry Cross Entropy Loss. In our paper, we follow CLIP [42] and propose a simple yet effective method, contrastive prototype-image adaptation (CoPA), to perform model adaptation on prototype and image representation pairs via contrastive learning. The loss is the symmetry cross entropy (SCE) loss, which is widely adopted in contrastive learning [40, 56] . To validate the effect of SCE loss, we conducted an ablation study on SCE loss respectively on URL and CoPA methods.To be specific, for URL, we replace the NCC loss with SCE loss and fine-tune the shared transformation head in the same way as URL. For CoPA, we substitute the SCE loss with NCC loss used in Table 9 : Effect of the number of parameters (Under "Train on all dataset" settings).According to the reported results in the table, although better performance is achieved on some datasets, such as Textures, URL+2Heads achieves comparable or even worse performance in other cases. In order to further examine the reason for such a phenomenon, we plot the test loss curves in Fig. 6 . According to the figures, we can observe that overfitting takes place during the adaptation phase. The overfitting phenomenon, to some extent, reflects that discriminative representations are learned via two different transformations. This phenomenon conforms to our intuition that the application of different transformations contributes to learning more discriminative representations.Thus, with all these aspects taken into consideration, we find that the different transformations help learn more discriminative representations respectively for prototypes and image instances. Meanwhile, the SCE loss contributes to improving generalization performance. The combination of these two techniques collaboratively facilitates achieving impressive empirical results on Meta-Dataset.In this section, we provide detailed proof of the theoretical results in the main paper.G.1 Proof of Theorem 3.1Proof. With Eq. ( 1), by replacing d(•, •) with cosine similarity, we have:L(θ) = - 1 n n i=1 log e z ⊤ i •cc N C j=1 e z ⊤ i •cj = - 1 n n i=1 z ⊤ i c c - 1 n n i=1   -log N C j=1 e 1 |C j | z ′ ∈C j z ⊤ i z ′   = - 1 n n i=1 z ⊤ i c c - 1 n n i=1   -log 1 N C N C j=1 e 1 |C j | z ′ ∈C j z ⊤ i z ′ -log N C   ≥ - 1 n n i=1 z ⊤ i c c + 1 n n i=1 N C j=1 z ′ ∈Cj 1 N C |C j | z ⊤ i z ′ ,where z ′ is the independent copy of z, C j denotes the set of sample representations of class j. The inequality sign is derived from Jesen's Inequality. Then, with a constant that satisfies 0 ≤ α < 1N C |Cj |for ∀j, we further have: Proof. We first expand the v ⊤ Θ term:L(θ) ≥ - 1 n n i=1 z ⊤ i c c + α n n i=1 z ′ ∈Z z ⊤ i z ′ .v ⊤ Θ 2 2 = v ⊤ Θ 1 , Θ 2 , ..., Θ d 2 2 = v ⊤ Θ 1 , v ⊤ Θ 2 , ..., v ⊤ Θ d 2 2 = d i=1 v ⊤ Θ i 2 = ∥v∥ 2 2 d i=1 Θ i 2 2 cos 2 (v, Θ i ).Then, we can obtain the lower and upper bound of v ⊤ Θ 1 |D T | z∈Z z - 1 N C c∈C c 2 2 = 1 |D T | x∈D T f ϕ * (x) - 1 N C N C b=1 1 |C b | x ′ ∈C b f ϕ * (x ′ ) ⊤ Θ 2 2 .Then, with Lemma G.1, we have the bounds of the gap between the prototype and image instance representations:m ∥Θ∥ 2 F ⃗ ∆ emb 2 2 ≤ 1 |D T | x∈D T z - 1 N C c∈C c 2 2 ≤ M ∥Θ∥ 2 F ⃗ ∆ emb 2 2, where ⃗ ∆ emb = 1|D T | |D T | i=1 f ϕ * (x i ) -1 N C N C b=1 1 |C b |x ′ ∈C b f ϕ * (x ′ ) denotes the gap between embeddings, m = min 1≤i≤d cos 2 ( ⃗ ∆ emb , Θ i ) and M = max 1≤j≤d cos 2 ( ⃗ ∆ emb , Θ j ).Proof. We first calculate the first term of Eq. ( 4). Then, we have:- 1 |D T | |D T | i=1 log exp(z ⊤ i c i ) |D T | j=1 exp(z ⊤ i c j ) = - 1 |D T | |D T | i=1 z ⊤ i c i + 1 |D T | |D T | i=1 log |D T | j=1 exp(z ⊤ i c j ) = - 1 |D T | |D T | i=1 z ⊤ i c i + 1 |D T | |D T | i=1 log 1 |D T | |D T | j=1 exp(z ⊤ i c j ) + log |D T | ≥ - 1 |D T | |D T | i=1 z ⊤ i c i + 1 |D T | 2 |D T | i=1 |D T | j=1 z ⊤ i c j + log |D T |.Since each prototype is expanded by Y Y ⊤ , there are |C k | same vectors for each class k, where C k denotes the set of samples of class k. Thus, we further have:- 1 |D T | |D T | i=1 log exp(z ⊤ i c i ) |D T | j=1 exp(z ⊤ i c j ) ≥ - 1 |D T | |D T | i=1 z ⊤ i c i + 1 |D T | |D T | i=1 N C k=1 |C k | |D T | z ⊤ i c kSimilarly, for the second term, we can obtain the same results. In detail,- 1 |D T | |D T | i=1 log exp(c ⊤ i z i ) |D T | j=1 exp(c ⊤ i z j ) = - 1 |D T | |D T | i=1 c ⊤ i z i + 1 |D T | |D T | i=1 log |D T | j=1 exp(c ⊤ i z j ) = - 1 |D T | |D T | i=1 c ⊤ i z i + 1 |D T | |D T | i=1 log 1 |D T | |D T | j=1 exp(c ⊤ i z j ) + log |D T | ≥ - 1 |D T | |D T | i=1 c ⊤ i z i + 1 |D T | 2 |D T | i=1 |D T | j=1 c ⊤ i z j + log |D T | = - 1 |D T | |D T | i=1 c ⊤ i z i + 1 |D T | N C k=1 |D T | j=1 |C k | |D T | c ⊤ k z j .Combining the two results, since the support data size is n, we then have: L SCE ≥ - 2 n n i=1 z ⊤ i c i + 2 n n i=1 N C k=1 |C k | n z ⊤ i c k .† Correspondence to Bo Han (bhanml@comp.hkbu.edu.hk).For fairness, the results of URL, TSA, TA-Net, and our proposed CoPA methods are reproduced with 5 random seeds, and we report the average of the 5 reproductions in the table. Particularly, although the reported performance of URL is lower than that in the original paper, the reproduction results are consistent with those reported on their project website. The ranks are calculated only with the first 10 datasets and only with the methods mentioned above.The results on URL, TSA, TA-Net and our proposed methods are reproduced with 5 random seeds and reported as the average of the 5 reproduction. The ranks only consider the first 10 datasets and are calculated only with the methods in the table. 


Decision: The paper received 6/4/7 ratings. The majority of the concerns come from the validity of the major claim of the paper. The authors provided a rebuttal. R1 is convinced with the explanation, while R2 still feels it questionable. R2 also mentioned that he can go either way with the paper's decision. After reading the paper, reviews, and rebuttal, the AC recommends accepting the paper. The findings and solution will benefit the community.

Reviews: review: paper_summary: paper_summary: This paper investigates the gap between the prototype and image instance embeddings under the setting of cross-domain few-shot classification. It shows that applying the same transformation to these embeddings will shrink their gap and constrain the exploration of optimal representation distributions. Based on a series of empirical and theoretical analyses, this paper proposes a prototype-image contrastive learning adaptation method, namely CoPA. The experimental results on Meta-Dataset indicates the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.	For the empirical analysis in Section 3.2, there is an interesting phenomenon that “appropriately enlarging the gap between the prototypes and image instances contributes to achieving better generalization performance”. It may be better to provide some discussions of the reasons for these these observed results (Figure 3(a)) to explain the influence of “enlarging the gap” on the “generalization ability”.
2.	The competing methods used in experiments appear to be somewhat outdated. The main baseline URL is published in 2021. It may be better to compare with some recently proposed adaptation-based methods. 
3.	Some equation numbers are missing. Some references are not correctly formatted, the conference or journal names as well as page numbers are missing.
questions: questions: It is a good paper, I personally like this paper and just have two small question:

1.	In Theorem 3.1, optimizing the first term in the equation between the 206th and 207th rows requires to maximize the similarities between instances and their corresponding protypes. So, would enlarging the protype-image gap negatively affect this optimization process?

2.	According to the equations between the 218th and 219th rows, I can understand the differences of the first term in these two equations are removed, when the shared transformation is adopted. But why this may lead to the drop of “the discriminative information in gradients”? Could the authors explain this more clearly?
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper claims that there exists gap between prototypes and image instance embeddings in cross-domain few-shot classification models. 
URL[29], a representative work of cross-domain few-shot classification, proposes to fast fine-tune linear classifier on top of a frozen back bone with the nearest centroid classifier loss. 
In URL, the embeddings of an image instance encode the instance-level information from the given image, while the embedding of a prototype contain some abstract and high-level information among image instance. 
This paper proposes Contrastive Prototype-image Adaptation (CoPA), to adapt different transformations for prototypes and image embeddings to adjust the modality gap.
strength_weakness: strength_weakness: The CoPA looks only improving the performance by adding complexity of models and losses. 
- CoPA have two linear heads that is more complicated than URL. This paper argues that discriminative information of gradients is removed when applying the same transformation. This claim seems to be about that when one common classifier is used, gradient does not consider the model parameters of two linear classifier (Discission of Theorem 3.1) However, it is obvious that the model that uses one linear classifier does not consider difference of two classifiers in gradients.  
- There is no clear reason that enlarging the gap between the prototypes and instance embeddings improve performance (though is empirically reducing the validation loss.) 
CoPA is based on such a uncertain phenomenon.
- Also CoPA have a temperature parameter $\gamma$ in the symmetric cross entropy, meanwhile the NCC objective does not. It seems to be unfair, as the temperature influence the gap [32].
questions: questions: This paper discuss the gap between prototype of class c, and image instances. However, I could not understand why such large gap exist. Since the mean vector is the average of image instances, the prototypes should be within the convex full of image instances. Thus, I feel strange that instances and prototypes are in the separate regions as shown in Fig.1 .

The number of samples per classes will influence the gap $\Delta$. Could you justify this issue?
limitations: limitations: N/A
paper_summary: paper_summary: The authors of this paper found that in cross-domain few-shot classification, there naturally exists a gap, which resembles the modality gap, between the prototype and image instance embeddings. Thus, this paper proposes a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformations for prototypes and images similarly to CLIP by treating prototypes as text prompts. Extensive experiments on Meta-Dataset demonstrate that CoPA achieves state-of-the-art performance more efficiently. Meanwhile, further analyses also indicate that CoPA can learn better representation clusters, enlarge the gap, and achieve the minimum validation loss at the enlarged gap.
strength_weakness: strength_weakness: One concern might be the novelty of this paper: the proposed CoPA is a classical method that has been proven to be effective in many previous works. However, I think the authors did a good job using old algorithms to solve new questions. Overall, I think this is a good paper that should be accepted.
questions: questions: See weaknesses.
limitations: limitations: One potential limitation of our proposed CoPA method is the symmetric cross-entropy. Symmetric cross-entropy loss is commonly applied in contrastive learning. In contrastive learning, the batch size of the data samples is an important issue for better downstream tasks. Larger batch size usually means better downstream performance. We also notice this phenomenon in this paper. Although CoPA can still achieve relatively better performance, the performance tends to degrade when the size of the data batch becomes small. Thus, this constrains the application of CoPA to some extent.

Comments: title: Look forward to your feedback
Further responses to your new concerns
One day remaining for the discussion period
Thanks a lot for your support!
Thanks for your support!
Do our responses solve your concerns?
Do you have any further concerns or questions?
Do you have any further concerns or questions?
Do you have any further concerns or questions?
comment: Dear Reviewer eGag,

Thanks for your time and efforts in reviewing our work.

We have posted responses to your new concerns. Would you mind checking the responses and confirming whether the updated responses solve your concerns? We are glad to answer any concerns and questions you may have before the deadline.

__As a gentle reminder, the discussion period will end in about 12 hours__. We hope to resolve your concerns about our work.

Moreover, if your concerns have been resolved, would you mind reconsidering your rating of our work based on the updated understanding?

Best wishes,

Authors
Thanks for your reply. We are glad that our responses partially solved your concerns. 

For your concern, the enlargment of the gap and the generalization performance are still open problems in the literatures. Currently, reseachers, including us, only empirically observe such phenomena, and find that generalization performance (zero-shot, few-shot etc.) is associated with the gap. All of us try to figure out and explain these problems from different perspectives. 

In the following, we will try to address your new concerns, based on the insightful discussions with Reviewer Ms7f.

> Gap enlargement & generalization performance

For your concern about why enlarging the gap improves the generalization performance, __inspired by the [comments from Reviewer Ms7f](https://openreview.net/forum?id=JWLiK3kKWQ&noteId=HRvuDHQQfk#:~:text=Based%20on%20the,different%20transformation%20heads)__, we find that overfitting and regularization may be a good perspective to explain. Specifically, __enlarging the gap reaches a balance between learning discriminative representations and achieveing better generalization performance__.

According to Fig. 3(a), it is easy to observe that the enlargement of the gap is constrained in a small range. Two cases are worth discussion here:
- If the gap is narrowed, the similarity between prototypes and instances increases. However, the validation loss also increases. __Such a phenomenon indicates that narrowing the gap may result in overfitting__. As a demonstration, we can also observe the overfitting phenomenon happens in some cases from Fig. 27 (f), (g), and (j) (__the orange line__). 
- If the gap is infinitely enlarged, the validation loss will also increase since the similarity between the prototypes and instances decreases.

Thus, from this perspective, __the enlargement of the gap can be seen as a regularization that drives the model to explore a balance between learning better representations and achieving better generalization performance__.

> Why NCC + 2 heads does not work well and how symmetric cross-entropy loss address this issue.

Based on the above findings, we find that we can give a more reasonable explaination to this issue as well. For your concern about why NCC loss along with 2 different transformation heads does not work well, we further conduct an experiment here to see if 2Heads+NCC has an overfitting issue. For simplicity, we only applied two different transformation heads on URL method with the random seed 42 and ImageNet dataset (due to the time constraint, but we will include more experiments for this in the updated version of our paper). 

|Methods|Train Acc|Test Acc|
|---|---|---|
|__URL+2heads+NCC__|94.93|56.67|
|__CoPA__|90.87|57.92|

According to the results, in the "URL+2heads+NCC" case, we observe __higher training accuracy and lower test accuracy__ compared to CoPA. __This is a phenomenon of overfitting__ (compared to CoPA). However, applying the symmetric cross-entropy loss alleviates such a phenomenon to some extent (see the training and test accuracy of CoPA).

Thus, based on the explanation above (thank Reviewer Ms7f for this valuable insight!), __the symmetric cross-entropy loss itself might implicitly regularize the model during the learning process and achieve a balance between the discriminability and the transferability of the learned representations__. 

In summary, we think that these explanations are interesting, and we will merge these discussions in our updated version.
Thank you for the response. 

Some of my concerns have been solved. For examples, it is not strange that gap exist if the instances and centroids are normalized.

I understood that authors followed the observation of [32], but why enlarging the gap between centroid and instances leads to improving the performance is not clear for me. Also, why different model parameters lead to enlarge the gap is not clear. I understand the discriminative information means the difference of gradients of model parameters for prototypes and instance embeddings. However, flexible model can both possible to enlarge or reduce the gap. 

I understand that this work is inspired form the modality gap [32] of CLIP, and thus using the symmetric cross-entropy of CLIP. I have noticed that Table 10 of Appendix compare loss functions. However, this paper does not explain why using only two linear projections on NCC loss does not work well and how the symmetric cross entropy of CLIP solves this problem.
Dear Reviewer eGag,

Thanks again for your time in reviewing our work. 

This is a gentle reminder that the Discussion period will end on __13 Aug 11:59 pm AoE__. There is __only one day remaining__. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors
Dear Reviewer PX84,

Thanks for maintaining 7! We appreciate a lot for your positive comments and support to our work!

Best wishes,

Authors
Dear Reviewer Ms7f,

Thanks a lot for your appreciation and support to our work! The opinion you proposed is interesting and can be used to explain CoPA from another perspective. 

We agree with your opinion that there exists overfitting in previous works. Two aspects can demonstrate this.
- On the one hand, as we have mentioned, closing the gap increases the similarity between the prototypes and image instances. However, according to Fig. 3(a), the validation loss also increases. This indicates the overfitting phenomenon to some extent. 
- On the other hand, we can also observe the overfitting phenomenon happens in some cases from Fig. 27 (f), (g), and (j) (__the orange line__). 

Both phenomena demonstrate the existence of the overfitting problem in previous work (i.e. URL). Thus, from this perspective, enlarging the gap can be viewed as __exploring a balance between learning better representations ("discriminability") and achieving better generalization performance ("transferability")__. According to the data clusters (Fig. 5(b)) and Fig. 27 (__the blue line__), our proposed CoPA does __achieve such a balance__.

We will merge this discussion in our updated version.
Thanks for your response,

I have no more questions, and will not reduce the score. Good luck!
Thank you for your response! 

Based on the authors' reply, I think that the "enlarging the gap between prototype and image instance representations" is more like a regularization that prevent the image representations from overfitting to the class semantic information rather than an alignment between text and image features as in CLIP. Specifically, in my own opinion, the enlargement operation in this paper is to balance the "discriminability" and the "transferability" of the learned representations by using two different transformation heads, while the "alignment" is to pull closer the features with the same semantics and push away those with different semantics. So it maybe better for the authors to provide more clear explanations about this. Nevertheless, I personally like this paper and will maintain the rating score of "weak accept".
Dear Reviewer eGag,

Thanks again for your time and efforts in reviewing our work. Do you still have any other concerns or questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors
Dear Reviewer PX84,

Thanks again for your time and efforts in reviewing our work. We appreciate your valuable and insightful comments on our work. We believe your expertise will help improve the quality of our work.

This is a gentle reminder that the Discussion period will end on 13 Aug 11:59 pm AoE. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors
Dear Reviewer eGag,

Thanks again for your time and efforts in reviewing our work. 

This is a gentle reminder that __the Discussion period will end on 13 Aug 11:59 pm AoE__. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors
Dear Reviewer Ms7f,

Thanks again for your time and efforts in reviewing our work. We appreciate your valuable and insightful comments on our work. We believe your expertise will help improve the quality of our work.

This is a gentle reminder that __the Discussion period will end on 13 Aug 11:59 pm AoE__. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors