Mind the Gap Between Prototypes and Images in Cross-domain Finetuning
Authors: Hongduan Tian
Feng Liu
Zhanke Zhou
Tongliang Liu
Chengqi Zhang
Bo Han
Venue: NeurIPS 2024 poster (2024)
Keywords: few-shot classification
cross domain adaptation
representation gap
computer vision
deep learning
Primary area: evaluation

Abstract: In _cross-domain few-shot classification_ (CFC), recent works mainly focus on adapting a simple transformation head on top of a frozen pre-trained backbone with few labeled data to project embeddings into a task-specific metric space where classification can be performed by measuring similarities between image instance and prototype representations. Technically, an _assumption_ implicitly adopted in such a framework is that the prototype and image instance embeddings share the same representation transformation. However, in this paper, we find that there naturally exists a gap, which resembles the modality gap, between the prototype and image instance embeddings extracted from the frozen pre-trained backbone, and simply applying the same transformation during the adaptation phase constrains exploring the optimal representation distributions and shrinks the gap between prototype and image representations. To solve this problem, we propose a simple yet effective method, _contrastive prototype-image adaptation_ (CoPA), to adapt different transformations for prototypes and images similarly to CLIP by treating prototypes as text prompts. 
  Extensive experiments on Meta-Dataset demonstrate that CoPA achieves the _state-of-the-art_ performance more efficiently. Meanwhile, further analyses also indicate that CoPA can learn better representation clusters, enlarge the gap, and achieve the minimum validation loss at the enlarged gap.

Mind the Gap Between Prototypes and Images in Cross-domain Finetuning
Hongduan Tian', Feng Liu', Zhanke Zhou', Tongliang Liu², Chengqi Zhang', Bo Han
'TMLR Group, Department of Computer Science, Hong Kong Baptist University
'TMLR Group. University of Melbourne 'Sydney Al Center, The University of Sydney *Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University {cshdtian, cszkzhou, bhanm1]@comp @ccomp. hkbu. edu edu.hk, f fengliu. ml @g@ggaml..cmm tongliang. wwalliinn...iussye liu@sydney. eeridddeyydduu.... edu. au, chengqi zhang@polyu.en edu. hk
Abstract
In cross-domain few-shot classification (CFC), recent works mainly focus on adapting a simple transformation head on top of a frozen pre-trained backbone with few labeled data to project embeddings into task-specific metric space where classification can be performed by measuring similarities between image instance and prototype representations. Technically, an assumption implicitly adopted in such framework is that the prototype and image instance embeddings share the same representation transformation. However, this paper, we find that there naturally exists gap, which resembles the modality gap, between the prototype and image instance embeddings extracted from the frozen pre-trained backbone, and simply applying the same transformation during the adaptation phase constrains exploring the optimal representations and shrinks the gap between prototype and image representations. To solve this problem, we propose simple yet effective method, contrastive prototype-imag adaptation (CoPA), to adapt different transformations respectively for prototypes and images similarly to CLIP by treating prototypes as text prompts. Extensive experiments on Meta-Dataset demonstrate that CoPA achieves the state-of-the-ari performance more efficiently. Meanwhile, further analyses also indicate that CoPA can learn better representation clusters, enlarge the gap, and achieve minimal validation loss at the enlarged gap.
The project repository of CoPA is available at:htps://gtthuh.com wow/mmlr-group/Cooooo
1 Introduction
Cross-domain few-shot classification [49] (a.k.a. CFC) aims at learning to perform elassification on tasks sampled from previously unseen domains with only few labeled data. Recent works [12, 34, 29, 30), which aims fast adapting light model on top of the frozen pre-trained backbones for feature selection or transformation, has shown impressive generalization capability CFC tasks.
Typically, URL [29], a representative work of this genre, proposes to fast fine-tune a linear transfor- mation head on top of frozen pre-trained backhone with the nearest centroid classifier loss [47]. The ultimate goal of URL to project embeddings into space where classification can be performed via measuring the similarity between representations of prototypes and image instances. Technically, an assumption implicitly adopted in URL is that the prototype and image instance embeddings share the same representation transformation (see the upper subfigure of Fig. 2) during the adaptation phase. Intuitively, however, prototype and image instance embeddings depict different levels of information. Specifically, the embeddings of an image instance encode the instance-level information from the given image, while the embeddings of a prototype contain some abstract and higher-level information that is commonly shared among image instances within the corresponding class yet discriminative from other classes. Thus, applying the same representation transformation as URL might constrain models from exploring the optimal representations respectively for prototypes and image instances.
"Correspnndence to Bo Han wwwtm//@oom..hhb..c.o
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we first conduct an analysis to validate the intuition that prototypes and image instances describe different levels of information. Specifically, following Liang et al. [32], we visualize the distributions of prototype and instance embeddings extracted from a frozen pre-trained backbone and mea- sure the Euclidean distance between the cen- troids of the normalized prototype and image instance embeddings. According to Fig. ((a), the UMAP [37] visualization indicates that there naturally exists a gap, which resem- bles the modality gap demonstrated by Liang et al. [32] in visual language models (V (VLMs, e.g. CLIP [42]), between prototype and im- age instance embeddings. However, after applying the same representation transforma- tion to both prototype and image instance embeddings as done in URL [29], we ob- serve that the gap is numerically shrunken (see Fig. 1(b)). Moreover, by visualizing the cluster distribution of data samples (see Figs. 3(b) and 3(c)), we further find that applying the same representation transformation to both prototype and image instance embeddings may fail to learn compact and clear representation clusters. With all these empirical results taken into consideration, two aspects can be summarized here: (1) there does exist gap between the prototype and image instance embeddings extracted from the frozen backbone; (2) the shared representation transformation tends to shrink the gap between the learned prototype and image instance representations and may potentially result in the failure of learning clear and compact representation clusters for classes in the task. Our further investigation into the URL framework in Section 3.3 indicates that applying the same transformation, on the one hand, removes the discriminative information in gradients. On the other hand, our empirical results further reveal that applying the shared transformation constrains learning representations where the gap between prototypes and image instances is preserved.
Inagaa
11-
probolypes
20-
46-
-10
-0.
-60
-07
-36
-15
-45 00
95
(a) |||||
= 0.22
(b) || = 0.12
There naturally exists a gap between pro- totype and image instance embeddings, but apply- ing the same transformation shrinks such a gap. Fig.(a) shows that there naturally exists a gap, which resembles the "modality" gap in visual language mod- els, between prototype and image instance embed- dings extracted from a frozen pre- pre-trained backbone. However, Fig.(b) shows that the gap between the repre- sentations of prototypes and image instances is shrunk after applying the same representation transformation to both the image instance and prototype embeddings.
Figure 1:
Previous work [32] has demonstrated that contrastive loss helps preserve the modality gap and further contributes to improving the generalization performance of downstream tasks. Thus, in this paper, we propose a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformations for prototype and image embeddings similarly to CLIP [42] by treating prototypes as text prompts. In this way, the discriminative information in gradients can be preserved in different parameters, and the optimal transformations, where the gap between prototype and image representations is held, can be explored in the function space constructed by the two transformations.
Extensive experiments on the representative Meta-Datase benchmark [49] under several task settings demonstrate that CoPA can achieve the state-of-the performance with less time consumption and learn more clear and compact representation clusters. In addition, further analysis reveals that the global minimum of the validation loss is achieved at the enlarged gap. which implies the enlargement of the gap is process of exploring representation distributions for better generalization performance.
Our Contribution. In this paper, our contributions can be summarized in the following 4 parts:
We validate there naturally exists gap, which resembles the modality gap, between prototype and image instance embeddings extracted from a frozen backbone in Section 3.2.
Our analyses in Section 3.2 reveal that the shared representation transformation shrinks the gap between prototype and image instance representations. Our investigation into the URL in Section 3.3 indicates that applying the same transformation removes the discriminative information in gradients and constrains learning representations where the gap is preserved.
To solve this problem, in Section 4, we propose a simple yet effective method, contrastive prototype- adapation (CoPA), to adapt two different transformations for prototypes and image instances as done in CLIP via substituting text prompts with prototypes.
Extensive results reveal that CoPA can achieve the stute-of-the-ari performance on Meta- Dataset [49] (Section 5.1), enlarge the gap to achieve the global minimum of the validation loss (Section 5.2), and learn more compact image representation clusters (Section 5.2).

---

2 Preliminary
Few-shot Task Generation. Consider meta dataset 8 (St ) 1==1 where is the number of
sub-datasets in S. The sub-datasets in meta dataset satisfy that S a SS = Ø, for VS,, Sj E S.
Each sub-dataset S; split into three disjoint subsets, which are training set D validation set Dimal and test set Di Consistent with typical supervised learning paradigm, the meta dataset is divided into two disjoint parts re- spectively for model training and evaluation. The datasets, in which training sets are used for pre-training, are called seen domains while the remaining datasets, which are only available for evaluation, are called unseen domains. Thus, the meta dataset can be alternatively expressed as S S Sunseen S Stnssonn Ø where the notations S and Sunsern respec- tively denote the seen and the unseen domains.
URL Pipeline
Σ by (e
Prototype
Shared Transforation Head
Representation
Imput
Backbone
Embedding
Instance Representation
CoPA Pipeline
[G]
Σ (w)
Prototype Transformation
Prototype Representation
Input
Backbone
Embadding
Transformation
Head
Representation
Instance
The upper subfigure shows the URL pipeline which applies the same transformation to both prototype and image instance embeddings. The bottom subfigure shows the pipeline of CoPA, which tries to adapt two different representation transformation heads respectively for prototypes and image instances in the way of CLIP via substi- tuting text prompts with prototype embeddings.
Figure 2:
Typically, a cross-domain few-shot classifica- tion task with vary-way vary-shot is randomly sampled in an episodic way. To be specific, at the beginning of each learning episode, new task T
{Dr, Qr is sampled from specific sub-dataset of the meta dataset S, S, where
DT= = { 1/4) ) denotes the support set of the sampled task, which is used for model training and QT {(0) 9 12T denotes the query set of the task that is used for model evaluation.
Problem Formulation. In this paper, we follow the pre-training framework adopted in previous works [12, 34, 29, 30] and build our method upon a frozen pre- pre-trained backbone.
Consider a frozen pre-trained backbone fø- Id + Rdout that is parameterized with a set of optimal parameters o² and trainable fine-tuning module he Rd - Rddu that is parameterized with 6 Given a set of support data Dy = {(z,, y/s 1 we then can obtain a set of representations 2 = {(2,, ! where 2; = ho(fs- (a::)), from the frozen pre-trained backbone and the fine-tuning module. The ultimate goal of the pre-training framework is learning a set of optimal task-specific parameters 0" from the given support data so that better generalization performance can be achieved on the query data in the task. To be specific, following previous works [34, 29], the learning problem is formulated minimizing a nearest centroid classifier (NCC) loss [47]:
Pr
1
[(0) =
Σ
log(p(y,=c/z3: 0))
DD
E=1
where p(y c)z;0) Ent ed(x.e') denotes the likelihood of a sample z belonging to its class c, d(---)
d(e.cc)
denotes a measure, such as negative Euclidean distance function or cosine similarity function, to describe the similarity between samples and prototypes. In this paper, we follow URL [29] and adopt cosine similarity as the measure. Moreover, c, denotes the prototype of class c, which is generated with all available samples the class e and formulated as: C.
[c2 Ezec, Z.C. = {zilyt = e).
3 Revisit the Previous Adaptation Strategy
In this section, we first revisit the adaptation strategy applied in the previous work and uncover an implicitly adopted assumption that the prototype and image instance representations are transformed with the same (linear) transformation. However, prototypes intuitively depict different levels of information from image instances. Then, we empirically demonstrate that there naturally exists a gap between prototype and image instance embeddings. Our further investigation into URL reveals that applying the same transformation to prototypes and images removes the discriminative information in gradients and constrains learning compact representation clusters where the gap is preserved.
3

---

3.1 An Assumption of Previous Adaptation Strategy
In previous works [29, 30] that are based on prototypes, cross-domain few-shot classification is typically formulated as a maximum likelihood estimation problem as mentioned in Eq. (1). The ultimate goal of this problem is learn set of representations such that classification can be performed by measuring the similarity between representations of image instances and prototypes.
Typically, the prototype of each class is calculated with all instances within the class (see Section 2 for concrete details). Assume that the transformation is defined as a linear transformation parameterized with θ dR Xdout, then the calculation of the prototype e of the class set C can be formulated as:
c
1 [C|
Σ #EC
(z)
Σ for (x:) O.
CCI
Instance Representations
DEC
Prototype Embeddings
Since the representation transformation usually linear, the average of image instance representations (the second part) is equivalent to the representation transformation of prototype embeddings (the third part). Thus, it is easy to observe that an assumption implicitly adopted in the URL framework:
Instance-level and prototype-level embeddings share the same representation transformation.
Intuitively, compared with image instance representations, the prototype representations contain higher-level information which is commonly shared among the instances within the class yet dis- criminative from that of other classes. Thus, there seems to exist gap between prototypes and images. Such an intuition motivates us to take a further step to explore whether such gap exists and what effect the shared representation transformation imposes on such gap. To this end, we perform a series of analyses on the prototype and image instance embeddings and their corresponding transformed representations. addition, we also investigate the mechanism of the adaptation strategy.
3.2 Empirical Analysis Prototype and Image Instance
In this section, we conduct an analysis on the prototype and image instance embeddings and their corresponding representations obtained from the same representation transformation. Specifically, the embeddings are extracted from a frozen pre-trained ResNet-18 18 [18] backbone, and the representations are transformed from the embeddings with a shared linear transformation model done in URL [29].
"Modality" Gap between Prototype and Image Instance. According to Liang et al. [32], there exist modality gaps between different modal embeddings, such as text and image embeddings extracted from visual language models (e.g. CLIP), and preserving such modality gaps facilitates improving the downstream performance. We notice that the prototypes, to prompts. Specifically, both text prompts and prototypes depict the common concepts shared among all images within the corresponding class yet discriminative from other classes. This observation motivates us to validate whether such a "modality" gap exists between prototypes and image instances.
extent, play the same role text
In this section, an empirical analysis is performed on the validation set of ImageNet [44]. Fol- lowing Liang et al. [32], we define the gap as the difference between the centroids of normalized prototype and image instance representations (embeddings) A = Pr Elerl
1 ENz Cj. No
where |Dy] denotes the size of support data representation set Z (z. |PTl and No denotes the number of classes. More detailed validation results on other datasets are available in Appendix B.
We visualize the distributions of prototype and image instance embeddings with UMAP [37] in Fig. ((a). As shown in the figure, is easy to observe that the prototype and image instance embeddings are located in two completely separate regions of the feature space, and there naturally exists a gap between prototype and image instance embeddings. Moreover, to verify the general existence of such gap, conduct analysis with 600 randomly sampled tasks on each of all 8 seen domain datasets in Meta-Dataset. The results are reported in Table in Appendix B. The results reveal that the gap also generally exists between prototype and instance embeddings in other datasets.
Larger Gap Facilitates Generalization. To further figure out the property of the prototype-image gap, we conduct the same embedding shift experiment done in previous work [32]. Specifically,

---

-2.00
originallaap:
Validation Loss
-4.00
-6. 6.00
8.00
-2
Euclidean Distance
30
(a) Validation loss
(b) Embedding clusters
(c) URL clusters
Figure (a). The global minimum validation loss is achieved when the "modality" gap is enlarged. Fig. (a) depicts the validation loss landscape w.r.t the changes of the "modality" gap between prototype and image instance embeddings. The validation loss fails achieve the global minimum at the original gap, and the global minimum can be achieved when the gap is enlarged. (b)-(c). The shared representation transformation fails to learn compact instance representation clusters. According to the visualization results of both prototype and image instance embeddings and their representations obtained with the same representation transformation, compared to the prototype and image instance embeddings extracted from the frozen pre-trained backbone (Fig. (b)), the shared transformation fails to learn image instance representations which are well clustered (Fig. (c)).
we manually shift prototype and image instance embeddings by scaling the gap between two sets of embeeddings/eerreentttiion to narrow or enlarge the gap and observe the change of validation loss. The results reported in Fig. 3()). The X-axis depicts the size of the gap where the negative value means the position of prototype and image embeddings are mutually replaced. From the figure, is easy to observe that the validation loss fails to achieve its global minimum the original gap between prototype and image instance embeddings. Interestingly, we can further observe that the global minimum be achieved by slightly enlarging the gap. This is consistent with the phenomenon observed by Liang et al. [32]. Thus, an intuitive insight is that appropriately enlarging the gap between the prototypes and image instances contributes achieving better generalization performance.
From our perspective, the phenomenon that the larger gap facilitates improving the generalization performance can be attributed to two aspects. On the one hand, each prototype is generated with all instances within the class. Thus, data samples are naturally closer/more similar to the prototypes of their own class. This potentially results in slight overfitting. Thus, when the gap is slightly enlarged, it is equivalent to decreasing the similarity between prototypes and images. Such an enlargement, in turn, helps alleviate the overfitting and further improve the generalization performance downstream tasks. On the other hand, as images and prototypes describe different levels of information, good generalization performance can only be achieved when the embeddings enlarging the gap can also be seen as exploring the optimal distributions of embeddings for alignment.
well aligned. Thus,
Drawbacks of Shared Transformation. In order determine the effect of the shared representation transformation on representation learning, we further analyze the prototype and image instance representations obtained from the existing SOTA baseline, URL [29]. The visualization results are respectively presented in Figs. 1(b) and 3(c). According to Fig. 1(b), we can observe that the gap between prototype and image instance representations is numerically shrunken after applying the same transformation to both prototype and image instance embeddings. Meanwhile, we can also observe from Fig. 3(c) that URL fails to learn well-clustered representations for classes in the task. All these results, to some extent, demonstrate the draw backs of applying the shared transformation.
3.3 Further Exploration on Shared Transformation
In this section, we conduct further investigation to explore the effect of the shared transformation. Theorem 3.1. Let the measure d(.,.) be the cosine similarity function. Given a set of normalized finite support data representation Z [(z,, y))r - where ||æ||2 = 1 for 4z Z and Ne classes are included, then we have a lower bound of the NCC-based loss in Eq. (1):
(()) 2
z
c
2,
n im
Z'EZ
where z' is an independent copy of samples in 2, C. denotes sets of sample representations C. {z y c}, and a is constant that satisfies 0 S < 1/(Nc|C, for Vj.
5

---

The complete proof is available in Appendix G.I. Theorem 3.1 provides a lower bound of Eq. (1). According to the bound, if we adopt the right side as the surrogate loss, solving Problem (1) is then equivalent to simultaneously maximizing the similarities between instances and their corresponding prototypes and minimizing the similarities among all samples. Moreover, since 2 2, < 2 2, for vj * i, the second term can then be approximated as Σzez = =T z by omitting the similarity term between different image instance representations. Then, we can reformulate Problem (1) as:
1 Tr (Jo 1( 11) [Dy]
a fo- (x- + [Dyl Tr (fTr((o())))
(X)) (x)
where X E R Ixdout and Y E RIPTIXNN respectively denote the support image instances and the corresponding one-hot labels, Op E R xdout and θ1 RRddus Xdo. denote the model parameters of linear transformation heads respectively for prototype and image instance embeddings, Tr(-) denotes the matrix trace operation. YYT fa- (X) E [R| (Pol denotes the prototypes which are expanded to the same size of instance embeddings. In this way, the gradients w.r.t. Op and I are:
1 [Dr]
Ve,,(Op, θ1)
0 fo- YY fø- (X),
2a
Ve, 01) [Dr] 0 0 (X)
YY Sa- (X)
so- S- Lo- (X). [Dr]
According to the gradients above, we can observe that the gradients of Op depict the similarity between instance representations and prototype embeddings while the gradients of O depict the similarity between prototype representations and instance embeddings. However, in the case of the shared transformation (i.c., Op = θ₃ = θ), such differences are removed. Thus, the shared transformation may potentially drop the discriminative information in gradients during the adaptation phase. Meanwhile, the shared transformation also constrains the complexity of the function space, and in turn, prevents exploring the optimal representation distributions for both prototypes and images.
(The shared transformation, Consider a support data DD= { 96) 3:)) 1 composed of Nc classes and frozen pretrained backbone fo- Rdu.. - Rd parameterized with the optimal parameters o' Let θ Raxd be shared linear transformation across the prototype and image instance embeddings. Then, we can obtain the image instance representations Ne, 2 = 1 1PTl [fa (a:)))[²r |PT and the prototype representations C {c:) where c; Σ Exec, for (x')0. Then we can obtain the bounds of the representation gap:
Theorem 3.2
1 Σ Exrec z' = [c.
m 11111 |cmb/| 1/3 <
1
Σ #E2
x-
1 Eo
S M 101
A [emb 1
1
[Dy]
Nc
CEC
where Kamb PDT EMED, (x)) No b==
[co] Exrec, So (x') denotes the gap between
prototype and image embeddings, m = minigigd cos? (Aemb; Q¹) denotes the minimum value of cos? (Aemb: O¹), and M = maxigi5a ( (Bemb: O²) denotes the maximum cos? ( A)).
The proof is provided in Appendix G.2.1. In Theorem 3.2, we derive the bounds of the gap between prototype and image rep- resentations learned from shared linear transformation. For simplicity, we only consider the unnormalized representations. However, in practice, the gap is calculated with normalized representations. According to the theorem, we find that the bounds are closely related to the Frobenius norm of the trans- formation matrix and the angle between the embedding gap vector and the column vectors of the transformation matrix.
livvre 2012
||01|s
0.72
sircraft
cu, bu_birds
0.70
dtd
quickdraw
fungt
0.68
yes_flower
steps
Theoretically, it is intractable to directly analyze the upper bound of the gap between prototype and image representa- tions. Thus, to determine the reason for the gap shrinkage, we empirically track the change of the scale VM || Fr on 600 randomly sampled tasks. The insight here is that the represen- tation gap will not be larger than the embedding gap if the scale is smaller than 1.0. According to curves in Fig. 4, we observe that the scale consistently smaller than 1.0 all cases. Thus, shared transformation cannot preserve the original gap between prototype and image instance embeddings.
Figure 4: The change of the scale of the upper bound of URL representa- tion gaps during the adaptation.

---

Table 1: Results Meta-Dataset under the "train on all datasets" setting. Under the "train on all datasets" setting, the first 8 datasets are treated as "seen domians" while the last 5 are treated as "unseen domains".. Mean accuracy and 95% confidence interval are reported.
Main Results
More Learning Modules
Datasets
CNAPS
S-CNAPS
Tr-M
FLUTE
TSA
TA
CoPA
TSA
ImageNet
508-111
58.4
586
518
57.3
57.8
57.4
11
57.5
57.8
1.1
Omniglot
91.7:20.5
91.6
0.6
91.1
0.1
912
1
0.1
92.0
93.2
1:
0.5
94.1
0.4
94.3
0.5
94.7
0.4
9
0.4
94.5
0.4
Aircraft
8.
82.0
- 0.7
85.5
1 0.5
838
0.05
R2
0.7
87.
1 0.5
88.2
+ 0.5
88.8
0.5
88.9
1 0.5
89.0
- 0.5
89.3
: 0.5
Birds
13.6=0.9
74.8
71.0
762
08
753
4
792
+
80.2
50.8
808
0.8
80.7
0.8
81.2
0.8
Textures
71.
716
712
68.
76.2
77.8
77.1
76.9
0.7
77.8
Quick Draw
74.7=0.88
76.5
108
818
06
R2.4
06
77.3
0.
79.5
0.7
82.2
2 0.6
828
82.2
1 0.6
82.2
0.6
82.7
0.6
Fungl
50,2+111
46.6
+ L0
64.3
+ 09
64.0
+
10
48.5
+
10
58.1
+ 1.
68.7
+ 1.0
69.5
1.0
67.4
+ 10
68.1
+ 1.0
69.0
+: 1.0
VGG Flower
88.9:00.5
90.5
8299
1
57.9
1
0.6
90.5
1
0.5
91.6
1:
91.9
1:
92.7
0.5
92.5
0.5
92.4
0.5
93.0
0.5
Traflic Sign
16.5
57.2
51.
482 1
63.0
1:
1.0
SK4
1.1
63.3 :
1.2
66.6
1.
83.5
4 0.9
88.3
2: 0.8
88.5
:: 0,9
MSCOCO
11.
$2.0
51.5
$2.8
50.0
1-2
$7.9
1.0
MNIST
94.6
0.4
94.3
1
0.4
90.6
1
0.5
96.2
1:
95.6
1:
94.7
1:
95.2
0.4
96.7
0.4
97.0
0.4
97.5
0.4
CIFAR-10 CIFAR-100
74,9 = 0.7
66.5
+ 0.9
67.0
1.08
75.4
+ 0.8
7866
:: 0.7
71.9
0.8
73.0
0.8
NO.3
+ 0.8
76.6
=: 0.9
78.7
: 0.8
61.3
1.1
$6.9
1.1
57.3
62.0
+
10
67.1
+ 1.0
62.9
= 1.0
63.4
1.01
70.6
+ 1.0
64.5
1.2
70.9
019
Average Secn
71.6
75.7
75.9
77.4
76.2
76.2
79.9
80.6
80.1
80.2
$0.7
Average Unseen
69.4
Average All
71.2
71.3
71.8
73.8
73.8
75.8
79.2
Average Rank
For faimess, resuiss of URL, ISA, 1A Net, and proposed CoPA methods reproduced with random seeds, and report the average the reprodicciins in tubllc Partiulluty, although the reposted perforeincce URI, lower than that original poper, the reproduction results consistent with those reposted ou their project website. The miks are cslculited only with the first 10 dezasets and only with the methods meniiond chove.
Table 2: Results on Meta-Dataset under the "train on ImageNet only" setting. Under the "train on ImageNet only" setting, only ImageNet is treated as "seen domain" while the remaining as "unseen domains". Mean accuracy and 95% confidence interval are reported.
Main Results
More Learning Modules TA
Datusets
Fincture
ProtoNes(lirg)
BOHA
FP-MAMI.
AFP-MAMI.
FLUTE
URI.
CoPA
TSA
COPA-TSA
ImageNet
458
53.71111
51.91111
49.11.11
$28:11
469:111
$7.31111
57.7:1111
57.77
L1
57.4
L.1
57.5 + LI
Omniglot
669116
6..5113
67.62:122
634113
61,91155
616611
69.111
7..9
73.5
1.2
72.8
1.2
73.3
: 1.2
Aircraft
68.74-133
$8.0++110
54.1
56.0--110
63.4
45.5+110
57.61-19
61.6110
65.1
11
63.5
1.0
64.9
+ L.1
Birds
57.3-1133
71.11099
70,7::0.9
68.71:10
69.:111
479:1:10
72.91009
74.2:009
74.0
1 0.9
73.8
0.9
74.7
1 0,9
Testurss
6901:09
68.8108
68.320.88
66.1108
70.8:09
618108
75210.7
77.0207
0.7
0.7
77.6
Quick: Druw
42.61:1.2
53.3+t1.0
1..
515+-10
29.2-1-.2
57.5+110
57.9119
61.3-1.
64,6
1.0
63.9
1.0
64.7
1.0
Fungi
38.2:::100
40.7:1:12
41.4111
40.0:1111
4..21:2
318:110
46.21:10
48.0-1.1
46.8
1 L1
47.8
2 L1I
48,3
1 L1
VGG Flower
85.5--0.7
51.0.0.7
01.3-0066
87-24-0-7
36.0-0088
80.1+:9
86.9-005
98.9-066
0.6
0.6
90.6
Traffic Sign
66.81113
18.1
5..8
60.8-1-.3
61.24:12
36,7
MSCOCO
34.91100
1.17.111
48.021.0
43.71:1.1
48.1::111
41441:0
53.01:100
16.1
55.8
1.0
51.3
$7.4
1.0
MINIST
50.++00.8
86.24-0.7
0,73-007
93.6
0.6
94.7
0.5
95.1
+ 0.6
CIFAR-10 CIFAR-100
65.4:0.8.
69.5:10.88
0,.4-008
79.6
0,8
0.9
76,8 1
12.7.:1.11
62.01:100
62.7:11.0
Average
45.8
53.7
49.5
$2.8
46.9
$7.7
57.7
$7.5
$7.5
Average Unseen
56.5
66.6
68.7
72.7
71.9
73.2
Average All
55.8
65.9
67.7
71.6
70.8
72,0
Average Rank
9.3
7.2
8.0
9.0
7.1
10.1
5.3
4.1
2.5
3.2
2.2
TTee results URRI, TSA, TA Not our preposed methods are repreduced with random scads and regortal the average of reproducciion The ranks only cunsider the first 10 datzeets and are calcultted only with the methuds in the table.
4 Contrastive Prototype-Image Adaptation
In this section, to address the problem mentioned above, we follow CLIP [42] and propose a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformation modules respectively for the prototypes and image instances in similar way of contrastive learning.
Backbone Pre-training, We pre-train a 18 [18] in the same way as URL [29] the backbone. Specifically, 8 Nomain-specific backbones respectively for all seen domains are firstly pre-trained. Then. universal encoder distilled from these backbones and frozen during the meta-test phase.
Contrastive Prototype- Image Adaptation. Briefly, CoPA aims to adapt two different transformation models respectively for prototype and image instances by optimizing the symmetric cross-entropy loss. The entire pipeline of CoPA resembles that of CLIP [42]. CoPA replaces the text prompts with prototypes with the simple insight that both prototypes and text prompts depict some higher-level concepts that are common among samples within class yet discriminative from other classes.
Specifically, consider two representation transformation heads, hap and he, respectively, for pro- totypes and image instances. Given support set DT (YY)) where X < PPrlx de-
7

---

5.50
protstypes
12-
Validation Loss
5.25
4.75
-04
4.50
original gap: 1.38
-0.5
0.0
0.5
10
15
Euclidean Distance
|||||
1.38
(b) CoPA representation clusters
(c) CoPA validation loss gap
Figure 5: (a). The gap between prototype and image instance representations is enlarged from 0.22 to 1.38 by CoPA. Such a phenomenon is consistent with that demonstrated by Liang et al. [32]. (b). The clusters of image instance representations learned from CoPA. The more compact clusters reveal that CoPA learns better instance representations. (c). The validation loss achieves its global minimum at the gap learned by CoPA, which indicates that CoPA can improve the generalization performance.
notes the support data samples, while Y E R R \xNe denotes the corresponding one-hot labels, respectively obtain the prototype representations Zp = hop (YY fe- (X)) and image instance representations Z = ho, (fa- (X)). Then, CoPA learns the representations by minimizing the following symmetry cross entropy objective:
where Ne denotes the number of classes,
1 min [(Op, 01) Lcr( Zi ZiZp, ppouddo) + LCR( - Zp Y ppecudo); 0p,0:
(3)
where 7 is temperature coefficient, Y peeudo [0, I,... [DT 1] denotes the pseudo labels, and Ccg denotes the cross-entropy loss. The complete CoPA algorithm is presented in Algorithm
Discussion, Two aspects are worth noticing in our proposed CoPA. On the one hand, two different transformations adopted in proposed CoPA method. In this way, the discriminative information in gradients is preserved in different sets of parameters. Besides, introducing different transformations expands the function space, which further facilitates learning better prototype and image instance representations. On the other hand, the contrastive learning objective, which has been demonstrated facilitate preserving the modality gap, is adopted the learning objective. In particular, match this objective, we modify the prototype embeddings from YT J (X) ) to YY fo- (X) so that the size of the prototype embeddings is consistent with that of image instance embeddings. Such modification further indicates the cluster structure of the given support data set. Thus, by minimizing the symmetric cross entropy loss, the two transformations are encouraged align the prototype and image instance representations and learn more class-specific and discriminative representations.
5 Experiments
In this section, we evaluate CoPA on on ImageNet only" settings with series of tasks answer the following questions: (1) Does CoPA achieve better generalization performance? (2) Does CoPA benefit from extra learning modules? (3)
[49] under both "train on all datasets" and "train
Why does CoPA perform better? Detailed experimental settings are available Appendix
5.1 Main Results & Analysis
In this section, we evaluate the generalization performance of our proposed CoPA method on Meta-Dataset under both "train on all datasets" and "train on ImageNet only" settings. In order to demonstrate the effectiveness CoPA, we compare CoPA against several currently state-oftthe-ar baselines, including fo-Proto-MAML (FP-MAML) [49],
(AFP-MAML) [2],
CNAPs [43], Simple-CNAPs [3], SUR [12], URT [34], Tri-M [35], FLUTE [50], URL [29].
Train on All Datasets. We report the evaluation results under the "train on all datasets" setting in Table As shown in the table, easy to observe that CoPA achieves the best performance on out of 13 datasets among all approaches and outperforms URL in all cases. Specifically, compared with URL, our proposed CoPA method achieves 0. 0.7%, 1.5%, and 1.0% improvements on average respectively seen, unseen, and all domains. It is also worthwhile to notice that CoPA performs better on unseen domains. Specifically, our proposed CoPA method achieves 3.3%, %, and 1.1% improvements respectively on Traffic Sign, MSCOCO, and CIFAR 10 datasets. Meanwhile, also

---

notice that CoPA achieves comparable or even better results on seen domains compared with TSA [30] which plugs extra learning modules into the frozen backbone for more task-specific parameters.
Train on ImageNet Only. The results under the "train on ImageNet only" setting are reported in Table 2. Briefly, CoPA achieves the best performance on 11 out of 13 datasets and consistently outperforms URL in all cases. On average, CoPA achieves 0.4%. 2.1%, and 1.8% improvements respectively on seen, unseen, and all domains. Moreover, the results also reveal that CoPA consis- tently performs better on unseen domains. Specifically, CoPA achieve impressive improvements on Omniglot (1.5%), Aircraft (4%). Birds (1.3%). Textures (1.8%), Quick Draw (3.4%), Fungi (1. (1.8%), 1oG_elower (2%), Traffic Sign (2.6%), MSCOCO (3.1%). MNIST (1.1%), CIFAR (2.9% & 0. 0.7%).
Extra Learning Modules. We follow TSA [30] to plug extra trainable modules into the frozen backbone. The results are reported respectively in Table and 2. According to the table, we can observe that CoPA benefits from these extra modules and achieves better performance than previous state-of-the-art approaches (e.g., TSA and TA -Net). CoPA + TSA outperforms 11 out of 13 datasets under the "train on all datasets" setting. Similarly, CoPA + TSA consistently achieves better generalization performance on unseen domains. Specifically, CoPA + TSA achieves 5%, 2.6%, and 0.8% improvements respectively on Traffic Sign, MSCOCO, and MNIST datasets. Under the "train on ImageNet only" setting, CoPA + TSA achieves the best performance on 7 out of 13 datasets. Compared with TSA, CoPA TSA achieves better performance on Birds (0. 7%). Textures (0.8%), Fungi (1.5%), 0lower (0.8%), Traffic Sign (4.5%), MSCOCO (1.6%) and MNIST (1.5%).
5.2 Discussion: Why CoPA Performs Better?
In previous experiments, we have demonstrated that CoPA can achieve better generalization per- formance. In this section, we aim to validate whether CoPA preserves the gap and learns better representations. Further, we would like to explore why CoPA performs better than previous works.
More Results. To verify the effectiveness of CoPA, we conduct more analyses. Firstly, we investigate the representation shift experiment to observe the gap between prototype and image representations. As shown in 5(a), in contrast to URL, the gap between prototype and image representations learned via CoPA is enlarged. Such a phenomenon demonstrates that CoPA can hold the gap between prototypes and image instances. In addition, we further check the image instance representations learned with CoPA. According to Fig. 5(b), the image representations are more clearly and compactly clustered than Fig. 3(c). This case reveals that CoPA can facilitate learning more class-specific representations.
5.2.1 Ablation Study
The results reported above indicate that CoPA can enlarge the representation gap and learn well- clustered image representations simultaneously. In order to further figure out why CoPA is able to improve performance, in this section, we propose to conduct ablation studies on the two important components of CoPA: the different transformations and SCE loss. The results are reported in Table 10.
Ablation:
Different Transformation Modules. As we have discussed in Section 3.3, the shared trans- formation may drop the discriminative information in gradients. Thus, we propose to substitute the
255
Tost Loca
2.0
shared transformation with the different transfor- mations adopted in CoPA. As shown in the table, although 2 achieves slightly better re- sults in some cases, its performance drops in the remaining cases. By plotting the test loss curves (cf. Fig. 6). we can observe that overfitting takes place.
20
30
30
(a) ImageNet
(b) CIFAR 100
Figure 6: Comparison of test losses of URL and
Ablation: SCE LOss. From the table, we can ob- serve that URL with SCE loss achieves comparable performance the URL baseline most datasets. Meanwhile, URL with SCE achieves significantly better results on some datasets, such as Fungi and MSCOCO, in which URI. baseline tends to overfit (cf. Figs. (g) and 27()). This indicates that SCE loss can improve generalization performance. In addition, we also replace the SCE loss with the NCC loss on CoPA. According to the results of "CoPA + NCC shown in the table, it is easy to find that the performance in all cases drops drastically, For example, the performance on Fungi and MSCOCO respectively decreases by 11.6% and 18.7%. All these empirical results demonstrate that SCE loss facilitates improving generalization performance.
2 on ImageNet and CIFAR 100.

---

5.2.2 Further Discussion
Effectiveness of Different Transformations. As aforementioned, an advantage of CoPA is applying two different transformations respectively to prototype and image instance embeddings. The goal of applying two different transformations in CoPA is to preserve the discriminative information of gradients different sets of parameters. Meanwhile, applying two different transformations also increases the complexity of the hypothesis space, which contributes to reducing the approximation error and learning representations in a more flexible way. However, simply increasing hypothesis space complexity may also result in overfitting (ef. Fig. 6). According to Fig. 27, CoPA, which is equipped with SCE loss, benefits from the increased complexity and achieves better performance.
Essence of Gap Enlargement. In Fig. 3(a), we observe that the global minimum of the validation loss can be achieved by slightly enlarging the gap between prototype and images. Meanwhile, according to Fig. 5(c), the validation loss achieves its global minimum at the enlarged gap learned with CoPA. As aforementioned, we conjecture that the reasons for the phenomenon that gap enlargement facilitates generalization mainly include two aspects: representation alignment and overfitting alleviation.
We first start with representation alignment. Typically, representation alignment is performed with SCE loss. Specifically. we omit the temperature coefficient T and simply rewrite Eq. (3) as:
ID+l
[Drl
cxp(z, ci) Pr Ej=1 exp(z,
cxp(c, zi)
LSCE = DD
log
log
(4)
[DT]
|Dr exp(c, 2,) -Ø=1
Theorem 5.1, Given a set of normalized finite support data representation 2 = {z v:) bim and a set of normalized protolype representations C {ci]i=1 where ||=||2 = 1 for V/2 Z and |c||2 = 1
for Vc E C, then we are able to oblain a lower bound of SCE loss in Eq. (4):
ESCE 2
2 Σ
2
2
ΣΣ
11
No
Call
ck. 2Tc
2,T
T2
12
n
where C₃ denotes the set of support data of the class is and Ne denotes the number of classes.
The proof is in Appendix G.3. According to Theorem 5.1, we find that the lower bound of SCE loss plays a similar role to that of NCC loss. In detail, the lower bound aims at maximizing the similarity between each sample and its corresponding prototype while minimizing the similarities between the sample and all prototypes. When minimizing the lower bound as the surrogate loss, the second term, which measures the similarities between image and prototype representations, is minimized. Thus, the Euclidean distances between representations of prototypes and images (i.e., the gap) are enlarged.
An interesting point is that the similarities between images and prototypes are minimized with the for all cases. Thus, the similarities between samples and the prototypes involving more samples will be significantly reduced. Consequently, the gap between images and prototypes tends to be enlarged.
weights calculated based on the size of C.. Differently, NCC loss, the weights are fixed to No
1
For the overfitting, on the one hand, the potential overfitting discussed in Section 3.2 is mitigated by applying different transformations. On the other hand, the high similarities between samples and prototypes may also result in overfitting, For example, in Fig. 3(a), the validation loss increases when the gap is narrowed. However, as we have discussed above, minimizing SCE loss tends to reduce such similarities. Thus, the gap enlargement also functions regularization alleviate overfitting.
Therefore, with all aspects taken into consideration, the essence of CoPA representation alignment of prototypes and images, in which a balance between learning discriminative representations and achieving better generalization performance, is explored in a more flexible hypothesis space.
Conclusion
In this paper, we validate that there naturally exists a gap, which resembles the modality gap in visual language models, between prototype and image instance embeddings, and uncover that the shared transformation shrinks such gap and constrains the learning of well-clustered representations. In order to solve these problems, we propose CoPA to adapt different transformations respectively for prototype and image instances via optimizing SCE loss. Empirical results on Meta-Dataset reveal that CoPA can achieve state-of-the-ar performance. Our further analyses indicate that the essence of CoPA is a representation alignment, where a balance between learning discriminative representations and achieving better generalization performance is explored in more flexible hypothesis space.
10

---

Acknowledgement
HDT, ZKZ and BH were supported by Guangdong Basic and Applied Basic Research Foundation Nos. 2022A151501 1652 and 2024A151501 2399, NSFC General Program No. 62376235, HKBU Faculty Niche Research Areas No. RC-FNRA-IG/2-22SSC/// and HKBU CSD Departmental Incentive Scheme. FL is supported by the Australian Research Council (ARC) with grant numbers DP230101540 and DE240101089, and the &SF&CSIRO Responsible AI program with grant number 2303037. TLL was partially supported by the following ARC projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint 1607.0665, 2016.
[2] Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta- learning with adaptive hyperparameters. NeurIPS, 2020.
[3] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. CVPR, 2020.
[4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
[5] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
[6] Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Yu-Chiang Frank Wang, and Jia- Huang. A closer look at few-shot classification. ICLR, 2019.
[7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0429) 2020.
[8] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. CVPR, 2005.
[9] Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. CVPR, 2014.
[10] Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot transfer. NeurlPS, 2020.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
[12] Nikita Dvoriík Cordelia Schmid, and Julien Mairal. Selecting relevant features from multi- domain representation for few-shot classification. ECCV, 2020.
[13] Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. EMNLP, 2019.
[14] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leariing for fast adapta- tion of deep networks. ICMI., 2017.
[15] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your latent-a new approach self-supervised learning. NeurIPS, 2020.
[16] Yurong Guo, Ruoyi Du, Yuan Dong, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Task-aware adaptive learning for cross-domain few-shot learning. ICCV, 2023.
11

---

[17] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. CVPR, 2006.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
[20] Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. IJCNN, 2013.
[21] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Lc, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. ICML, 2021.
[22] Jonas Jongejan, Rowley Henry, Kawashima Takashi, Jongmin Kim, and Fox-Gieg Nick. The quick, draw! a.i. experiment. https://quickkrraw with 2016.
[23] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeurlPS, 2020.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, 2009.
[26] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, (3006666661332-133333 2015.
[27] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-22224 1998.
[28] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021.
[29] Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for few-shot classification. ICCV, 2021.
[30] Wei-Hong Li, Xialei Liu, and Hakan Bilen. Cross-domain few-shot learning with task-specific adapters. CVPR, 2022.
[31] Yazhe Li, Roman Pogodin, Danica Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. NeurIPS, 2021.
[32] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS, 2022.
[33] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[34] Lu Liu, William Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. A universal representation transformer layer for few-shot image classification. NewrIPS, 2020.
[35] Yanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, and Yi Yang. A multi-mode modulator for multi-domain few-shot classification. ICCV, 2021.
[36] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint 1rX1v:100..5551, 2013.
12

---

[37] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03425 2018.
[38] Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-op:: Simple and effective postpro- cessing for word representations. ICLR, 2018.
[39] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, 2008
[40] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: arXiv: 180.00370..0. 2018.
[41] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. AAAI, 2018.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021.
[43] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. NeurIPS, 2019.
[44] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015.
[45] Brigit Schroeder and Yin Cui. flngg_uunnnpnpa 2018.
Fgvex
fungi
classification challenge.
[46] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embedding for face recognition and clustering. CVPR, 2015.
[47] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NIPS, 2017.
[48] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. NIPS, 2016.
[49] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. ICLR, 2020.
[50] Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning universal template for few-shot dataset generalization. ICML, 2021.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 2017.
[52] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. NIPS, 2016.
[53] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001 2011.
[54] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. arXiv preprint arXiv:21099 14084, 2021.
[55] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. ICML, 2021.
[56] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In MIHC. PMLR, 2022.
13

---

Appendix
Limitations
15
Broad Impact
15
Related Work
15
B Detailed Study on Gaps between Prototypes and Images
17
Detailed Study on Prototype Selection
19
C.1 Introduction of Three Prototypes
19
C.2 Analysis Settings & Results..
20
Detailed Task Settings
D.1 Introduction of Meta-Dataset
21
D.2 Brief Introduction of Vary-Way Way-WWa Vary-Shot Task Settings
21
Detailed Implementation Settings
22
E.1
Pre-trained Backbones
22
E.2
Architecture of Visual Transformer
................................
23
E,3
Module Initialization
E,4
Optimization
23
E.5
Hardware & Random Seeds
23
E,6
Task Specific Adapter Setting
23
E.7
Reproduction of State-of-the-art Methods
24
More Experimental Results
24
F.
CoPA with More Complex Modules
24
F.2
Fewer Shot Classification Tasks,
24
E.2.1
Fewer-shot Experiments
24
F.2.2
Study on Effect of Support Data Size
27
F3
Results of Analysis regarding Number of Parameters
27
F4
Efficiency of CoPA.
28
F55 Ablation Study
28
G Proof Results
30
G.1 Proof of Theorem 3.1
30
G.2 Proof of the Bound of the Gaps
30
G.2.1 Proof of Theorem 3.2
31
G.3 Proof of Theorem 5.1
31
14

---

Limitations
In this paper, in order to solve the problem that the shared transformation adopted in previous works potentially hurts the discriminative information in gradients and constrains learning representations where the gap is preserved, we propose to fine-tune two different transformations respectively for prototype and image instances in the same way of CLIP by treating the prototypes as text prompts.
One potential limitation of our proposed CoPA method is the symmetric cross-entropy. Symmetric cross entropy loss is commonly applied in contrastive learning. In contrastive learning, the batch size of the data samples is an important issue for better downstream tasks. Larger batch size usually means better downstream performance. We also notice this phenomenon in this paper. A Although CoPA can still achieve relatively better performance, the performance tends to degrade when the size of the data batch becomes small. Thus, this constrains the application of CoPA to some extent.
Broader Impact
In this paper, we find that there exists a gap, which resembles the modality gap in visual language models, between prototype and instance embeddings. An intuition of such a phenomenon is that prototypes are generated with all data samples within a specific class and consequently can be treated as an abstract concept of all data samples in the class. From this perspective, the prototype information plays the same role as text prompts in visual language models to some extent. Thus, our work provides direction that we can extract some abstract concept information from the available single modal data and then train/adapt models with such information in some appropriate way, such as contrastive learning, to learn more discriminative and useful representation for downstream tasks. Since visual information is more diverse than texts, we think it will be a promising and feasible way of developing general models for various kinds of visual tasks. Based on this, there many potential societal consequences of our work, none of which we feel must be specifically highlighted.
A Related Work
Cross-domain few-shot classification. Cross-domain few-shot classification is a learning paradigm that aims at learning to perform classification on previously unseen data and domains by merely fast training model on several labeled training data. Generally, the cross-domain few-shot classification problem can be seen as an extension of conventional few-shot classification problem [52, 14, 47, 6]. The main difference between conventional and cross-domain few-shot classification lies in the domains for evaluation. In the conventional few-shot classification setting, a model is trained and evaluated on the same dataset (domain), where the distribution is invariant. However, in the cross- domain few-shot classification setting, a model is trained on the training sets of several datasets with different distributions and then evaluated on the test sets of datasets that are used for training as well as the datasets that have never been observed during the training/adaptatioon phase. Thus, compared with the conventional few-shot classification task, the cross-domain few-shot classification task is much more challenging since there exist distribution gaps between the training and previously unseen test datasets. Currently, all existing works regarding cross-domaie few-shot classification are mainly based on conventional meta-learning methods, such as Prototypical Networks [47] and MAML [14]. The former famous for its strong ability to learn effective inductive bias while the latter is impressive for its flexibility of adaptation to new tasks. Generally, the existing works can be divided into two different types according to their ways of training the backbones and classifiers
Some of the existing works train the model from scratch with sequence of few-shot tasks in the same way as conventional few-shot learning frameworks. A typical case of this kind of learning paradigm is Proto-MAML [49]. Specifically, Proto-MAML proposes to train a model with the modified Prototypical loss in the same way as MAML take advantage of the strengths of the two frameworks. We would like to note that Prototypical loss plays an important role in current cross-domain few-shot classification research. The reason behind this that the tasks adopted in cross-domain few-shot classification tasks are usually sampled with varied numbers of classes and shots [49]. In this case, it infeasible to train a classifier in the same way as the conventional meta-learning framework, where a parameterized classifier is initialized with a fixed number of classes. In addition to Proto-MAMI more works are proposed to solve cross-domain few-shot classification tasks. CNAPs [43] proposes to generate parameters with feature- wise linear modulation (a.k.a., FiLM [41]) for task-specific modules
15

---

in their model. However, a potential problem of CNAPs is that the Euclidean distance adopted in the loss implicitly assumes that each cluster is distributed according to unit normal. This may
further result in misclassification of data points. Based on this observation, SimpleCNAPs [3] further proposes a non-parametric classifier from the perspective of clas--oovariance-base distance metric to improve the classification performance of the original CNAPs method. Moreover, Doersch et al. [10] notice that the representations learned via the Prototypical Network framework may only represent the observed data yet discard the information that might contribute to the recognization of out-of- distribution classes. Thus, in order to solve this problem. they further propose CrossTransfone leverage query image information in pixel-level during the adaptation steps via an attention module.
to
Different from the learning frameworks that train the entire pipeline from scratch, the other kinds of methods usually respectively pre-train one or several Nomain--secinn backbones on all available datasets during the training phase in conventional supervised learning paradigm. Then, during the evaluation phase, the prior knowledge in these pre-trained backbones is transferred to the new tasks through representation fusion or representation adaptation. In cross-domain few-shot classification community, SUR [12] firstly proposes to select relevant features from the 8 pre-trained backbones to represent the data of the given new task via a vector. Later, URT [34] further proposes to quickly train multi-head transformer on top of 8 pre-trained backbones to learn to select features for unseen data. Besides, Triantafillou et al. [50] proposes to treat the convolutional layers in a model as universal templates while treating the batch normalization layers as the domain-specciff modules. Thus, FLUTE is proposed to pre-train& a set of universal weights and several Nomain-speccife modules and then train small model to learn to combine these BN layers and transfer the prior knowledge to the new tasks. Although leveraging pre-trained backbones is able to take advantage of high-quality representations learned from previous domains, it is quite time-consuming to perform several forward passes during both training and evaluation phases. In order to solve this concern, Li al. [29] proposes URL to distill the prior knowledge of all pre-trained backbones in one single backbone. Then, during the test phase, a simple linear layer is fast fine-tuned on top of the distilled multi-domain backbone to transfer the priori to the unseen domains. Later, TSA [30] is proposed to
BN
enhance URL via plugging more learnable residual modules into the frozen pre- pre-trained backbone to learn more task-specific features for unseen data. Meanwhile, recent work T [16] notices that different tasks prefer different learnable modules and proposes to train an Action Generation Network in the way of reinforcement learning learn to select the optimal module for each task.
Contrastive loss and contrastive learning. The ultimate goal of contrastive learning is to learn dis- criminative and robust representations at the instance level by comparing two augmented counterparts of an image. In practice, the goal is realized through contrastive loss which focuses on maximizing the similarity between positive pairs while minimizing the similarity between negative pairs.
Compared with early contrastive loss [8, 17, 46], the current contrastive loss is constructed upon batch of data, where the number of negative samples is larger than that of positive samples. Such contrastive loss was first proposed by Sohn [48] multi-class N-pair loss and then improved and further explored by further works [5, 19, 7, 15, 55, 4]. Based on the InfoNCE, Li et al. [31] further explores an alternative loss from the perspective of the Hilbert-Schmidt Independence Criterion measure. Different from conventional unsupervised contrastive learning. Khosla et al. [23] proposes a fully supervised contrastive learning paradigm. SupCon, via leveraging label information. To be specific. in conventional unsupervised contrastive learning, only the augmented counterparts of an image are treated as positive images. However, in supervised contrastive learning, all images that share the same "general" concept are treated as positive samples. The ultimate goal of SupCon is to pull normalized embeddings from the same class closer than embeddings from different classes. Then, with the emergence of pre-trained vision language models, such as CLIP [42], embeddings that are generated from other modalities are treated as supervised information and have shown impressive power in multi-modality learning fields. From our perspective, SupCon and CLIP share the same core. The key difference that SupCon learns representations by comparing each sample with all other samples within the same class while CL.IP learns to align the representations of each image sample with the representations of text information that are more "general" than a set of images.
Multi-modal contrastive representation learning. Multi-modality learning [21, 28, 54, 56] is a learning paradigm that aims at aligning representations extracted from different data modalities, such as text and images. Since the emergence of CLIP [42] has shown impressive performance on downstream tasks, more and more attention has been attracted to this research topic. Recently, Liang
16

---

Images prototypes
0.4
Images prototypes
0.0
mmages prototypes
0,9
-044
-0.4
-0.4
-088
-0.
1.0
(a) Aircraft Emh: ||||| 0.14
(h) Aircraft URL: ||| | 0.04
(c) Aireraft CoPA: ||||| = 1.77
0.8
Images prototypes
0.4
0.4
0.0
0.0
0.4
Images prototypes
Images
0.4
-0.8
prototypes
-0.5
-0.75
0.00
(d) DTD Emb:
|||| = 0.22
(e) DTD URL: || ||| = 0.06
(f) DTD CoPA: ||||| 0.63
images prototypes
Images prototypes
prototypes
-0.5
-0.60
-0.55
-2225
6.25
1.00
-0.75
0.00
(g) Qraw Emb:
||||| = 0.15 (h) QuickDraw URL: || = 0.04 (i) QuickDraw CoPA: |||||
1.33
images prototypes
Images prototypes
prototypes
0.8
0.0
-0.4
0.8
-0.50 -0.25 0.00
(i) Fungi Emb: |||| = 0.20
(k) Fungi URL: ||ق||| = 0.08
(l) Fungi CoPA: || = 1.75
Figure 7: "Modality" gaps between prototypes and images some other datasets. We selected a part of the visualization results regarding the "modality" gap between embeddings and learned representations of prototypes and image instances. It is easy to observe that URL shrinks the gap while CoPA enlarges the gap between the prototype and image instance representations.
al. [32] argues that there exists modality gap between different modal data and theoretically
demonstrates this from the perspective of cone effect [38, 13). Further, Liang et al. [32] demonstrates that such modality gap contributes to the downstream performance zero-shot learning tasks. In this work, following Liang et al. [32], we find that such kind of "modality" gap also exists between prototype and instance embeddings in the context of single modality data, and applying the same transformation to both of these two kind of embeddings shrinks such a gap. In order to solve this problem, we propose CoPA adapt different transformation heads respectively for prototypes and instances in the way of CLIP where text prompts are substituted with prototype embeddings.
B Detailed Study on Gaps between Prototypes and Images
As we have mentioned in the previous section, prototype is typically calculated with all available data samples (images) within the class. Thus, compared to image instances, the prototypes contain representative representations, which are commonly shared among image instances within the class. From this perspective, we notice that the prototypes play a similar role to the text prompts adopted in visual language models (c.g.. CLIP [42]). The text prompts describe the general and representative
17

---

Table 3: Average gaps between prototype and image instance embeddingrreprssenaaiions The results are the average value of the gaps of 600 random tasks under "train on all datasets" settings.
Gaps
ImageNet
Omniglot
Aircraft
Birds
Textures
QuickDraw
Fungi
|| Aembeal
0.19
0.11
0.12
0.12
0.14
0.14
0.14
0.14
|
0.17
0.09
0.08
0.07
0.07
0.06
0.06
0.06
ACoPA
0.60
0.42
0.55
0.54
0.54
0.54
0.56
0.59
information of a category. Thus, we conjecture that the prior information respectively contained in prototypes and image instances is at different levels. To be specific, image instance representations encoded with instance-level information closely related to the contents in the given image while prototype representations are a set of class-specific representations. According to the visualizations in [32], the image and text prompt embeddings extracted from visual language models are located in completely separated areas of the feature space. In that work, the reason for such a phenomenon is attributed to the different kinds of modalities of the input data. Thus, the gap between the clusters of the extracted text and image emheddings is correspondingly called the modality gap-
As aforementioned, the prototype resembles the text prompt to some extent since both of them describe some high-level information commonly shared among images within a class. Thus, similar to the modality gap, there might also exist some type of gap between prototype and image representations. To further determine our intuition, in this section, we propose to conduct an analysis to figure out whether the gap between the prototype and image instance embeddiingsrrppeeentatiinn exists.
Analysis Settings. To examine the gap between prototypes and image instances, we follow Liang et al. [32] to measure the Euclidean distance between the prototypes and image instances embed- dings/representtiions. To be specific,
the analysis, tasks are first randomly sampled from the validation set of a specific dataset, such ImageNet [44], and the corresponding embeddings are obtained by feeding support data of the tasks into a frozen pre-trained ResNet-18 backbone. Then, the prototype for each class in the task is calculated
mean of all support data within the class following previous works [47, 49, 43, 3, 12, 34, 29, 30j. By respectively applying URL [29] and our proposed CoPA methods on the extracted embeddings, the corresponding representations are obtained. Finally, we reduce the dimension of the prototype and image representations via some dimension reduction tools, such UMAP [37], and visualize the results. The results are visualized in Figs. 1, 5(a), 7, and 19. According to the visualization results, there are several interesting phenomena:
According to Figs. I(a), 7(a), 7(d), 7(g). 7(), and the results of other datasets (Fig. 19), it is easy to observe that the prototype and image instance embeddings extracted from a frozen pre-trained backbone tend to be located in different regions of the feature space. This phenomenon consistent with what happens visual language models [32]. also demonstrates the aforementioned intuition that prototype and image instance data describe the information at different levels and should be treated diffcrently during the adaptation.
Then, according to Figs. (b). 7(b), 7(e) and 7(k), we can observe that the "modality" gap is shrunken after applying URI., which imposes an assumption that the same represen- tation transformation is applied to both prototype and image instance embeddings. This phenomenon demonstrates that sharing the same transformation between prototypes and image instances potentially damages the distribution differences between the prototypes and image instances and in turn removes the differences of semantic information between them.
Finally, according to Figs. 5(a), 7(c), 7(f) and 70), we can observe that the representations learned from our proposed CoPA method preserve the "modality" gap and the gap is further enlarged compared with the gap between the embeddings. As claimed in Liang et al. [32], the enlargement of the gap contributes to the improvements in generalization performance. According to our empirical results in Tables and 2, CoPA achieves the SOTA performance and significantly outperforms URL and URL + TSA. which is consistent with the conclusion. Thus, we can further say that it is the gap between prototype and image instance representations that facilitates CoPA to achieve better empirical performance.
In addition to these visualization results, we take a further step to validate whether this phenomenon is common among all tasks of all seen domain datasets. To this end, we collect the results of the gaps of the extracted embeddings and the representations respectively learned from URL and CoPA among 600 randomly sampled tasks under the "train on all datasets" task setting with the
18

---

17.3
oegatine
56.0
56.0
56.0
15
Mix
Max Sample
Arg 0.4
2.5 05
62
0.4
0.6
0.6
04
0.6
0.8
50
Average Accuracy Figure 8: Analysis results of three kinds of prototypes: "Max", "Max Sample" and "Average" prototypes ImageNet dataset. (a). Average accuracy of URL with three kinds of prototypes ImageNet dataset. The results are obtained by averaging 5 reproductions with different random seeds (i.e. 41-45). (b). The distribution of both positive and negative similarities with the "Max" prototype. (c). The distribution of both positive and negative similarities with the "Max Sample" prototype. (d). The distribution of both positive and negative similarities with the "Average" prototype.
(b) "Max"
"Max Sample"
"Average"
random seed 42 on the validation sets of seen domain datasets. The results are reported in Table 3. The gaps are calculated following Liang et al. [32]. Given a set of normalized support data embeedings/represenaaiiss {= !2r and set of normalized prototypes generated from the support data {- No. the gap calculated as the difference between the averaged prototypes and images:
Drl
Ne
1 Nc
Σ C
Zi
Dj
Thus, this way, with the normalized prototype and image instance embeedings/vepresenatiion, the "modality" gap is further defined as the Euclidean distance between the two sets of embeddings/repre- sentations || where
denotes the Euclidean distance.
According to the results reported in Table is easy to observe that URL consistently shrinks the gaps between prototypes and image instances over all datasets while CoPA consistently enlarges the gaps. These results indicate that URL undermining the "modality" gaps while CoPA preserving and enlarging the gaps is not occasional case. It commonly happens across all domains.
C Detailed Study on Prototype Selection
In this paper, in order to select the most appropriate prototypes for CoPA, we conduct an analysis to perform prototype selection. Generally, the prototype for each class ought contain the most general and discriminative information regarding the class. To this end, three kinds of prototypes are selected candidates. We will provide detailed descriptions and analyses of them in the following parts.
C.1 Introduction of Three Prototypes
Max Prototype. The "Max" prototype is defined as the sample that is composed of the largest features in each dimension among all support data embeddings extracted from frozen backbone. The pseudo codes are provided the following:
import torch
def max eax_ppototppe (X: torch. tensor, t:torch, tensor):
Args:
X: shape: [n_samples, n_dims], the support data embeddings;
Y: shape: [n_samples, ,J, the support data labels.
Return:
max_protos: torch. tensor, shape [n_ n_dims].
n a_way len (Y. unique ())
max_protos torch zeros (n_way, I. shape [-1]). (X. for i in range (n_way):
max_proto [i],
X[(Y == i), :] max (dim=0)
return max_protos
19

---

Max Sample Prototype. The "Max Sample" prototype is the data sample that has the highest similarities over all other samples within its class set. The pseudo codes are provided in the following:
import torch
def mxx_sampeo.rroooype (X:torch tensor, Y: torch. torchh. tensor):
Args:
I: shape: [n_samples, n_dims], the support data embeddings;
Y: shape: [n_samples,]] ,], the support data labels.
Return:
maxsample.protos: torch. tensor, shape: [n_classes, n_dims].
n_way = len (Y. un.uuuuuueuee
eaxsample_protos torch. zeros (n_ vay, vay, x. shape (-1)) type_as (X. dtype
range range (n_way):
class_samples X[(Y i), :1
num_ num_samples = class_samples shape [0]
normed_sample class_ samples class_ norm (dim=1,
keepdim-1)
similarity_mat normed_samples normed_samples t()
max_proto class class_samples samples (torch argmax (eggaax(simmaaiiin mat sum (1
) :]
return
Average Prototype. The "Average" prototype is the typical prototype adopted in previous few-shot classification works and calculated as the average vector of all support data embeddings extracted from a frozen pre-trained backbone within the given class. The pseudo codes are as the following:
import torch
import torch nn functional
def avg- avg-prototype (x: torch. tensor, Y: Y:torch, tensor): ,,,
Args:
x: shape [n_samples, n_dims], the support data embeddings;
Y: shape: [n_samples,] JJ, the support data labels.
Return:
avg-protos: torch. tensor, shape [n_classes, n_dims].
one_hot_labels F.oneeeooteee
# shape: [n_samples, n_ classes avg-protos Y.t t. (dim=1, keepdim=True)
return avg-protos
C.2 Analysis Settings & Results
Settings. The analysis regarding prototype selection is performed the support data embeddings (of tasks sampled from the validation set of seen domain datasets) extracted from the frozen pre-trained backbone. The results are obtained from 600 randomly sampled with the random seed 42. Specifically, for each sampled task, the support data are fed into the backbone for the corresponding embeddings. Then, the prototype embeddings for each class caleulated respectively. With the support data and prototype embeddings, we measure two kinds similarity. The first one is the similarity between each support data sample and the prototype that it belongs to. We call such kind of similarity "positive similarity". The other one is the similarity between each sample and the prototypes of other classes. We call such kind of similarity "negative similarity". The density of both kinds of similarity of the three prototypes (over all seen domain datasets) are visualized Fig. 8 and Figs. 12- 18.
Results. As aforementioned, the prototype should contain information that is commonly shared among samples within the class while distinctive from other classes. In other words, such a prototype has high similarity with the samples within the class while low similarity with the samples from
20

---

other classes. Thus, for prototypes that depict the representative features of the classes, the positive similarity should be as large as possible while the negative similarity should be as small as possible.
According to the results on ImageNet dataset shown in Fig. 8, the comparison of the three kinds of prototypes mainly includes two parts. On the one hand, the numerical results of the few-shot classification task are evaluated respectively with the three prototypes. On the other hand, the distributions of "positive similarity" and "negative similarity" are respectively measured.
From the perspective of numerical results, it is easy to observe that the "Average" prototype achieves the best empirical generalization performance on ImageNet and outperforms the "Max" and the "Max Sample" " prototypes. In addition, the "Average" prototype also achieves the best results in positive similarity distribution with an average positive similarity of 0.79 (see Fig. 8(d)). Interestingly, the "Max Sample" prototype, which owns the highest similarity over all other samples, obtains the lowest positive similarity, even smaller than the "Max" prototype. We conjecture the reason for this phenomenon is that the "Max Sample" prototype does not contain the common features shared in its class though owns the highest similarity among all other samples within its class. Thus, with all aspects taken into consideration, we select the "Average" prototype for our proposed CoPA.
D Detailed Task Settings
In this section, we intend to provide comprehensive and concrete details about the task settings adopted in the experiment section of our paper. Specifically, we mainly introduce the dataset (i.e. Meta-Dataset [49]), and the vary-way vary-shot classification task setting in this section.
D.1 Introduction of Meta-Dataset
Meta-Dataset is dataset composed of several datasets that are easy and free to access. It spans variety of visual concepts with different degrees of fine grain (e.g. from hand-writing to natural sce- narios). It was first proposed by Triantafillou et al. [49] for cross-domain few-shot classification tasks and has been a popular and mainstream benchmark in this field. Originally. Meta-Dataset only con- tained 10 datasets, which are ILSVRC 2012 (a.k.a. ImageNet) [44]. Omniglot [26], FGVC_ Aircraft (Aircraft) [36], CUB_ 200-201 1 (CU_ (CU_Birds) [53], Describable Textures (DTD/Textures) [9], Queik Draw [22], FGVCx Fungi (Fungi) [45], VGG Flower [39], Traffic Sign [20], MSCOCO [33]. Later, MNIST [27], CIFAR-10 [25] and CIFAR-100 [25] are further contained by Bateni et al. [3].
In practice, there are two mainstream settings widely applied in the existing works [49, 43, 10, 12, 34, 29, 30J. The first setting is the "train on all datasets" setting. In this setting, 8 datasets, including ImageNet, Omniglot, Aircraft, Birds, Textures, Quick Draw, Fungi, and VGG_ are selected as seen domains while the remaining datasets are treated as unseen domains. In contrast, the other setting, the "train ImageNet only" setting, only takes ImageNet the seen domain while all other datasets are treated as unseen domains. In this paper, we report results under both task settings. In particular, if there is no further declaration, the default task setting the "train all datasets" setting.
D.2 Brief Introduction of Vary-Way Vary-Shot Task Settings
Vary-way vary-shot classification task setting is first proposed in Meta- Mata-Dataset [49] to mimic the most common situations, where the number of classes and the number of samples in each class randomly determined, in the real world. Thus, the samples in these tasks are imbalancedly distributed. Such task setting widely adopted in many previous works [43, 3, 10, 50, 12, 34, 29, 30, 16). Different from the conventional few-shot classification task setting, where the numbers of classes and shots are invariant in each task, vary-way vary-shot task setting adopted in cross-domain few-shot classification is much more challenging due to the imbalanced data and the uncertain task structure.
Roughly, the generation process of vary-way vary-shot task mainly includes two steps: class sampling and data point sampling. To be specific, the first step is sampling classes from a specific dataset. Given dataset, the number of classes is randomly determined with some probability distribution in the interval [5, Mmxx, where Nmax denotes the maximum number of classes of the dataset. In the context of Meta-Dataset, the Nmax is usually set either 50 or many classes there are available.
After the number of classes determined, a set of classes uniformly sampled from the given dataset. Then, the number of data points for each class is assigned in a random way. Concretely, the
21

---

Table 4: Comparisons of CoPA respectively with linear transformation head and visual Transformation under both "train on all datasets" and "train on ImageNet only" settings. Mean accuracy and 95% % confidence interval reported. All results are the average of reproductions with seeds 41-45.
train on all datasets CoPA + VIT
train on ImageNet only
Datasets
CoPA + Linear
CoPA + Linear
CoPA ViT
ImageNet
57.8
57.711111
57.7
17.411.1
Omniglot
94.3:10.5
94.3:10.4
70,9:112
7.,5+1.2
Aircraft
88.810.5
88.810.5
61.611.0
59.7+1.0
Birds
80.8+0.8
80.8+0.8
74.2+0099
74.0+1.0
Textures
77.8+0.77
77.4+0.77
77.0+0.77
76.6+0.77
Quick Draw
82.8+0.66
82.8+0.6
61.3++10
58.7+1.0
Fungi
69.5.1.0
69.4111.
48.0.1..1
47.211.11
VGG Flower
92.710.5
92.610.55
88
88.310.7
Traffic Sign
66.6.11.1
68.811111
63.8
68.2.1.11
MSCOCO
56.3+1111
55.8+1.11
16.1
1.1
MNIST
95.2++04
95.2++..5
87.3+0.7
89.7+0.7
CIFAR-10
73.0
72.7
72.4+0.8
72.1
CIFAR-100
63.4+.10
63.24:1.0
62.7+110
62.3+1.0
Algorithm 1 CoPA Algorithm.
Input: pre-trained backbone for, number of inner iterations n, learning rate 7), linear transforma- tion heads hep and ha, temperature coefficient 7.
Output: the optimal parameters for linear transformation heads 6%. and 07.
# Sample task
Sample new support data set DJ = (X,
Generate pseudo labels pseuddo {0, ..., DT 1}: Performing contrastive prototype- iiaaee-imeeeeee adaptation
for i 1 to do
Obtain the prototype and instance representations:
Zp hop (YY for (X));
Z = he, (fo- (X)
Compute SCE loss L(Ar, 01) in Eq. (3);
Update parameters:
Op Op Vy L(0p, 0t);
01 + 01 Vo, C(0p, 0r);
end for
first step data point sampling is determining the number of query data. the vary-way vary-shot task setting proposed in Meta-Dataset [49], the number of query data for each class is fixed to the same number (e.g. 10). The number of query data ought to be no more than half of all available data in the given class so that at least 50% of the data can be preserved as support data. Then, based on the number of query data, the support data are sampled from the remaining data with the constraint that the total number of the entire support data (of all classes) should be no more than 500. The number of support data for each class is determined by the size of support data and a set of random scalars sampled from given interval. In this way, task with random numbers of classes and shots sampled from given dataset. More concrete details are available in the Meta-Dataset article [49].
E Detailed Implementation Settings
In this section, provide detailed implementation settings, including backbone pre- mre-training, model structures, adaptation module initialization, hardware information, and other hyperparameter settings (e.g. learning rate & weight decay), for the convenience of reproducing our proposed CoPA method.
E.1 Pre-trained Backbones
In this paper, our proposed CoPA method is built upon a ResNet-18 backbone [18] as done in previous works [12, 34, 29]. For the pre-trained backbone used in the "train ImageNet only" setting, the
22

---

backbone is pre-trained in the same way proposed in SUR [12]. Specifically, the backbone is trained with stochastic gradient descent optimizer and cosine annealing with a learning rate 3 10 The momentum is set to 0.9 and the weight decay is set to 7 x 10-1. ' In addition, the batch size is 64 and the total number of learning iterations is 480,000. All images are reshaped to the size of 84 x4.
Moreover, for the universal backbone that is used in the "train on all datasets" setting, we follow URL [29] to distill a ResNet-18 backbone from models that are respectively pre-trained on 8 single datasets. The distilled model is also pre-trained with SGD optimizer (learning rate is set to 3 x 10-2 and weight decay is set
10-4) and cosine annealing as well with 240,000 learning iterations. The details of single domain backbone pre-training are available in appendices of URL [29].
E.2 Architecture of Visual Transformer
In addition to the linear transformation head, we also evaluate our method on visual Transformer (i.e. ViT) [11] to figure out whether the proposed CoPA method benefits from complex and powerful learning modules. In our experimental setting, the visual Transformer we adopted is composed of a single self-attention block. Specifically, each visual Transformer model only contains one attention block followed by LayerNorm layer [1] and linear layer. All layers in the ViT, including query. key, value heads, and the linear transformation head, are formulated as 512 x 512 matrices.
E.3 Module Initialization
During the meta-test phase, the adaptation modules (i.e. trainable learning modules) are re-initialized for the adaptation of new tasks at the beginning of each episode. Specifically, all linear layers will be re-initialized an identity matrix with the size of 512 x 512. Besides, in the visual Transformer case, the weight and bias of the LayerNorm layer are respectively set to 1.0 and 0.0, all class tokens are re-initialized 0.0, and all position embeddings are re-initialized with a normal distribution (00, 0.02). In particular, in CoPA TSA case, the trainable learning modules plugged into the pre-trained backbone are re-initialized to all-ones matrices with a scale coefficient 4.
E.4 Optimization
In our method, the optimizer adopted during adaptation is Adam optimizer [24]. For each adaptation episode, the total number of iterations is 50 for linear transformation and 40 for visual Transformer. For linear head, the learning rate is set to 5e-3 for Traffic Sign and MNIST datasets while 1e-3 for the remaining datasets. The weight decay is set to 0.1 for all datasets except for Traffic Sign and MNIST. For the visual Transformer, the learning rate is set to le-3 for Traffic Sign and MNIST and le-4 for other datatsets. The weight decay set to 0.1 for all datasets except Traffic Sign and MNIST.
E.5 Hardware & Random Seeds
All experiments in this paper are conducted with an NVIDIA GeForce RTX 3090 GPU. To make sure that all experimental results are reproducible and the comparisons are fair, all results of our proposed CoPA method and other reproduced methods are the average of 5 reproductions with the random seeds 41, 42, 43, 44, 45. The GPU memory that is required for running our method is about 10 GB.
E.6 Task Specific Adapter Setting
Among all existing methods, as far we know, the best results on cross-domain few-shot classification tasks with respect to Meta-Dataset are reported by TSA [30]. Besides, recent work, TA2-Not [16], can be seen as a further extension of TSA by adaptively selecting learning modules plugged into the pre-trained backbone. Due to the extra trainable modules plugged into the frozen pre-trained backbone, more task-specific knowledge can be learned to achieve better generalization performance.
In this paper, although our proposed CoPA method is able to achieve better generalization performance than TSA on seen domains without having to add any extra learning modules on the rreitriine backbone, there still exist obvious performance gaps on unseen domains. We conjecture that the improvements over unseen domains significantly rely on the extra learning modules. Thus, to verify this conjecture, we followed TSA to plug some extra learning modules into the pre-trained backbone to combine our proposed CoPA with TSA strategy. However, since TSA consumes large amounts
23

---

of time compared with URL in practice (about 30s/iter), concerning efficiency, we only plugged learning modules on the second and third residual blocks of the pre-trained ResNet- 18 backbone.
In our implementation of CoPA + TSA case, we optimize the learning modules and transformation heads with Adam optimizer as well. To be specific, the learning rate of the transformation heads is set to 5e - 3 for Traffic Sign, MNIST, CIFAR-10, 10, and CIFAR-IO0 datasets while 1e 3 for the remaining datasets. For learning modules plugged into the pre-trained backbones, the learning rate is set to 2.5e - 3 for Traffic Sign, MNIST, CIFAR-10 and CIFAR- 100 datasets while le - 4 for the remaining datasets. Except for Traffic Sign and MNIST, the weight decay is set to 0.1.
E.7 Reproduction of State-of-the-art Methods
In this paper, for fair comparisons between our proposed CoPA method and current otate-of-the-art methods, such as URL, TSA, and TAt-Net, we ran both CoPA and those existing methods with the same random seeds and reported the average of 5 reproduction experiments as the final results. All results of these methods reported in our paper can be reproduced by setting the seed the bash file.
F
More Experimental Results
F.
CoPA with More Complex Modules
Similar to CLIP [42], which performs contrastive learning via two Transformers [51, 11] with text and image data pairs, the proposed CoPA method in this paper fine-tunes two different modules respectively for prototype and image instance embeddings by substituting the text prompts with prototypes. However, the modules adopted in the main results are two linear transformations. It is not clear whether applying more complex modules contributes the performance. Thus, in this section, we similarly evaluate the performance of CoPA with two visual Transformer (ViT) heads [11].
The implementation details of VIT are available in Appendix E.2. The comparison of performance between CoPA + VIT and CoPA + Linear is reported in Table 4. According to the results in the table, can observe that the performance changes between CoPA + Linear and CoPA ViT are slight under both "train on all datasets" and "train on ImageNet only" settings. Such a phenomenon reveals that the proposed CoPA method is not sensitive the complexity of model architectures.
F.2 Fewer Shot Classification Tasks.
In this section, we follow previous works [49] to evaluate our proposed CoPA method on fewer- shot classification tasks. Different from the vary-way vary-shot task setting mentioned above, the fewer-shot classification task contains fewer data samples in each task and thus is more challenging.
F.2.1 Fewer-shot Experiments
In addition to the vary-way vary-shot task setting adopted in the main results section, some other task settings, such as 5-way 1-shot and vary-way 5-shot, are also widely adopted in previous works [14, 47, 49, 29, 30]. Compared with the vary-way vary-shot task setting, 5-way 1-shot and vary-way 5-shot tasks are more challenging since fewer data samples are included the support set of each task. For example, in the vary-way 5-shot task setting, although the number of classes is still randomly determined the beginning of each episode, the number of data samples (i.e. shot) is fixed to 5. Similarly, in the 5-way 1-shot task setting, both the numbers of classes and data samples in each class are respectively fixed to 5 and 1. In this section, in order to further explore the capability of CoPA, we validate the performance of CoPA on both task settings aforementioned under the "train all datasets" setting for all datasets in Meta-Dataset, All results are reported in Tables 5 and 6.
Vary-way 5-shot. As shown in Table 5, compared with previous works, such as SimpleCNAPs [3]. SUR [12], URT [29] and URL [29], CoPA achieves the best performance on 7 out of 13 datasets and ranks 3.2 among all approaches. In particular, CoPA outperforms URL, which CoPA is based on, on out of 13 datasets, and achieves 0.2%, 0.5%, and 0.3% improvements on average respectively on seen, unseen, and all domain cases. To be specific, CoPA respectively achieves better performance on Omniglot (0.2%), Aircraft (0.4%), CU Birds (0.2%), VGG_Flowe Flower (0.5%) and Traffic Sign (2.1%) datasets. Mcanwhile, we also notice that CoPA achieves better results than TSA, where extra trainable
24

---

Table 5: Results under vary-way 5-shot task setting (under "Trained on all datasets" setting). setting)) Mean accuracy, 95% confidence interval reported.
Vary-way 5-shot
Datasets
Sim-CNAPS
SUR
URT
URL
CoPA
TSA
CoPA + TSA
ImageNet
47.2
46.71:10
48.611.0
17.811.0
47.0 47.61:10
47.1 = 1.0
Omniglot
95.1
95.8100.
96.0=0.3
95.820.3
96.01003
9.3
96.4 0.3
Aireraft
74.6:10.6
82.110.6
81.2:10.6
83.910.5
84.310..
84.6
84.6 0.5
Birds
69.6+0.77
62.8+0.9
71.2+0.7
76.1+0.7
76.3+-0.6
76.6 7.73
Textures
57,5+0.7
60.2+0.77
65.2++07
66.8+0.6
66.9++0.6
6.66
66.7 0.6
Quick Draw
70.9+0.6
79.0+0.5
79.2+0.5
78.3+0.5
78.6+0.5
0.5
78.3 0.5
Fungi
50.3.11.0
66.5110.8
66.910.9
68.7.0.9
68.5 68.8.1..9
68.5 0.9
VGG Flower
86.510..4
76.91006
82.410.5
88.510.4
89.0.1.44
1
89.3
Traffic Sign
55.2+:0.8
44.940.99
45.1
56.7+0.8
71.5 58.8
80.0 0.5
MSCOCO
49.2
48.1
52.3+0.9
51.34.0.8
50,9+0.8
52.0 0.9
52.4 0.8
MNIST
88,9+0.4
90.1
86,5
88,5#0.4
89.5+0.4
92.2 0.3
94.8 + 0.3
CIFAR-10 CIFAR-100
66.1
50.3+.100
61.4-10.7
59.6+0.77
59.6+0..7
66.3 + 0.7
65.7 0.7
53.8++0.9
46.4+10.
52.5+
55.8
55.3+0.8
62.2 + 0.8
63.1 0.7
Average Seen
69.0
71.2
73.8
75.7
75.9
75.8
75.9
Average Unseen
62.6
56.0
59.6
62.3
62.8
68.8
71.2
Average
66.5
65.4
68.3
70.6
70.9
73.1
74.1
Rank
5.6
5.9
3.9
3.2
2.8
2.2
Both the results on URL. TSA and CoPA are average random seeds (41 45).
Table 6: Results under 5-way 1-shot task settings (under "Trained on all datasets" setting). Mean accuracy, 95% confidence interval reported.
5-way 1-shot URL
Datasets
Sim-CNAPS
SUR
URT
CoPA
TSA
CoPA + TSA
ImageNet
42.6+t0.9
40.7++10
47.4+1.00
16.
46.5+
1.0
46.3
Omniglot
93.1
93.011.7
95.610.5
95.51:.5
95.510.5
9.5
95.7 0.5
Aircraft
65.81009
67.11114
77.910.9
78.6:10.
78.4110.9
7.9
78.9 0.9
Birds
67.910.99
59.21110
70.910.9
76.2.1.9
76.2:10.9
75.8 0.9
7
Textures
42.2+t0.8
42.5+0.88
49.4+009
52.0+0.9
52.2+009
0.99
52.0
Quick Draw
70.5+00.9
79.8+0.9
79.6+0.9
79.1
79.1+0099
0.9
79.0
Fungi
58.3+0.1
64.8+1.11
71.0=1..
71.4+10
71.4+1.0
71.4 1.0
VGG Flower
79.910.7
65.0.11.0
72.711.0
80.3
80.3
80.5 1 0.8
80.6 1 0.8
Traffic Sign
55.310.9
44.6.10.9
52.7100.9
57.4.1.99
58.4.10.9
$7.3 1 0.9
$7.8 1 1.0
MSCOCO
48.8++0.9
47,8+1.11
56.9-1.1
52.1
51.7=#0.9
52.9 1.0
52.1 1.0
MNIST
80.1+00.9
77.1
75.6
73.310.8
75.1 73.410..
76. 0.8
CIFAR-10 CIFAR-100
50.3+0.99
35.840.8
47.3+0.9
48.620.8
48,41008
49.2 0.8
54.5 + 0.8
53.8+t0.9
42.9+100
54.9
61.5++100
61.3++1.0
62.4 + 1.0
62.6 1.0
Average Seen
65.0
64.0
70.6
72.5
72.5
72.3
72.5
Average Unseen
57.7
49.6
57.5
58.4
58,6
59.4
60.6
Average All
62.2
58.5
65.5
67.1
67.1
67.3
67.9
Rank
5.5
5.9
4.1
3.2
3.2
3.5
2.3
Both the results on URL., TSA and CoPA are the average of random seeds (41 45).
learning modules are plugged into the frozen pre-trained backbones. Concretely, CoPA outperforms TSA respectively on ImageNet (0.6%), Textures (0.5%). Quick Draw (0.4%), and Fungi (0.3%).
Moreover, we also evaluate the performance CoPA + TSA on vary-way 5-shot tasks. According to the results in the table, CoPA + TSA achieves the best performance on 8 out of 13 datasets and ranks 2.2 among all approaches. In general, CoPA + TSA obtains the similar performance on seen domains to CoPA and TSA while significantly outperforms all other methods on unseen domains. Specifically, compared with the state-of-the-arr method TSA, CoPA + TSA achieves 8.5%, 0.4%, 2.6%, and 0.9% improvements respectively Traffic Sign, MSCOCO, MNIST, and CIFAR- 100 datasets.
25

---

95.275
79.0
shot) CoPA (1 shot)
Accuracy (%)
46.0
(%6
95.250
(%)
CoPA + TSA (1 shot)
URL (1
78.8
95.225
CoPA [1 shot)
45.8
URL (1 shot)
95.200
CoRA + TSA (2 shot)
78.6
Copn (1 shot]
45.6
shot)
95.175
78.4
20 Steps
30
10
20 Steps
30
40
Steps
(a) ImageNet
(b) Omniglot
(c) Aircraft
71.7
76.4
52.4
Accuracy 1%6)
71.6
76.3
URL (1 shot]
Accuracy
52.2
71.5
76.2
CoPA TSA (1 shot)
52.0
shot)
71.4
URL (1
76.1
CoPA shot)
CoPA TSA (1 shot)
71.3
CoPA TSA (1 shot)
CoPA (1 shot)
51.8
10
20 Steps
30
40
0
10
20
30
40
10
20
30
40
Steps
Steps
(d) Birds
(e) Textures
(f) Fungi
53.0
URL (1.hot)
Accuracy (%)
CoPA shot)
58.0
Accuracy (%)
79.8
Accuracy
52.5
79.6
CoPA TSA (1 shot)
URL (1 shot)
52.0
CoPA (1 shol)
79.4
CoPA TSA (1 shot)
0
10
20 Steps
30
40
0
10
20 Steps
30
40
0
10
20
30
40
Steps
(g) VGG Flower
(h) Traffic Sign
(i)
MSCOCO
76
62
Accuracy (%6)
48.5
Accuracy (%)
(1 shot)
CaPA (1 shat)
Accuracy
48.0
74
CoPA TSA (1 shot)
47.5
URL (1 shot]
73
CoPA TSA (1 shot)
(1 shot)
60
CoPA (1 shot)
CoPA T5A (1 shot)
47.0
40
10
20
30
40
10
20
40
Steps
Steps
Steps
(i) MNIST
(k) CIFAR-10
(I) CIFAR-100
Figure 9: Comparisons of test accuracy curves between URL, CoPA and CoPA + TSA under
5-way 1-shot task setting. The figures depict the comparison of validation accuracy between URL, CoPA and CoPA TSA under the 5-way 1-shot setting during the adaptation process. According to these figures, it is easy observe that overfitting tends to take place in all cases in URL and several cases in CoPA during the adaptation process. The main reason is that the prototype of each class under the 5-way 1-shot setting is calculated with only one data sample. Thus, such prototypes too biased to contain the comprehensive and useful information regarding the classes they belong to.
All these results demonstrate that proposed CoPA is robust and able to achieve better generalization performance compared with the existing approaches on CFC tasks even with fewer data samples.
5-way 1-shot. As to 5-way 1-shot task, it is special case in the few-shot classification task since the prototype of each class in this task setting is calculated with only one data sample. Thus, the obtained prototypes may be more biased compared with those obtained in the vary-way vary-shot and vary-way 5-shot task settings. Consequently, the 5-way 1-shot task is much more challenging.
According to the results reported in Table 6, it is easy to observe that our proposed CoPA achieves comparable results compared with URL and ranks 3.2 among all approaches. To be concrete, CoPA achieves the similar performance to URL on the seen domains while only outperforms URL on Traffic Sign dataset among the unseen domains. As for TSA, we find that plugging more extra trainable learning modules into the pre-trained backbone fails to improve the generalization performance on the seen domains. However, the performance on the unseen domains benefits from these extra learning modules. Specifically, although TSA fails to outperform our proposed CoPA on Traffic Sign
26

---

Table 7: Study of the size of support set (Under "train on all dataset" settings).
Methods
ImageNet
Omniglot
Aireraft
Birds
DTD
QuickDraw
Fungi
VGG_Flowwer
Max
498
165
497
494
498
497
494
497
Min
Avg
364.9
49.7
340
312.3
282.3
428.0
252.5
276.3
dataset, it achieves better performance on other unseen domain datasets compared with URL and CoPA. Moreover, in order to further explore our proposed CoPA method on 5-way 1-shot tasks, we further evaluate the combination of CoPA and TSA (i.e. "CoPA + TSA" the table). According to the results reported in the table, we find that CoPA + TSA achieves the best performance on average among all approaches and ranks 2.3. Compared with the original TSA, CoPA + TSA improves the generalization performance On unseen domains, CoPA + TSA performs better than TSA. For example. CoPA + TSA increases the performance on Traffic Sign, MNIST and CIFAR- 10 respectively by 0.5%, 1.0%, and 5.3%.
the seen domains and obtains comparable results to URL and CoPA.
One conjecture for the phenomenon that CoPA fails to outperform other approaches evidently in most cases is that the prototype of each class generated under the 5-way 1-shot setting fails to contain comprehensive and useful information that is representative enough to depict the general features of the class since it is calculated with only one data sample. In this case, contrastive learning is performed between two identical data samples and overfitting may take place. In order to validate our conjecture, we plot the accuracy curves of URL, CoPA, and CoPA + TSA. The visualization results are presented in Fig. 9. According to these figures, we can see that is quite hard to perform adaptation tasks with extremely few labeled data samples and overfitting tends to take place. For example, For datasets like Omniglot (Fig. 9(b)) and Aircraft (Fig. 9()),, overfitting is obvious on both URL and CoPA. Meanwhile, the application of CoPA even deteriorates the performance.
F.2.2 Study on Effect of Support Data Size
In this paper, due the consideration of preserving the gap between prototypes and image instances, we propose to adopt symmetric cross-entropy loss as the learning objective. Typically, the symmetric cross-entropy loss commonly applied in the contrastive learning framework.
An important issue in the conventional contrastive learning framework is the size of the data. As demonstrated in previous works [5], the large batch size is the key to learning useful representations for downstream tasks. Thus, we are also interested in the effect of the batch size of the data. However, due the task settings, such as the vary-way vary-shot setting, it is intractable to manipulate the batch size via task sampling directly. Thus, we can approximately study the problem by comparing the performance of CoPA respectively on vary-way vary-shot tasks and fewer-shot tasks since the latter contains much fewer data samples in the support set.
First of all, we count the number of samples in 600 tasks randomly sampled from the validation set of seen domain datasets (with the random seed 42). The results are reported in Table 7. According to the table, we can observe that the difference between the minimum and the maximum of the support set size is large. On average, in most cases, the average size of the support set is more than 200. These results indicate that the number of data samples under vary-way vary-shot settings is not too small.
Then, we compared the performance of CoPA respectively on vary-way vary-shot tasks, vary-way 5-shot tasks, and 5-way 1-shot tasks. In vary-way vary-shot setting, which contains the most samples, CoPA evidently achieves the best results. However, in vary-way 5-shot tasks, CoPA can still achieve the best results average, though it fails to outperform on some datasets. On 5-way 1-shot tasks, we can obviously observe that CoPA achieves similar results to URL. without absolute advantages.
Thus, we can see that the rule is not changed and the large batch size is still preferred in our proposed method. Thus, a potential limitation of our proposed method is that CoPA may fail achieve good generalization performance with extremely few data samples.
F.3 Results of Analysis regarding Number of Parameters
In this paper, the original CoPA aims at fast fine-tuning two linear heads respectively for prototype and image instance embeddings top of the frozen pre-trained backbones in the same way contrastive
27

---

learning applied in CLIP [42]. Compared with previous works, such as URL [29] on which CoPA is based, CoPA utilizes more model parameters. To be specific, the number of parameters contained in the trainable modules of CoPA is twice those contained in the trainable module in the URI.. Thus, in this case, concern that may be raised whether the improvements in generalization performance on CoPA result from more trainable parameters contained in the two linear transformation heads.
In order to figure out this question, we conduct an analysis to explore the effect of the number of model parameters. To be specific, we compare CoPA with variant of URL that includes the same number of trainable model parameters under the "train on all datasets" setting. In the URL, only simple linear layer with the size of 512 x 512 is trained on support data for further few-shot classification on query data. In order to increase the number of the number of parameters, we propose to substitute the original linear transformation head in the URL with a simple two-layer MLP. The MLPP is designed with the structure of Sannerr-Batch Normalization-Lien in the MLP model are with the size of 512 x 512. Then, the number of parameters is equal to CoPA.
All linear layers
The experimental results are reported in Table According to these results, we can observe that the combination of URL and MLP fails to outperform CoPA and even achieve worse results than URL where fewer parameters are contained. These results demonstrate that simply adding more parameters in the linear transformation head does not positively affect the generalization performance since the implicit assumption about the shared transformation not removed. The success of CoPA is mainly based on the mechanism of CoPA, where the prototype and image instance representations are explored respectively with different transformations by optimizing symmetric cross entropy loss.
F44 Efficiency of CoPA.
Due to the two linear transformation heads, the number of pa- rameters in CoPA is twice of that in URL. Assume that there are N data samples with the dimension of N2, the number of classes is Nc. then the size of linear head is Na x Na- For instance, in our experimental settings, N₂- = 512 and the size of the linear head is 512 x 512. Based this, list the number of parameters and the computational complexity of forward pass (including representation transformations with linear heads and inner products between samples and prototypes) in Table 8. Notice that we did not take the com- putation of feature extraction from the frozen backbone into consideration. However, shown in Fig. 10, CoPA is more efficient than URL. The main reason for this phenomenon is that CoPA calculates the prototypes the beginning of each episode and does not have to calculate prototypes repeatedly in each iteration as done in URL.
Running Time (s/itar)
1.0
CoPA
URL
0.8
0.6
0.4
0.2
nove 2012
Areratt Omougiet
Binds
suickdrg draw
lung! vo Bamic Sen macoca
mous
cifar20g
Figure 10: The comparison of run- ning time between URL and CoPA.
Table Comparison of the number of parameters, the computational complexity of URL and CoPA.
Methods
Params
Computational Complexity
URL
N3
N(Ne + NN) Na)
CoPA
2N3
(N + No + 2N3) Na)
E55
Ablation Study
In this section, we perform a series of ablation studies to evaluate the abilities of our proposed CoPA method and determine the functions of the components, such as the SCE loss, adopted in the CoPA.
Symmetry Cross Entropy Loss. In our paper, we follow CLIP [42] and propose simple yet effective method, contrastive prototype-image adaptation (CoPA), to perform model adaptation on prototype and image representation pairs via contrastive learning. The loss is the symmetry cross entropy (SCE) loss, which is widely adopted in contrastive learning [40, 56]. To validate the effect of SCE loss we conducted an ablation study on SCE loss respectively URL and CoPA methods.
To be specific, for URL,, we replace the NCC loss with SCE loss and fine-tune the shared transfor- mation head in the same way as URL. For CoPA, substitute the SCE loss with NCC loss used in
28

---

Table 9: Effect of the number of parameters (Under "Train on all dataset" seftings).
Datasets
URL
URI + MLP
CoPA
# Params
2
2
249
ImageNet
57.3+
56.1+1.1
57.8
Omniglot
94.14-0.4
94.6+-04
94.3++0.5
Aircraft
88.2110.5
86.610.5
88
Birds
80.210..7
79.410.7
80.8
Textures
76.2110.7
72.610.7
77.8-0.7
Quick Draw
82.2+0.6
81.7=0.6
02.
Fungi
68.7+1.0
66.7+1.0
69.5=110
VGG Flower
91.9+0.5
91.5=00.5
92.7==055
Traffic Sign
63.3111.2
54.51111
6..6
MSCOCO
5..2.11.0
53.1
5..3
MNIST
94.7+10.4
92.5=0.4
95.2=004
CIFAR-10 CIFAR-100
71.910.8
70.9
73.0=
62.9:1.00
60.6
63.4
Table 10: Ablation study on symmetry cross entropy loss and different transformation heads (Under the "train on all dataset" setting).
Datasets
URL
URL SCE
URL 2Heads
CoPA
CoPA + NCC
ImageNet
17.341.1
17.2
16.5
57.8:1
$2.5+1.0
Omniglot
94.11:0.4
94.11:0.4
94.1
94.3
93.8:10.5
Aircraft
88.240.5
87.610.5
88.3+0.5
88.8-0..8
85.610.5
Birds
80.240.7
80.1
80.0
80.810.88
78.2+0.7
Textures
76.2400.
75.910.7
76.8+0.7
77.8:10.7
71.110.7
Quick Draw
82.2+0.6
82.2+0.6
82.1+0.66
82.8-t0.6
80.9+0.6
Fungi
68.7+1.0
69.4+1.0
64.0+1.0
69.5++
57.9+0.8
VGG Flower
91.9+0.5
91.9++05
91.3++0.5
92.7++0.5
85.0+006
Traffic Sign
63.3111.
62.311.2
62.011.1
66.6.1..1
41.511.2
MSCOCO
54.2.11.0
55.911.0
$1.81100
56.3.11.1
37.611.11
MNIST
94.7+t0.4
94.9+1.4
94.7=0.5
95.2++0.4
93.4
CIFAR-10
71.9+0.8
72.5+0.8
71.7
73.0
65.0++07
CIFAR-100
62.9+1.0
62.8+1.0
62.7+1.0
63.4+1.1
58.4+0.9
previous frameworks 30] and evaluate the invariant algorithm. Both experiments are conducted under the "train on all datasets" setting on Meta-Dataset. The hyperparameter settings in the variant CoPA algorithm are consistent with those in the CoPA. For fairness, all the results the average of 5 experiments with different random seeds. The numerical results are reported in Table 10.
According to the table, on the one hand, URL+SCE loss achieves comparable performance on most datasets, such as ImageNet and Omniglot. An interesting phenomenon URL+SCE achieves significantly better performance on some difficult datasets, such as Fungi and MSCOCO, where URL tends to overfit the data. This observation indicates that SCE loss can alleviate the constraints some extent, but the assumption of the shared representation transformation still constrains its capability. We think such a phenomenon reasonable. typical contrastive learning, such as SimCLR [5] and B BOO [15], siamese networks are adopted to respectively perform representation encoding in case the collapse of the networks. Thus, the shared transformation may be naturally not good at performing representation learning two sets of data. Mcanwhile, the performance of CoPA + NCC drops significantly compared to CoPA. Thus, both two observations demonstrate that SCE loss facilitates achieving better generalization performance in transformation model adaptation.
Different Representation Transformations. As aforementioned, previous works [29, 30] implicitly assume that the same representation transformation is shared between prototype and image instance embeddings. However, in this paper, both empirical results and analyses indicate that such an assumption constrains the transformation models from learning desirable representations. To solve problem, we propose to simply apply different transformations respectively for prototypes and images. In this part, we perform ablation study to validate the effect of the different transformations.

---

According to the reported results in the table, although better performance is achieved on some datasets, such Textures, URL+2Heads achieves comparable even worse performance other cases. In order to further examine the reason for such phenomenon, we plot the test loss curves in Fig. According to the figures, we can observe that overfitting takes place during the adaptation phase. The overfitting phenomenon, to some extent, reflects that discriminative representations are learned via two different transformations. This phenomenon conforms to our intuition that the application of different transformations contributes to learning more discriminative representations.
Thus, with all these aspects taken into consideration, we find that the different transformations help learn more discriminative representations respectively for prototypes and image instances. Meanwhile, the SCE loss contributes improving generalization performance. The combination of these two techniques collaboratively facilitates achieving impressive empirical results on
G Proof Results
In this section, we provide detailed proof of the theoretical results in the main paper.
G.1 Proof Theorem 3.1
Proof. With Eq, (I), by replacing d(:,)) with cosine similarity, we have:
1
e²| cc
(())= =
log
n i=1
No Σ -=
e²!
NC
jcjl Exec, x/
2
Ce
log
n2
i=1
1=1
3=1
Ne
Ce
Σ
loog
p.j
E Eerce, =T= - log Nc
π1
Nc j=1
i=1
1=1
Ne
ΣΣΣ JE Nc
2 TTZ
Co
+
where is the independent copy of 2, C, denotes the set of sample representations of class j. The incquality sign is derived from Jesen's Inequality. Then, with constant that satisfies 0 I a < Ne for Vjj, we further have:
1 n
a EI Σ 2 ce +
C(0)
2
n
n
2'
G.2 Proof the Bound of the Gaps
Lemma G.I. Consider square matrix θ = [el. 02, ...., Od] E Bdddd where O' E IRd is the i-th
column of O and vector to E R.d. Then, we have:
min cos"(v, O6) 1|0 < To Isidd
max cos (2, θ²) || |0ll2 15j5d
where ||- and || ll2 respectively denote the Frobenius and L-z norm and cos(:.) denotes the cosine similarity function.
30

---

Proof. We first expand the o term:
1 e1 = || (0) 0 ".......7 00 1|
= -I[ @ v o² ..... v o
Σ (u 0t)
o 11 cos (v, 0¹).
Then, we can obtain the lower and upper bound of ||0 T
d Σ
min cos (o, 1gidd
2
|0
| <
max cos 18i
θ 1
1=1
1=1
Since
Σ
||0||3
Σ E (0 = |el we further have:
min cos 09 |012 | s T max cos (v. 04) |111 1sisd
1535d
G.2.1 Proof of Theorem 3.2
Proof. First of all, observe that [Dyl Exez and No Then, can expand the representation gap:
Ecec are d-dimension vectors.
1 DDT
Σ= sEZ
Ee eEC
|
E
(x)
Nc
Σ Sa
(ax')
)
+
Nc
Nc
Cb.
2
BEDT
b=1
w'ECs
Then, with Lemma G.I, we have the bounds of the gap between the prototype and image instance representations:
||01 Aemb
1
Σ
Σc
M 0 A Lemb 2
[T
Nc
DEDr
2
where Aumb 1 PTl
12 E før (x:)) No
1 EN
Ne
Exrec for (x')) denotes the gap between
[c,l
embeddings, m sigd cos2 (eemh, Q¹) and M = maxisied cos (Remb, O²).
G.3 Proof Theorem 5.1
Proof.
We first calculate the first term of Eq. (4). Then, we have:
exp(2, c.) log
Pr
Pr
Dr
2 Cj
+
log
exp(z] cj)
D
exp(2 (2. c;)
DDT
DT
i=
==1
Drl
Drl
Drl
T +
1
log
exp(2] Cj) + log DD
DD
|DT
'DT
1=1
1=1
j=1
Pr
1
1
DT
2:
T
DD
2
2 c, + log [TT]
1=1
1=
31

---

Since each prototype is expanded by YY", there are |Cx| same vectors for each class k where Cj denotes the set of samples of class k. Thus, we further have:
1 DD
P
exp(z, Gi)
1 cxp(2]T,) Dr
Pr
1
Pr
Ne
Cp.
log
2
2,
Ck.
[DT
DT
i=1
Similarly, for the second term, we can obtain the same results. In detail,
exp(c, zi)
Prl
1
P
Drl log
log
C 21
exp(c 2j)
DDT
Σラ= exp(c 2z))
iDT
[DT
i=1
1=1
pry
iryl
Drl
C 21 +
log
exp(c, xj) + log (D
[Dy]
|Dr
i=1
j=1
PPT
1
C 21 +
C xj + log [DT
PDT
[D
F=1
1=1 )=1
Drr
No Dr
1
Cx
C 21 +
C₁ Zj.
Dj
D
Dr
k=
Combining the two results, since the support data size is n, we then have:
LSOR
2 2 n i=1
Σ 2, C:
Nc
CMl
T 2, Ck-
n
32

---

positive Gooeth Smm negetave Co 63, m0,05)
postiive lau.00.7% S.m0
10-
6-
Density
Density
Denaity
nugative (ivs 0,03)
poultive (v 80 Sp -8,00
4
2-
0
0.6
Similarity
Similarity
Similarity
(a) "Max" Prototype
(b) "Max Sample" Prototype
(c) "Average" Prototype
Figure 11: Comparison of three prototypes ImageNet.
10
prsiitive 60 9,,00.77 07)
Rumm0.67)
m0,
(Soy dog
6-
Density
6-
Density
4-
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
1.0
0.2
0.4
0.6
1.0
Similarity
Similarity
Similarity
(a) "Max" Prototype
(b) "Max Sample" Prototype
(c) "Average" Prototype
Figure 12: Comparison of three prototypes Omniglot.
positive Usw -0.5, R0000.0))
positive (p..0.75, -2.00)
posltive (Su -0.33 (..0)
negative lo -0,8s. -0.08)
nugetive Coup -0 3t (.09)
6-
repotive lou 0,87, Suy 0.000
Density
Density
Donatty
2-
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Similarity
Similarity
Similarity
(a) "Max" Prototype
(b) "Max Sample" Prototype
(c) "Average" Prototype
Figure 13: Comparison of three prototypes on Aircraft.
positive Use. 6678 Sau 0.077 negative lnc =8007)
positiin Uso -.84 0.10)
negative 1.1
Demsity
Donstty
posiiive Gu 78,
10)
negabive (oig +0.3% 56 0010)
2
0.2
0.4
0.6
1.0
0.2
0.4
0.6
0.0
1.0
0
0.2
0.4
0.6
0.8
1.0
Simllarity
Similarity
Similarity
(a) "Max" Prototype
(b) "Max Sample" Prototype
(c) "Avcrage" Prototype
Figure 14: Comparison of three prototypes on CU Birds.
pacitive Usen 6.77, Ss. (8.77 negative li 00,00, 2u +0,07)
6- pacitiue (s 0,82, Swail 6.10)
neputive loy 0,50, =0.10
6
Density
Density
Dunsity
4-
positive 60. -0.10)
negadive (Sup 55
(6110
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Similarity
Similarity
Similarity
(a) "Max" Prototype
(b) "Max Sample" Prototype
(c) "Average" Prototype
Figure 15: Comparison of three prototypes on DTD.
33

---

pasttive
positive
pasitivo
Density
Density
0.2
Similarity
Similarity
Similarity
(a) "Max" Prototype
(b) "Max Sample" Prototype
(c) "Average" Prototype
Figure 16: Comparison of three prototypes Quick Draw.
potitive Di 0.6 0.09
pacitiun Osr (h11 Suc A11]
negative ln -0.60 Su (0.09)
repotive lou
Density
Density
Densty
Cup negative (oop P..
0.2
0.4
0.6
0.0
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6 Similarity
0.0
1.0
Similarity
Similarity
(a) "Max" Prototype
(b) "Max Sample" Prototype
(c) "Average" Prototype
Figure 17: Comparison of three prototypes on Fungi.
posttive Uo -077 Qu
pasitive (so -8.82 So. 0.11)
negative la 00.1, Cuy (8.8))
negative Loy 0.50 2. 1)
Density
Density
Denstty 2
augetive (o -0.111
0.2
0.4
0.5
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Similarity
Similarity
Similarity
(a) "Max" Prototype
(b) "Max Sample" Prototype
(c) "Average" Prototype
Figure 18: Comparison of three prototypes on VGG Flower.
34

---

0.4
Images prototypes
0.4
prototypes
0.3
0.
-0.4
images
-0.4
-A4
prototypes
-6.2
a25
0.75
(a) Omniglot Emb: ||
0.06
(b) Omniglot URL: ||
0.02
(c) Omniglot CoPA: ||||| = 1.40
Images prototypes
0.4
prototypes
prototypes
0.0
-0.4
0.4
0.4
0.5
(d) Birds Emb: ||ق||| = 0.14
(c) Birds URL: || = 0.06
(f) Birds CoPA: ||
1.66
0.4
0.0
images prototypes
prototypes
protatypes
0.9
0.8
0.7
0.6
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.6
0.5
0.0
0.5
1.0
(g) Flower Emb: |||||
0.15
(h) Flower URL: ||
0.05
Flower CoPA: |||||
1.84
Figure 19: "Modality" gaps between prototypes and images on some other datasets.
12
11
fu8
(a) Extracted embedding clusters
(b) URI. representttinn clusters
(c) CoPA representation clusters
Figure 20: Embedding and representation clusters on Omniglot dataset.
18
12
5-
3-
0-
-7.5
-5.0
-2.
0.0
2.5
5.0
7.5
-10
-5
10
15
20
-10
10
15
(a) Extracted embedding clusters
(b) URL representation clusters
(c) CoPA representation clusters
Figure 21: Embedding and representation clusters on Aircraft dataset.
35

---

16
12-
0-
-2.5
0.0
2.5
5.0
7.5
10.0
1.0
15
20
25
15
(a) Extracted embedding clusters
(b) URL representation clusters
(c) CoPA representation clusters
Figure Embedding and representation clusters on CU Birds dataset.
10
(a) Extracted embedding clusters
(b) URL representation clusters
(c) CoPA representation clusters
Figure 23: Embedding and representation clusters on DTD dataset.
15
10
(a) Extracted embedding clusters
(b) URL representation clusters
(c) CoPA representation clusters
Figure 24: Embedding and representation clusters on Quick Draw dataset.
12
10
5.0
16
(a) Extracted embedding clusters
(b) URL representation clusters
(c) CoPA representation clusters
Figure 25: Embedding and representation clusters on Fungi dataset.
12
4-
-6-
10
10
15
10
20
(a) Extracted embedding clusters
(b) URL representation clusters
(c) CoPA representation clusters
Figure 26: Embedding and representation clusters VGG Flower dataset.
36

---

94.4
URL
URL
CoPA
(%)
Accuracy
Accuracy
8B
URL
94.0
CoPA
TSA
20 Steps
30
0
10
20
30
40
20 Steps
30
Steps
(a) ImageNet
(b) Omniglot
(c) Aircraft
78
URL
Accuracy (%)
81.0
CoPA
82.6
Accuracy
76
Accuracy
82.4
URL
URL
82.2
TSA
80.0
75
CoPA
82.0
74
30
40
20 Steps
30
Steps
Steps
(d) CU_Birds
(e) DTD
(f) Quick Draw
69.5
92.5
Accuracy (%6
69.0
CoPA TSA
CoPA
(%6
80
CoPA TSA
CoPA
68.5
Accuracy
70
60
URL
67.5
TSA
91.0
50
40
20
30
Steps
Steps
Staps
(g) Fungi
(h) VGG Flower
(i) Traffic Sign
Accuracy (%)
Accuracy (50
96
77.5
CoPA
CoPA TSA
94
54
72.5
CoPA
92
CoPA
70.0
52
CoPA TSA
CoPA TSA
10
20
30
40
0
10
20
30
40
10
20
30
40
Steps
Steps
Steps
(j) MSCOCO
(k) MNIST
(l) CIFAR 10
URL
CoPA + TSA
Accuracy
65
60
0
10
20
30
40
Steps
(m) CIFAR 100
Figure 27: Generalization accuracy curves of Meta-Dataser with respect to the steps under 'Train on All Datasets' setting. As shown in figures, CoPA and CoPA + TSA evidently achieve better learning process and convergence performance compared with URL baseline.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly claimed our contributions and the research scope in the Section I.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitation of our proposed CoPA method in Section 6.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We have provided the proof's of the lemma and theorems used in this paper in Appendix G.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have contained all necessary information to reproduce the results in this paper in Appendix D and We also provide concrete algorithm details in Algorithm
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The codes are submitted as supplementary materials along with this paper.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the concrete details of the training and evaluation in the Appendix E.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the 95% confidence interval in our numerical results, done in previous works in this field.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
38

---

Justification: We have provided details of computational resources required to reproduce our method in the appendices.
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /ttp///eerrppppeeeee htc/ppblic/Ethicsuuiiiiiiieceeei
Answer: [Yes]
Justification: We conform every aspects.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the broad impact in Section 6.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does pose such risks.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: have cited the paper where the codes and datasets are created.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
39


Reviews:
[{"review_id": "A631JoAu46", "reviewer": "Reviewer_Ms7f", "summary": "This paper investigates the gap between the prototype and image instance embeddings under the setting of cross-domain few-shot classification. It shows that applying the same transformation to these embeddings will shrink their gap and constrain the exploration of optimal representation distributions. Based on a series of empirical and theoretical analyses, this paper proposes a prototype-image contrastive learning adaptation method, namely CoPA. The experimental results on Meta-Dataset indicates the effectiveness of the proposed method.", "questions": "It is a good paper, I personally like this paper and just have two small question:\n\n1.\tIn Theorem 3.1, optimizing the first term in the equation between the 206th and 207th rows requires to maximize the similarities between instances and their corresponding protypes. So, would enlarging the protype-image gap negatively affect this optimization process?\n\n2.\tAccording to the equations between the 218th and 219th rows, I can understand the differences of the first term in these two equations are removed, when the shared transformation is adopted. But why this may lead to the drop of “the discriminative information in gradients”? Could the authors explain this more clearly?", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well organized, easy to follow and free of typos. \n2.\tExtensive empirical and theoretical analyses are provided, so the proposed contrastive learning method seems to be technically sound.\n3.\tThe implementation details are clearly stated, the algorithm and source code are provided, ensuring the reproducibility of the method.\n4.\tExperiments and ablation studies are adequate, the results under different experimental settings are convincing and promising.", "weaknesses": "1.\tFor the empirical analysis in Section 3.2, there is an interesting phenomenon that “appropriately enlarging the gap between the prototypes and image instances contributes to achieving better generalization performance”. It may be better to provide some discussions of the reasons for these these observed results (Figure 3(a)) to explain the influence of “enlarging the gap” on the “generalization ability”.\n2.\tThe competing methods used in experiments appear to be somewhat outdated. The main baseline URL is published in 2021. It may be better to compare with some recently proposed adaptation-based methods. \n3.\tSome equation numbers are missing. Some references are not correctly formatted, the conference or journal names as well as page numbers are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your effort and time in reviewing our work. We appreciate a lot for your insightful and valuable comments towards our work. Now, we post our responses regarding your concerns in the following:\n> __Weakness 1__: For the empirical analysis in Section 3.2, there is an interesting phenomenon that “appropriately enlarging the gap between the prototypes and image ...”. It may be better to provide some discussions of the reasons for these these observed results (Figure 3(a)) to explain the influence of “enlarging the gap” on the “generalization ability”.\n\n__Answer__: Thanks for your interesting question. As we have mentioned in the paper, we found that the prototypes play the similar role to the text prompts used in multi-modal domains. __Specifically, a prototype is the \"common features\" shared within a class of images, and the text prompt summarizes the \"common sense\" of a class of images__. From this perspective, there should exist some kind of semantic differences between prototypes/text prompts and image instances. As shown in Fig. 1 and other visualization results in Appendix B, there do exist a gap between the prototype and image instance embeddings. Such a gap can be seen as the difference between two sets of data.  \n\nAccording to previous works, such as CLIP, to achieve good performance, the representations of images and texts should be aligned. Thus, as we argued in Section 5.2, we think __the essence of the gap enlargement is a kind of alignment between prototype and image instance representations, in which the optimal distributions of prototype and instance representations are explored__. Thus, for a pretrained model, where representations are well aligned, better generalization performance can be obtained.\n\n>__Weakness 2__: The competing methods used in experiments appear to be somewhat outdated. The main baseline URL is published in 2021. It may be better to compare with some recently proposed adaptation-based methods.\n\n__Answer__: Thanks for your suggestion. In fact, we have compared CoPA with some recent representative adaptation-based baselines in our paper. Specifically, we considered TSA (CVPR 2022) and TA$^2$-Net (ICCV 2023), which achieved better empirical performance by leveraging extra adaptation learning modules, in Table 1 & 2. According to the results, CoPA outperforms these baselines evidently. We will continue to review other works and consider adding them as references in our updated version.\n\n>__Weakness 3__: Some equation numbers are missing. Some references are not correctly formatted, the conference or journal names as well as page numbers are missing.\n\n__Answer__: Thanks for your suggestions. We will add equation numbers to all equations and modify the reference format in our updated versions.\n\n\n>__Question 1__: In Theorem 3.1, optimizing the first term in the equation between the 206th and 207th rows requires to maximize the similarities between instances and their corresponding protypes. So, would enlarging the protype-image gap negatively affect this optimization process?\n\n__Answer__: Thanks for your insightful question. We would like to discuss this question via our existing empirical results. First of all, according to Fig. 3(a), it is easy to observe that __the gap cannot be enlarged infinitely__ and __the enlargement is limited in a small range__. The validation loss tends to increase when the gap is larger than a threshold. This is reasonable since significantly enlarging the gap, as you mentioned, reduces the similarities between instances and their prototypes. Meanwhile, narrowing the gap (increasing the similarity) also results in the increase of validation loss, as excessively maximizing the similarity may lead to generalization problems (e.g., overfitting).\n\nMoreover, according to Fig. 5(c), the minimal validation loss is achieved at the learned representation gap. In this case, both enlarging and narrowing the gap increase the validation loss. Since SCE can align different sets of representations and preserve the gap between two sets of data, we argued that __the essence of the enlargement of the gap between image and prototype representations is exploring the optimal function space for representation alignment__ (see Section 5.2). Thus, we think such a slight enlargement will not negatively affect the optimization process.\n\nThis question is interesting. We will add this discussion in our updated version.\n\n\n>__Question 2__: According to the equations between the 218th and 219th rows, I can understand the differences of the first term in these two equations are removed, when the shared transformation is adopted. But why this may lead to the drop of “the discriminative information in gradients”? Could the authors explain this more clearly?\n\n__Answer__: Thanks for your question. When two transformations are applied, the gradients for image instance and prototype transformations are respectively calculated with the equations mentioned in the paper. As you can see, the two gradients are different. \n\nHowever, when applying a __single__ transformation ($\\Theta_{\\rm I}=\\Theta_{\\rm P}=\\Theta$), the gradient of $\\Theta$ is $\\nabla\\_{\\Theta}\\mathcal{L}(\\Theta)=-\\frac{2}{|\\mathcal{D}\\_{\\mathcal{T}}|}\\Theta^{\\top}f\\_{\\phi^*}(\\boldsymbol{X})^{\\top}YY^{\\top}f\\_{\\phi^*}(\\boldsymbol{X}) + \\frac{2\\alpha}{|\\mathcal{D}\\_{\\mathcal{T}}|}\\Theta^{\\top}f\\_{\\phi^*}(\\boldsymbol{X})^{\\top}f\\_{\\phi^*}(\\boldsymbol{X})$, where\n- the first term is the combination of the first terms of $\\nabla_{\\Theta_{\\rm P}}\\mathcal{L}(\\Theta_{\\rm P}, \\Theta_{\\rm I})$ and $\\nabla_{\\Theta_{\\rm I}}\\mathcal{L}(\\Theta_{\\rm P}, \\Theta_{\\rm I})$ with $\\Theta_{\\rm P}=\\Theta_{\\rm I}=\\Theta$, \n- the second term is the second term of $\\nabla_{\\Theta_{\\rm I}}\\mathcal{L}(\\Theta_{\\rm P}, \\Theta_{\\rm I})$ with $\\Theta_{\\rm I}=\\Theta$. \n\nIt is easy to find that the discriminative information of gradients is damaged since the gradient information regarding prototypes and image instances is mixed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UauaZxqfkU", "reviewer": "Reviewer_eGag", "summary": "This paper claims that there exists gap between prototypes and image instance embeddings in cross-domain few-shot classification models. \nURL[29], a representative work of cross-domain few-shot classification, proposes to fast fine-tune linear classifier on top of a frozen back bone with the nearest centroid classifier loss. \nIn URL, the embeddings of an image instance encode the instance-level information from the given image, while the embedding of a prototype contain some abstract and high-level information among image instance. \nThis paper proposes Contrastive Prototype-image Adaptation (CoPA), to adapt different transformations for prototypes and image embeddings to adjust the modality gap.", "questions": "This paper discuss the gap between prototype of class c, and image instances. However, I could not understand why such large gap exist. Since the mean vector is the average of image instances, the prototypes should be within the convex full of image instances. Thus, I feel strange that instances and prototypes are in the separate regions as shown in Fig.1 .\n\nThe number of samples per classes will influence the gap $\\Delta$. Could you justify this issue?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-  This paper show the existence of gaps between prototype and image embedding. \n- CoPA achieves the state-of-the-art performance on Meta-dataset. \n- CoPA can enlarge the gap to achieve the global minimum validation loss and learn more compact representation clusters.", "weaknesses": "The CoPA looks only improving the performance by adding complexity of models and losses. \n- CoPA have two linear heads that is more complicated than URL. This paper argues that discriminative information of gradients is removed when applying the same transformation. This claim seems to be about that when one common classifier is used, gradient does not consider the model parameters of two linear classifier (Discission of Theorem 3.1) However, it is obvious that the model that uses one linear classifier does not consider difference of two classifiers in gradients.  \n- There is no clear reason that enlarging the gap between the prototypes and instance embeddings improve performance (though is empirically reducing the validation loss.) \nCoPA is based on such a uncertain phenomenon.\n- Also CoPA have a temperature parameter $\\gamma$ in the symmetric cross entropy, meanwhile the NCC objective does not. It seems to be unfair, as the temperature influence the gap [32].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We post our responses regarding your concerns in the following:\n>__Weakness 1__: CoPA has two ... one linear classifier does not consider difference of two classifiers in gradients.\n\n__Answer__: According to your concern, you also believe that a single transformation cannot consider the difference of gradients. That is what we wanted to highlight. The shared transformation is incompetent in CFC adaptation tasks.\n\nIn fact, __the improvements of CoPA cannot be simply attributed to the complexity of models and loss__.\n\nIn this paper, \n- we first noticed that prototypes play a similar role to the text prompts in CLIP. \n- Then, we empirically demonstrated that there exists a gap between the image and prototype embeddings, which is similar to the gap found in CLIP, and slightly enlarging the gap helps improve the generalization performance. \n- According to further analyses, we found that the gap may be narrowed since applying a single transformation may damage the discriminative information in gradients. \n\nThis naturally motivated us to consider using two different transformations and substituting the text prompts with prototypes, as done in CLIP, to treat prototypes and images independently and perform representation alignment for better generalization performance. These points are highly appreciated by other reviewers.\n\nPer your concern about complexity, as we have discussed in Section 5.2 (Line 330-338), __applying decoupled transformations does increase the complexity of the hypothesis space, but it reduces the approximation error and is more flexible in representation learning (preserving the discriminative information)__. Meanwhile, we conducted several ablation studies in Appendix F3 & F5. As shown in Table 9, we perform URL with an MLP head (two 512$\\times$512 layers). However, the performance drops evidently. This indicates that __it is the flexible hypothesis space, instead of the model capacity, that facilitates the generalization performance__. Besides, as shown in Table 10, although equipped with SCE loss (same as CoPA), URL fails to achieve similar performance to CoPA. This reveals that __merely using SCE is insufficient for better generalization performance__. Thus, the success of CoPA cannot be simply attributed to the complexity of the model and loss.\n\n>__Weakness 2__: There is no clear reason that ... such a uncertain phenomenon.\n\n__Answer__: Per your concern, according to the paper, we noticed that prototypes play a similar role to the text prompts. Specifically, a prototype is the \"common features\" of a class of images, and a text prompt summarizes the \"common sense\" of a class of images. Thus, some semantic differences should exist between images and prototypes. According to empirical results, a gap does exists between prototypes and images. Such a gap describes the differences between prototypes and instances. \n\nMeanwhile, as demonstrated in CLIP, for good generalization performance, representations of images and texts should be aligned. Thus, in Section 5.2, we argued that the essence of the gap enlargement is a kind of alignment between prototype and image instance representations, in which the optimal distributions of prototype and instance representations are explored. For a pre-trained model, where representations are well aligned, better generalization performance can be obtained.\n\n>__Weakness 3__: Also CoPA have a temperature parameter ... seems to be unfair...\n\n__Answer__: Per your concern, the role of the temperature coefficient is tuning how concentrated the features are in the representation space. __In fact, in practice, the NCC objective also adopts temperature coefficient, though it is not mentioned in the paper__. \n\nIn the source code of URL (see Line 149 in `./models/losses.py` file of our submitted supplementary file), a temperature coefficient 10 is multiplied to the logits. When reproducing URL, we __adopted all original settings for fairness__.\n\n>__Question 1__: This paper discuss the gap ... Thus, I feel strange that instances and prototypes are in the separate regions as shown in Fig.1. The number of samples per classes will influence the gap $\\Delta$. Could you justify this issue?\n\n__Answer__: Per your concern about the gap, we discuss it from two different angles. Firstly, if __representations are not normalized__, the prototype is within the convex of instances and influenced by the number of samples. Specifically, when the number of samples in each class is equal, the gap is 0. \n\nHowever, according to [32], the calculation of the gap is based on __normalized embeddings/representations__ (we have highlighted this in line 174-175). Here, we take the embedding gap as an example. \n\nConsider a set of __unnormalized enbeddings__ $\\{f\\_{\\phi^*}(\\boldsymbol{x}\\_i)\\}\\_{i=1}^{|\\mathcal{D}\\_{\\mathcal{T}}|}$, the prototype for class $c$ is $\\boldsymbol{c}\\_{c}=\\frac{1}{|\\mathcal{C}\\_c|}\\sum\\_{f\\_{\\phi^*}(\\boldsymbol{x})\\in\\mathcal{C}\\_c}f\\_{\\phi^*}(\\boldsymbol{x})$, where $\\mathcal{C}\\_c=\\\\{f\\_{\\phi^*}(\\boldsymbol{x}\\_i)|y\\_i=c\\\\}$. Then, the gap vector is calculated as $\\vec{\\Delta}=\\frac{1}{|\\mathcal{D}\\_{\\mathcal{T}}|}\\sum\\_{i=1}^{\\mathcal{D}\\_{\\mathcal{T}}}\\frac{f\\_{\\phi^*}(\\boldsymbol{x}\\_i)}{||f\\_{\\phi^*}(\\boldsymbol{x}\\_i)||\\_2}-\\frac{1}{N\\_C}\\sum\\_{j=1}^{N\\_C}\\frac{\\boldsymbol{c}\\_j}{||\\boldsymbol{c}\\_j||\\_2}$ (see P7 of [32]). It is easy to see that the embeddings will be normalized when calculating the gap. In this case, __the prototype may not be in the convex of images, and the relationship between the number of samples of each class and the gap is undetermined__. \n\nTo further solve your concern, we checked the gap between prototype and instance embeddings under the 'vary-way 5-shot' setting, where the number of samples of each class is equal. It is easy to see that __the gap still exists__.\n|Methods|ImageNet|Omniglot|Aircraft|Birds|DTD|QuickDraw|Fungi|Flower|\n|---|---|---|---|---|---|---|---|---|\n|__Gap__|0.15|0.10|0.10|0.10|0.11|0.12|0.12|0.12|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p6ZTRtDg1G", "reviewer": "Reviewer_PX84", "summary": "The authors of this paper found that in cross-domain few-shot classification, there naturally exists a gap, which resembles the modality gap, between the prototype and image instance embeddings. Thus, this paper proposes a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt different transformations for prototypes and images similarly to CLIP by treating prototypes as text prompts. Extensive experiments on Meta-Dataset demonstrate that CoPA achieves state-of-the-art performance more efficiently. Meanwhile, further analyses also indicate that CoPA can learn better representation clusters, enlarge the gap, and achieve the minimum validation loss at the enlarged gap.", "questions": "See weaknesses.", "limitations": "One potential limitation of our proposed CoPA method is the symmetric cross-entropy. Symmetric cross-entropy loss is commonly applied in contrastive learning. In contrastive learning, the batch size of the data samples is an important issue for better downstream tasks. Larger batch size usually means better downstream performance. We also notice this phenomenon in this paper. Although CoPA can still achieve relatively better performance, the performance tends to degrade when the size of the data batch becomes small. Thus, this constrains the application of CoPA to some extent.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and visualized which is easy to read and understand.\n2. The paper systematically analyzes and validates the modality gap between prototype and image instance embeddings in previous methods and proposes a simple yet effective method, contrastive prototype-image adaptation (CoPA), to adapt two different transformations for prototypes and image instances as done in CLIP via substituting text prompts with prototypes.\n3. The experiments are sufficient and the results validate the theory in the previous section.\n4. The supplementary material provides complete proof and results which are good.", "weaknesses": "One concern might be the novelty of this paper: the proposed CoPA is a classical method that has been proven to be effective in many previous works. However, I think the authors did a good job using old algorithms to solve new questions. Overall, I think this is a good paper that should be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your effort and time in reviewing our work. We appreciate a lot for your positive evaluation towards our work! \n\nFor your concern about the novelty of our paper, we think that our contributions can be summarized as following:\n\n- We first noticed that prototypes play a similar role to the text prompts adopted in the multi-modal domain, and conjectured that the phenomenon that there is a gap between the prototype and image instance embeddings/representations may exist.\n\n- Then, we __empirically validated the gap between prototypes and image instances__. According to our empirical results, we found that \n    - applying a single transformation head as done in previous work may narrow the gap,\n    - the generalization performance can be improved by slightly enlarging the gap between the prototype and image instance embeddings.\n\n- To further explore the gap, we __conducted a series of theoretical analyses__. The results reveal that \n    - applying the same transformation potentially removes the discriminative information in gradients during the adaptation phase;\n    - the upper bound of the representation gap limits the change of the gap when applying a single transformation head.\n\n- Based on the observations and inspired by the previous work, we followed CLIP and proposed to use two decoupled transformations and substitute the text prompts in the SCE loss with prototypes to preserve the gap between the prototypes and image instances. Thus, we proposed an efficient and effective CFC adaptation method, CoPA. \n\n- According to our experimental results, CoPA is able to\n    - improve the generalization performance without hurting the efficiency，\n    - preserve the gap between prototype and image instance representations,\n    - explore the optimal distributions (representation space) to align the representations for better performance.\n\nA potential novelty of our proposed CoPA method is that __CoPA provides a new perspective of representation learning__. Specifically, we can consider manually designing some kinds of data that contain different levels of information from the original data. Then, by performing representation alignment based on the flexible hypothesis space, we can learn better representations and achieve better performance without extra data. In this way, we can take advantage of the existing data more sufficiently and effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received 6/4/7 ratings. The majority of the concerns come from the validity of the major claim of the paper. The authors provided a rebuttal. R1 is convinced with the explanation, while R2 still feels it questionable. R2 also mentioned that he can go either way with the paper's decision. After reading the paper, reviews, and rebuttal, the AC recommends accepting the paper. The findings and solution will benefit the community.
Decision: Accept (poster)