Title: CROSS-DOMAIN FEW-SHOT CLASSIFICATION VIA LEARNED FEATURE-WISE TRANSFORMATION

Abstract: Few-shot classification aims to recognize novel categories with only few labeled images in each class. Existing metric-based few-shot classification algorithms predict categories by comparing the feature embeddings of query images with those from a few labeled images (support examples) using a learned metric function. While promising performance has been demonstrated, these methods often fail to generalize to unseen domains due to large discrepancy of the feature distribution across domains. In this work, we address the problem of few-shot classification under domain shifts for metric-based methods. Our core idea is to use feature-wise transformation layers for augmenting the image features using affine transforms to simulate various feature distributions under different domains in the training stage. To capture variations of the feature distributions under different domains, we further apply a learning-to-learn approach to search for the hyper-parameters of the feature-wise transformation layers. We conduct extensive experiments and ablation studies under the domain generalization setting using five few-shot classification datasets: mini-ImageNet, CUB, Cars, Places, and Plantae. Experimental results demonstrate that the proposed feature-wise transformation layer is applicable to various metric-based models, and provides consistent improvements on the few-shot classification performance under domain shift.

1. INTRODUCTION:

Few-shot classification (Lake et al., 2015) aims to recognize instances from novel categories (query instances) with only few labeled examples in each class (support examples). Among various recent approaches for addressing the few-shot classification problem, metric-based meta-learning methods (Garcia & Bruna, 2018; Sung et al., 2018; Vinyals et al., 2016; Snell et al., 2017; Oreshkin et al., 2018) have received considerable attention due to their simplicity and effectiveness. In general, metric-based few-shot classification methods make the prediction based on the similarity between the query image and support examples. As illustrated in Figure 1 , metric-based approaches consist of 1) a feature encoder and 2) a metric function. Given an input task consisting of few labeled images (the support set) and unlabeled images (the query set) from novel classes, the encoder first extracts the image features. The metric function then takes the features of both the labeled and unlabeled images as input and predicts the category of the query images. Despite the success of recognizing novel classes sampled from the same domain as in the training stage (e.g., , both training and testing are on mini-ImageNet classes), Chen et al. (Chen et al., 2019a) recently raise the issue that existing metric-based approaches often do not generalize well to categories from different domains. The generalization ability to unseen domains, however, is of critical importance due to Figure 1 : Problem formulation and motivation. Metric-based meta-learning models usually consist of a feature encoder E and metric function M . We aim to improve the generalization ability of the models training from seen domains to arbitrary unseen domains. The key observation is that the distributions of the image features extracted from tasks in the unseen domains are significantly different from those in the seen domains.

the difficulty to construct large training datasets for rare classes (e.g., , recognizing rare bird species in a fine-grained classification setting). As a result, understanding and addressing the domain shift problem for few-shot classification is of great interest.

To alleviate the domain shift issue, numerous unsupervised domain adaptation techniques have been proposed (Pan & Yang, 2010; Chen et al., 2018; Tzeng et al., 2017) . These methods focus on adapting the classifier of the same category from the source to the target domain. Building upon the domain adaptation formulation, Dong and Xing (Dong & Xing, 2018) relax the constraint and transfer knowledge across domains for recognizing novel category in the one-shot setting. However, unsupervised domain adaptation approaches assume that numerous unlabeled images are available in the target domain during training. In many cases, this assumption may not be realistic. For example, the cost and efforts of collecting numerous images of rare bird species can be prohibitively high. On the other hand, domain generalization methods have been developed (Blanchard et al., 2011; Li et al., 2019) to learn classifiers that generalize well to multiple unseen domains without requiring the access to data from those domains. Yet, existing domain generalization approaches aim at recognizing instance from the same category in the training stage.

In this paper, we tackle the domain generalization problem for recognizing novel category in the few-shot classification setting. As shown in Figure 1 (c), our key observation is that the distributions of the image features extracted from the tasks in different domains are significantly different. As a result, during the training stage, the metric function may overfit to the feature distributions encoded only from the seen domains and thus fail to generalize to unseen domains. To address the issue, we propose to integrate feature-wise transformation layer to modulate the feature activations with affine transformations into the feature encoder. The use of these feature-wise transformation layers allows us to simulate various distributions of image features during the training stage, and thus improve the generalization ability of the metric function in the testing phase. Nevertheless, the hyper-parameters of the feature-wise transformation layers may require meticulous hand-tuning due to the difficulty to model the complex variation of the image feature distributions across various domains. In light of this, we develop a learning-to-learn algorithm to optimize the proposed feature-wise transformation layers. The core idea is to optimize the feature-wise transformation layers so that the model can work well on the unseen domains after training the model using the seen domains. We make the source code and datasets public available to simulate future research in this field. 1We make the following three contributions in this work:

• We propose to use feature-wise transformation layers to simulate various image feature distributions extracted from the tasks in different domains. Our feature-wise transformation layers are method-agnostic and can be applied to various metric-based few-shot classification approaches for improving their generalization to unseen domains.

• We develop a learning-to-learn method to optimize the hyper-parameters of the featurewise transformation layers. In contrast to the exhaustive parameter hand-tuning process, the proposed learning-to-learn algorithm is capable of finding the hyper-parameters for the feature-wise transformation layers to capture the variation of image feature distribution across various domains.

• We evaluate the performance of three metric-based few-shot classification models (including MatchingNet (Vinyals et al., 2016) , RelationNet (Sung et al., 2018) , and Graph Neural Networks (Garcia & Bruna, 2018) ) with extensive experiments under the domain generalization setting. We show that the proposed feature-wise transformation layers can effectively improve the generalization ability of metric-based models to unseen domains. We also demonstrate further performance improvement with our learning-to-learn scheme for learning the feature-wise transformation layers. 

2. RELATED WORK:



3. METHODOLOGY:



3.1. PRELIMINARIES:

Few-shot classification and metric-based method. The few-shot classification problem is typically characterized as N w way (number of categories) and N s shot (number of labeled examples for each category). Figure 1 shows an example of how the metric-based frameworks operate in the 3-way 3-shot few shot classification task. A metric-based algorithm generally contains a feature encoder E and a metric function M . For each iteration during the training stage, the algorithm randomly samples N w categories and constructs a task T . We denote the collection of input images as X = {x 1 , x 2 , • • • , x n } and the corresponding categorical labels as and

Y = {y 1 , y 2 , • • • , y n }. A task T consists of a support set S = {(X s , Y s )} and a query set Q = {(X q , Y q )}.

The support set S and query set Q are respectively formed by randomly selecting N s and N q samples for each of the N w categories.

The feature encoder E first extracts the features from both the support and query images. The metric function M then predicts the categories of the query images X q according to the label of support images Y s , the encoded query image E(x q ), and the encoded support images E(X s ). The process can be formulated as

EQUATION

Finally, the training objective of a metric-based framework is the classification loss of the images in the query set,

EQUATION

The main difference between various metric-based algorithms lies in the design choice for the metric function M . For instance, the MatchingNet ( Problem setting. In this work, we address the few-shot classification problem under the domain generalization setting. We denote a domain consisting of a collection of few-shot classification tasks as

T = {T 1 , T 2 , • • • , T n }. We assume N seen domains {T seen 1 , T seen 2 , • • • , T seen N

} available in the training phase. The goal is to learn a metric-based few-show classification model using the seen domains, such that the model can generalize well to an unseen domain T unseen . For example, one can train the model with the mini-ImageNet (Ravi & Larochelle, 2017) dataset as well as some public available fine-grained few-shot classification domains, e.g., CUB (Welinder et al., 2010) , and then evaluate the generalization ability of the model on an unseen plants domain. Note that our problem formulation does not access images in the unseen domain at the training stage.

3.2. FEATURE-WISE TRANSFORMATION LAYER:

Our focus in this work is to improve the generalization ability of metric-based few-shot classification models to arbitrary unseen domains. As shown in Figure 1 , due to the discrepancy between the feature distributions extracted from the task in the seen and unseen domains, the metric function M may overfit to the seen domains and fail to generalize to the unseen domains. To address the problem, we propose to integrate a feature-wise transformation to augment the intermediate feature activations with affine transformations into the feature encoder E. Intuitively, the feature encoder E integrated with the feature-wise transformation layers can produce more diverse feature distributions which improve the generalization ability of the metric function M . As shown in Figure 2 

EQUATION

We then compute the modulated activation ẑ as

EQUATION

where ẑc,h,w ∈ ẑ and z c,h,w ∈ z. In practice, we insert the feature-wise transformation layers to the feature encoder E at multiple levels.

3.3. LEARNING THE FEATURE-WISE TRANSFORMATION LAYERS:

While we can empirically determine hyper-parameters θ f = {θ γ , θ β } of the feature-wise transformation layer, it remains challenging to hand-tune a generic set of parameters which are effective ? ? ?

% " & $'( ! " # $'( ? ? ?

) *+ , - ./0 , , 2 ./0

EQUATION

, ∀9

Pseudo-seen (ps):

mini-ImageNet

Pseudo-unseen (pu) We illustrate the process in Figure 2 (b) and Algorithm 1. In each training iteration t, we sample a pseudo-seen domain T ps and a pseudo-unseen domain T pu from a set of seen domains

CUB % " & $ : ∈ < =×0×0 ? ∈ < =×0×0

Feature-wise transformation layer:

@ A (B) = {@ E , @ F } H (6) ∈ < =×I×J K H (6) (Eqn. 5) (Eqn.

{T seen 1 , T seen 2 , • • • , T seen N }.

Given a metric-based model with feature encoder E θ t e and metric function M θ t m , we first integrate the proposed layers with hyper-parameters θ t f = {θ t γ , θ t β } into the feature encoder (i.e., E θ t e ,θ t f ). We then use the loss in equation 2 to update the parameters in the metric-based model with the pseudo-seen task T ps = {(X ps s , Y ps s ), (X ps q , Y ps q )} ∈ T ps , namely

EQUATION

)

where α is the learning rate. We then measure the generalization ability of the updated metric-based model by 1) removing the feature-wise transformation layers from the model and 2) computing the classification loss of the updated model on the pseudo-unseen task T pu = {(X pu s , Y pu s ), (X pu q , Y pu q )} ∈ T pu , namely

EQUATION

Finally, as the loss L pu reflects the effectiveness of the feature-wise transformation layers, we optimize the hyper-parameters θ f by

EQUATION

)

Note that the metric-based model and feature-wise transformation layers are jointly optimized in the training stage. For the GNN approach, we integrate the official implementation for graph convolutional network into Chen's implementation. 3 In all experiments, we adopt the ResNet-10 (He et al., 2016) model as the backbone network for our feature encoder E.

We present the average results over 1, 000 trials for all the experiments. In each trial, we randomly sample N w categories (e.g., 5 classes for 5-way classification). For each category, we randomly select N s images (e.g., 1-shot or 5-shot) for the support set X s and 16 images for the query set X q . We discuss the implementation details in Appendix A.2.

Pre-trained feature encoder. Prior to the few-shot classification training stage, we first pre-train the feature encoder E by minimizing the standard cross-entropy classification loss on the 64 training categories in the mini-ImageNet dataset. This strategy can significantly improve the performance of 

4.2. FEATURE-WISE TRANSFORMATION WITH MANUAL PARAMETER TUNING:

We train the model using the mini-ImageNet dataset and evaluate the trained model with four other unseen domains: CUB, Cars, Places, and Plantae. We add the proposed feature-wise transformation layers after the last batch normalization layer of all the residual blocks in the feature encoder E during the training stage. We empirically set θ γ and θ β in all feature-wise transformation layers to be 0.3 and 0.5, respectively. Table 1 shows the metric-based model trained with the featurewise transformation layers performs favorably against the individual baselines. We attribute the improvement of generalization to the use of the proposed layers for making the feature encoder E produce more diverse feature distributions in the training stage. As a by-product, we also observe the improvement on the seen domain (i.e., mini-ImageNet) since there is still a slight discrepancy between the feature distributions extracted from the training and testing sets of the same domain. It is noteworthy that we also compare the proposed method with several recent approaches (e.g., Lee et al. ( 2019)) in Table 8 and Table 9 . With the proposed feature-wise transformation layers, the GNN (Garcia & Bruna, 2018) model performs favorably against the state-of-the-art frameworks on both the seen domain (i.e., mini-ImageNet) and unseen domains.

4.3. GENERALIZATION FROM MULTIPLE DOMAINS:

Here we validate the effectiveness of the proposed learning-to-learn algorithm for optimizing the hyper-parameters of the feature-wise transformation layers. We compare the metric-model trained with the proposed learning procedure to the model trained with the pre-determined feature-wise transformation layers. The leave-one-out setting is used to select one domain from the CUB, Cars, Places, and Plantae as the unseen domain for the evaluation. The mini-ImageNet and the remaining domains serve as the seen domains for training the model. Since we select the training iteration according to the validation performance on the mini-ImageNet domain for evaluation, we do not consider the mini-ImageNet as the unseen domain. We present the results in Table 2 . We denote FT and LFT as applying pre-determined feature-wise transformation layers and those layers optimized with the proposed learning-to-learn algorithm, respectively. The models optimized with proposed learning scheme outperforms those trained with the pre-determined feature-wise transformation layers since the optimized feature-wise transformation layers can better capture the variation of feature distributions across different domains. Table 1 and Table 2 show that the proposed feature-wise transformation layers together with the learning-to-learn algorithm effectively mitigate the domain shift problem for metric-based frameworks.

Table 2 : Few-shot classification results trained with multiple datasets. We use the leave-one-out setting to select the unseen domain and train the model as well as the feature-wise transformation layers using Algorithm 1. FT and LFT indicate applying the pre-determined and learning-to-learned feature-wise transformation, respectively. Note since the proposed learning-to-learn approach optimizes the hyper-parameters via stochastic gradient descent, it may not find the global minimum that achieves the best performance. It is certainly possible to manually find another set of hyper-parameter setting that achieves better performance. However, this requires meticulous and computationally expensive hyper-parameter tuning. Specifically, the dimension of the hyper-parameters θ γ and θ β is c i × 1 × 1 for the i-th feature-wise transformation layer in the feature encoder, where c i is the number of feature channels. As there are n feature-wise transformation layers in the feature encoder E, we need to perform the hyperparameter search in a (c

1 + c 2 • • • + c n ) × 2-dimensional space.

In practice, the dimension of the the search space is 1920.

Visualizing feature-wise transformed features. To demonstrate that the proposed feature-wise transformation layers can simulate various feature distributions extracted from the task in different domains, we show the t-SNE visualizations of the image features extracted by the feature encoder in the RelationNet (Sung et al., 2018) model in Figure 3 . The model is trained with 5-way 5-shot classification setting on the mini-ImageNet, Cars, Places, and Plantae domains (i.e., corresponding to the fifth block of the second column in Table 2 ). We observe that the distance between features extracted from different domains becomes smaller with the help of feature-wise transformation layers. Furthermore, the proposed learning-to-learn scheme can further help the feature-wise transformation layers capture the variation of feature distributions from various domains, thus close the domain gap and improve the generalization ability of metric-based models.

Visualizing feature-wise transformation layers. To better understand how the learned featurewise transformation layers operate, we show the values of the softplus(θ γ ) and softplus(θ γ ) in the feature-wise transformation layer. Figure 4 presents the visualization. The values of scaling terms softplus(θ γ ) tend to become smaller in the deeper layers, particularly for those in the last residual block. On the other hand, the depth of the layer does not seem to have an apparent impact on the distributions of the bias terms softplus(θ β ). The distributions are also different across different metric-based classification methods. These results suggest the importance of the proposed learningto-learn algorithm because there does not exist a set of optimal hyper-parameters of the feature-wise transformation layers which work well with all metric-based approaches.

5. CONCLUSIONS:

We propose a method to effectively enhance metric-based few-shot classification frameworks under domain shifts. The core idea of our method lies in using the feature-wise transformation layer to simulate various feature distributions extracted from the tasks in different domains. We develop a learning-to-learn approach for optimizing the hyper-parameters of the feature-wise transformation layers by simulating the generalization process using multiple seen domains. From extensive experiments, we demonstrate that our technique is applicable to different metric-based few-shot classification algorithms and show consistent improvement over the baselines.

Table 4 : Ablation study on pre-trained metric encoder. We conduct leave-one-out setting to select the unseen domain to study the effectiveness of pre-training the feature encoder E on the mini-ImageNet dataset. to the fourth and fifth block of the second column in Table 2 ). Our proposed learning-to-learned feature-wise transformation layers are capable of improving the generalization of metric-based models to the unseen domain under various numbers of ways in the testing stage.

Pre-determined hyper-parameters of feature-wise transformation layers. We demonstrate the difficulty to hand-tune the hyper-parameters of the proposed feature-wise transformation layers in this experiment. Different from the setting described in Section 4.2, we set the hyper-parameters θ γ and θ β in all feature-wise transformation layers to be 1. The model is trained under 5-way setting using the mini-ImageNet domain, and evaluate it on the other domains. We report the 1shot and 5-shot performance in Table 6 . We denote applying feature-wise transformation layers with {θ γ , θ β } = {0.3, 0.5} as FT and those with {θ γ , θ β } = {1, 1} as FT*. We observe that the metric-based models applied with FT perform favorably against to those applied with FT*. In several cases, applying FT* even yields inferior results compared to the original training without the feature-wise transformation layers. This suggests the difficulty of hand-tuning the hyper-parameters and the importance of the proposed learning-to-learn scheme for optimizing the hyper-parameters of the feature-wise transformation layers.

Hyper-parameter initialization for learning-to-learn. For all the experiments, we initialize the parameters θ γ and θ β to 0.3 and 0.5, which we empirically determine in Section 4.2, to train the feature-wise transformation layers. In practice, we find that the cross-domain performance is not sensitive as long as the initialized values are within the same order (e.g., 0.1 and 0.3). Here we report the results of training the RelationNet with the initialization {θ γ , θ β } = {0.1, 0.3}. In this experiment, we use the CUB dataset as the unseen domain for evaluation and conduct the training Table 7 : Few-shot classification results by applying the learning-to-learn approach trained with a single seen domain. We attempt to conduct the proposed learning-to-learn trainin with as singe seen domain, denoted as LFT*. We train the model using the mini-ImageNet dataset and report the 5-way 5-shot classification accuracy. described in Algorithm 1. The 5-way 5-shot classification accuracy on the CUB dataset is 64.79 ± 0.55%, which is similar to the one we report in Table 2 ( i.e., 64.99 ± 0.54%).

Learning-to-learn using a single domain. The proposed learning-to-learn method requires multiple domains for training. Here we apply the learning-to-learn method based on one single domain.

More specifically, we randomly sample two different tasks from the mini-ImageNet dataset in each iteration of the training process described in Algorithm 1. One task serves as the pseudo-seen task, while the other one serves as the pseudo-unseen task. We train the RelationNet model with the above-mentioned setting on 5-way 5-shot classification using the mini-ImageNet dataset only. As shown in Table 7 , the performance improvement of the RelationNet model is not significant compared to the model trained with pre-determined hyper-parameters {θ γ , θ β } = {0.3, 0.5}. This suggests that utilizing one single domain for learning the feature-wise transformation is not as effective as that using multiple domains (demonstrated in Table 2 ). The reason is that during the training phase, the discrepancy of the feature distributions extracted from the psudo-seen and pseudo-unseen tasks is not as significant since these two tasks are sampled from the same domain. As a result, the hyper-parameters {θ γ , θ β } cannot be effectively optimized to capture the variation of feature distributions sampled from various domains.

Comparison with the state-of-the-art few-shot classification on the mini-ImageNet. We compare the metric-based frameworks applied with the proposed feature-wise transformation layers to the state-of-the-art few-shot classification methods in Table 8 . In this experiment, we train the model with the pre-determined hyper-parameters of feature-wise transformation layers on the training set of the mini-ImageNet (Ravi & Larochelle, 2017) dataset. Note that we do not use the learned version of the feature-wise transformation layers in the training to ensure fair comparison. Combining Table 4 and Table 8 , we observe that the metric-based frameworks train with 1) pre-trained feature encoder, and 2) feature-wise transformation layers with carefully hand-tuned hyper-parameters can demonstrate competitive performance.

:

https://github.com/hytseng0509/CrossDomainFewShot

https://github.com/wyharveychen/CloserLookFewShot

https://github.com/vgsatorras/few-shot-gnn

https://github.com/wyharveychen/CloserLookFewShot

https://github.com/hytseng0509/CrossDomainFewShot

https://github.com/kjunelee/MetaOptNet

Figures and Tables:

(b), we insert the feature-wise transformation layer after the batch normalization layer in the feature encoder E. The hyper-parameters θ γ ∈ R C×1×1 and θ β ∈ R C×1×1 indicate the standard deviations of the Gaussian distributions for sampling the affine transformation parameters. Given an intermediate feature activation map z in the feature encoder with the dimension of C × H × W , we first sample the scaling term γ and bias term β from Gaussian distributions,

Figure 2: Method overview. (a) We propose a feature-wise transformation layer to modulate intermediate feature activation z in the feature encoder E with the scaling and bias terms sampled from the Gaussian distributions parameterized by the hyper-parameters θ γ and θ β . During the training phase, we insert a collection of feature-wise transformation layers into the feature encoder to simulate feature distributions extracted from the tasks in various domains. (b) We design a learningto-learn algorithm to optimize the hyper-parameters θ γ and θ β of feature-wise transformation layers by maximizing the performance of the applied metric-based model on the pseudo-unseen domain (bottom) after it is optimized on the pseudo-seen domain (top).

Learning-to-Learn Feature-Wise Transformation. Require: Seen domains {T seen • • • , T seen n }, learning rate α Randomly initialize θ e , θ m and θ f while training do Randomly sample non-overlapping pseudo-seen T ps and psuedo-unseen T pu domains from the seen domains Sample a pesudo-seen task T ps ∈ T ps and a pseudo-unseen task T pu ∈ T pu // Update metric-based model with pseudo-seen task: Obtain θ t+1 e , θ t+1 m using equation 5 // Update feature-wise transformation layers with pseudo-unseen task: We validate the efficacy of the proposed feature-wise transformation layer with three existing metricbased algorithms (Vinyals et al., 2016; Sung et al., 2018; Garcia & Bruna, 2018) under two experimental settings.First, we empirically determine the hyper-parameters θ f = {θ γ , θ β } of the featurewise transformation layers and analyze the impact of the feature-wise transformation layers. We train the few-shot classification model on the mini-ImageNet (Bousmalis et al., 2017) domain and evaluate the trained model on four different domains: CUB (Welinder et al., 2010), Cars (Krause et al., 2013), Places (Zhou et al., 2017), and Plantae (Van Horn et al., 2018). Second, we demonstrate the importance of the proposed learning-to-learn scheme for optimizing the hyper-parameters of feature-wise transformation layers. We adopt the leave-one-out setting by selecting an unseen domain from CUB, Cars, Places, and Plantae domains. The mini-ImageNet (Bousmalis et al., 2017) and the remaining domains then serve as the seen domains for training both the metric-based model and feature-wise transformation layers using Algorithm 1. After the training, we evaluate the trained model on the selected unseen domain. Datasets. We conduct experiments using five datasets: mini-ImageNet (Ravi & Larochelle, 2017), CUB (Welinder et al., 2010), Cars (Krause et al., 2013), Places (Zhou et al., 2017), and Plantae (Van Horn et al., 2018). Since the mini-ImageNet dataset serves as the seen domain for all experiments, we select the training iterations with the best accuracy on the validation set of the mini-ImageNet dataset for evaluation. More details of dataset processing are presented in Appendix A.Implementation details. We apply the feature-wise transformation layers to three metric-based frameworks: MatchingNet (Vinyals et al., 2016), RelationNet (Sung et al., 2018), and GNN (Garcia & Bruna, 2018). We use the public implementation from Chen et al. (Chen et al., 2019a) to train both the MatchingNet and RelationNet model. 2

Figure 3: T-SNE visualization of the image features extracted from tasks in different domains. We show the t-SNE visualization of the features extracted by the (a) original feature encoder E, (b) feature encoder with pre-determined feature-wise transformation layers, and (c) feature encoder with learning-to-learned feature-wise transformation.

Figure 4: Visualization of the feature-wise transformation layers. We show the quartile visualization of the activations softplus(θ γ ) and softplus(θ β ) from each feature-wise transformation layer that are optimized by the proposed learning-to-learn algorithm.

the proposed feature-wise transformation layers. The application context, however, differs from prior work as we focus on recognizing novel category from unseen domains in few-shot classification. The goal of this work is to make few-shot classification algorithms robust to domain shifts.

Table content: <table><tr><td>Domain generalization. In contrast to the domain adaptation frameworks, domain generaliza-</td></tr><tr><td>tion (Blanchard et al., 2011) methods aim at generalizing from a set of seen domains to the un-</td></tr><tr><td>seen domain without accessing instances from the unseen domain during the training stage. Before</td></tr><tr><td>the emerging of learning-to-learn (i.e., meta-learning) (Ravi &amp; Larochelle, 2017; Finn et al., 2017)</td></tr><tr><td>approaches, several methods have been proposed for tackling the domain generalization problem.</td></tr><tr><td>Examples include extracting domain-invariant features from various seen domains (Blanchard et al.,</td></tr><tr><td>2011; Li et al., 2018b; Muandet et al., 2013), improving the classifiers by fusing classifiers learned</td></tr><tr><td>from seen domains (Niu et al., 2015a;b), and decomposing the classifiers into domain-specific and</td></tr><tr><td>domain-invariant components (Khosla et al., 2012; Li et al., 2017a). Another stream of work learns</td></tr><tr><td>to augment the input data with adversarial learning (Shankar et al., 2018; Volpi et al., 2018). Most</td></tr><tr><td>recently, a number of methods apply the learning-to-learn strategy to simulate the generalization</td></tr><tr><td>process in the training stage (Balaji et al., 2018; Li et al., 2018a; 2019). Our method adopts a similar</td></tr><tr><td>approach to train</td></tr><tr><td>Among these three classes, metric-based methods have attracted considerable attention due to their</td></tr><tr><td>simplicity and effectiveness. Metric-based few-shot classification approaches consist of 1) a feature</td></tr><tr><td>encoder to extract features from both the labeled and unlabeled images and 2) a metric function</td></tr><tr><td>that takes image features as input and predict the category of unlabeled images. For example,</td></tr><tr><td>MatchingNet (Vinyals et al., 2016) applies cosine similarity along with a recurrent network, Pro-</td></tr><tr><td>toNet (Snell et al., 2017) utilizes euclidean distance, RelationNet (Sung et al., 2018) uses CNN</td></tr><tr><td>modules, GNN (Garcia &amp; Bruna, 2018) employs graph convolution modules as the metric func-</td></tr><tr><td>tions. However, these metric functions may fail to generalize to unseen domains since the distribu-</td></tr><tr><td>tions of the image features extracted from the task in various domains can be drastically different.</td></tr><tr><td>Chen et al. (Chen et al., 2019a) recently show that the performance of existing few-shot classification</td></tr><tr><td>methods degrades significantly under domain shifts. Our work focuses on improving the general-</td></tr><tr><td>ization ability of metric-based few-shot classification models to unseen domains. Very recently,</td></tr><tr><td>Triantafillou et al. (Triantafillou et al., 2020) also target on the cross-domain few-shot classification</td></tr><tr><td>problem. We encourage the readers to review for a more complete picture.</td></tr><tr><td>Domain adaptation. Domain adaptation methods (Pan &amp; Yang, 2010) aim to reduce the domain</td></tr><tr><td>shift between the source and target domains. Since the emergence of domain adversarial neural net-</td></tr><tr><td>works (DANN) (Ganin et al., 2016), numerous frameworks have been proposed to apply adversarial</td></tr><tr><td>training to align the source and target distributions on the feature-level (Tzeng et al., 2017; Chen</td></tr><tr><td>et al., 2018; Hsu et al., 2020) or on the pixel-level (Tsai et al., 2018; Hoffman et al., 2018; Bous-</td></tr><tr><td>malis et al., 2017; Chen et al., 2019b; Lee et al., 2018). Most domain frameworks, however, target at</td></tr><tr><td>adapting knowledge of the same category learned from the source to target domain and thus are less</td></tr><tr><td>effective to handle novel category as in the few-shot classification scenarios. One exception is the</td></tr><tr><td>work by Dong and Xing (Dong &amp; Xing, 2018) that address the domain shift issue in the one-shot</td></tr><tr><td>learning setting. Nevertheless, these domain adaptation methods require access to the unlabeled</td></tr><tr><td>images in the target domain during the training. Such an assumption may not be feasible in many</td></tr><tr><td>applications due to the difficulty of collecting abundant examples of rare categories (e.g., rare bird</td></tr><tr><td>species).</td></tr></table>

Few-shot classification results trained with the mini-ImageNet dataset. We train the model on the mini-ImageNet domain and evaluate the trained model on another domain. FT indicates that we apply the feature-wise transformation layers with empirically determined hyperparameters to train the model.

Table content: <table><tr><td>5-way 1-Shot</td><td>FT</td><td>mini-ImageNet</td><td>CUB</td><td>Cars</td><td>Places</td><td>Plantae</td></tr><tr><td>MatchingNet</td><td>-</td><td>59.10 ± 0.64%</td><td>35.89 ± 0.51%</td><td>30.77 ± 0.47%</td><td>49.86 ± 0.79%</td><td>32.70 ± 0.60%</td></tr><tr><td/><td/><td>58.76 ± 0.61%</td><td>36.61 ± 0.53%</td><td>29.82 ± 0.44%</td><td>51.07 ± 0.68%</td><td>34.48 ± 0.50%</td></tr><tr><td>RelationNet</td><td>-</td><td>57.80 ± 0.88%</td><td>42.44 ± 0.77%</td><td>29.11 ± 0.60%</td><td>48.64 ± 0.85%</td><td>33.17 ± 0.64%</td></tr><tr><td/><td/><td>58.64 ± 0.85%</td><td>44.07 ± 0.77%</td><td>28.63 ± 0.59%</td><td>50.68 ± 0.87%</td><td>33.14 ± 0.62%</td></tr><tr><td>GNN</td><td>-</td><td>60.77 ± 0.75%</td><td>45.69 ± 0.68%</td><td>31.79 ± 0.51%</td><td>53.10 ± 0.80%</td><td>35.60 ± 0.56%</td></tr><tr><td/><td/><td>66.32 ± 0.80%</td><td>47.47 ± 0.75%</td><td>31.61 ± 0.53%</td><td>55.77 ± 0.79%</td><td>35.95 ± 0.58%</td></tr><tr><td>5-way 5-Shot</td><td>FT</td><td>mini-ImageNet</td><td>CUB</td><td>Cars</td><td>Places</td><td>Plantae</td></tr><tr><td>MatchingNet</td><td>-</td><td>70.96 ± 0.65%</td><td>51.37 ± 0.77%</td><td>38.99 ± 0.64%</td><td>63.16 ± 0.77%</td><td>46.53 ± 0.68%</td></tr><tr><td/><td/><td>72.53 ± 0.69%</td><td>55.23 ± 0.83%</td><td>41.24 ± 0.65%</td><td>64.55 ± 0.75%</td><td>41.69 ± 0.63%</td></tr><tr><td>RelationNet</td><td>-</td><td>71.00 ± 0.69%</td><td>57.77 ± 0.69%</td><td>37.33 ± 0.68%</td><td>63.32 ± 0.76%</td><td>44.00 ± 0.60%</td></tr><tr><td/><td/><td>73.78 ± 0.64%</td><td>59.46 ± 0.71%</td><td>39.91 ± 0.69%</td><td>66.28 ± 0.72%</td><td>45.08 ± 0.59%</td></tr><tr><td>GNN</td><td>-</td><td>80.87 ± 0.56%</td><td>62.25 ± 0.65%</td><td>44.28 ± 0.63%</td><td>70.84 ± 0.65%</td><td>52.53 ± 0.59%</td></tr><tr><td/><td/><td>81.98 ± 0.55%</td><td>66.98 ± 0.68%</td><td>44.90 ± 0.64%</td><td>73.94 ± 0.67%</td><td>53.85 ± 0.62%</td></tr></table>

LFT 61.41 ± 0.57% 43.08 ± 0.55% 64.99 ± 0.59% 48.32 ± 0.57% LFT 64.99 ± 0.54% 43.44 ± 0.59% 67.35 ± 0.54% 50.39 ± 0.52%

Table content: <table><tr><td>5-way 1-Shot</td><td/><td>CUB</td><td>Cars</td><td>Places</td><td>Plantae</td></tr><tr><td colspan="2">MatchingNet -</td><td>37.90 ± 0.55%</td><td>28.96 ± 0.45%</td><td>49.01 ± 0.65%</td><td>33.21 ± 0.51%</td></tr><tr><td/><td>FT</td><td>41.74 ± 0.59%</td><td>28.30 ± 0.44%</td><td>48.77 ± 0.65%</td><td>32.15 ± 0.50%</td></tr><tr><td/><td colspan="5">LFT 43.29 ± 0.59% 30.62 ± 0.48% 52.51 ± 0.67% 35.12 ± 0.54%</td></tr><tr><td>RelationNet</td><td>-</td><td>44.33 ± 0.59%</td><td>29.53 ± 0.45%</td><td>47.76 ± 0.63%</td><td>33.76 ± 0.52%</td></tr><tr><td/><td>FT</td><td>44.67 ± 0.58%</td><td>30.38 ± 0.47%</td><td>48.40 ± 0.64%</td><td>35.40 ± 0.53%</td></tr><tr><td/><td colspan="4">LFT 48.38 ± 0.63% 32.21 ± 0.51% 50.74 ± 0.66%</td><td>35.00 ± 0.52%</td></tr><tr><td>GNN</td><td>-</td><td>49.46 ± 0.73%</td><td>32.95 ± 0.56%</td><td>51.39 ± 0.80%</td><td>37.15 ± 0.60%</td></tr><tr><td/><td>FT</td><td>48.24 ± 0.75%</td><td>33.26 ± 0.56%</td><td>54.81 ± 0.81%</td><td>37.54 ± 0.62%</td></tr><tr><td/><td colspan="5">LFT 51.51 ± 0.80% 34.12 ± 0.63% 56.31 ± 0.80% 42.09 ± 0.68%</td></tr><tr><td>5-way 5-Shot</td><td/><td>CUB</td><td>Cars</td><td>Places</td><td>Plantae</td></tr><tr><td colspan="2">MatchingNet -</td><td>51.92 ± 0.80%</td><td>39.87 ± 0.51%</td><td>61.82 ± 0.57%</td><td>47.29 ± 0.51%</td></tr><tr><td/><td>FT</td><td>56.29 ± 0.80%</td><td>39.58 ± 0.54%</td><td>62.32 ± 0.58%</td><td>46.48 ± 0.52%</td></tr><tr><td>RelationNet</td><td>-</td><td>62.13 ± 0.74%</td><td>40.64 ± 0.54%</td><td>64.34 ± 0.57%</td><td>46.29 ± 0.56%</td></tr><tr><td/><td>FT</td><td>63.64 ± 0.77%</td><td>42.24 ± 0.57%</td><td>65.42 ± 0.58%</td><td>47.81 ± 0.51%</td></tr><tr><td/><td/><td/><td/><td/><td>Seen domains</td></tr><tr><td/><td/><td/><td/><td/><td>mini-ImageNet</td></tr><tr><td/><td/><td/><td/><td/><td>Cars</td></tr><tr><td/><td/><td/><td/><td/><td>Places</td></tr><tr><td/><td/><td/><td/><td/><td>Plantae</td></tr></table>

Few-shot classification results under various numbers of ways in testing stage. We compare the 5-shot performance under various number of ways in the testing phase. The CUB dataset is select as the testing (unseen) domain. All the models are trained with 5-way 5-shot setting.

Table content: <table><tr><td>5-Shot</td><td/><td>CUB 2-way</td><td>CUB 5-way</td><td>CUB 10-way</td><td>CUB 20-way</td></tr><tr><td colspan="2">MatchingNet -</td><td>78.46 ± 0.78%</td><td>51.92 ± 0.80%</td><td>38.22 ± 0.38%</td><td>26.17 ± 0.24%</td></tr><tr><td/><td>FT</td><td>80.74 ± 0.77%</td><td>56.29 ± 0.80%</td><td>41.09 ± 0.39%</td><td>29.19 ± 0.24%</td></tr><tr><td/><td colspan="5">LFT 83.88 ± 0.72% 61.41 ± 0.57% 45.69 ± 0.39% 32.81 ± 0.23%</td></tr><tr><td>RelationNet</td><td>-</td><td>84.25 ± 0.72%</td><td>62.13 ± 0.74%</td><td>47.15 ± 0.40%</td><td>34.52 ± 0.24%</td></tr><tr><td/><td>FT</td><td>85.48 ± 0.69%</td><td>63.64 ± 0.77%</td><td>48.35 ± 0.38%</td><td>35.30 ± 0.24%</td></tr><tr><td/><td colspan="5">LFT 85.44 ± 0.72% 64.99 ± 0.54% 49.90 ± 0.40% 37.20 ± 0.25%</td></tr></table>

Few-shot classification results by applying different pre-determined hyper-parameters of feature-wise transformation layers. We train the model on the mini-ImageNet with a different set of pre-determined hyper-parameters of feature-wise transformation layers. FT and FT* indicate that we apply the feature-wise transformation layers with hyper-parameters {θ γ , θ β } to be {0.3, 0.5} and {1, 1}, respectively.

Table content: <table><tr><td>1-Shot</td><td/><td>mini-ImageNet</td><td>CUB</td><td>Cars</td><td>Places</td><td>Plantae</td></tr><tr><td>MatchingNet</td><td>FT</td><td>58.76 ± 0.61%</td><td>36.61 ± 0.53%</td><td>29.82 ± 0.44%</td><td>51.07 ± 0.68%</td><td>33.48 ± 0.50%</td></tr><tr><td/><td>FT*</td><td>51.66 ± 0.64%</td><td>31.74 ± 0.51%</td><td>27.08 ± 0.41%</td><td>45.04 ± 0.64%</td><td>28.73 ± 0.42%</td></tr><tr><td>RelationNet</td><td>FT</td><td>58.64 ± 0.85%</td><td>44.07 ± 0.77%</td><td>28.63 ± 0.59%</td><td>50.68 ± 0.87%</td><td>33.14 ± 0.62%</td></tr><tr><td/><td>FT*</td><td>57.45 ± 0.66%</td><td>40.20 ± 0.53%</td><td>29.15 ± 0.45%</td><td>49.40 ± 0.64%</td><td>33.21 ± 0.47%</td></tr><tr><td>GNN</td><td>FT</td><td>66.32 ± 0.80%</td><td>47.47 ± 0.75%</td><td>31.61 ± 0.53%</td><td>55.77 ± 0.79%</td><td>35.95 ± 0.58%</td></tr><tr><td/><td>FT*</td><td>62.63 ± 0.76%</td><td>44.61 ± 0.66%</td><td>31.56 ± 0.52%</td><td>53.39 ± 0.74%</td><td>36.73 ± 0.57%</td></tr><tr><td>5-Shot</td><td/><td>mini-ImageNet</td><td>CUB</td><td>Cars</td><td>Places</td><td>Plantae</td></tr><tr><td>MatchingNet</td><td>FT</td><td>72.53 ± 0.69%</td><td>55.23 ± 0.83%</td><td>41.24 ± 0.65%</td><td>64.55 ± 0.75%</td><td>41.69 ± 0.63%</td></tr><tr><td/><td>FT*</td><td>64.93 ± 0.60%</td><td>42.83 ± 0.61%</td><td>32.19 ± 0.48%</td><td>59.47 ± 0.63%</td><td>39.61 ± 0.49%</td></tr><tr><td>RelationNet</td><td>FT</td><td>73.78 ± 0.64%</td><td>59.46 ± 0.71%</td><td>39.91 ± 0.69%</td><td>66.28 ± 0.72%</td><td>45.08 ± 0.59%</td></tr><tr><td/><td>FT*</td><td>72.79 ± 0.64%</td><td>59.18 ± 0.57%</td><td>40.54 ± 0.54%</td><td>65.73 ± 0.52%</td><td>43.64 ± 0.49%</td></tr><tr><td>GNN</td><td>FT</td><td>81.98 ± 0.55%</td><td>66.98 ± 0.68%</td><td>44.90 ± 0.64%</td><td>73.94 ± 0.67%</td><td>53.85 ± 0.62%</td></tr><tr><td/><td>FT*</td><td>82.40 ± 0.58%</td><td>66.33 ± 0.73%</td><td>47.63 ± 0.64%</td><td>75.48 ± 0.65%</td><td>51.92 ± 0.59%</td></tr></table>

Peer review (iclr2020):
This paper proposes a feature-wise transformation layer to augment the image features, which is a new regularization of neural networks and leads to better generalization ability of the features. And the proposed method performs well in the few-shot classification problem. Furthermore, the paper develops a learning-to-learn model in the cross-domain setting to choose optimal hyper-parameters of the feature-wise transformation layers. Which leads to consistent improvement in the cross-domain leave-one-out setting. Although I vote weak accept for this paper, I still have several concerns: * In equation , how to calculate the gradient of w.r.t in the condition that the is calculated after removing the feature-wise transformation layers from the model? In my understanding, The is not related to after the removal of the feature-wise transformation layers. * In Section 4.2, why not choose the Prototypical networks? According to the results in (Chen et al., 2019a), it performs much better in the mini-ImageNet to CUB setting. * In Section 4.3, what*s the initial value of and ? Is the initial value sensitive to the performance? * In Section 4.2, actually, you can learn the and automatically even in a single domain. For example, use a different batch to update and . What*s the performance under this setting? * There is no direct comparison to the state-of-the-art methods. Overall, the paper is well written and the figures are well illustrated. The experiments show the effectiveness of the proposed feature-wise transformation layers and the learning-to-learning approach. But the above concerns should be addressed.