Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a multi-task paradigm for deep transfer learning via cross-domain mixup. Specifically, authors develop a strategy to transfer the knowledge from source to target tasks more efficiently by selecting the auxiliary samples from the source
dataset and augments training samples via the simple mixup strategy. The paper is well written and easy to follow. Experiments on six datasets show the efficacy of the proposed method over existing alternatives.

Overall, I vote for rejecting the paper due to limited novelty and lack of convincing experiments. The proposed approach simply combines different existing well-known techniques such as sample selection and mixup without any significant changes. Mixup has been widely used as an augmentation technique to improve the performance of a model. Similarly, as mentioned by the authors, source sample selection is also not new in transfer learning and has shown to be effective in Seq-Train. The only difference with Seq-Train is that the proposed approach uses a multi-task learning set up instead of a two phase approach for transfer learning. Low-cost mixup has also been proposed in Zhang et. al, 2018. Thus, I fail to understand the major contributions of the paper beside combination of the two simple strategies for transfer learning.

Authors mention about reviewing exiting solutions as a contribution in the paper. However, apart from a couple of paragraphs, there is no analysis on the catastrophic forgetting, negative transfer and efficiency. How does the proposed method is superior in terms of catastrophic forgetting and negative transfer compared to prior multi-task learning approaches? How prior methods are affected by catastrophic forgetting is not clear? More experiments and analysis is essential to consider this as a major contribution of the paper. 

Mixup across different domain samples has been used. in domain adaptation. E.g., Adversarial Domain Adaptation with Domain Mixup, Dual Mixup Regularized Learning for Adversarial Domain Adaptation. Besides the difference in application, how is XmixUp different from these cross-domain mixup strategies? Please explain on this.

Authors compare few existing method in experiments. However, there are few very related recent methods that should be compared in the experiments to verify the effectiveness of the proposed approach. E.g., Why comparison with Delta: Deep learning transfer using feature map with attention for convolutional networks, ICLR 2019 is missing in the paper? Although DELTA is based on regularization, it is one of the recent transfer learning approach that has much better results compared to L2-SP. Comparison with recent baselines are essential to show the advantage of the proposed approach in transfer learning.

How is the proposed method different from Learning to Transfer Learn: Reinforcement Learning-Based Selection for Adaptive Transfer Learning, ECCV 2020? L2TL reweighs source samples in a multi-task learning set up which is very close to the current approach. Authors should compare and discuss L2TL in the paper.

How is the proposed method comparable to Pay Attention to Features, Transfer Learn Faster CNNs, ICLR 2020? AFDS is much better than L2SP and should be compared in the experiments to verify the effectiveness of the proposed transfer learning approach.

I would like the authors to do more works related to experiments to improve the quality of the paper. I think the paper needs significant changes including new experiments and discussions before being accepted to any major conference.
review_title: Recommend to Reject due to Limited Novelty and Missing Comparisons

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

This paper proposes a method to address the transfer learning in the multi-task learning setup. The proposed method has two main components: (1) finding a one-to-one mapping from source dataset classes to the target dataset classes (2) applying mixup between the samples from the source and target class pairs.

Reasons for score:
Overall, I vote for rejecting the paper. 

1.The main idea of this paper is to compute the one-to-one pairing of classes in source and target domain ( i.e. for a given class in target dataset, find the closest class in the source dataset) I think this idea of having one-to-one mapping is flawed in principle. For example, let us consider that  the source dataset consists of 100 classes which represent different dog breeds and the target dataset has 10 classes representing different cars. In this scenario, one to one pairing between a dog class and the car class has no benefit for transfer learning. The authors should clearly state the underlying assumptions about the source dataset and the target datasets for the proposed method to work.

2. The technical novelty of the paper is very limited. The closest related work to this paper is Cui et. al. which selects K classes from source domain which are closest to the target dataset classes. In this work, the authors propose instead to use one-to-one pairing, without explicit explanation of why this is a good idea. Applying mixup between the source and target dataset samples is not novel by itself, without making explicit why it helps to facilitate better transfer learning. 

3.The paper claims that “XMixup however further mixes up the target training set with
auxiliary samples and fine-tunes the pre-trained model with the data in an end-to-end manner, rather than using a two-step approach for fine-tuning Cui et al. (2018).” As far as I understand this is the biggest contribution of this paper ( to make the learning efficient), yet the authors do not make an attempt to elaborate  on this point of “two-step approach of Cui et. al vs their approach) 

4.Overall, the tone of the paper is not scientific at many places ( examples below) 

Pros:
The paper studies a very relevant problem of transfer learning.
It seems that the authors have tried to do robust experimental evaluation. 

Other minor comments:

(A)The sentence formation is incorrect at many places:
“Later, the work Chen et al. (2019);Wan et al. (2019) introduces new algorithms that prevent the regularization from the hurts to transfer learning,”
“Given two classes c s and c t in the source and target domains respectively, we consider the similarity between the two classes as the potentials for knowledge transfer, while XMixup measures the similarity between the two classes using the cosine measures between the centroids of the two classes”
“Note that P ∗ refers to the optimal mapping that potentially exists to minimize the overall distances, while XMixup solves the optimization problem using a simple Greedy search”

(B)“While vanilla mixup augments the training data with rich features and regularizes
the stochastic training beyond empirical risk minimization (ERM),” saying “rich features” is not a technical description of the related work. 

C. ”According to the empirical study in Ge & Yu (2017a), a threshold value of the auxiliary dataset size is required to guarantee the effect. In our implementation, we repeat the one-to-one pairing procedure until the size of the selected subset reaches a specific threshold. The value is 100,000 for Stanford Dogs and 200,000 for other datasets. ” This seems like an important hyperparameter, yet it is not obvious where it is in the method section.

review_title: Method has limited technical novelty and it is not well motivated.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes XMixup, a strategy for improving transfer learning in neural networks. Specifically, XMixup consists of mixup applied between target samples and source samples from the class pre-determined to be closest to target sample’s class. Experiments conducting transfer learning from pre-trained ImageNet to 6 smaller image classification datasets demonstrate XMixup to outperform the baseline approaches.

Pros:
1.	The proposed method is conceptually simple and easy to understand/implement. Since source and target information are combined into a single sample, only a single gradient needs to be backpropagated through the network. Additionally, the mixup operation is simple (draw from beta distribution, then a convex combination of the two images + labels), so overall XMixup is fairly cheap computationally, once class feature centroids have been computed (see Cons #2).
2.	The experiments demonstrate that XMixUp is fairly effective for the target image classification datasets shown, moderately outperforming the baseline methods on the 6 datasets, which have varying degree of overlap/similarity with ImageNet (source dataset).
3.	The authors made an effort to demonstrate why XMixup is effective, with many different kinds of experiments, including several I wouldn’t have thought of. See Cons #1, however.

Cons:
1.	The novelty of XMixup is moderately low; for the most part, it’s a fairly standard application of mixup, applied to transfer learning with a simple pairing of source and target classes. This could be fine if the experiments were particularly thorough or convincing. However, all experiments are in image classification, transferring from ImageNet as the source dataset. I would have liked to see experiments on more settings (e.g. NLP) or problem types (e.g. source: classification -> target: object detection). Additionally, several ablation studies that would shed more light on how XMixup works (see 2. below) are missing.
2.	While training is fairly efficient, XMixup does require computing the centroids for all source and target samples first, which can be somewhat expensive (for example, if ImageNet pre-training is the source, then features for all 1M ImageNet samples must be computed). It’s not clear how important this class-matching step is though. An ablation study without class matching target samples with the closest source class would be helpful. It’s also unclear how essential that the mapping between source and target classes be one-to-one; again, an ablation would have been helpful.
3.	One of the central arguments made is that XMixup prevents catastrophic forgetting, with experiments (Table 2) providing further evidence. Why does catastrophic forgetting matter for transfer learning though? “Forgotten” features are most likely to be those from the source pre-training task that have little relevance to the target task. For example, if fine-tuning ImageNet for aircraft classification, why should one care if the network can still distinguish dog breeds, or foods?
4.	The writing quality is not up to publication standards; please give this draft another careful round of edits. A non-exhaustive list of corrections:
o	Pg 1: Backgrounds -> background
o	“Later, the work Chen et al. (2019); Wan et al. (2019) introduces new algorithms that prevent the regularization from the hurts to transfer learning.” 
o	“two phase” -> “two-phase”
o	“large scale” -> “large-scale”
o	Both “pre-train” and “pretrain” appear in the text
o	“each of which containing 100 examples” -> “each containing 100 examples”
o	“standard division” -> “standard deviation”

Questions:
-	Using centroids of the features for each class (Eq 1) to determine class similarity makes several assumptions about the shapes of the class distributions in the feature space. Have you inspected the feature space of the pre-trained network to verify if these assumptions are reasonable?
-	What is the dimensionality of the model output? Assuming m source classes and n target classes, is the output dimensionality m, or m+n? Figure 1 implied the latter, but I didn’t see it stated anywhere in the text.
-	Figure 3: Why would there be a drop in performance (especially for Stanford dogs) when the auxiliary dataset gets large? Why is it surprising that XMixup would outperform fine-tuning when using the full source dataset?
-	Pre-training is not only used just for classification. Object Detection, segmentation, VQA, and many other types of problems also rely on pre-trained networks. Can XMixup be used for other problem types? What about for NLP (e.g. BERT fine-tuning)?

Miscellaneous:
-	Unless using it like a noun, citations should be parenthetical (\citep).
-	x_t.class is awkward notation
-	Table 1: It would be useful to see plain naïve fine-tuning (no regularization, no XMixup, etc.) as well. Also, consider bolding the best values; makes the table easier to read.

Decision:
Despite mediocre novelty, XMixup seems to be an effective and simple technique. However, although the authors include several experiments aimed at understanding XMixup’s efficacy, several key ablations are missing, and there’s a lack of diversity in experimental settings (all ImageNet -> smaller image classification set). The writing is also not quite at publication standards and could use some work. As such, while I think there may be potential, I don’t think this work is ready at this time, and vote for rejection. I think this work can be significantly improved with increased experimental setting variety, better ablations, and a careful round of edits.


=======
Post-rebuttal
=======

I thank the authors for their comments. The current draft isn't a bad start, but in agreement with the other reviewers, it needs more work before it's ready for publication. Please carefully consider our recommendations for your next draft.
review_title: Simple and moderately effective method for transfer learning, but not quite ready for publication

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

This paper proposes a simple variant for the mixup training mechanism for transfer learning problems: cross-domain mixup (XMixup). The key idea is to mix up the training samples from both domains where the samples are generated by nearest-center assignment in each class. Experiments on several datasets have shown its effectiveness in transfer learning compared to some SOTA methods.

Pros:

1. A simple but effective method for mixup in transfer learning.
2. The method is easy to implement with improvements in experiments.
3. The experiments are fair enough.

Cons:

1. The main disadvantage is its limited novelty. Computing the class center of each class and then initiating an assignment between each source and target class is a common strategy and also has been used by several existing works (see references below). In addition, the experiments on arbitrary sample selection (table 3) also show that even using random samples to form the mixup training samples, the performance is also good and has a very small gap to that in table 1. Therefore, I think this method can only be seen as a moderate extension to mixup for transfer learning problems.
2. The second disadvantage is that even if the experiments in table 1 show some positive results, the improvement is rather marginal, and all the datasets are rather small. For example, the largest dataset in this paper is Stanford Cars with 16,185 images where XMixup only achieves an accuracy improvement of 0.6%, which is not significant. Why not use other larger datasets such as Caltech-101, Caltech-256, Sun397, STL, and CIFAR-100? Since the method itself is simple, you should experiment on more larger datasets to show that it really works.
3. This method is problematic in real applications. It is not easy to use as it requires to store and access the source domain data (e.g., ImageNet), which is almost not possible. Therefore, even if XMixup is simple and doable in the idea level, the efficiency should be considered by not only the running time, but also the data storage and access cost. To this end, the regularized learning methods in table 1 are more applicable since they do not have to access the original source data. In addition, only marginal improvements are achieved by storing and accessing the large volume of source data. Therefore, the method in its current form is not helpful in real applications.
4. One of the claims in this paper is that XMixup can prevent catastrophic forgetting, i.e., "remember" both the source and target tasks. Why do we care about the forgetting phenomenon in transfer learning since its goal is to transfer the knowledge to the target domain to ensure that best performance can be achieved in the target domain? Do you have any evidence to show that preventing the catastrophic forgetting can somehow benefit the performance on the target?
5. Continued from point four, the efficiency of this method is not validated in experiments to compare with both regularized learning and multitask learning methods. So, it is hard to evaluate its time efficiency.
6. Finally, there lacks detailed analysis of why this simple mixup strategy would work apart from the accuracy results. For instance, what kind of representations did this method learn by mixing up that help it generalize better to target domain? Why it achieves comparable results with random selection? 
7. Application scenario: All images in this paper are natural images that should remain similar in general. Will this method work for distant domain images such as ImageNet->medical images? It remains unclear when to use this method. A doable application scenario is needed. 

To sum up, this is a good paper, but not appropriate for ICLR in its current form due to limited novelty, poor analysis, and marginal improvements.

References:

[1] Snell J, Swersky K, Zemel R. Prototypical networks for few-shot learning[C]//Advances in neural information processing systems. 2017: 4077-4087.

[2] Pan Y, Yao T, Li Y, et al. Transferrable prototypical networks for unsupervised domain adaptation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 2239-2247.

[3] Wang J, Chen Y, Yu H, et al. Easy transfer learning by exploiting intra-domain structures[C]//2019 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2019: 1210-1215.

[4] Hu D, Liang J, Hou Q, et al. PANDA: Prototypical Unsupervised Domain Adaptation[J]. arXiv preprint arXiv:2003.13274, 2020.

[5] Sun Q, Liu Y, Chua T S, et al. Meta-transfer learning for few-shot learning[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2019: 403-412.

[6] Choi J, Hwang S J, Sigal L, et al. Knowledge Transfer with Interactive Learning of Semantic Relationships[C]//AAAI. 2016: 1505-1511.
review_title: Limited novelty and marginal improvements

Meta-review: All reviewers recommend rejection due to limited novelty and insufficient experimental analysis. The author’s response has addressed several other questions raised by the reviewers, but it was not sufficient to eliminate the main concerns about novelty (as the method is a combination of existing techniques) and missing comparisons to justify the effectiveness of the proposed approach.
Decision: Reject