Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers a new task, knowledge distillation using out-of-domain data, and proposed a novel method, MosaicKD. Leveraging the shared common local patterns, MoscaiKD is able to effectively learn from the OOD data.  The paper is well organized and presented, and experimental results demonstrated the effectiveness of the proposed solution. Even I have some questions about the experiments, the quality of the paper is high.
main_review: Originality: In this submission, a novel problem is presented: Out-of-Domain Knowledge Distillation, which could be considered as a simple combination of two sub-problems: domain adaptation and knowledge distillation. To deal with this new problem, a new method MosaicKD is proposed, which also could be considered as a combination of KD and GAN methods (adapted to patches, local patterns). The main idea is presented cleverly, but the contribution is not significant. Combining two sub-problems (DA and KD) and proposing a plausible insight (local pattern)  cloud be a good choice for publication, but not an outstanding work. Related work is adequately cited.

Quality: the submission is technically sound, and the claims are well supported. This work is a complete piece. 
Questions about Equations (3) and (4): How can we evaluate the effectiveness of the first term (about the patch) in Equations (3) and (4)? Since the key insight behind this submission is the shared common local patterns, we should validate the effectiveness of the first term. Maybe this could be answered by results in Table 4.?
I understand the insights behind Fig.1, but I could not get helpful information from Fig. 1 as there seems no overlapping local patterns between large images.

Clarity: The submission is clearly written and well organized. The writing is good. Some errors:
Line 197: The first term refers to the entropy …    => should be “The second”?

Significance and  Experiments: Experimental results demonstrate the effectiveness of the proposed methods. However, the comparison is not fair. In Table 1, the authors mainly compared with KD [18] Balanced [35] FitNet [41] RKD [38] methods, which are not relatively new. 
In Table 2, first, if the FLOPs indicate the FLOPs of the student model? If so, it is unnecessary to present the FLOPs here since all student models are the same. Also, why use mIoU on ImageNet?
An important experiment is about the patch size (Table 4). We see these datasets using a resolution of 32\times 32 (cifar10 and SVHN, not sure about the Places365). But the best result is achieved given 18 * 18 patch (aka., 1/4 image area) which seems not reasonable. 
limitations_and_societal_impact: Yes. No critical points are missing.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method for knowledge distillation with out-of-domain data (OOD-KD). The key idea is to dismantle out-of-domain images into patches and assemble in-domain data by training a generative adversarial network. It is based on observation that different domain may still share common local patterns from which new (in-domain) categories can be re-assembled. The proposed method is developed undert the framework of the so called Distributionally Robust Optimization (DRO), restricted by a ball space centered at the OOD patch distribution.
main_review: Overall, I like the idea of this paper. First, I think the proposed method is a clear and novel way for knowledge distillation from out-of-domain data. Theoretically, the method is reasonable, since it is an extension of DRO framework for distant distributions (OOD and ID data). The empirical evaluation sufficiently validates the authors' claim. I think the proposed method can be useful in practice.

On the other hand, the main weakness of this paper lies in the patch learning on local patterns. Specifically, the proposed method trains a generator on all patches directly and it is not very clear what kind of local patterns is useful for the target domain. Maybe the aligning loss L_{align} can help select task-related patterns for synthesis but is not clear enough in the method part. I think the author should take a further step on this point and provide more discussion on this topic as it is the core of the method.
limitations_and_societal_impact: The paper conveys the motivation and idea nicely but the presentation needs to be improved. After going through the paper and appendix, I have some questions about the proposed MosaicKD: 

1) Adversarial attack adds a small perturbation to the image and is also able to modify the category of OOD data. What is the difference between samples obtained from MosaicKD and adversarial attacks? 

2) In Figure 3, why the FID of MosaicKD in some categories is a bit worse than the OOD data? 


3) some errors and typos need to be fixed: 
- Line 178: The symbol V(G) and V(X’) is a little confusing. 
- Line 148: diverged => divergent.
- Line 167: V(X^prime) => V(X^\prime)
- Line 197: The first term refers to … => The second term refers to …
- Line 207,: a redundant bracket for x ~ P_G in Eqn 6. 
- Line 221: Equation 4.1 => Equation 4. Besides, I think the derivation of equation 6 and equation 7 should be moved from appendix to the main paper for better presentation.
- Line 298: We conduct ablation … => we conduct ablation study… 
- Line 1 in Algorithm 1: D(z; \theta_d) => D(x; \theta_d).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies knowledge distillation in out-of-domain settings and proposes a generative method called MosaicKD to handle the agnostic domain gap between OOD transfer set and original training data. The core idea of MosaicKD lies in the observation that different domain may still share some local patterns, which can be re-assembled for cross-domain synthesis. The author introduces a four-player adversarial training method to learn a locally-authentic and globally-legitimate distribution for knowledge distillation.
main_review: **[Strengths]:**

The motivation of this work is well presented and the idea of patch learning is novel. In my opinion, knowledge distillation with DRO is reasonable and practical for OOD settings as the goal is to optimize the risk on an unknown distribution. And I agree with the statement that conventional DRO framework would be problematic if the target distribution is far away from the OOD data. To address this problem, the author further introduces a patch-restricted method to make DRO applicable for OOD settings and effectively improves the performance of learned student models on OOD data. The experiments in table 1 are very interesting. It seems that MosaicKD not only works on similar data like CIFAR10, but also works on extremely mismatched data like SVHN. Besides, extensive experiments are conducted in this work for evaluation. Table 1 benchmarks existing methods on several datasets, which would be helpful for future works. 

**[Weaknesses]:**

Here are my questions: 
1) To my knowledge, some KD methods are also designed for large-scale unlabeled data, like PU Compression [50]. Why they are not used as baselines in this work? 
2) In some settings, data-free KD is also a strong competitor (table 1, resnet34 to resnet18). What is the advantage of the data-driven approach proposed in this paper compared to the data-free approach that does not require data? 
3) There is some room for improvement in the selection of patch size, such as adaptive patch partition or selective patch generation.

**[Additional Feedback]:**

In addition to the above issues, there are some suggestions for improvement: 
1) Some equations should be presented in more appropriate ways, such as patch distribution in Eqn 3 and the entropy term in Eqn 4. 
2) A brief description of the algorithm can be included at the beginning of the Sec. 4.2 to make the whole algorithm clearer. 
3) In L205, the citation to the equivalence between GAN objective and JS divergence is missing.

**Update: post rebuttal**

I have read the authors' response and other reviewers' comments. Overall I think the rebuttal addresses most of my concerns and I will increase my rating to *7*.
limitations_and_societal_impact: Part of these have been discussed in the paper.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a knowledge distillation algorithm for out-of-domain data, where local similarity between different domains is leveraged to transfer knowledge across models. In particular, the authors first present a basic method based on distributionally robust optimization and discuss the its problem in OOD settings. Then the authors improve the basic method by introducing a local loss and an aligning loss. The proposed method, named MosaicKD aims at synthesizing locally-authentic and globally-legitimate samples for KD. The authors demonstrate that MosaicKD can be applied to various OOD data and outperforms several data-free and data-driven methods (e.g. ZSKT, DFQ and CRD)
main_review: pros: 
It is the first attempt to study knowledge distillation with out-of-domain data. The author utilizes the local similarity to alleviate the domain gap between OOD data and source training data. The idea of crafting in-domain data from low-level patches is novel and practical. The author also demonstrate that patch learning can help the generator escape from the OOD distribution and synthesize correct semantic, as the global structure is freed by the patch-level training.  

cons: 
It is not clear to me how the authors determine the patch size (the receptive fields of Patch Discriminator) and the overlap between patches (the stride applied on discriminator’s logits) for different OOD data. According to the code in supplementary materials, the stride is set to 1 by default. Please provide more details about the design of patch discrimination. Besides, some equations and derivations are not clearly presented in the main paper, such as the Equation 7 and its derivations. 

========= post rebuttal ===========
I have read the rebuttal. The rebuttal addresses all my concerns. I will change my score to 8.
limitations_and_societal_impact: I have some questions about the proposed method: 
1) what is the relation between MosaicKD and existing data augmentation methods, like mixup, or perturbation. In the literature of KD, some works adopts augmented data for training [1,2]. Please make a clarification for their difference.
2) In table 4, a too small patch size leads to `best ` FID but worse student performance. Please provide a discussion about this phenomenon. In particular, what kind of information is missing in small patches which degrades student’s accuracy.
[1] Heo, B., Lee, M., Yun, S., & Choi, J. Y. (2019, July). Knowledge distillation with adversarial samples supporting decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 3771-3778).
[2] Fu, J., Geng, X., Duan, Z., Zhuang, B., Yuan, X., Trischler, A., ... & Dong, H. (2020). Role-wise data augmentation for knowledge distillation. arXiv preprint arXiv:2004.08861.
review_title: null

Meta-review: The paper studies a new setting, called out-of-domain knowledge distillation, where the teacher network and the student network are trained on different datasets. The proposed new setting is interesting and more practical than existing ones. In terms of the solution, the authors propose a mosaic idea to synthesize in-domain data by imitating the local patterns from real world OOD data.  The key technical idea is to use minimax optimization to ensure the synthesized data could fool the discriminator. The solution is novel and technically sound. Reviewers all agreed that the proposed problem is interesting and the solution is novel.Thus, I am recommending acceptance. 

Of course, the authors need to carefully revise the manuscript according to reviewers’ comments. The answers to Q3 and Q4 of Reviewer yT4j must be included in the final version.

Decision: Accept (Poster)