Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper focuses on a class of NP-hard combinatorial optimization problems called Vehicle Routing Problems (VRP) which is of wide practical interest. In the deep learning community there is a wide interest in replacing the domain and expert dependent heuristics with data-driven methods like reinforcement learning. However, as the paper notes, most current approaches focus on one particular distribution of problem rather than trying to generalize across distributions, which limits their practical applicability. The paper builds on the ideas of knowledge distillation widely applied in practical deep learning applications to train a light-weight student model to learn from this variety of individual distributions focused methods. This allows demonstration of competitive results on both unseen in-distribution as well as OOD instances while requiring cheaper compute resources.
strengths_and_weaknesses: ### Strengths

A simple and widely used idea of knowledge distillation is demonstrated useful in a new domain with wide practical applications. Given the simplicity the method is likely simple to implement and generalize to real world problems. The shown improvements are fairly consistent along a variety of different axes of comparisons and are compared against a fair number of strong baselines. They also confirm that their method can improve further with more compute resources for their student model.

### Weaknesses

Main weakness as the authors acknowledge as future work is generalizing across different/larger problem sizes. The idea of knowledge distillation is straightforward and not necessarily novel, but this will likely be a strong baseline for the community to build upon.

questions: Would be useful to understand the computational tradeoff for the larger student models as well. What's their resulting size and time taken? 
limitations: The paper adequately mentions its limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a knowledge distillation framework named Adaptive Multi-Distribution Knowledge Distillation (AMDKD) for transferring various knowledge from multiple expert teachers to a generalist lightweight student model.  Firstly, AMDKD performs teacher pre-training to obtain expert teachers for different distributions.  Then in each epoch of the knowledge distillation process, AMDKD selects a specific distribution and its teacher with an equal probability. The probability of each distribution being selected is computed according to the performance of the student model on this distribution.  With the selected teacher model, AMDKD performs a general single-teacher knowledge distillation and updates the student model. Several experiments on TSPLIB and CVRPLIB datasets are conducted and AMDKD shows competitive results compared with state-of-the-art methods.
strengths_and_weaknesses: Strengths：
1.	Benefitting from a concise description of the overall framework, the motivation and training process is clear and easy to read.
2.	The idea of selecting a specific teacher model on the weakness of the student model is interesting, which converts a multi-teacher knowledge distillation to a single-teacher knowledge distillation in each epoch, avoiding the profitless influence of averaging the outputs from multiple teacher models.
3.	Sufficient experiments are conducted to prove the effectiveness and the generalization of AMDKD, especially the experiments on the unseen in-distribution and out-of-distribution instances.

Weaknesses:
1.	The probability of selecting specific distribution and its teacher is updated with each epoch, which will take a lot of computation cost to evaluate the performance of the student model. 
2.	The selection of teachers is decided by the performances on the validation datasets, which makes the experimental results rely on the quality of the validation datasets

questions: See the weaknesses.
limitations: See the weaknesses.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a knowledge distillation (KD) approach for learning generalizable models for vehicle route planning (VRP). One key challenge in VRP is identified, that is, cross-distribution generalization. The proposed KD approach (a.k.a. AMDKD) leverages multiple specialized teacher models from different distributions to distill a generalist student model that can handle multiple distributions at the same time and generalizes well to OOD instances. Overall, the paper is well-written, the proposed method is clearly explained, and the experiments demonstrate promising results of the hybrid distillation scheme for VRP.

strengths_and_weaknesses:  Strengths:
1. This paper presents a novel exploration of KD or multi-teacher KD for VRP;
2. The proposed AMDKD approach is simple but effective and could be easily applied to help improve the generalizability of existing models by training and distilling on multiple distributions;
3. The paper is very well written and easy to read.

Weaknesses:
1. There is some confusion about the time analysis.
2. Could provide more understanding of AMDKD.


questions: 1. Table 2 shows the time* of AMDKD+EAS methods is much slower than existing methods, but the authors claimed that the proposed approach can help attain a more efficient student model. Why is that?
2. Maybe add more understanding of why AMDKD is better than # models. Why training on mixed data is not as good as training separately then distilling? Any deeper understanding in the context of VRP?
3.  Why do the three distributions (Figure 1c-1e) generalize better to other distributions? Given a real-world application scenario where generalization needs to be improved, is there a strategy for including more distributions with AMDKD in addition to the existing distribution?
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to enhance the cross-distribution generalizability of deep learning models for the vehicle routing problem. It proposes a novel method that is based on knowledge distillation and uses several teacher networks to guide learning a student network that is both light-weight and of strong generalizability. Moreover, it proposes an adaptive strategy to select the teacher models that are more informative. Experiments on two datasets verify the effectiveness of the proposed method by adding it on two existing methods and achieves performance gains. 
strengths_and_weaknesses: Strengths
1) The proposed method is technically sound. 
2) Experiments on two datasets verify the effectiveness of the proposed method by comparing with existing methods. 
3) The proposed method is overall well presented.


Weaknesses:
1) From the implementation details, it seems that the proposed method has quite a few task-specific designs, e.g., training policies, architectures, hyper-parameters, etc. It is thus doubtful if the proposed method is generalizable enough when applied in practice. 

2) In Table 1, there are many 0.00%, especially for methods LKH and Gurobi, all their result numbers are 0.00%. I am not sure if it has some special implications. If it is, there should be some explanations about it; otherwise, it could be presented in a better way, or maybe just leave it unfilled. 

3) What is the performance variance with respect to the hyper-parameters.   
questions: See the weakness above. 
limitations: Limitations are well addressed.
review_title: null

Meta-review: metareview: All the reviewers are in agreement to accept the paper. The paper tackles vehicle route planning via a knowledge distillation framework using student teacher models. The ideas in the paper are appreciated by all the reviewers. There are minor criticisms, especially regarding scalability with the problem size, selection of teachers, and the fact that the knowledge distillation is relatively known. 

Given the minor criticisms from the original three reviewers, I asked for another expert in the field to look at it and the reviewer agreed with the rest of the pool. Additionally, I took a good look at the paper as well and I am happy to recommend accepting the manuscript.

confidence: Certain
Decision: Accept