Kati8/ICLR_2023_2025 huggingface dataset:
authors: Kaiqiang Tang
authors: Huiqiao Fu
authors: Jiasheng Liu
authors: Guizhou Deng
authors: Yuanyang Lu
authors: Yiming Qi
authors: Chunlin Chen
reviews: 
Comments: This paper proposes a dynamic attention network for dynamic capacitated vehicle routing problem.
Strengths: This paper proposes a dynamic attention network for dynamic capacitated vehicle routing problem.
Weaknesses: This work seems to be not novel enough. The similar ideas of the proposed dynamic attention network can be found in previous literature [1]. The batch training for sequences with different length is a common issue in the field of natural language processing, and this issue can be easily tackled by padding.

The experiments are not sufficient and not convincing.

[1] Solving Dynamic Traveling Salesman Problems With Deep Reinforcement Learning, IEEE TNNLS 2023.
Recommendation: 3: reject, not good enough
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 2 fair
reviews: 
Comments: This paper introduces a DRL-based framework for solving the dynamic CVRP, enabling the model to adapt to real-time customer location updates. The results achieved with the DRL-based model surpass those of the benchmark solutions. However, the presentation of the paper prevents readers from fully grasping the ideas and the implementation details. The mathematical formulation of the problem and the RL algorithm also needs significant work. This makes the validity of the outcomes and the reproductivity of the study questionable.
Strengths: This paper offers innovative insights into the integration of DRL within the dynamic CVRP.
The paper demonstrates good performance compared to benchmark solutions.
Weaknesses: I am concerned about the level of innovation presented in this paper. It appears that the only innovation is to set the input as dynamic disclosed customer locations in a mask fashion. Other than that, it follows a standard and straightforward pipeline that uses DRL to solve the static VRP problem, which has been well-studied in many other publications. 

While I believe that DRL is used in this study, the paper provides very limited details / technical explanations related to DRL implementation and how DRL is specifically tailored to solve CVRP function, e.g., what are the rewards, what are the states, and how state transition is defined.

Moreover, the mathematical formulation in this paper is unclear and not rigorous.  For instance, the rationale behind the introduction of equations 1 to 5 is not clearly explained. Constraint 3 appears to define t_i^{cur} under various scenarios, but it requires further clarification.


Clarity, Quality, Novelty And Reproducibility

I had a hard time understanding the notations, often having to refer back to previous pages or struggling to find clear definitions. Examples include mha, sha, D', X', and π_{j} (is it a set or a sequence of visited nodes?). 

Besides, the paper also contains multiple grammar issues, just to name a few:
Page 3-4: “However, the current DRL models that utilize the attention model rely on a fixed number of customer points throughout the solution and cannot adapt to these dynamic changes.”
Page 5: “The point with the highest probability il+1 is selected as the target point, which the vehicle travels directly to from the current point.“

Finally, I have concerns about the reproducibility of the paper due to the significant modifications made to transform the static VRP instance into a dynamic one.
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: A deep reinforcement learning approach for the dynamic capacitated vehicle routing problem (DCVRP) is proposed and evaluated on uniformly randomly generated instances. The main contributions of the approach are a "dynamic attention network" for addressing the DCVRP. They develop two methods to solve the DCVRP based on LKH and OR-Tools. The paper also proposes a mathematical model for the DCVRP.
Strengths: The DCVRP has indeed seen less attention than offline problems, and considering dynamic problems with learned methods definitely presents an exciting research opportunity.

There is an argument to be made that attention networks must be adjusted to handle this type of problem. This paper does not make the contribution (1) in this respect so clear. What is new and what is old is very hard to determine; the overall method from Wu is not described.
Weaknesses: I do not consider the last two contributions to be significant contributions for ICLR for two reasons. First, these are operations research (OR) contributions that ought to be evaluated by the OR community and not ICLR. But even if we accept them as thematically relevant, which they aren't, neither would be accepted in any OR venue in their current state. 

Regarding contribution (2): the methods are not even close to the state-of-the-art. In the NeurIPS meets EURO competition the organizers provide a method for solving such dynamic problems (admittedly with time windows, but removing those would be trivial) based on Vidal's Hybrid Genetic Search (HGS), which is the state of the art. It's relatively simple to extend the offline problem in a naive way to support the dynamic version. The approaches offered here are a decade behind the current state of the art. Another option would be to use Ulmer's approach to dynamic routing problems. The author's cite his work, so I am not sure why it is then forgotten by the time we get to the experiments.

Regarding contribution (3): The mathematical model is not structured in a way that any OR journal would accept it without a serious revision. One could call it unconventional at best, i.e., the model is not a mixed-integer linear program (even though the problem is linear). There are good reasons for the standard formats used in the OR community, namely that they are easy to read and interpret, which this model is not.

The experiments are also rather weak: only randomly generated instances, and even the algorithm run times seem to be randomly chosen. Please see Accorsi et al. (2022) "Guidelines for the computational testing of machine learning approaches to vehicle routing problem". Furthermore, the extent of the experiments is very limited. There's no investigation of parameter sensitivity, ablation, etc. It is all missing.
Recommendation: 3: reject, not good enough
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: This paper proposes a deep RL-based framework for dynamic capacitated vehicle routing problem. Because of the dynamic nature, the algorithm is able to serve customers even if they are added later. The paper then compared the proposed approach with the existing algorithms.
Strengths: 1. The main strengths seem to be that this work can adaptively address the dynamic customer addition problem, using deep RL algorithm it can route the vehicles in an adaptive manner.

2. The algorithm's performance is better compared to the existing algorithms.
Weaknesses: 1. The paper is not well-written to judge the main contributions. For example, deep RL-based approaches have been proposed [A1,A2]. The algorithm has not made any comparison compared to the other existing deep RL-based work.

2. The algorithm has not explained many things to get proper insights of the algorithm. We should not judge any algorithm's performance truly based on its performance rather how it is being developed (and the intuitions behind that). The reviewer did not get any intuition behind the algorithm. For example, the paper repeatedly mentioned about applying deep RL. What are the parameters for this deep RL problem? What is the reward? What are the transition probability models? How do those change dynamically? 

3. The paper uses attention model, what is the logic behind using it.

4. The optimization problem in (1)--(5) is not clear. The notations have not been introduced before proposing the problem, hence, it is difficult to parse what it is doing. Besides, it contains another optimization problem as a constraint (in (2)), hence it is a bi-level optimization problem. How do the authors propose to solve this bi-linear optimization problem?


[A1]. Joe, Waldy, and Hoong Chuin Lau. "Deep reinforcement learning approach to solve dynamic vehicle routing problem with stochastic customers." In Proceedings of the international conference on automated planning and scheduling, vol. 30, pp. 394-402. 2020.

[A2]. Hildebrandt, Florentin D., Barrett W. Thomas, and Marlin W. Ulmer. "Opportunities for reinforcement learning in stochastic dynamic vehicle routing." Computers & operations research (2022): 106071.
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 1 poor
Clarity: 1 poor