Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper finds its motivation in scheduling for semiconductor manufacturing systems. It proposes a deep RL algorithm for tackling the highly delayed feedback usually encountered in this type of dynamic system. The authors introduce an algorithm called Predictron Deep Q-Network (PDQN) that plugs a predictron architecture into the loss of the Q-network. The efficiency of PDQN is tested on two scheduling problems and challenged against 3 natural baselines, CR, FIFO, and DQN, the first two baselines being standard for scheduling. 
main_review: *Strengths*

- This work raises an important issue that has hardly been tackled in the RL literature, namely high delays in the reward signal. It has a strong motivation, and the proposed solution is sound in my opinion. 
- In terms of contribution, I think that making a connection between a typical operations research problem and a deep RL method constitutes a contribution in and of itself, as I am not aware of any previous work that does so. 
- The paper is clearly written, in a comprehensive manner that makes it easy to follow, may it be for an RL expert reader or for a scheduling one. I have minor comments in that regard, which I will make at the end of this review. 

*Weaknesses / Questions to the authors*

- The authors cite Stricker et al. (2018) as an RL paper tackling scheduling tasks with Q-learning. Why is their method not used as a baseline in that paper? Indeed, PDQN is also a Q-learning algorithm and aims to solve the same type of problem. Another Q-learning method for delayed rewards is proposed in [3]...
- The authors motivate the predictron submodule by « highly delayed rewards » (Sec. 3.3 and a few other places). How much is it « high »? Although I guess it is stochastic, is it bounded? Another concern is regarding the way time steps are set in the MDP: according to Eq. (8) time is discretized in order for the reward signal to fit with the MDP framework. But what is a typical value of $\Delta_i:= t_i - t_{i-1}$? Would not it affect that of the reward delay, i.e. high $\Delta_i$ implies lower delay value? Similarly, what is a typical value for $h$ in Sec. 3.3.? I guess this should depend on an average delay value. How would a high $h$ value affect the computational time of PDQN (higher $h$ implies long-term inference)? 
- How different are Eq. (11) and (12) from Eq. (5) and (6), respectively? If these are in some sense equivalent, the authors can synthesize their notation and refer to (5) and (6) in Sec. 3.3. instead of rewriting them.
- Sec. 2 looks like a Preliminary section providing the problem setting, more than a Related work section. In that regard, a review of delayed RL works is missing. More precisely, high delay in the signal results in some challenging issues: although one could perform embedding and retrieve a non-delayed MDP as in [1], this greatly increases computational complexity. In my opinion, the predictron has a major advantage over augmentation methods that may be even less tractable than MCTS (such drawback of MCTS is indeed mentioned in Sec. 3.3).
- In [2], the authors use a forward model to tackle execution delay issues. How does the predictron compare to this type of model? The use of eligibility traces seems to have similar effects. 
-  In Figs. 2-3, DQN seems to perform decently against PDQN. This discredits a bit the use of the predictron. How could we explain such good performance of DQN which, by the way, does not account for delay at all? Unexpectedly, although they are fashioned precisely to solve such scheduling problems, FIFO and CR perform much worse than those two deep RL methods. How can this be interpreted? 
summary_of_the_review: Although this work is well-motivated, I have two main concerns leading me to a « weak reject » notation. 
- First, on the theoretical side, the problem of delayed reward is not tackled frontally, but rather implicitly through the predictron. As such, delay is not formalized, although intrinsic to the problem considered. It weakens this paper’s contribution in my opinion, as one could question whether PDQN would necessarily work in a different type of domain (e.g., healthcare, as mentioned in Sec. 1), where the reward function has other dynamics. 
- Second, delayed reward and delay in general (delayed execution; delayed observations), have been addressed in RL. This span of relevant works is never mentioned nor discussed. I list below a few references that are welcome to be discussed/compared with PDQN.

*Relevant references (all of them are missing in this paper)*

[1] Konstantinos V Katsikopoulos and Sascha E Engelbrecht. Markov decision processes with delays
and asynchronous cost collection. IEEE transactions on automatic control, 48(4):568–574, 2003.

[2] Derman, Esther, Gal Dalal, and Shie Mannor. "Acting in Delayed Environments with Non-Stationary Markov Policies." ICLR 2021.

[3] Jeffrey S Campbell, Sidney N Givigi, and Howard M Schwartz. Multiple model q-learning for stochastic asynchronous rewards. Journal of Intelligent & Robotic Systems, 81(3-4):407–422, 2016.

[4] Thomas J Walsh, Ali Nouri, Lihong Li, and Michael L Littman. Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18(1):83, 2009.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tackles the dynamic scheduling problem in semiconductor manufacturing using an RL approach. Due to the processing actions taking different amount of time to complete, the problem has a long reward delay issue, which the paper addresses through using predictron to estimate the targets in DQN. Simulation experiments were performed on domains constructed using real-world data and against common heuristic baselines and DQN. Results show that the proposed method outperforms the baselines in terms of cumulative lateness of the parts. 
main_review: Strengths
- The paper tackles an important decision/scheduling problem in manufacturing industry, which seems to be a very promising real-world application of deep RL.
- The proposed method outperformed the common heuristics used in industry in terms of cumulative lateness.
- The use of model-based approach to address the long reward delay issue in this problem makes sense to me.


Weaknesses and other comments
- The paper should formally define the dynamic scheduling problem in manufacturing in mathematical terms. The current level in the introduction is not sufficient for a scientific paper. The true objective of interest should also be clearly defined. The current choice of reward is the total time past due over the parts during a unit time period, but later in the experiments, it appears that the throughput is also a metric of interest. The choice of reward should be motivated by the ultimate objective of the problem, not by certain behavior of the policy that we want to encourage.
- Before the conclusion, it is mentioned that "It is interesting, however unexpected, that FIFO is better than CR on almost all parameters in the G20 setup, and also achieves lower mean lateness in the B20 setup." Is CR typically expected to outperform FIFO? Does this observation mean that the simulator requires some calibration before being used for training the agent?
- The baselines for the benchmarking should be more properly selected. For example, Section 2.1 discussed several recent works based on RL for the same problem. Why is not one of them selected as a baseline? The proposed method is basically DQN with the targets estimated by predictron based on rolling out a simulator. To justify this combination of techniques (especially the use of predictron), benchmarking should be performed against a version of DQN, say, with targeted estimated simply by MC rollouts. 
summary_of_the_review: This paper tackles an interesting and promising industrial application of RL. To address the long reward delay issue, a combination of DQN and predictron (for estimating targets) is proposed to train the agent, which makes sense. Empirical results are presented to show the advantage of the proposed method over common heuristics and vanilla DQN. The paper needs significant improvement in formal problem/objective definitions and the choice of baselines in benchmarking for a more convincing case. The simulation environment is not a standard one, so some calibration results should be presented to ensure the validity of the results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study a deep reinforcement learning approach to the problem of scheduling machines in a semiconductor factory. The authors' main contribution is a deep Q-learning based system that uses a predictron to estimate the value function (instead of max_a Q(s',a)). The intuition behind the domain is that there are many machine groups that can process parts. The goal is to select the order that the machines should process parts. The authors compare a double DQN to their predictron DQN (PDQN) to baselines that are currently in use: critical ratio and first-in-first-out, with the objective of minimizing lateness. They build simulators based on real data: one that is proprietary and one that they release. They find the RL systems reduce lateness by substantially (20-50%) relative to the baselines and that their PDQN performs better by around 30% compared to DQN. However, they note that the RL systems complete substantially fewer parts, as much as 15% fewer for PDQN and 25% for DQN, suggesting that lateness may not be a good reward function for this task.
main_review: Strengths:
- The authors develop a realistic, public shareable environment for manufacturing dispatching that is suitable for RL methods research.
- The PDQN framework is novel and offers better performance for these tasks.

Weaknesses:
- The main baseline for the work is quite old—the authors use the 2015 vanilla DQN. The authors say they tried various improvements from Hessel et al., and they didn't seem to help, but they don't enumerate which improvements they tried (see question 1 below). From a methods perspective, I find the evaluation of the PDQN to not be rigorous enough—the authors only evaluate in the new environment they constructed and they only compare against vanilla DQN.
- From an application perspective, the issue of sim2real problems does not receive _any_ attention that I can see. The authors do not go into detail about how they build their simulated environment—specifically how they choose incoming parts that need to be processed. They also do not discuss a real-world test.
- The throughput issue seems like a substantial real-world problem. I can't imagine wanting to implement this system if throughput (which translates directly to the bottom line) could decrease by 15%.

Questions:
1. The architecture the authors propose seems quite similar to a dueling DQN in the way that the Q-function estimation and the value function estimation are split—the difference being that the dueling DQN does not use a predictron and that the dueling DQN has a different aggregation architecture. The authors mention they tried the improvements of Hessel et al., but did they try the dueling architecture specifically? It would also be easy to integrate a predictron into the value-estimation network in a dueling DQN. (It would be interesting to know how much the predictron structure is specifically contributing.)
2. The use of L2 regularization in the predictron is interesting—did the authors find regularization improved performance at convergence or did it decrease sample complexity? There has been recent interest in the use of L2 regularization, e.g., https://arxiv.org/abs/1810.00123

Detailed comments:
- Eq. 1 is often called the Mean Squared Bellman Error because it measures the extent to which Q satisfies the Bellman equations
- “Q-learning is considered an off-policy method” -> “Q-learning is an off-policy method”. It’s definitional—doing Q-learning would not make - sense because there would be no exploration.
- Figure 1 is many pages after it is referenced

Post-rebuttal: It seems that the rebuttal discussion has been productive—the paper has been improved in many relevant ways. There are still outstanding issues—it seems like throughput and baselines are the largest of these.

summary_of_the_review: From a methodological perspective, I found the evaluation to not be rigorous enough. As an application perspective, there are critical application-specific issues that are not addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: 
 PDQN - A Deep Reinforcement Learning Method for Planning with Long Delays: Optimization of Manufacturing Dispatching
In this paper, the authors applied a reinforcement learning algorithm to the problem of manufacturing dispatching.
They used a model-based algorithm that is inspired on the algorithm of predictron.

main_review: *Main Review

Pros:
1. The problem is very interesting. And it is a real life application that could generate good real-life values. 
2. The code is released and has good reproducibility.

Cons:
1. It is not clear why the authors decide to make the presented modifications to the original predictron algorithm.
And it is hard to understand the intuition of the algorithm.
The modifications are not justified.

2. There’s no proper reinforcement learning baselines.
The algorithm is only compared with some toy methods and the RL baselines DQN is quite outdated. If I didn’t misunderstand, at least predictron should be compared against?
And there are also some new algorithms that seem to be applicable to the problem.


summary_of_the_review: 
*Summary Of The Review
I am worried about the novelty of the algorithm, and the claims are not supported by experiments.
review_title: null

Meta-review: This paper deals with solving the problem of scheduling machines in a semiconductor factory using an RL approach. As the different actions take a different amount of time to complete, the authors propose to use a predictron architecture to estimate the targets in DQN. The experimental results show that the proposed method outperforms the considered baselines on two scheduling problems.

After reading the authors' feedback and discussing their concerns, the reviewers agree that this paper is still not ready for publication.
In particular, the main issues are about the novelty/similarity with respect to related works, the lack of theoretical insights and formal definitions, the effectiveness of the presented benchmarking, lack of analysis of some unexpected results.

I encourage the authors to take into consideration the concerns raised by the reviewers when they will work on the updated version of their paper.
Decision: Reject