Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: For certain industrial sequential decision problems, this paper proposes to decompose the conventional MDP transition into two steps, state-dependent stage and input-dependent stage, and learn separate value functions. The state-dependent stage focuses on security or safe constraints and the input-dependent stage is for capturing the non-stationarity of the environment. The authors design a dynamic-aware and constraints-confined reinforcement learning based on this two-stage MDP formulation. On bin packing problems and elevator group scheduling problems, the proposed method demonstrate superior performance compared to rule-based algorithms and learning based algorithms. 
strength_and_weaknesses: ### Strength

The proposed two-stage MDP framework is novel and empirical results show that it perform well in non-stationary industrial benchmark tasks.

### Weaknesses

The clarity needs improvement. See detailed comments below.
clarity,_quality,_novelty_and_reproducibility: ### Clarity

Sec 3.1 is important for the audience to understand the contributions of this work. However, it’s currently hard to follow. Please consider revising this part. It would be helpful to directly compare the two stage framework with the standard one-stage framework in Figure 1 and provide some formal analysis. The statement that existing RL methods suffer from high estimation bias is too vague. 

### Quality

There are some grammar issues and typos. Please carefully proofread the draft. 

### Novelty

The proposed method is somewhat novel. 


summary_of_the_review: This paper studies the important problem of applying RL in industrial applications and the proposed method demonstrates strong empirical performance. I could be willing to accept this paper if the clarity could be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to solve industrial sequential decision making tasks. The paper makes two observations about the transition dynamics for industrial scale decision making tasks and proposes a two-stage approach that leverages the underlying structure of industrial problems. Using the observation the paper proposes a bi-critic framework (DACC) which leverages this underlying structure. Finally, empirical results are provided for the proposed method.
strength_and_weaknesses: Strength:
- The topic covered by the paper is intriguing and significant for enabling wider adoption of RL algorithms.
- The illustrations provided in the paper help better illustrate the core ideas

Weaknesses:
- The paper can be hard to understand at times. For instance, in Section 3.1, it took several reads to realize what the authors mean by "two-stage" framework. I also personally believe that writing $S_t = (S_t^p, d, F_t)$ could better represent this fact. It is also relatively unclear to me what **exactly** $F_t$ represents: in the paper immediately above eqn (1) it is mentioned that $F_t$ relates to the marginal distribution of future boxes and is related to a series of distributions. Are we saying that $F_t$ is some function that could approximate the future distributions? Approximate in what sense? Are we saying that $F_t$ is a model that is able to predict $f_t(d)$ using information obtained up to step $t$?
- While the figures contain error bars but standard errors are not reported in the tables. The number of runs used to obtain the error bars is not reported. 
- The authors use a self-developed simulator for the elevator task. It is hard to assess what the simulator does by simply reading Appendix B without seeing the underlying source code.
clarity,_quality,_novelty_and_reproducibility: - Clarity. See the previous section on weaknesses.
- Quality. The claims made in Section 3 are reasonable and should be correct. The experiment results cannot be verified but seem plausible.
- Novelty. The paper focuses on a novel setting and provided new insights into industrial-scale decision making problems
- Reproducibility. The paper cannot be reproduced at its current stage.
summary_of_the_review: The topic studied in the paper is novel and interesting. The paper would significantly benefit from improved presentation and more instructions on how to reproduce the results. Further details on the experiments are also be greatly appreciated.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This  paper  proposes  a  new  Markov  Decision  Process  (MDP)  definition  to account  for  the  non-stationary near-predictable  tasks  that  are  common  in  industrial  application.   These tasks require the agents to learn the constraints and rules of environment and the strategy to get high reward in the dynamic environment.  The author propose a two-stage MDP in which the agent learns constraints of the environment and the near-predictable reward separately in two steps.  A RL model DACC is designed to fit in the two-stage MDP setting.  With two critic networks, one of which learns the constrained transition of environment states and the other learns the transition of state reward that are regular to some extent.  The experiments compare DACC to other methods on two representative applications.  Two models (A2C and DACC w/o rules) are included in experiments as an ablation study trying to support the superiority of DACC’s dual critic design. These models outperform others.
strength_and_weaknesses: Strengths:
1. The problem setting and methods are well-defined.  The two example tasks are representative for the application environment targeted in this paper.
2.  The two-stage  MDP  proposed is technically correct as its two stages learn the constraints of the non-stationary environment and the task-specific or time-dependent values respectively.
3.  The experiments compare the proposed methods  (DACC)  with  4  heuristic methods and  2  other learning-based methods as well as two learning-based models for ablation studies in two example tasks.

Weaknesses:
1.  The DACC models use a marginal latent variable F(t) that is dependent on the recent experiences to account for the ”near predictable” feature of the problem.  However, the agent might only exploit recent regularity instead of the long-term regularity that is motivated by the authors (e.g., in elevator scheduling)
2.  Due to the recent experience-dependent F(t), how does the model respond to change of input pattern (e.g., is there a delay, can things go wrong during the transition?)
3.  The two critic networks in DACC use each other’s estimation of value function to update themselves. Does this create issues? What is the impact of the KL divergence term—does changing the weight of the KL divergence term affect the outcome?
4. The safety mask is not present in A2C and "DACC w/o rules", the two models in the paper's ablation study. I think the DACC w/o rules has more parameters to train and hence should be more powerful than A2C. But the experiment data shows that A2C has better performance. So if the extra critic in DACC w/o rules doesn't necessarily increase the performance, then the safety mask might be more important than the extra critic. Can A2C be tested with a safety mask?
clarity,_quality,_novelty_and_reproducibility: Clarity: The structure of the paper and the mathematics are fine. The flow of arguments leaves something to be desired, e.g., why exactly what the authors are doing is necessary. The paper seems to be combining a lot of stuff, but it is not really clear how the combination works (and, in the end, the performance is only a little better than A2C?).

Quality: The work is justified by intuition and experiments. I would prefer to see more thorough experiments that analyze the pieces of DACC (see weaknesses).

Novelty: The source of innovation in this paper comes from the two-stage MDP, a new interpretation for the non-stationary near predictable environment. The DACC uses two critic networks to learn environment constraints and task-related value functions respectively.

Reproducibility: The algorithms are described clearly. It is unclear if the authors will release code.
summary_of_the_review: The framework is interesting and the model is novel.  However, the experiment design failed to prove that the superiority of DACC comes from its two-stage design but not safety mask that can significantly reduce the action space. The dual critic set up raises convergence concerns.

Post author revisions:
The clarity of the paper is improved. A more inclusive ablation study supports the superiority of the model DACC and the two-stage MDP. The detailed training algorithm helps to explain how the critics are updated during training. The ablation also clarifies that A2C w/o rules performs worse than DACC w/o rules, while A2C with rules performs better than DACC w/o rules.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Reinforcement learning can be used in many industrial decision-making problems due to its potential to outperform heuristics.  To avoid issues around scale, current state-of-the-art practical algorithms are simple rule-based strategies which are tuned for improved performance.  In contrast, reinforcement learning is a powerful technique that can be used to learn near optimal policies.  Important to industrial models is the fact that the dynamics are near-predictable (i.e. the decision maker has additional knowledge on the transition distribution).  This is observed in elevator scheduling and bin-packing, where the demands are essentially the only unknown in the problem formulation.  Additional information like this can be used to obtain improved performance.  At a high level, the authors first present these problems as a two-stage MDP, allowing the algorithm to reduce state transition uncertainty.  They then design DACC, a framework for learning input dynamics and making decision with guidance of problem-specific rules to satisfy constraints.  The authors then complement these results with numerical experiments on bin packing and elevator scheduling problems.

More concretely - the authors first state the two-stage MDP model.  In this model the transition can be decomposed into two stages: state-dependent stage and input-dependent stage.  In the first stage, the algorithm picks action $a_t$ based on current arrival $d_t$, leading to a deterministic next state.  In the second stage, the new arrival $d_{t+1}$ is sampled according to a state-independent distribution.

Based on this model, the authors note that the value function decomposes over the exogenous arrival $d_t$ (although this is never proved explicitly, and the authors do not give a general MDP formulation for their model). To complement this reduction they propose a framework for learning the value function estimates (and hence a policy) over this decomposition.

## Questions
- Should it be $F_{t+1}$ and $F_t$ in equation 1?
- Do you believe that learning a good-enough predictor of the demand sequence is reasonable in these settings? See "Protean: VM Allocation Service at Scale" which highlights how demand patterns are highly correlated and impossible to predict.
- What is the justification for "online interaction" in the training versus a historical dataset?

## Related Work
The authors should consider the following papers and how they fit into their model and discussions:
- "OR-Gym: A Reinforcement Learning Library for Operations Research Problems" - considers impact of "action masking" on learning a policy.  The authors seem to suggest that the fact that the problems have constraints make RL in input driven environments more complicated.  However, the constraints that they consider are just the fact that the feasible actions are state-dependent.  Simple action masking (as the authors do in practice) seems to alleviate that issue.
- "Hindsight Learning for MDPs with Exogenous Inputs" - provides a general model for input driven MDPs similar to one studied here - and highlights a decomposition of the value function over the exogenous demand variables (similar to Equation 2)
- "Sample-Efficient Reinforcement Learning in the Presence of Exogenous Information" - Considers RL with extra exogenous information
- "Learning Compact Models for Planning with Exogenous Processes" - similar algorithmic framework for learning a mask in input driven MDP models
- "Markov Decision Processes with Exogenous Variables" - value function decomposition
- "Variance Reduction for Reinforcement Learning in Input-Driven Environments" - This paper was already cited but the authors should include a detailed distinction between their MDP model and the one considered here
- 
strength_and_weaknesses: ## Strengths
1. The authors consider an important problem of learning to plan and schedule with empirical results in bin packing and elevator scheduling.

## Weaknesses
1. The writing needs to be improved in order for the main points of the paper to be better described.
2. There are no theoretical justification for the advantages of considering the two-stage model over just the naïve model
3. The related work is insufficient, and the authors do not connect their model to other existing works in the area
clarity,_quality,_novelty_and_reproducibility: ## Quality and Novelty

The authors provide a novel algorithmic framework for understanding reinforcement learning in these input-driven models with empirical results in bin packing and elevator scheduling problems.  However, the model is well studied and the authors provide no new theoretical justification of their algorithm.

## Clarity

The submission is poorly written. The authors should read through the text to fix the mistakes and typos.  For some high level writing comments:
- The "two-stage" model is not described in the introduction
- In the introduction it is mentioned that the demand is highly-predictable and non-stationary, but then later mention that it is near predictable.  This distinction makes sense later on once the model is described - but could be included earlier
- Section 3.1 needs to include a full description of the underlying MDP - i.e. defining $S,A,P$ in this model.  The full transition distribution is never fully described, and the value functions are introduced without ***any*** context
- The discussion on page 4 frequently refers to "bias" but this is never described mathematically.  The remarks that are broken out are also not technical or theoretically justified.

And some minor corrections:
- "especially" twice in the abstract
- "behind GPU computing" unclear in abstract
- "two-stage" MDP in abstract not defined
- "focuses" twice in first paragraph
- "zheng" first paragraph citation
- "decision algorithms" first paragraph
- Last sentence on top of page 3 is unclear
- space after $f_t(d)$ on bottom of page 3
- Paragraph before section 3.2 is unclear
- $i_t$ not defined in 4.1.1
- "set other" in page 7
- $h_t$ space on page 8
summary_of_the_review: The authors omit a detailed discussion on the relationship of the proposed research to prior work on exogenous decomposition in MDPs.  Moreover, the writing quality and clarity needs to be improved.  The authors provide no theoretical justification for the benefits of the two-stage decomposition of the process (and similar ideas have been presented in prior work).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an RL framework motivated by many industrial applications, where we can isolate the stochasticity arising from state-independent distribution. The experiments were done on two simulated industrial tasks, bin packing and elevator dispatching.

The manuscript has improved considerably after the initial review phase. Unfortunately, it seems that the paper is essentially re-discovering a classical technique in RL. The proposed formulation seems essentially the same as the afterstate representation (Sutton & Barto the RL book) or the post-decision state representation (Powell 2007). How the proposed framework is different or related to this classical technique should be adequately addressed through literature review.

justification_for_why_not_higher_score: The proposed formulation is essentially a classical technique for representation / modeling in RL, but unfortunately the authors do not seem to know this.
Decision: Reject