Towards Solving Industrial Sequential Decision-making Tasks under Near-predictable Dynamics via Reinforcement Learning: an Implicit Corrective Value Estimation Approach
Authors: Jianyong Yuan
Jiayi Zhang
Junchi Yan
Venue: Submitted to ICLR 2023 (2023)

Abstract: Learning to plan and schedule is receiving increasing attention for industrial decision-making tasks for its potential for outperforming heuristics, especially under dynamic uncertainty, as well as its efficiency in problem-solving, especially with the adoption of neural networks and the behind GPU computing. Naturally, reinforcement learning (RL) with the Markov decision process (MDP) becomes a popular paradigm. Rather than handling the near-stationary environments like Atari games or the opposite case for open world dynamics with high uncertainty. In this paper, we aim to devise a tailored RL-based approach for the setting in the between: the near-predictable dynamics which often hold in many industrial applications, e.g., elevator scheduling and bin packing, as empirical case studies tested in this paper. We formulate a two-stage MDP by decoupling the data dynamics from the industrial environment. Specifically, we design a bi-critic framework for estimating the state value in stages according to the two-stage MDP.

Under review as a conference paper at ICLR 2023
REINFORCEMENT LEARNING OF INDUSTRIAL SE- QUENTIAL DECISION-MAKING TASKS UNDER NEAR- PREDICTABLE DYNAMICS: A BI-CRITIC VARIANCE REDUCTION APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Learning to plan and schedule is receiving increasing attention for industrial decision-making tasks (partly) for its potential to outperform heuristics, especially under dynamic uncertainty, as well as its efficiency in problem-solving, particu- larly with the adoption of neural networks and the behind GPU computing. Nat- urally, reinforcement learning (RL) with the Markov decision process (MDP) be- comes popular paradigm. Instead of handling the near- near-stationary environments like Atari games or the opposite case for open world dynamics with high uncer- tainty, in this paper, aim to devise tailored RL--based approach for the practice setting in the between: the prar-predictable dynamics which often hold many industrial applications, e.g., elevator scheduling and bin packing, as two empiri- cal case studies investigated in this paper. Specifically, we propose two-stage MDP to decouple the state transition uncertainty caused by the data dynamics and constrained action space in the industrial environment. A bi-critic framework is then devised for amortizing the uncertainty and reducing the variance of value estimation according the two-stage MDP. Experimental results show that our engine can adaptively handle different dynamics data tasks and outperform recent learning- bearning-based based models and traditional heuristic algorithmss.
INTRODUCTION
The advent of Industry 4.0 has put forward demanding requirements for resolving the sequential decision-making task in the industry. The task that involves planning and scheduling has been researched for decades for its commercial value. The planning task, like the bin packing problem (BPP) (Zhao et al., 2020; Zhu et al., 2021; Duan et al., 2022; Zhao et al., 2022; Zhao & Xu, 2022), involves series of discrete objects under certain constraints optimize an objective function. While the scheduling task focuses on allocating limited resources to multiple objects to optimize performance indicators under certain constraints, such the elevator group scheduling problem (EGSP) (Crites & Barto, 1998; Zheng et al., 2013; Wei et al., 2020), the vehicle routing problem (VRP) (Nazari al., 2018), and the job scheduling problem (JSP) (Chen & Tian, 2019).
With the increase of the problem scale and variety, methods for finding the optimal solution with effectiveness and joint applicability are becoming more and more attractive yet challenging. Tradi- tional solutions in the industry are often rule-based, tuning a score function with expert experience for specific tasks but can hardly be generalized to others. Others try to formulate the tasks as combi- natorial optimization problems and then apply heuristics algorithms (often due to its NP-hardness) - -such as search algorithm (ELA, 2019; TUR, 2020) and greedy algorithm (Ramalingam al., 2017)-but lack real-time response and scalability. Many learning-bass based methods (Wei et al., 2020; Zhao et al., 2020) are developed, e.g., RL showing its remarkable advantages in sequential decision- making problems. However, emerging learning- based works still often fall behind the industry stan- dards, which can be partly attributed to the lack of real-world training data, and the unavailability of strong simulators to provide rich and realistic data for training.
We consider two aspects for addressing industrial sequential decision fecision-making tasks with RL. The first and mainly addressed issue in work is to better utilize the character of the environment

---

Under review as a conference paper at ICLR 2023
dynamics in the industrial pipeline. Existing efforts (Hadoux et al., 2014; Chandak et al., 2020; Chen et al., 2021) assume general non-stationary environments and develop classical RL algorithms to learn structural features of the environment dynamics, including Meta-RL (Xu et al., 2020; Chen et al., 2021) and context detection RI. (Padakandla et al., 2020). However, we argue that, in fact, the environment often bears some inherent regularities, and sometimes it is nearly predictable, e-g.- in BPP cases, items of similar shape and size usually appear in a batch. The above existing works neglect such potential near-predictable dynamics and leave space for more tailored algorithmic de- velopment. Another practical aspect is to strictly obey hard constraints, which are common in the industry for specific reasons, e.g., safety. Although Chen et al. (2021) and Wei & Luo (2021) further consider the Constrained Markov Decision Process (CMDP) (Altman, 1999) and robust constrained Markov decision process (RCMDP) (Russel et al., 2020) for safe RL (Hewing et al., 2020), they tolerate constraints violations and are not up to industry standard. In particular, enforcing hard con- straint often increases the high state transition uncertainty (Mao et al., 2018) and further leads to the high variance problem of value estimation. Thus, we argue that the near- near- pear-predicttbblity must be more carefully considered to mitigate the challenge of the hard constraints.
In this paper, we propose Dynamic-Awand Aware and Constraints-Confined (DACC) RL framework for industrial sequential secision-making tasks. Unlike previous RL-based efforts for industrial cases that formulate the problem as non-stationary CMDP, we first identify the non-stationary but near- predictable environmental dynamics and reformulate these tasks as a two-stage MDP (Kim et al., 2019) for its potential to distinguish the effects of environment dynamics (exogenous variables) and constrained action space (endogenous variables) in the state transition. Furthermore, the value estimation based on a two-stage MDP reduces the variance of value estimates without introducing bias, as proved by (Mao et al., 2019). Specifically, we design DACC, bi-critic framework for perceiving the dynamics and making decisions under hard industrial constraints with the guidance of heuristic rules, respectively. By estimating the state value in two stages with our bi-critic framework, we reduce the state transition uncertainty and state value estimation variance caused by the mutually adverse effects of dynamic variability and hard constraints. To evaluate our method's effectiveness and generalization, we conduct experiments on two typical industrial sequential decision tasks: 3D bin packing and elevator group scheduling. For the latter case for which there lacks realistic simulator, we improve the open-source simulator by adding more constraints, business rules, and logic, and (will) release more realistic one based our first-hand engagement with top-tier lift manufacturer to benefit the community. The highlights of this paper are:
1) Though many sequential decision-making tasks in the industry often require strict constraints, increasing the high state transition uncertainty to challenge RL-based methods. Fortunately, in this paper, we identify that in many cases, the environment often pear-predictable such that allows for more tailored MDP model development, which is largely ignored by existing methods.
2) We innovatively separate the state transition process of these industrial tasks into two stages. We derive theoretical solutions embodied by two-stage MDP to the high variance problem of value estimation appearing in the single-stage settings. We then propose a bi-critic framework called Dynamic-Aware and Constraint-Confined (DACC), to capture the regularity of dynamics and makes decisions under hard industrial constraints.
3) We apply our framework to two represennative yet challenging real-world cases: 3D bin packing and elevator group scheduling problems. Results show that our methods outperform conventional rule-based and state-of-the-art learning-based models. Further comparisons with the Meta-RL meth- ods verify our framework's superiority in capturing the inherent regularities in these dynamic indus- trial scenes. Generalization experiments on the untrained data show that the model generalizes well.
2 RELATED WORK
Many works extend the Markov decision process model. Constrained Markov decision process (CMDP) (Altman, 1999) is suitable for constrained physical systems, such as avoiding obstacles or unsafe parts in space. Robust Markov decision process (RMDP) Petrik & Russel (2019) is suitable for scenarios where transition probabilities or rewards are unclear. And robust constrained Markov decision process (RCMDP) (Russel et al., 2020) merges both CMDP and RMDP. Time- -eee-nepnddn MDP (TMDP) (Boyan & Littman, 2000) considers both stochastic state transitions and stochastic, dime-dependent action duration. A two-stage MDP task is designed by (Kim et al., 2019) to differ- entiate the effects of state transition uncertainty and state-space complexity the brain's arbitration
2

---

Under review as a conference paper at ICLR 2023
between model-based and model-free learning. In this paper, process state transition caused by exogenous (environmental dynamics) and endogenous variables in two stages. RL algorithms are recently devised for environmental dynamics, such as Meta-RL (learn to learn) (Finn et al., 2017; Nagabandi et al., 2018; Xu et al., 2020) and context detection. Meta-RI. learns numerous tasks and acquires prior knowledge learn new tasks faster. The popular Meta-RI. models model- based methods suitable for mainly state-dependent simulation scenarios like maze games rather than input-dependent scenarios to some extent. The context detection RL methods (da Silva al., 2006; Padakandla et al., 2020; de Oliveira et al., 2006) assess whether the MDP functions have changed based on sequential observations, creating, updating, and selecting one among several partial models of the environment. Whether to generate new context depends on expert experience, and the model consumes much memory. CASRL (Chen et al., 2021) considers both meta and context-detection. Compared with these works for general dynamics, our efforts are devoted to a tailored approach to better capture the industry dynamics, which are often near predictable.
PROPOSED APPROACH
We propose the Dynamic- Aware and Constraint-Confinel (DACC) RL framework for industrial sequential decision-making tasks under the typical setting: near-predictable dynamics and hard con- straints. We first reformulate it as a two-stage MDP and show its inherent properties of reducing state transition uncertainty in $ 3.1. In $ 3.2, we design our bi-critic framework based on our refor- mulation, which captures the near-predictable dynamies and confines the hard constraints.
3.1 REFORMULATION AS TWO-STAGE MDP
Preliminaries. Many industrial decision makinn tasks can be described as a two-stage process: first, the environment generates new input for decision-making (e.g., new-coming item to be placed in BPP, a new-coming passenger to be assigned an elevator in EGSP); second, the RI. agent takes plamning/scheduling actions under real-world constraints for the new input.
Some Remarks. As the new input's generation is stochastic while the effect of an action on the state deterministic after the input's generation, the significant uncertainty of environment dynamics falls into the first stage (i.e., the stochastic generation of the input). The second stage focuses on the uncertainty caused by the restricted action space according to sources of uncertainty during the state transition process pose significant challenges of high variance of value estimation (explained in Appendix C) for existing RI. algorithms, which formulate these tasks as single-stage MDPs and mix the two sources of uncertainty together. To avoid this problem, we reformulate them as two-stage MDPs and process the two sources of uncertainty separately.
hard constraints. These two
Notations. We define a two-stage MDP for industrial sequential decision-making tasks as tuple < SSD,, C, 7 >, where S is a finite set of states, is a set of the new coming inputs (e.g.. new coming items in BPP, new coming passengers in EGSP), fe(d) is the distribution of the new coming input at time t, A(aayss dz, C,) is a finite set of actions under constraints C,, r(5t, ae de) is the reward function, and 7 is the reward discount factor.
Corresponding to the two-stage process of industrial decision- -making tasks, we named the two stages MDP as the ieput-dependent stage and the -tate-dep stage, respectively. Without loss of generality, we use the bin packing problem to describe our approach, especially the two-stage state transition process in Fig. 1. In the input-dependent stage, the generation of new coming input d, fol- lows a certain time-varying item generating distribution fz(d), which usually presents task-related and near- prediccaiee patterns over time (i.e.. independent of the current state st). For instance, in many industrial BPP cases, items of similar shape and size usually appear in a batch, thus the distance between (d) and fe+1(d) small, and the environment dynamics are vear-predie Hence, the state transition probability of this stage is
P (St, de I st, f(dd) = P (de (d)) = f((d)
(1)
In the state-dependent stage, the constraint C, is calculated based on the current state & (representing the placement state of previous input items) and the pending current de. For instance, positions where de will topple over are forbidden. Then, the current de is picked to the position indicated by
the action a,, which the RL agent gives out according to (st, Ce, Ce. ft41:00), where fi41:00 is the features of future item generating distributions that are related to f+41(d), fet2(d), ..., Setoc (d).
3

---

Under review as a conference paper at ICLR 2023
fi(d)
Input- dependent
State-
fres (4)
dependent
0.1
stage
stage
0.1
fift
Constaints Co
box3
Generate de
Placed, atd,
box3
Figure The two-stage state transition process of the BPP. In the input-dependent stage, new coming item d, is generated with the current item generating distribution (d).. Here fi(d) denotes the generation probability of three types of items. In the state-dependent stage, the pending d, is picked to the position indicated by the action under hard constraints C,.
State dependent Stage
Env
Constraints Module
St
Actor
dt
State Encoder
ap
State Critic
V.
fr+1:00
take action a
Dynamics Aware
d
4-=T L-T-1
Input = dependent Stage
Env
Dynamics Encoder
Dynamics Critic
St+1
2
c+1
The proposed bi-critic framework. In the input-dependent stage, the dynamics critic focuses handling the value estimation based task-related and time-varying data. In the state- dependent stage, the state critic focuses on handling the value estimation depending on security constraints. And the actor makes decisions based on the learned features.
Figure 2:
The transition process of this stage is deterministic such that:
if 0e leads 8r4-1 11,
f(+1()) st. de, ax)
0, otherwise
3.2 BI-CRITIC FRAMEWORK FOR RL UNDER NEAR- PRAR-PREDICTABL ENVIRONMENTS
Based on the reformulation, we design dynamic- and constraints constraints-confine (DACC) reinforce- learning framework for industrial sequential decision-making tasks under pradiccedieealee dy- namics. Dealing with the two challenges in two stages separately, we overcome the high variance value estimation problem caused by two sources of state transition uncertainty.
Value Functions. In actor-critic-basee algorithms (Sutton et al., 1999), for a policy æ II. the value function is defined as:
Vi(5) = E
Σ r(Sr ar ds
720
where the expectation is taken over both the randomness of in policy and the input d (de [e(d))
Overall Framework. The overall framework is shown in Fig. 2. For clarity, we plot the state- dependent stage of the current state transition and the input-dependent stage of the next state transi- tion. We use two critic networks V* and / to estimate the state value for the state-dependent stage and input-dependee tent features fr+1:0 of future data with historical data d,, d,-1, dr-r, where T is the length of historical data used to predict ft Then, for the -tateddependent stage, we use the state encoder encode (sc, de, Je+1:00) and output latent vector (denoted 2i), which is multidimensional
stage, respectively. We first use the dynamics aware module to extract the la-
4

---

Under review as a conference paper at ICLR 2023
Gaussian distribution. After that, estimate the current state value Vi² with the state critic network and use the actor network to calculate the action vector T T z2) under the hard constraints of indus- trial tasks. For the input-dependent stage, use the dynamics encoder to extract input-dependent features 4 (similar to zf) from 8+41 and J1+1:00- Then we use the dynamics critic network to es- timate the input-dependent state value V/441. We can readily resort to off-the-shelf actor-critic-based algorithms (e.g., A2C as used in our experiments for its simplicity and effectiveness, or others like A3C (Mnih et al., 2016), PPO (Schulman et al., 2017)) to train our model. With the derivation of variance reduction shown in Appendix D, we calculate the advantage function of A2C as
A, aT, (V f(+1=e)) (* de f(+1:0))
(3)
From the state transition probability of the iaput-dependent stage (shown in Eq. 1), we get that after both critic networks converge, the relationship between V and V* is as follows:
(s ft:00) = Σ
fu+1:00)
(4)
To achieve this and reduce the variance caused by the uncertainty of inputs, we introduce an addi- tional loss function L to minimize the Kullhack-Leibh Divergence between 2/41 and zi:
2
2410 + (o2,0)) + (Mi.o) - (2411,0)) 2.
LKE
log
(5)
2(0'+1.00)
2
where () and (o) represent the jth component of the mean vector and variance vector of zú. respectively, while 1(+1.0)) and (of (0(+1.0))² represent those of zi+1. respectively, and l: is the dimension of oi and 2/+1- Thus the overall loss function is defined as
== A,2 A, log ((ae) 2 + x(a (z,)) log x (a, |2, + LKL
(6)
The overall training algorithm is shown in Alg. Appendix
Dynamics Aware Module. To capture the trend of data and learn the features of the marginal distribution of future data, we propose a dynamics aware module, which takes into the historical data dz-r, de-T+1, d, and outputs the feature vector ft+1: As the distribution of data (d) is independent of the current state, we can pre-train the dynamics aware module beforehand. feasible alternative is to build the dynamics aware module with long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) or Attention (Vaswani et al., 2017) networks and set a pre-training task using the last T data predict the next three data. For instance, de-T, di-T+1, ... d--1 to the dynamics aware module and make it predict the de, di+1, de42- After that, we the pre-trained dynamics aware module to output the hidden feature vector f141:00- The detailed pre-training algorithm is shown in Alg. in Appendix E.
input the data sequence of
Constraints and Rules. To better satisfy strict constraints and make of task-related heuristic rules in industrial applications, we design constraints module and heuristic-based reward shaping. For constraints, the technique is the hard action mask. Concretely, it uses task-related constraints to evaluate the feasibility and quality of each action in the current state and the pending input. With its output action mask M, we then get an adjusted policy function: n' + T(alzz M. By ex- plicitly changing the action probability with task melated constraints, we avoid excessive exploration of inefficient or illegal actions (e.g., actions that violate constraints) without introducing excessive training complexity. In practical usage, we can set the chosen probability of illegal actions close to zero to enforce the security constraints. Detailed settings of this module for the two industrial cases can be found in $ 4. The other technique use is heuristic-based reward shaping for its advantage of amortizing the complexity of long-term credit assignment and leveraging heuristic algorithms to al., 2021). Heuristic functions that estimate the expected state value are ubiquitous in industrial scenarios. For example, we can use the ETA algorithm to estimate the average waiting time of existing passengers as the heuristic state value h((s) in EGSP. With such heuristic functions, we can introduce heuristie guidance to the RL agent by changing the rewards and lowering the discount factor of the original MDP:
accelerate RL's training(Chee)
(s d, a) re(s, d, a) + (L-A)yBwj.ad Ih(s)) and 5 ;= A7,
where À E [0,1] 1] is the mixing coefficient that increases iteratively along the training process.
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS WITH TWo CASE STUDIES
We show the implementations and experiments on two real-world NP-hard industrial scenarios: the bin packing problem (BPP) in $ 4.1, and the elevator group scheduling problem (EGSP) in $ 4.2.
4.1 3D BIN PACKING PROBLEM
Problem Settings. The 3D bin packing problem is the essence of mixed palletizing in logistics and warchousing, For a given set of items and containers of a fixed size, the goal is to find the assembly method with the highest space utilization and the least number of containers required. Here we consider the online scenario. Given sequence of rectangular items I with size x; X y; X 2; (for each í E I) and standardized pallet with size L X W and maximum height limit H, the palletization agent, e.g., robot, only sees the upcoming item and selects position to place the item. Environment Dynamics. Elhedhli et al. (2019) analyzes the actual industrial data and finds that the characteristics of depth/width ratio, height/width ratio, and repetition follow specific distributions. The average item volume of different industries and categories is different.
Realistic Constraints. The placement process is subject to the following constraints as widely used et al., 2013; Zhao et al., 2022): i) no space overlap between items on the pallet; ii) while sliding new item to the assigned position, no collision with the items already placed: iii) all placed items are physical of stability.
in literature
4.1.1 IMPLEMENTATION COMPONENTS
MDP. Specifically, the state is represented by the pallet's height, 5t = H., The action is the position selection for item placing. An in- valid action mask M, filters out feasible posi- tions and guides exploration. In line with the standard practice in literature, the reward ry for each successful placing step is set proportional to an X., Xz, ExW*H H. and Ti = 0 if the placement fails.
(a)
(b)
Network Architecture We use attention CNN (CBAM) (Woo et al., 2018) state encoder, long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) as dynamics aware module, and fully-connected networks (FCN) as Critic net and Actor net. Considering that the representation of the pallet is fine-graines and the state dimension is relatively large, we lever- age the attention mechanism to increase focus on key regions. The input of the pallet state is height map H, E F. And an action mask M, represents security constraints for each grid. the item rotate (two rotations: zyz and yæ.2). Since the action mask calculation takes extended time, we decouple the action space into multiple sub-regions to reduce mask calculation. Details of the hierarchical architecture are given in Appendix F. We use an invalid action mask to guarantee two constraints (stability and collision avoidance), a corner point rule to speed up the computation of the invalid action mask and guide the placement of items, and a suitable heuristic to guide placement and speed up algorithm convergence. For stability, we use the convex hull and maintain a solid mask. As Fig. 3(a) and 3(b) show, when placing the yellow item, the centroid of the yellow item must fall within the convex hull of the bottom surface in contact with the lower item. To avoid bad cases like Fig. 3(c) (satisfy that the centroid falls within the convex hull of the contact base but is unstable). we also maintain solid global mask. Only the intersection of the convex hull and the supporting surface is set to solid, and the item centroid should fall within the solid mask. To avoid collisions, the placement process is to push the item laterally to a particular position so that even if a collision occurs, it will only cause a slight movement of adjacent items. The four sliding directions for selection are shown in Fig. 3(d), including left-back to right-front, left-front to right- right-front to left-back, and right-back to left-front. To speed up the invalid action mask calculation, in terms of rules, when selecting actions,
(c)
(d)
Figure 3: Constraints satisfaction for the BPP
task. (a) and (b) shows the convex hull calcula- tion, (c) shows bad case resulting from consider- ing only the convex hull constraint, and (d) shows the four sliding directions we defined.
The size of H, and M, is doubled while allowing
Constraints and Rules.
6

---

Under review as a conference paper at ICLR 2023
emphasize the selection of corner points, increase the weight of corner points and only consider sub-regions with corner points, which can significantly reduce the number of solutions, speed up action mask calculation, and improve space utilization (Crainic al., 2008; Martello et al., 2000; CRA, 2009). To deal with heuristic-hased reward shaping, we select the hest-performing heuristic on the dataset from four heuristics in the compared baselines guide the RI. agent.
4.1.2 EXPERIMENTAL RESULTS
Datasets and Hyperparameters. We test our model two datasets generated differently and containing multiple subsets with different distributions. Pybullet, a physics engine that supports 3D) collision detection, has verified the test results. Mixed-item Dataset (MI Dataset) is a collection of mixed items with a great variety and few identical items. The data generation scheme follows the realistic 3D-BPP instance generator proposed in (Elhedhli al., 2019). The training set has 10 million items, with 36,294 species, and occurrences vary from to 7,037. The testing set has 10 thousand items, with 4,886 species, and occurrences vary from 1 to 15. We set pallet dimensions to the size often used in practice: L = 120, W 100, and H = 100, allowing two packing rotation directions. Large-item Dataset (LI Dataset) is a collection of large items and comes from (Martello et al., 2000), including five classes from (Martello & Vigo, 1998) and three class from (Berkey & Wang. 1987). The pallct dimensions are set to packing rotation directions. The training set has 39,500 items, with 24,889 species, and occurrences vary from to 12. The testing set has 500 items, with 401 species total, and occurrences vary from to 4. We initialize λo 0.95 in Eq. and set = 1, 1 in Eq. 3 and learning rate 1) 1 x 10-5
100, W = 100, and H = 100, allowing two
Compared Baselines. We compare our ap- proach with four heuristics and two learning- based methods, presented in Appendix A. For heuristics, we compare with four classic heuristics, which select free spaces represented by Empty Maximum Space (EMS): Bottom- Left heuristic (BL) (Chazelle, 1983), Deepest- Bottom-Left with Fill heuristic (DBLF) (Kang et al., 2012), Best Match First Packing heuristic (BMF) (Li & Zhang, 2015). Online BPH (Ha et al., 2017). As for learning methods, compare with an outstanding RL BPP model PCT (Zhao & Xu, 2022) in continuous setting and a model-based Meta-RL model CASRL (Chen al., 2021). Besides, model A2C (Mnih et al., 2016) (with rules), A2C (w/o rules) and DACC (w/o rules) are for ablation study; A2C represents DACC without bi-critic archi- tecture and (w/o rules) means no rule guidance.
60%
Space utility
(ith neea!
BACC
CASRL
30%
20%
10000
200CO Episodes
30030
40000
50000
Figure 4: Training rewards of learning-based
models on MI Dataset for 3D bin packing.
Result. Table shows online bin packing results on two representative datasets. Our model performs well on both mixed-item and large-item datasets, with the flexibility to adapt to different pallet sizes and high-precision placement requirements. Our model achieves higher space utility rates than traditional rule-based algorithms, RL-based models, and the meta-RL model. For learning- based models, the reward curves over training episodes are shown in Fig. 4, whereby the confidence interval is 85. As can be seen from the Fig. 4, our model converges faster than other learning models.
Ablation Study. Compared to A2C (without bi-critic structure) without rules, the A2C with rules, and DACC without rules, DACC can converge to stable performance earlier with higher reward, which means the bi-critic structure and the rule guidance are beneficial. After adding rules (in constraints and rewards). the superiority of DACC over A2C is further enlarged, which means that DACC can better adapt to rule guidance.
4.2 ELEVATOR GROUP SCHEDULING PROBLEM
Problem Setting, The elevator group scheduling problem (EGSP) mainly studies assigning ele- vators respond to flood of passengers in buildings equipped with multiple elevators. Among various EGSP systems, we consider a prevalent one: at every time step, stochastic passengers ap- pear and give hall call requests with up/down directions; the system assigns elevator for every
7

---

Under review as a conference paper at ICLR 2023
Table 1: Performance comparison between the baselines and DACC in MI Dataset and LI Dataset.
Dataset
LI Dataset
Method
Space utility
Avg. items
Space utility 1
Avg. items
BL (Chazelle, 1983)
22.55%
9.32
27.97%
2.04
DBLF (Kang et al., 2012)
26.54%
10.51
27.18%
1.98
BMF (Li Zhang, 2015)
29.67%
11.47
27.28%
1.99
Online BPH (Ha et al., 2017)
24.56%
10.82
24.77%
CASRL (Chen ct al., 2021)
11.18%
3.75
17.92%
2.22
PCT-continues (Zhao & Xu, 2022)
56, 36.59%
19.50
41.18%
3,22
A2C (w/o rules) (Mnih et al., 2016)
48.89%
16.53
37.04%
2.97
A2C (with rules) (Mnih al., 2016)
55.24%
19.15
40.77%
3.11
DACC (w/o rules)
54.55%
18.40
38.71%
3.03
DACC
58.37%
19.75
43.49%
3.25
hall call request without knowing the passengers' destination floors until they get the elevator; also, passengers do not know the assigned elevators so that they can be reassigned at any time.
Environment Dynamics. Passenger flow in the EGSP often shows specific traffic daily patterns, which can be divided into three major periods (Jansson & Uggla Lingvall, 2015): up-peak traffic in morning rush hours, two-way traffic at lunchtime, and down-peak traffic in evening rush hours.
There are some commonly-used constraints for the elevator assignment and movement. I) The EGSP system is required dispatch exactly one elevator for every hall call request; 2) The assigned elevator must respond to the hall call request while completing the car calls generated by passengers on car call indicates the passenger's 's destination floor and is registered at the elevator after the passenger enters it; 3) The elevator can not change its service direction until there is no hall call or car call the current direction.
Realistic Constraints.
4.2.1 IMPLEMENTATIO COMPONENTS
Mull-HHad Attention
Simulator: To mimic the real-world behav- ior of EGSP, based on the only available open- source simulator Liftsim (Wang et al., 2020), we further improve it into a more realistic EGSP simulator that satisfies the three realis- tic constraints mentioned in the last paragraph. Additionally, our simulator can simulate pas- senger flow following the three traffic patterns. Detailed comparisons among existing simula- tors are explained in Appendix J.
GCN
FCN
GCN
Up
Constralnts Mocule
Figure 5: DACC of EGSP. We use multi-head attention network to extract features from histor- ical hall call data, graph convolutional networks (GCN) state and dynamics encoders, and fully- connected networks (FCN) as critics and the actor.
MDP. The simulator formulates EGSP as two-stage MDP: in every simulation time step, new passengers are generated with a probabil- ity matrix corresponding to the specific pattern; the agent receives the state information from the simulator and makes a decision to assign elevators for existing unprocessed hall calls. As shown in Fig. 4.2, the state includes the elevator state 8, and the hall call state h4 The elevator state includes the position, speed, service direction, door status, and registered car call of all elevators. And the hall call state includes the unprocessed requests (marked orange in Fig. 4.2), the existing hall call requests, and their corresponding assigned eleva- tors. The action is defined as assigning an elevator for each unprocessed hall call, and the objective is to reduce the average waiting time (denoted as Taw) and the average transmitting time (denoted as Tat) of all passengers. At the same time save energy consumption. Here the waiting time passenger is the time taken from the moment he presses the hall call button on a floor to the instant he picked up. The transmitting time is the time taken from when one is picked up to the instant one reaches the destination floor. Thus the reward function is defined
= - (3 + E|N.1 + E),
where N₁. N₂ is the number of waiting passengers in the hall, and those in the elevators, respectively, weighted by hyperparameter B, E. F. is the energy consumption measured by electricity cost.
8

---

Under review as a conference paper at ICLR 2023
Table Evaluation of the baselines and DACC in three traffic patterns in terms of two criteria: Tas
(the sum of Taw and Tau) and T'' (a criteria that integrates Tos and energy consumption E).
Two-way Taxt
Up-peak Tast
Down-peak
Method
TEL
TE4
Taxt
TE
ETA (Rong et al., 2003)
92.50
110.98
119.83
142.47
65.30
70,50
SFM (Ramalingam et al., 2017)
90.99
110.67
119.27
141.98
70.63
75.37
Robert (Crites & Barto, 1998)
159.31
261.70
126,70
247.87
177.42
209.21
DRL-EGC (Wei et al., 2020)
112.93
134.89
123.60
151.39
82.21
86,40
Context-QL (Padakandla et al., 2020)
91.14
111.18
114.89
137.03
66.34
71.82
DACC
89.75
108.14
110.21
132.15
65.83
70.35
Network Architecture. As shown in Fig. 4.2, we use a multi-head attention network (Vaswani et al., 2017) as the dynamics aware module to capture the trend of passenger flow and learn the hidden features from historical hall call data. We record the number of generated hall calls on each floor every minute during the whole simulation process. For the current time step t, we forward the collected hall call data in the last 30 minutes to the attention network and use a two-layer FC network output the latent feature vector distribution ft+ 1:00 We use two four-layer graph convolutional networks (GCN) (Kipf& & Welling, 2016) as the state encoder and the dynamics encoder, respectively. And we use two-layer FC networks as the critic networks and the nctor-network, respectively.
Constraints and Rules. We develop an action mask function to avoid choosing actions that violate the realistic constraints mentioned above, And we incorporate the estimated time of arrival algorithm (ETA) as heuristic rule to guide the exploration of the RL agent, which can reduce the occurrence of inefficient actions such as assigning an elevator running oppositely to a hall call.
4.2.2 EXPERIMENTAL RESULTS
Dataset and dyperparameters. We use passenger flow data in 30 days respectively for two-way traffic, up-peak, and down-peak patterns, provided by a world-renowned elevator manufacturer, from a real-world fo-floor and 4-elevator building. with nearly 50 persons residing on each floor except the first floor. For each pattern, we calculate probability matrix P, that represents the generating probability of passenger who appears at floor i and targets floor j at time / and use P, to generate passengers randomly. We initialize Åg = 0.95 in Eq. 7 and set a = 0.01, 7 = 0.99 in Eq. 3, B = 1, & 0.6 in the reward Eq. 8, and the learning rate as 17 2 10
Compared Baselines. We compare our model with traditional EGSP algorithms, including Round- Robin, estimated time of arrival (ETA) (Rong et al., 2003), and genetic algorithm (GA), among which ETA empirically achieves the best performances in most cases. The SFM (Ramalingam et al., 2017) algorithm for EGSP is combinatorial optimization-baxt by commercial elevator companies. Learning-bai & Barto, 1998) (using Q-Learning) and DRL-EGC (Wei et al., 2020) (using the A3C (Mnih al., 2016) algorithm). Additionally, we train a context detection method for non-stationary environments Context Q-learning (Padakandla et al., 2020) in our environment as bascline.
algorithm known to be adopted algorithms for EGSP include Robert (Crites
Results. As shown in Table 2, our framework outperforms all existing methods for EGSP in all of the three traffic patterns terms of the TE criteria, which takes the average waiting time Tats the average transmitting time Ta and energy consumption E into consideration. For the Ta criteria, which only considers the average waiting time and transmitting time, our framework outperforms all existing methods in the two-wat traffic and up-peak patterns and achieves comparable performance with the best algorithm in the dn- dn-peak pattern.
5 CONCLUSION
Identifying the pradtpredictability of many industrial dynamic environments, we have devised a tailored RL-based framework using bi-critic to handle the typical industrial sequential decision- making tasks with two successful case studies for 3D bin packing and elevator group scheduling. We also develop a more realistic simulator compared with current open-source simulators.
Limitation & Future Work. As our approach is tailored to the near predictaaee setting, it may fail in cases when the dynamics are more uncertain, as shown in our failure case study in Appendix I.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ts2pack: A two-level tabu search for the three-dimensional bin packing problem. European Journal of Operational Research, 195(3):744-766) 2009. ISSN 0377-2217.
A modified squirrel search algorithm based on improved best fit heuristic and operator strategy for bin packing problem. Applied Soft Computing, 82:105565, 2019. ISSN 1568-4946.
Hyper-heuristic local search for combinatorial optimisation problems. Showledge-Basedd Systems, 205: 106264, 2020. ISSN 0950-7051.
Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
Judith O Berkey and Pearl Y Wang. Two-dimensional finite bin-packing algorithms. Journal of the operational research sociery, 38(5):423 429, 1987.
Justin Boyan and Michael Littman. Exact solutions to time-dependent mdps. In T. Leen, T. Diet- terich, and Tresp (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
Yash Chandak, Scott Jordan, Georgios Theocharous, Martha White, and Philip Thomas. Towards safe policy improvement for non-stationary mdps. Advances in Neural Information Processing Systems, 1591156-918,, 2020.
Bernard Chazelle. The bottomn-left bin-packing heuristic: An efficient implementation. IEEE Transactions on Computers, C-32:697-707, 1983.
Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Liang Li, and Ding Zhao. Context-aware safe reinforcement learning for non-stationary environments. In 2021 IEEE In- lernational Conference on Robotics and Automation (ICRA), pp. 10689-10695, 2021. doi:
10.1109//CRA4450062220 2021440060222..00 200.9066221119611555 99561593.
Xinyun Chen and Yuandong Tian. Learning to perform local rewriting for combinatorial optimiza- tion. In NeurlPS, 2019.
Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learn- ing. Advances in Neural Information Processing Systems, 34: 14411500-13563. 2021.
Rohan Chitnis and Tomás Lozano-Pésn -érez. Learning compact models for planning with exogenous processes. In Conference Robot Learning, pp. 813-822. PMLR, 2020.
Teodor Gabriel Crainic, Guido Perboli, and Roberto Tadei. Extreme point-based heuristics for three- dimensional bin packing. INFORMS Journal on Computing, 10:368-384, 08 2008
Robert H Crites and Andrew G Barto. Elevator group control using multiple reinforcement learning agents. Machine learning, 33(2):235-262, 1998.
Bruno C. da Silva, Eduardo W. Basso, Ana L. C. Bazzan, and Paulo M. Engel. Dealing with non- stationary environments using context detection. Proceedings of the 23rd International Con- ference Machine Learning, ICML '06. pp. 217-224, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832.
Denise de Oliveira, Ana LC Bazzan, Bruno Castro da Silva, Eduardo W Basso, Luis Nunes, Rosaldo Rossetti, Eugénio de Oliveira, Roberto da Silva, and Luis Lamb. Reinforcement learning based control of traffic lights in non-stationary environments: A case study microscopic simulator. In EUMAS, 2006.
Jiahui Duan, Xialiang Tong, Fei Ni, Zhenan He, Lei Chen, and Mingxuan Yuan. data-driven column generation algorithm for bin packing problem in manufacturing industry, 2022.
Samir Elhedhli, Fatma Gzara, and Burak Yildiz. ftrce-dimensional bin packing and mixed-case palletization. INFORMS Journal Optimization, 1(4):323-352, 2019.
10

---

Under review as a conference paper at ICLR 2023
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp- 1126-1135. PMLR, 06-11 Aug 2017.
Chi Trung Ha, Trung Thanh Nguyen, Lam Thu Bui, and Ran Wang. An online packing heuristic for the three-dimensional container loading problem in dynamic environments and the physical internet. In Giovanni Squillero and Kevin Sim (eds.), Applications of Evolutionary Computation, pp. 140-155, Cham, 2017. Springer International Publishing.
Emmanuel Hadoux, Aurélie Beynier, and Paul Weng. Sequential Decision-Making under Non- stationary Environments via Sequential Change-point Detection. In Learning over Multiple Con- texts (LMCE), Nancy, France, September 2014.
Lukas Hewing, Kim P Wabersich, Marcel Menner, and Melanie N Zeilinger. Learning-based model predictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3:269-296, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Anton Jansson and Kristoffer Uggla Lingvall. Elevator control using reinforcement learning to select strategy, 2015.
Kyungdaw Kang, Ilkyeong Moon, and Hongfeng Wang. A hybrid genetic algorithm with new packing strategy for the dimenssonll bin packing problem. Applied Mathematics and Com- putation, 219(3):127--1999 2012.
Dongjac Kim, Geon Yeong Park, Sang Wan Lee, et al. Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning. Nature communica- tions, 1011):1-14, 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609,02977 2016.
Xueping Li and Kaike Zhang. A hybrid differential evolution algorithm for multiple container loading problem with heterogeneous containers. Computers & Industrial Engineering, 90:305- 313, 2015.
Eunice Lôpez-Camachi Gabriela Ochoa, Hugo Terashima-Marinn and Edmund K Burke. An ef- fective heuristic for the two-dimensional irregular bin packing problem. Annals of Operations Research, 206( (00(())241-22)) 264, 2013.
Hongzi Mao, Shaileshh Bojja Venkatakrishnan, Malte Schwarzkopf, and Mohammad Al- izadeh. Variance reduction for reinforcement leamingg in input-driven environments. CoRR, abs//800702235 2018.
Hongzi Mao, Shaileshh Bojja Venkatakrishnan, Malte Schwarzkopf, and Mohammad Alizadeh. Variance reduction for reinforcement learning in input-driven environments. In International Conference on Learning Representations, 2019.
Silvano Martello and Daniele Vigo. Exact solution of the two-dimensional finite bin packing prob- lem. Management science, 44(3):388-39) 1998.
Silvano Martello, David Pisinger, and Daniele Vigo. The three-dimensional bin packing problem. Operations research, 48(2):256 2i0:22,66666,00 2000.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR

---

Under review as a conference paper at ICLR 2023
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning, 2018.
Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takác. Reinforcement learning for solving the vehicle routing problem. In NeurIPS, 2018.
Sindhu Padakandla, Prabuchandran KJ, and Shalabh Bhatnagar. Reinforcement learning algorithm for non-stationary envirommett.. Applied Intelligence, 50(1 10311)3330-36 2020.
Peter. Elevate 8: a commercial-level elevator simulator. https: http///wwww...
com/ iiwdinndxxx.ii php/ 8- wwweeeeeeeeeeeeeeeeeeeeeeeeeeeee 58 elevatee-.
Marek Petrik and Reazul Hasan Russel. Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Srikumar Ramalingam, Arvind U Raghunathan, and Daniel Nikovski. Submodular function max- imization for group elevator scheduling. In Twenty-Seventh International Conference on Auto- mated Planning and Scheduling, 2017.
Aiying Rong, Henri Hakonen, and Risto Lahdelma. Estimated time of arrival (ETA) based elevator group control algorithm with more accurate estimation. Turku Centre for Computer Science, 2003.
Reazul Hasan Russel, Mouhacine Benosman, and Jeroen Van Baar. Robust constrained- mdps: Soft-constrained robust policy optimization under model uncertainty. arXiv preprint arXiv:2010.04870, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Scan R Sinclair, Felipe Frujeri, Ching-An Cheng, and Adith Swaminathan. Hindsight learning for mdps with exogenous inputs. arXiv preprint arXiv:2207.06272 2022.
George R Strakosch. Vertical transportation: elevators and escalators. Wiley, 1983.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Fan Wang, Bo Zhou, Yunxiang Li, and Kejiao Li. Liftsim: configurable lightweight simulator of elevator systems. https: //gittttus Paanm/aaalleePe addle metagym/ liftsim, 2020.
Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Conference on Learning Theory, pp. 4300-4354. PMLR, 2021.
Qinglai Wei, Lingxiao Wang, Yu Liu, and Marios M. Polycarpou. Optimal elevator group control via deep asynchronous actor-critic learning. IEEE Transactions on Neural Networks and Learning Systems, 31(1)):2455-555 2020. doi: 10.1 N0.1109/NNNLSS.200222266
Sanghyun Woo, Jongchan Park, Joon- Lee, and In So Kweon. Cham: Convolutional block attention module. In ECCV, September 2018.
Mengdi Xu, Wenhao Ding, Jiacheng Zhu, ZUXIN LIU, Baiming Chen, and Ding Zhao. Task- agnostic online reinforcement learning with infinite mixture of gaussian processes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Infor- mation Processing Systems, volume 33, pp.
Curran Associates, Inc., 2020.
12

---

Under review as a conference paper at ICLR 2023
Hang Zhao and Kai Xu. Learning efficient online 3d bin packing packing configuration trees. In International Conference on Learning Representations, 2022.
Hang Zhao, Qijin She, Chenyang Zhu, Yin Yang, and Kai Xu. Online 3d bin packing with con- strained deep reinforcement learning. arXiv preprint arXiv:2006. 149788, 2020.
Hang Zhao, Chenyang Zhu, Xin Xu, Hui Huang, and Kai Xu. Learning practically feasible policies for online 3d bin packing. Science China Information Sciences, 65(1))1-17, 2022.
Liu Zheng, Shu Guang, and Dong Hui. Research of elevator group scheduling system based on reinforcement learning algorithm. Proceedings of 2013 2nd International Conference on Mea- surement, Information and Control, volume 01, pp. 606-610, 2013.
Qianwen Zhu, Xihan Li, Zihan Zhang, Zhixing Luo, Xialiang Tong, Mingxuan Yuan, and Jia Zeng. Learning to pack: A data-driven tree search algorithm for large-scale 3d bin packing problem. In Proceedings of the 30th ACM International Conference on Information amp; Knowledge Man- agement, CIKM '21, pp. 4393-4402, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384469.
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL RELATED WORKS
Rule-based Algorithms for Sequential Decision-Making Tasks. Traditional practices usually utilize rule-based algorithms with expert experience for specific tasks. For BPP, the Bottom- Left heuristic (BL) (Chazelle,, 1983) searches for Empty Maximum Space (EMS) with the minimum bot- tom (x) and left (y) coordinates to place the current item; Deepest-Bottom-LLfff with Fill heuristic (DBLF) (Kang et al., 2012) sorts EMS in order of deepest coordinates (z) and uses bottom and left as tie-breakers; Best Match First Packing heuristic (BMF) (Li & Zhang, 2015) sorts EMS in order of distance to the container's deepest-bottom-let point; and Online BPH (Ha al., 2017) sorts EMS in order of minimum bottom, deepest, and minimum left. For EGSP, collective control (Strakosch, 1983) assigns the elevator to stop its nearest call in the running direction; and (Rong et al., 2003) allocates hall calls to the elevator that minimizes the estimated arrival time. Others create policies with matrices of "heuristic scores" and option decisions with the highest score. (Ramalingam et al., 2017) develops greedy algorithm based on submodularity to maximize its objective function.
Learning-based Algorithms for Sequential Decision-Making Tasks. Reinforcement learning (RL) excels at dealing with sequential decision-making problems. In real-world BPP applications, the agent usually can not foresee the upcoming items, which puts online decision requirements on BPP solvers. This variant of BPP is called online BPP, and Zhao et al. (2020) use RL and well- designed discrete representations to solve it. While Zhao & Xu (2022) treat the BPP problem as a decision in continuous space and formulate it into tree structure. The model combines RL and tra- ditional algorithms for flexible applications in various settings. As for EGSP, an early work (Crites & Barto, 1998) uses Q-learning to control elevators move up and down. A recent work (Wei et al., 2020) explores deep asynchronous actor-critic learning to decide each elevator's 's next target stop floor. Our model focuses on input-dependent industrial scenarios, where input data are collected from production line with regularity to follow.
B COMPARISON WITH EXISTING RL MODELS WITH EXOGENOUS INPUTS
Recent works investigate RL with exogenous inputs. (Mao et al., 2019) proposes meta- RL model that formally defines input-driven MDP and proves that an input-dependen baseline is bias- free and variance-reduced when input and action are conditionally independent. Our theoretical basis is the same as (Mao et al., 2019): considering the sequence of input values in value estimation can reduce estimation variance without introducing bias. Different from (Mao et al., 2019), to deal with class of industrial problems, we consider the input and constraints in two stages during the state transition, thereby designing a bi-critic network. Chitnis & Lozano-Pérez (2020) divides the state s, into endogenous component n, and exogenous component Z₁. Then split exogenous component 2, into m state variables wi . ...,. . and learn mask, a subset of the exogenous state variables. Chitnis & Lozano- Pérez (2020)'s 's decomposition of state transition is similar to ours, but the subsequent learning is completely different. Sinclair et al. (2022) samples trajectories over the exogenous trace from the history dataset and trains the model-based RL model offline. Given a fixed exogenous sequence & = ........ .......,Ez ) and policy π, Sinclair et al. (2022) decomposes the value function into an expectation of fixed exogenous trajectory V," (s. == Σ. T(a s s)QT (s, E2t). The training method and handling of value function are not the same as ours.
C DERIVATION OF REDUCING THE VARIANCE OF VALUE ESTIMATION CAUSED BY MIXING TWO SOURCES OF UNCERTAINTY
Previous works Mao et al. (2019) prove that subtracting an action-independenn baseline function help reduce value estimation variance for actor-critic-hast algorithms. However, the problem of high variance of value estimation is still exposed in industrial decision-making tasks because the two sources of uncertainty during the state transition process are mixed and processed together (shown in Fig. 6) by existing RL algorithms. Here we use the case of BPP to illustrate why. The same is true for other tasks. Without loss of generality to all actor-critic cridic--ii- algorithms, here we use the advantaged function of the A2C algorithm for illustration:
A, = + Ve+1 V
14

---

Under review as a conference paper at ICLR 2023
1
St+1
(d+1(d)
Single-
fd+2
stage
d
(d),
d d
Figure 6: The single-stage state transition process the BPP. It takes (st, de, f(+1) as the state in regular MDP formulation. And during the state transition process, it mixes the sources of uncertainty ((+i(d) and C.), leading to a high variance of value estimation.
Consider the state transition process from (s;,de, des f(+1) to ($1+1, ft+2) in the single-stage formulation, where state at time denotes the current placing state, d, denotes the current pending item, and f++1 denotes the generation distribution of the next coming item d,+1- Additionally, we use 100 to denote the marginal distribution of future items at time t, which is determined by 1 ... f And we denote the finite set item types as D, the finite action set as A, the policy as Ta. hard constraints at time i as Cr, the reward function as and the discount factor y Suppose that the value estimation function inputs at time are the st, de, and fi+1:0 The advantage function can be written
A, = rt + 1 V Se-1oso)
(10)
During the training process of the A2C algorithm with plenty of exploration, A, is approximate to its expectation:
E, [AJ] = [r + de41,
V(s de, f(+1:0))
D
Σ fi(ee)
Σ To(ac)s; dr, C,)
Σ fet1 ((ti(de)))
(11)
d++1
- V(s,, 10-1:
For simplicity, define the hard constraint as function prohibiting actions that cause the pending item to tip over:
10. if a, makes de tip over at 8,
((s
(12)
otherwise
Then we can simplify the policy function:
To(ar de. Ce) mo(a Ce) at)
(13)
15

---

Under review as a conference paper at ICLR 2023
And further get
P
E,,
[A]]
Σ f((d))
E To(ails, di))c(ss,, ai) d4
de41 - (s,. de, f1+1:00]
D
D (4) + Σ f((de)
A
Σ f((de) Σ
Σ
To (o(aa)ss,,0
di
d2
Ca(se, de; at)
Σ
(82s de, sct1s)))
de-1
D
Σ (Ad)))
Σ
To(ailoe, (st, de, at) a+)
de
41
(d()
(e(aelst,d))CCaa de, ax)
dot (42200))
Σ
de+1
Σ (d,)
az) fert f(+1
(14)
We focus the second term of Eq. 14. Notice that the generation of de+1 is stochastic and fol- lows the distribution of f++1 (d++1), which is unknown to the RL agent. Thus during each training episode, the term (V(+++1, Je+2:00) in Eq. 10 introduces lot of variance due to the uncer- tainty of de+1- Moreover, the constraint term C,(8,, de, 0z) applied to the policy function introduces variance due to the uncertainty of constraints. As the second term in Eq. 14 multiply Ce(se, de ar) by (St++1, de+1, f2+2:0)), it brings mutually adverse effect on variance and further confuse the RL agent during the training process, hindering the critic network from estimating the state value accurately and the actor-network from learning good policies.
D DERIVATION OF VALUE REDUCTION OF THE BI-CRITIC FRAMEWORK
In the two-stage MDP formulation, use V and V for value estimation of the state-dependent stage and the input-dependent stage, respectively. In the state-dependent stage, as the state transition process is deterministic (from Eq. 2) and does not care about the generation of d,41, the advantaged function A, is calculated by
A, + + (
V (e
(15)
16

---

Under review as a conference paper at ICLR 2023
Thus the expectation of A, with sufficient exploration during the training process is calculated by
Ex[AA] Ex(r: + (et1.
-4
D
Σ f(()))
(" d4) + f1+::0) - (s de, f141:00]]
D
D
Σ
Σ
d4
+ Σ S((d)) Σ To(aelse,d)
de
de
Ci de, (1 dests)))
((d,)
Σ (s) (,)
de
(de)
(1) de, (s44), fet1so))
Σ fe(de)
Telaelse (eldelss.ddCCeee.. de)Co(sr, de, az) V* fet1so)]
(16)
Comparing Eq. 14 and Eq. 16, we get that the major difference is in the second term, where we sub- stitute ('(4111, fe+1:00) for Ed..7 mulation. In other words, we expect reduce the variance by updating the advantaged function with the expectation of (( de41, ft+2:0)) (i.e., 11 in Eq. 15) instead of the ex- V(5e+1, (et4,,,441,,/ /1+2:00) (in Eq. 10) during the training process. And this goal is achieved by two independent stages in our bi-critic framework. First, the term Ce(st, de a (s f(+1:0)) does not bring excess variance caused by mixing the two sources of uncertainty. Second, introduce an additional loss function to minimize the distance between ('(+11, [( and
f(+2:0)) 1
in two-stage MDP for-
D Edyz Se+1 (de+1 ( 1, 2242:0)) in the training process, that we reduce the value timation variance caused by the uncertainty of de+1.
Hence, our bi-critic framework under the two-stage MDP formulation can reduce value estimation variance, promoting the convergence of the critic network and guiding the actor-network learn better policies stably.
E TRAINING ALGORITHMS
The overall training algorithm for the whole framework is shown in Alg. 1. Note that the input data distribution sequence fo(d)), (i())..... fN (dx) is only for the generation of input data d, and is unknown to the RL agent, which follows the setting of most industrial sequential decision decision-making scenarios.
17

---

Under review as a conference paper at ICLR 2023
Algorithm Training algorithm for our bi-critic framework
Input task-related heuristic function Hi;
Input a task-related data distribution sequence fo(d), fi (d). fx(();
Initialize bi-critic networks Vă. Vě, and actor Tw with random parameters 0s 0s,
Initialize state encoder E dynamics encoder E,,- dynamics aware module Fv., and the constraint module Mc with random parameters @2 ds. 19,
Initialize input buffer D;
Pretrain the dynamics module Fu with Alg. 2:
for episode = 0 to mar-episode do
Reset the environment and observe state so. reset D; Generate do with fu(do) and store du in D;
Update A: 4- Xg (1 An) episode//maz episode;
for 1 0 to N do
# state dependent stage
Get future data features 11+1:00 = =D);
Get the constrained action mask M, (M, de):
Get z = Eg, (st, de, ft+1:00), Vi" = Vø (zī) action de (arlz))) M,; Perform action de in the environment and then observe reward Y2 and next state 8141;
# imput dependent stage Get z E (st+1. fi+1:00), Vi+1 Voj (z(+1);
Generate de+1 with (d) and store de+1 in D;
# compute loss and update parameters Get advantage estimate with Eq. and Eq. 7:
A, = ar; + (1 (s) de, az)] + X 1+1 - V Update all parameters with loss function in Eq. 6;
Here we illustrate simple example for pre-training the dynamics module, which makes it predict next three input data with previous T input data at time l. To ensure the diversity the input data distributions, we add noise (0, 0.1) to them in each training iteration. The algorithm is shown in Alg. 2.
Algorithm 2 Pretrain algorithm for the dynamics aware module
Input a rask-related data distribution sequence fo)),, (i),, fx):): Initialize the dynamics module F& and predict network Pe with random parameters v/, yp: Initialize memory buffer B; for äter
maz, iter do
Perturb the input data distribution for each time step t: f((d) + f((d)+ 000.0.1)
for T 1 to N 3 do
Reset B;
for i to K do
Generate X, = di-T+1, de with ..).
Generate Y, d,+1, d242, d++3 with J(t1 (d), fe+2(d), Ses((d)
Store (X,, Yj) B:
Sample batch < X, > from B; (fo+1:0))
Get fi+1:00 (X),
Compute the loss function & Cross Entropyyt Y);
Update sh, & with L;
F BPP HIERARCHICAL ARCHITECTURE
The framework the BPP placement planner is a two-level hierarchical Actor-Critic with an atten- tion mechanism. Facing the Exploding action space, decompose the action space (whole pallet)
18

---

Under review as a conference paper at ICLR 2023
High-Levell Placement
Astor
M
Quarse Ab
Spesi-
Critte
02
Low-Leevel Pheeeent Planner
Critic
Sp12
Figure 7: Overview of the proposed hierarchical architecture for the BPP task. Since invalid action mask calculation takes extended time and is essential for decision constraints, we need to decouple the action space to reduce mask calculation. With hierarchical architecture, the entire space of the pallet (9*9) is divided into small regions (3*3), the high-level agent selects feasible subspace for the low-level, and the low-level agent selects a specific position to pack. Decoupling of action space reduces the size of invalid action masks (colored in yellow), thus reducing computation time.
Table Distribution Functions and Parameters for Item Characteristics Characteristic
Distribution
Parameters
Depth/width ratio
Normal (loc, scale)
(0.695, 0.118)
Height/width ratio
Lognormal (mean, sigma)
(- 0.664, 0.453)
Repetition
Lognormal (mean, sigma)
(0.544, 0.658)
Volumes
Lognormal (mean, sigma)
(2.568 x Voats 0.705)
Veat
Normal (loc, scale)
(2.5, 0.118)
Numbercat
Normal (loc, scale)
(1000, 0.118)
Width clip
[min width, max width]
[20, 60
Depth clip
min depth, max depth
20, 50
Height clip
min height, max height
10, 50
Vens clip
min Veat. max Vent
[1.4]
Table 4: The uniform distribution interval of the items' width, height and depth of each type/class.
Type/Class
Width
Height
Depth
Type 1
1, W
H H
D.
Type 2
2 W, W
1, 1
D
Type 3
W. W
3 H
1
33
3
Type 4
W. W
H, H
DD
Type 5
1, W
1,
1
Class
[10, W
10.
10,
Class
1, ZW
1
1
Class 8
[1 WT
[1
1.D
into a certain number of sub-regions. The high-level policy selects a sub-region with an attention mechanism for the low level achieve, and the low level selects specific packing position in the sub-region. As the action space decreases, the corresponding mask also decreases, and the amount of calculation becomes smaller.
G DATA GENERATION OF BPP
Mixed-item Dataset (MI Dataset) follows the realistic 3D-BPP instance generator proposed in (El- hedhli et al., 2019). The pallet dimensions are set to L= 120, W = 100, and H 100. The item char- acteristics are determined by fitting the real-life industry data from (Elhedhli al., 2019). We gen-
19

---

Under review as a conference paper at ICLR 2023
Table 5: Result on the untrained dataset with different location (loc) parameters of N Number
distribution and scale parameter of Veat distribution.
loc of Vumbercad scale Vent
Space utility f
Avg.items
500
0.118
52.36%
18.44
100
0.118
52.79%
19.00
10
0.118
53.91%
19.32
100
0.5
52.72%
19.93
100
1.0
52.78%
21.57
erate data considering the ratio of item depth and height to its width (depth/width and height/width ratio), volume, and frequency of occurrence (Repetition). The first two characterize the size of the item. The reason consider the proportions and volumes of item rather than its dimensions is construct items of different sizes whose dimensions are appropriately related to each other. In real- life industrial environments, the average volume of incoming items varies with each category. In order to simulate the volume changes of different categories. we generate the number (N umbercat) and average volume (Veat) of each category of items according the normal distribution and then generate the item volume (Volumes) within the category according to the log-normal distribution.
Large-item Dataset (LI Dataset) comes from (Martello et al., 2000) with eight classes of data, which generates dineedimensional instances from randomly generated width, depth, and height within preset interval. The pallet dimensions are set to L = 100, first five classes of data come from (Martello & Vigo, 1998). We generate five types of instances as Table 4. For Class kk = 1, .....,5), 5), each item is of type with probability 60%, while of the other four types with probability 10% each. The last three classes are generalization of the instances presented by (Berkey Wang, 1987).
100, and H 100. The
H GENERALIZATION EXPERIMENT ON BPP
Since our MI Dataset built on certain distributions for item characteristics, we can also evaluate the performance of DACC on different distributions from training data. First, we change the location parameter of uumerr.at 's distribution and the scale parameter of Veat 's distribution. The testing data includes 10000 items generated from different distributions. The result is presented in Table 5. Our method can still perform well while testing on the untrained dataset.
I FAILURE CASE STUDY
We increase the uncertainty and dynamic of data by changing some data generation parame- ters of the MI dataset. Then, we train DACC on datasets with different uncertainty. To achieve greater uncertainty and minimize the impact on the space utility, change location and scale parameter of Number catt scale parame- ter of Veat. clip parameter of Veat, width, depth, and height. We have generated a total of five datasets, and their dynamics are progressing. Here we discuss the changes of each dataset compared with the previous one. Dataset one (D1) change the Number location from 1000 to 500. Dataset two (D2) change the Numbercat's location from 500 to 100. Dataset three (D3) change the Numberroa'ss
0.4
Space
3 R
1000C
20000
30000
40000
50000
Episodes
Figure 8: Failure case study result on five datasets with progressing dynamics.
scale from 0.118 to 10. Dataset four (D4) change the Voat's 's clip from [1, 4] to [0.5, 8). Dataset five (D5) change the clip of width, depth, and height to [10.90]. The result failure case study is shown in Fig 8, when the increase of data uncertainty is limited, model DACC is stable and can converge to a better space utility. With the substantial increase of data uncertainty (D5), the model fails.
20

---

Under review as a conference paper at ICLR 2023
Table 6: Major differences among three simulators: LiftSim, Elevate and our self-developed engine which will be open-soureed.
Environment
LiftSim
Elevate 8
Ours
Meet realistic constraints
X
Realistic passenger data Support up-down mode Support destination mode Provide interface for RL Open source
X
V
X
V
V
Table 7: Comparison among different agents in two-way traffic pattern in terms of Tow and Tos We the agent's goal as optimizing the mean waiting time and the average transmitting time.
Agent
T Taw
Ta
DACC w/o DAM and rules
111.73
173.55
DACC w/o rules
65.44
128.02
DACC w/o DAM
35.13
91.08
DACC (our full version)
32.79
89.36
up call ream Boor <h coll num from flor 0 call mum foor call rum foor
up from
dn cal num from floor
0.7
upiccl num to fbor
dnicolliumm num fbord
passenger ger nure
0.5-
passergge
0.4
0.2
(b)
1.0
from mum foor
cell rur Boor
0.8
0.8-
rum
bassenger
0.6
0.4
0.0
0.0
10
20
30
40
60
10
20
30
40
50
60
time
time
(c)
(d)
Figure 9:
Average number of passengers per minute in one hour in two-way pattern: (a) Floor 0.
(b) Floor 4. (c) Floor 8. (d) Floor 12.
J OUR ENHANCED SIMULATOR OF THE EGSP
The existing EGSP simulator Liftsim (Wang et al., 2020) fails to meet realistic constraints mentioned in $ 4.1 and the commercial EGSP software Elevate8 (Peter) is not open-source for training RL agents. To better reflect the mechanism simulator based Liftsim, satisfying all the realistic constraints of the real-world EGSP. And implement both the up-down mode and destination mode for further study. The differences among I LiftSim, Elevate and our simulator are concluded in Tab. 6.
commercial elevator scheduler, we first develop our
21

---

Under review as a conference paper at ICLR 2023
call rum oin call rum Boor
floor dh coll num to floor
Daseener rum
15-
1,25
passenger
1.00
10-
0.75
0.90
0.25
0
10
20
50
60
0.00
10
40
50
60
time
time
(a)
(b)
up call fum foor
coll num from foor 12
di call num foor 0
1.76
call rum from fooo
numn Boor
call rum foor 12
flor
1.50
passanger num
1.26
1.00
1.00
0.76
0.50-
0.00
30
30
60
time
time
(c)
(d)
Figure 10: Average number of passengers per minute in one hour in up-peak pattern: (a) Floor O.
(b) Floor 4. (c) Floor 8. (d) Floor 12.
call rum from num from call num foor 0 call num to floor O
numm
num Bloo
1,50
up call mum to 4oor di cadi mum to Roor-
casgergee hum
1.25
15
passenger
1.00
10
0.25
0.00
50
(a)
(b)
up call num from foor un from Boor
1.6
col rum from Toor call ramm from floor
1.8
1.4
1.4
1.0
paseenger
1.0
0.8
0.8-
0.6
0.4
0.4
02
0.2
0.0
6.0
10
20
30
40
50
60
16
20
30
40
50
60
time
time
(c)
(d)
Figure 11: Average number of passengers per minute in one hour in down-peak pattern: (a) Floor 0. (b) Floor 4. (c) Floor 8. (d) Floor 12.
22

---

Under review as a conference paper at ICLR 2023
K DYNAMIC PASSENGER DATA OF THREE PATTERNS IN OUR SIMULATOR
As for dynamic passenger data, instead of randomly generating passenger data with fixed probability factors, we use real-world passenger data (provided by world-renowned elevator manufacturer) that conforms to different patterns from a 16-floor building. And we further augment the data with randomness for RL's training. The real-world passenger data of three different flow patterns (two- way pattern, up-peak pattern, and down-peak pattern) are illustrated in Fig. 9, Fig. 10 and Fig. 11, respectively. And for each pattern, we take four floors (floor 0, floor 4, floor 8, and floor12) as examples and show the average number of passengers per minute in one hour for each floor. And for each floor, we show the average number of passengers that request to go up from this floor, go down from this floor, go up this floor, and go down to this floor. From Fig. 9, can know that in the two-way pattern, the up call requests are mostly from floor O, and the down call requests are mostly to floor 0. From Fig. 10, O to other floors form the majority of all hall call requests. From Fig. 11, we can know that in the down-call requests from other floors to floor
know that in the up-peak pattern, up-call requests from floor
down-peak pattern, most of the hall call requests O. From all three figures, we also know that the total passenger flow change over time. The highest peak time of total passenger flow in an hour is 30 to 35 minutes in the two-way pattern, 35 to 40 minutes in the up-peak pattern, and 12 to 17 minutes in the down-peak pattern.
L ABLATION STUDY ON THE EGSP
We take an ablation study to evaluate the advantages of the dynamics aware module and combining rules. this experiment, we evaluate four agents on 120-min passenger data flow that follows the two-way traffic mode: the agent without the dynamics aware module (DAM) and rules, the agent without rules, the agent without the dynamics aware module, and the full version of DACC agent-assuming that the objective in this scenario is to reduce the average waiting time and the average transmitting time. From the experimental result shown Table 7, we can find that both the dynamics aware module and combining rules play essential roles. By combining the advantages of rules and dynamics aware, our framework, DACC, can inherit the controllability and reliability of rules and further optimize the customized objectives for different scenarios.
23


Reviews:
[{"review_id": "bZYt0HeSGK", "reviewer": "Reviewer_WYQB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For certain industrial sequential decision problems, this paper proposes to decompose the conventional MDP transition into two steps, state-dependent stage and input-dependent stage, and learn separate value functions. The state-dependent stage focuses on security or safe constraints and the input-dependent stage is for capturing the non-stationarity of the environment. The authors design a dynamic-aware and constraints-confined reinforcement learning based on this two-stage MDP formulation. On bin packing problems and elevator group scheduling problems, the proposed method demonstrate superior performance compared to rule-based algorithms and learning based algorithms. ", "summary_of_the_review": "This paper studies the important problem of applying RL in industrial applications and the proposed method demonstrates strong empirical performance. I could be willing to accept this paper if the clarity could be improved. ", "strength_and_weaknesses": "### Strength\n\nThe proposed two-stage MDP framework is novel and empirical results show that it perform well in non-stationary industrial benchmark tasks.\n\n### Weaknesses\n\nThe clarity needs improvement. See detailed comments below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mflpd0qmdeN", "reviewer": "Reviewer_EhZ3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to solve industrial sequential decision making tasks. The paper makes two observations about the transition dynamics for industrial scale decision making tasks and proposes a two-stage approach that leverages the underlying structure of industrial problems. Using the observation the paper proposes a bi-critic framework (DACC) which leverages this underlying structure. Finally, empirical results are provided for the proposed method.", "summary_of_the_review": "The topic studied in the paper is novel and interesting. The paper would significantly benefit from improved presentation and more instructions on how to reproduce the results. Further details on the experiments are also be greatly appreciated.", "strength_and_weaknesses": "Strength:\n- The topic covered by the paper is intriguing and significant for enabling wider adoption of RL algorithms.\n- The illustrations provided in the paper help better illustrate the core ideas\n\nWeaknesses:\n- The paper can be hard to understand at times. For instance, in Section 3.1, it took several reads to realize what the authors mean by \"two-stage\" framework. I also personally believe that writing $S_t = (S_t^p, d, F_t)$ could better represent this fact. It is also relatively unclear to me what **exactly** $F_t$ represents: in the paper immediately above eqn (1) it is mentioned that $F_t$ relates to the marginal distribution of future boxes and is related to a series of distributions. Are we saying that $F_t$ is some function that could approximate the future distributions? Approximate in what sense? Are we saying that $F_t$ is a model that is able to predict $f_t(d)$ using information obtained up to step $t$?\n- While the figures contain error bars but standard errors are not reported in the tables. The number of runs used to obtain the error bars is not reported. \n- The authors use a self-developed simulator for the elevator task. It is hard to assess what the simulator does by simply reading Appendix B without seeing the underlying source code.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BOkla6Ft2Rc", "reviewer": "Reviewer_7Ak6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Reinforcement learning can be used in many industrial decision-making problems due to its potential to outperform heuristics.  To avoid issues around scale, current state-of-the-art practical algorithms are simple rule-based strategies which are tuned for improved performance.  In contrast, reinforcement learning is a powerful technique that can be used to learn near optimal policies.  Important to industrial models is the fact that the dynamics are near-predictable (i.e. the decision maker has additional knowledge on the transition distribution).  This is observed in elevator scheduling and bin-packing, where the demands are essentially the only unknown in the problem formulation.  Additional information like this can be used to obtain improved performance.  At a high level, the authors first present these problems as a two-stage MDP, allowing the algorithm to reduce state transition uncertainty.  They then design DACC, a framework for learning input dynamics and making decision with guidance of problem-specific rules to satisfy constraints.  The authors then complement these results with numerical experiments on bin packing and elevator scheduling problems.\n\nMore concretely - the authors first state the two-stage MDP model.  In this model the transition can be decomposed into two stages: state-dependent stage and input-dependent stage.  In the first stage, the algorithm picks action $a_t$ based on current arrival $d_t$, leading to a deterministic next state.  In the second stage, the new arrival $d_{t+1}$ is sampled according to a state-independent distribution.\n\nBased on this model, the authors note that the value function decomposes over the exogenous arrival $d_t$ (although this is never proved explicitly, and the authors do not give a general MDP formulation for their model). To complement this reduction they propose a framework for learning the value function estimates (and hence a policy) over this decomposition.\n\n## Questions\n- Should it be $F_{t+1}$ and $F_t$ in equation 1?\n- Do you believe that learning a good-enough predictor of the demand sequence is reasonable in these settings? See \"Protean: VM Allocation Service at Scale\" which highlights how demand patterns are highly correlated and impossible to predict.\n- What is the justification for \"online interaction\" in the training versus a historical dataset?\n\n## Related Work\nThe authors should consider the following papers and how they fit into their model and discussions:\n- \"OR-Gym: A Reinforcement Learning Library for Operations Research Problems\" - considers impact of \"action masking\" on learning a policy.  The authors seem to suggest that the fact that the problems have constraints make RL in input driven environments more complicated.  However, the constraints that they consider are just the fact that the feasible actions are state-dependent.  Simple action masking (as the authors do in practice) seems to alleviate that issue.\n- \"Hindsight Learning for MDPs with Exogenous Inputs\" - provides a general model for input driven MDPs similar to one studied here - and highlights a decomposition of the value function over the exogenous demand variables (similar to Equation 2)\n- \"Sample-Efficient Reinforcement Learning in the Presence of Exogenous Information\" - Considers RL with extra exogenous information\n- \"Learning Compact Models for Planning with Exogenous Processes\" - similar algorithmic framework for learning a mask in input driven MDP models\n- \"Markov Decision Processes with Exogenous Variables\" - value function decomposition\n- \"Variance Reduction for Reinforcement Learning in Input-Driven Environments\" - This paper was already cited but the authors should include a detailed distinction between their MDP model and the one considered here\n- ", "summary_of_the_review": "The authors omit a detailed discussion on the relationship of the proposed research to prior work on exogenous decomposition in MDPs.  Moreover, the writing quality and clarity needs to be improved.  The authors provide no theoretical justification for the benefits of the two-stage decomposition of the process (and similar ideas have been presented in prior work).\n", "strength_and_weaknesses": "## Strengths\n1. The authors consider an important problem of learning to plan and schedule with empirical results in bin packing and elevator scheduling.\n\n## Weaknesses\n1. The writing needs to be improved in order for the main points of the paper to be better described.\n2. There are no theoretical justification for the advantages of considering the two-stage model over just the naïve model\n3. The related work is insufficient, and the authors do not connect their model to other existing works in the area", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5ML7Y8Ib_J", "reviewer": "Reviewer_CzjD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This  paper  proposes  a  new  Markov  Decision  Process  (MDP)  definition  to account  for  the  non-stationary near-predictable  tasks  that  are  common  in  industrial  application.   These tasks require the agents to learn the constraints and rules of environment and the strategy to get high reward in the dynamic environment.  The author propose a two-stage MDP in which the agent learns constraints of the environment and the near-predictable reward separately in two steps.  A RL model DACC is designed to fit in the two-stage MDP setting.  With two critic networks, one of which learns the constrained transition of environment states and the other learns the transition of state reward that are regular to some extent.  The experiments compare DACC to other methods on two representative applications.  Two models (A2C and DACC w/o rules) are included in experiments as an ablation study trying to support the superiority of DACC’s dual critic design. These models outperform others.", "summary_of_the_review": "The framework is interesting and the model is novel.  However, the experiment design failed to prove that the superiority of DACC comes from its two-stage design but not safety mask that can significantly reduce the action space. The dual critic set up raises convergence concerns.\n\nPost author revisions:\nThe clarity of the paper is improved. A more inclusive ablation study supports the superiority of the model DACC and the two-stage MDP. The detailed training algorithm helps to explain how the critics are updated during training. The ablation also clarifies that A2C w/o rules performs worse than DACC w/o rules, while A2C with rules performs better than DACC w/o rules.\n", "strength_and_weaknesses": "Strengths:\n1. The problem setting and methods are well-defined.  The two example tasks are representative for the application environment targeted in this paper.\n2.  The two-stage  MDP  proposed is technically correct as its two stages learn the constraints of the non-stationary environment and the task-specific or time-dependent values respectively.\n3.  The experiments compare the proposed methods  (DACC)  with  4  heuristic methods and  2  other learning-based methods as well as two learning-based models for ablation studies in two example tasks.\n\nWeaknesses:\n1.  The DACC models use a marginal latent variable F(t) that is dependent on the recent experiences to account for the ”near predictable” feature of the problem.  However, the agent might only exploit recent regularity instead of the long-term regularity that is motivated by the authors (e.g., in elevator scheduling)\n2.  Due to the recent experience-dependent F(t), how does the model respond to change of input pattern (e.g., is there a delay, can things go wrong during the transition?)\n3.  The two critic networks in DACC use each other’s estimation of value function to update themselves. Does this create issues? What is the impact of the KL divergence term—does changing the weight of the KL divergence term affect the outcome?\n4. The safety mask is not present in A2C and \"DACC w/o rules\", the two models in the paper's ablation study. I think the DACC w/o rules has more parameters to train and hence should be more powerful than A2C. But the experiment data shows that A2C has better performance. So if the extra critic in DACC w/o rules doesn't necessarily increase the performance, then the safety mask might be more important than the extra critic. Can A2C be tested with a safety mask?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject