# Hierarchical Reinforcement Learning for Crude Oil Supply Chain Scheduling
## 4 Hierarchical RL Scheduling Scheme
### The Variant on Discrete Action Space  
Exploring the optimal policy in a discrete action space with low dimensions for an RL agent is much easier and more stable than in a continuous action space. Moreover, we aim to solve the crude oil supply chain scheduling problem in the real world, which requires the algorithm to converge quickly in a stable manner. Therefore, we design a variant that combines 9 operators as the action space. The operators are shown in Table 3. The action of the RL agent is to select one operator for each corresponding node and one operator for the solving order \(or^{t}\), forming an action space of \(|V_{r}|+|V_{r}|+1\) dimensions. The crude oil supply chain receives crude oil from oil fields and import ports and provides diesel and gasoline on demand. Then, the inventory of each node can be stable if the production and demand are balanced. In other words, the end-stage inventory of nodes is a baseline inventory that needs to be dynamically adjusted in case the supply and demand are unbalanced. The demand for diesel and gasoline is uncertain at each time step; thus, we designed 3 naive operators:  
* To meet the upper safety limit, which means \(\hat{m}_{r}^{d,t}=u_{r}^{d}\) and \(\hat{m}_{r}^{g,t}=u_{r}^{g}\);
* To meet the lower safety limit, which means \(\hat{m}_{r}^{d,t}=l_{r}^{d}\) and \(\hat{m}_{r}^{g,t}=l_{r}^{g}\);
* To maintain the current inventory, which means \(\hat{m}_{r}^{d,t}=m_{r}^{d,t-1}\) and \(\hat{m}_{r}^{g,t}=m_{r}^{g,t-1}\). Since diesel and gasoline are produced proportionally, to control the crude inventory in refineries, we approximate the demand for crude oil accordingly:  
\[\hat{n}_{r}^{c,t}=\max\left\{\frac{n_{r}^{d,t}}{\rho_{r}^{d}},\frac{n_{r}^{g,t }}{\rho_{r}^{g}}\right\} \tag{20}\]  
The refinery could process crude oil into diesel and gasoline if its crude oil inventory is about to overflow, but it is hard to replenish the inventory when the crude oil supply is short; thus, the operators for the end-stage inventory of crude oil are designed as follows:  
* To meet periodic demand, which means \(\hat{m}_{r}^{c,t}=N\cdot\hat{n}_{r}^{c,t}\), where \(N\) is a length of time step period;
* To meet the upper safety limit, which means \(\hat{m}_{r}^{c,t}=u_{r}^{c}\). To avoid the sharp rise and fall of the inventory, the operators of transfer stations are designed as follows:  
* To increase by 10%, which means \(\hat{m}_{f}^{c,t}=1.1\times m_{f}^{c,t-1}\);
* To decrease by 10%, which means \(\hat{m}_{f}^{c,t}=0.9\times m_{f}^{c,t-1}\). In addition, we use another operator to control the solving order of the lower-level agent. The idea is that when the processing volumes \(q_{r}^{t}\) in refineries are settled, the amount of crude oil consumed at this time step is known, and the approximate range of the amount of crude oil that the refineries need to receive from the transfer stations can be deduced, facilitating the optimization of \(w_{e_{f}}^{t}\). On the contrary, if the transportation volumes \(w_{e_{f}}^{t}\) are settled first, there is a high probability that the refineries will not be able to meet the inventory requirements only by adjusting the processing volume of \(q_{r}^{t}\). However, mindlessly optimizing \(q_{r}^{t}\) first may cause inventory overflow of many transfer stations; thus, we use an operator \(or^{t}\) to control the optimization order:  
* To solve \(q_{r}^{t}\) and \(w_{e_{f}}^{t}\), simultaneously, which means \(or^{t}=0\);
* To solve \(q_{r}^{t}\) and \(w_{e_{f}}^{t}\), sequentially, which means \(or^{t}=1\). At each time step \(t\), the upper-level agent makes decisions in two steps: firstly, the RL agent selects one way of the combination of these operators, and secondly, \(\hat{m}_{f}^{t}\) and \(or^{t}\) are calculated according to the operators. Then, the lower-level agent will solve the \(LP\) problem as discussed in Section 4.3 according to the solving order \(or^{t}\).