# Hierarchical Reinforcement Learning for Crude Oil Supply Chain Scheduling
## 4 Hierarchical RL Scheduling Scheme
### Overall Architecture  
As described in Section 2, the main challenges of this problem are the large decision space and uncertainty. Traditional solvers are powerful tools to solve large-scale LP problems but cannot deal with uncertainty. In contrast, RL methods can solve problems with stochastic state transitions but are hard to train when the problem scale is enormous. Considering the property of this problem, we design an HRL-based optimization architecture, as shown in Figure 2 and Algorithm 1.  
```
1:Initialize model parameters \(\theta\)
2:while Not converged do
3: Observe initial state \(s_{0}\)
4: Initialize \(\tilde{U}_{q_{r}}^{0}=U_{q_{r}}\)
5:for t=1 to \(T\)do
6: Select action \(a_{t}\sim Q_{\theta}(s_{t},a_{t})\) by Section 4.2.2
7: Solve the LP problem Equation (18a), where \(m_{i}^{k}\gets a_{t}\), \(U_{q_{r}}\leftarrow\tilde{U}_{q_{r}}^{t}\), get \(w_{e_{f}}^{t}\), and \(q_{r}^{t}\)
8: Transit from \(s_{t}\) to \(s_{t+1}\) by Equations (11)-(16)
9: Set \(\tilde{U}_{q_{t}}^{t}\leftarrow\tilde{U}_{q_{r}}^{t-1}-q_{r}^{t}\)
10:endfor
11:endwhile
```  
**Algorithm 1** Hierarchical RL Scheduling Algorithm  
The entire crude oil supply chain scheduling problem involves \(T\) steps. The supply volume of crude oil and the demand volume for diesel and gasoline at each step is dynamic, forming a stochastic environment. Moreover, the transportation plan and processing volume plan decided in each step will affect future decisions. The impacts are reflected explicitly in two ways: the initial inventory of the future steps and the remaining available processing volume, since there is a constraint on total processing volumes \(U_{q_{r}}\). The latter can be dealt with shaped constraints at each time step with \(\tilde{U}_{q_{r}}^{t}\leftarrow\tilde{U}_{q_{r}}^{t-1}-q_{r}^{t}\) and \(\tilde{U}_{q_{r}}^{1}=U_{q_{r}}\). Then, suppose the initial inventory \(m_{i}^{k,t}\), the ending inventory \(m_{i}^{k,t+1}\), the supply volume \(w_{e_{f}}^{t},w_{e_{r}}^{t}\), \(m_{e_{f}}^{t}\), and \(w_{e_{f}}^{t}\), and the demand volume \(n_{i}^{k,t}\) at time step \(t\) are given; then, the problem for time step \(t\) is an LP problem, independent from other time steps. Therefore, we can use an RL agent to learn the inventory volume and use a solver to solve the LP problem.  
The upper layer is the RL layer. The RL agent takes the inventory volumes of each facility at the end of the time step \(t\) as the action, denoted as \(\tilde{m}_{i}^{k}\). The goal of the RL agent is to minimize the summation of the alert penalty for multiple steps and the transportation penalty, as defined in Equation (7a). However, using the hierarchical framework, the transportation penalty is received from the lower-layer agent.  
The lower layer is a traditional optimizer layer, which solves the LP problem Equation (7a) of one time step. At time step \(t\), the solver receives the end-stage inventory volumes \(\tilde{m}_{i}^{k}\), and the remaining processing volume of each refinery \(q_{r}^{t}\) from the upper-level agent uses them to form the one-step constraints which split from the multi-steps constraints in Equations (7b) and (7e). The lower layer takes the process volume plan \(q_{r}^{t}\) and the transportation plan \(w_{e_{f}}^{t}\) of the time step \(t\) as decision variables and minimizes the total transportation costs \(\mathcal{H}^{t}\) defined in Equation (6), which is passed back to the upper-level agent.  
Figure 2: Overall Architecture.  
In addition, based on experience, we know that RL performs better on small-scale discrete problems; thus, we propose a variant for our crude oil supply chain scheduling problem. We discretize the action space into several end-stage inventory values and add one dim, indicating the solving orders of the solver. Compared to the vanilla architecture, the lower layer of the variant architecture receives an extra signal, determining the solving order of \(w^{t}_{e_{f}}\), and \(q^{t}_{r}\).