# Hierarchical Reinforcement Learning for Crude Oil Supply Chain Scheduling
## 3 Problem Formulation
In each facility, \(v_{i}\), the inventory volume of petroleum kind \(k\) at time step \(t\) is denoted as \(m_{i}^{k,t}\), and the demand volume of the petroleum kind \(k\) at time step \(t\) is \(n_{i}^{k,t}\). For each refinery, \(v_{r}\), \(q_{r}^{t}\) represents the processing volume of crude oil at time step \(t\), and a fixed processing scheme \(pr_{r}\) determines that the \(\rho_{r}^{d}\) unit of diesel and \(\rho_{r}^{g}\) unit of gasoline can be produced from 1 unit of crude oil. * Define the set of roads \(\mathbf{E}=\{E_{of},E_{or},E_{pf},E_{rr},E_{fr}\}\), where \(E_{ij}\in\mathbf{E}\) is the set of transport roads from facility \(v_{i}\) to facility \(v_{j}\) (i.e., \(j=o,p,f,r\) is the facility type). Each \(e_{ij}\in E_{ij}\) represents one road from \(v_{i}\) to \(v_{j}\), and the set of roads starting from facility \(v_{i}\) is denoted as \(E_{i}\). In addition, the transportation volume on road \(e_{ij}\) at time step \(t\) is denoted as \(w_{e_{ij}}^{t}\), and if there is no road from \(v_{i}\) to \(v_{j}\), \(w_{e_{ij}}^{t}=0\). * The constraints of the crude oil supply chain scheduling problem are defined in \(\mathbf{C}\), which contains seven sets: \(LV=\{l_{i}^{k}|v_{i}\in\mathbf{V},k\in\mathbf{K}\}\) and \(UV=\{u_{i}^{k}|v_{i}\in\mathbf{V},k\in\mathbf{K}\}\) are the lower and upper safety imprints of facility \(v_{i}\) for petroleum kind \(k\), \(PV=\{ph_{i}^{k}|v_{i}\in\mathbf{V},k\in\mathbf{K}\}\) is the upper physical inventory limit of facility \(v_{i}\) for the petroleum kind \(k\), \(LQ=\{l_{q_{r}}^{t}|v_{r}\in V_{r},t=1,\ldots,T\}\) and \(UQ=\{u_{q_{r}}^{t}|v_{r}\in V_{r},t\)=1,...,T\}\) are the lower and upper bound of the processing volume in refinery \(v_{r}\) at time step \(t\), \(TQ=\{U_{q_{r}}|v_{r}\in V_{r}\}\) is the upper bound of total processing volume in refinery \(v_{r}\) for the whole \(T\) steps, and \(UE=\{u_{e_{ij}}|e_{ij}\in\mathbf{E}\}\) is the upper limit of the transportation volume on road \(e_{ij}\). * The constant coefficients related to the system cost are denoted as \(\mathbf{L}=\{\mathbf{\zeta},\mathbf{\lambda}\}\), where each \(\zeta_{i}\in\mathbf{\zeta}\) is the unit cost of facility \(v_{i}\) when its inventory volume \(m_{i}^{k,t}\) exceeds upper limit \(u_{i}^{k}\) or lower safe limits \(l_{i}^{k}\), and each \(\lambda_{e_{ij}}\in\mathbf{\lambda}\) is the unit cost of the road \(e_{ij}\) as \(w_{e_{ij}^{t}}\
eq 0\). At each time step \(t\), the amount of crude oil from oil fields and import ports is randomly generated from the environment and is observed in real-time. Denote \(w_{e_{ef}}^{t}\), \(w_{e_{ef}}^{t}\), \(w_{e_{ef}}^{t}\), and \(w_{e_{ep}}^{t}\) as the supplied volume from oil field \(v_{o}\) to transfer station \(v_{f}\), oil field \(v_{o}\) to refinery \(v_{r}\), import port \(v_{p}\) to transfer station \(v_{f}\) and import port \(v_{p}\) to refinery \(v_{r}\). Meanwhile, the demand volume \(n_{i}^{k,t}\) of each facility \(v_{i}\) is observed from the environment. The processing volume plan \(q_{r}^{t}\) and transportation plan \(w_{e_{fr}}^{t}\) are controlled by our system, then the corresponding inventory volume \(m_{i}^{k,t}\) of each facility can be deduced as follows:  
\[ \begin{split} m_{f}^{c,t}=& m_{f}^{c,t-1}+\sum_{e_{ ef}\in E_{ef}}w_{e_{ef}}^{t}+\sum_{e_{ef}\in E_{pf}}w_{e_{pf}}^{t}\\ &-\sum_{e_{fr}\in E_{f}}w_{e_{fr}}^{t}-n_{f}^{c,t}\\ m_{r}^{c,t}=& m_{r}^{c,t-1}+\sum_{e_{or}\in E_{w}}w_{e_{ cor}}^{t}+\sum_{e_{pr}\in E_{pr}}w_{e_{pr}}^{t}-n_{r}^{c,t}\\ &+\sum_{e_{fr}\in E_{f}}w_{e_{fr}}^{t}-q_{r}^{t}\\ m_{r}^{d,t}=& m_{r}^{d,t-1}+\rho_{r}^{d}q_{r}^{t}-n_{r} ^{d,t}\\ m_{r}^{g,t}=& m_{r}^{g,t-1}+\rho_{r}^{g}q_{r}^{t}-n_{r} ^{g,t}\end{split}