# Hierarchical Reinforcement Learning for Crude Oil Supply Chain Scheduling
## 3 Problem Formulation
 \tag{1}\]  
Equation (1): The transfer station \(v_{f}\) receives crude oil \(c\) from all oil fields to which the import ports connected with the amount of \(w_{e_{ef}}^{t}\) and \(w_{e_{pf}}^{t}\) (\(w_{e_{fi}}=0\) if \(v_{i}\) and \(v_{f}\) are disconnected) sends crude oil \(c\) to all connected refineries with the amount \(w_{e_{fr}}^{t}\) and provides a \(n_{f}^{c,t}\) amount of crude oil \(c\) on demand. Equation (2): Different from the transfer station, a refinery \(v_{r}\) also receives crude oil \(c\) from all connected transfer stations with amount \(w_{e_{f}}^{t}\), and processes the amount \(q_{r}^{t}\) of crude \(c\) into gasoline and diesel. Equation (3): At time step \(t\), the \(\rho_{r}^{d}q_{r}^{t}\) amount of diesel is processed out, and the \(n_{r}^{d,t}\) amount of it is deducted from the refinery on demand. Equation (4) has the same logic as Equation (3). When the inventory volume exceeds safety limitations \(LV\) and \(UV\), the alert penalty at time step \(t\) is  
\[\mathcal{P}^{t}= \sum_{v_{i}\in\
abla V\in K}\zeta_{i}(m_{i}^{k,t}-u_{i}^{k}) \mathbb{1}[m_{i}^{k,t}>u_{i}^{k}]\] \[+\sum_{v_{i}\in\
abla V\in K}\zeta_{i}(l_{i}^{k}-m_{i}^{k,t}) \mathbb{1}[m_{i}^{k,t}<l_{i}^{k}] \tag{5}\]  
Moreover, the total transportation cost of the system at time step \(t\) is  
\[\mathcal{H}^{t}=\sum_{e_{f_{r}}\in E_{fr}}\lambda_{e_{fr}}w_{e_{fr}}^{t} \tag{6}\]  
This problem aims to minimize the total cost of \(T\) steps. At the same time, the system keeps all petroleum volumes under physical limitations (Equations (7b) and (7c)) and processing volumes under capacity limitations (Equations (7d) and (7e)):  
\[\max_{\mathbf{w}_{e_{f}},\mathbf{\mathcal{H}}} J=\sum_{t=1}^{T}-\alpha_{1}\mathcal{P}^{t}-\alpha_{2}\mathcal{H}^{t} \tag{7a}\] \[\text{s.t.}0\leq m_{i}^{k,t}\leq p\mu_{i}^{k}\quad\forall v_{i} \in\mathbf{V},k\in K,t=1,\ldots,T\] (7b) \[0\leq w_{e_{ij}}^{t}\leq u_{e_{ij}}\quad\forall e_{ij}\in E,t=1, \ldots,T\] (7c) \[l_{q_{r}}^{t}\leq a_{r}^{t}\leq u_{q_{r}}^{t},\quad\forall v_{r} \in V_{r},t=1,\ldots,T\] (7d) \[\sum_{t=1}^{T}q_{r}^{t}=U_{q_{r}},\quad\forall v_{r}\in V_{r} \tag{7e}\]  
where \(\alpha_{1}\) and \(\alpha_{2}\) are weights. The difficulties of this problem lay in two aspects: (1) Large problem scale. We aim to solve the crude oil supply chain scheduling problem in the real world, which includes hundreds of facilities and thousands of roads, and the decision period lasts about 30-time steps. These form a decision space of about \(10^{7}\) with continuity. (2) Uncertain environment. Environment variables are not deterministic but are generated in real-time, which means the variables at time step \(t+1\) cannot be observed at time step \(t\). Suppose the environment variables of \(T\) steps are known in advance; in that case, the problem can be formulated as a Linear Programming (LP) problem which can be solved by powerful solvers (e.g., [PERSON] [34]). However, this kind of method cannot be applied to our problem directly due to uncertainty. On the other hand, as the one-step problem is also an LP problem, an intuitive way might be solving the multi-step problem step by step. However, traditional solvers are unable to take the future influence of the current solution into consideration, thus leading to sub-optimal solutions when dealing with multi-step problems, and even failing to find a feasible solution after several steps (as shown in Section 6).