# Hierarchical Reinforcement Learning for Crude Oil Supply Chain Scheduling
## 3 Problem Formulation  
In this section, we formulate the problem of crude oil supply chain scheduling. As shown in Figure 1, the system involves four facilities: oil fields, import ports, transfer stations, and refineries. The crude oil supply chain scheduling problem is a sequential decision-making problem. As crude oil is dynamically provided by oil fields and import ports at each time step and transported to transfer stations and refineries, the processing and transportation plans should be designed to satisfy system constraints such as the inventory and processing capacity. In order to facilitate the correspondence between symbols and definitions, Table 1 lists the notations defined throughout the paper. The crude oil supply chain scheduling problem is formally defined as \(M=(T,\mathbf{K},\mathbf{V},\mathbf{E},\mathbf{C},\mathbf{L})\), in which \(T,\mathbf{K},\mathbf{V},\mathbf{E},\mathbf{C}\), and \(\mathbf{L}\) represent the total time steps and the set of petroleum materials, facilities, roads, constraints, and constants, respectively. To be specific,  
* The petroleum material set \(\mathbf{K}\) contains crude \(c\), diesel \(d\), and gasoline \(g\). Let \(k\in\mathbf{K}\) represent one of the petroleum materials. \begin{table}
\begin{tabular}{l|l} \hline \hline
**Symbols** & **Definitions** \\ \hline \(M=(T,\mathbf{K},\mathbf{V},\mathbf{E},\mathbf{C},\mathbf{L})\) & A crude oil supply chain scheduling problem \\ \(t,T\) & A time step and the number of total time steps \\ \(k,\mathbf{K}\) & A material kind and the set of material kinds \\ \(c,d,g\) & The material kind of crude, diesel, and gasoline \\ \(v,V,\mathbf{V}\) & A facility, facility set of a set, and facility set of all types \\ \(e,E,E\) & A road, road set related to a facility, and the set of all roads \\ \(o,p,f,r\) & Oilfield, import port, transfer, and refinery \\ \(\mathbf{C}\) & The set of constraints \\ \(LV=\{l_{i}^{k}|v_{i}\in\mathbf{V},k\in\mathbf{K}\}\) & The lower safety inventory limits \\ \(UV=\{u_{i}^{k}|v_{i}\in\mathbf{V},k\in\mathbf{K}\}\) & The upper safety inventory limits \\ \(PV=\{ph_{i}^{k}|v_{i}\in\mathbf{V},k\in\mathbf{K}\}\) & The physical inventory limits \\ \(LQ=\{l_{q}^{t}|v_{i}\in V_{r},t=1,\dots,T\}\) & The lower bound of processing volumes \\ \(UQ=\{u_{q}^{t}|v_{r}\in V_{r},t=1,\dots,T\}\) & The upper bound of processing volumes \\ \(TQ=\{U_{q}|v_{r}\in V_{r}\}\) & The upper bound of total processing volumes \\ \(\mathbf{U}E=\{u_{e_{ij}}|e_{ij}\in\mathbf{E}\}\) & The upper limits of transportation volumes \\ \(\mathbf{L}\) & The set of constant coefficients \\ \(\mathbf{\xi}=\{\zeta_{i}|v_{i}\in\mathbf{V}\}\) & The unit cost of oil in facilities \\ \(\mathbf{\lambda}=\{\lambda_{e_{ij}}\in\mathbf{E}\}\) & The unit cost of oil on roads \\ \(m,n\) & The inventory volume and the demand volume in facilities \\ \(pr,\rho\) & The processing plan and the proportions in processing plan \\ \(q\) & Decision of the processing volume plan in refineries \\ \(w\) & Decision of the transportation volume on roads \\ \(\mathcal{P},\mathcal{H},\mathcal{R}\) & The alert penalty, the transportation cost, and the total cost \\ \(s,a,R\) & The state, action, and reward of a Markov decision process (MDP) \\ Q & The Q-function in Q-learning algorithm \\ \(\mathbf{or}\) & The solving order \\ \(\mathbf{J}\) & The objective function \\ \(x\) & The decision variables \\ \(\alpha,\beta,\epsilon,\mu\) and \(N\) & Hyperparameters \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Notations. Figure 1: Crude oil supply chain scheduling problem. * Define the facility set \(\mathbf{V}=\{V_{o},V_{p},V_{f},V_{r}\}\), where \(V_{o}\) is the set of oil fields, \(V_{p}\) is the set of import ports, \(V_{f}\) is the set of transfer stations, and \(V_{r}\) is the set of refineries. For convenience, let \(V_{i}\in\mathbf{V}\) be the set of facilities of type \(i\), where \(i=o,p,f,r\), and \(v_{i}\in V_{i}\) is a single facility of type \(i\).