## 2 Materials and Methods
### Hybrid DRL/CBR Method for Marine Oil Spill Emergency Response  
In this research, a marine oil spill emergency response is assumed as an MDP, and the policy is trained by the DQN algorithm using CBR. The CBR system provides an environment for reinforcement learning (RL) agent exploration. Many RL algorithms have been developed to learn approximations of an optimal action based on agent experience in a given environment. The return function is defined in the MDP as follows:  
\[V_{\gamma}^{\pi}(x)=\mathbb{E}_{\pi}\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}r_{t+ 1}\mid x_{0}=s_{0}\Bigg{]}, \tag{3}\]  
where future rewards are discounted by a factor \(\gamma\) per time step \(t\) with a start state \(s_{0}\in S\). State \(S\) is a vector composed of features representing a marine oil spill scenario, where \(r\) is the reward for the current emergency response action. The DQN uses experience to learn value functions that map state-action pairs to the maximal expected reward that can be achieved for a given state-action pair. The optimal action value function \(Q^{*}(s,a)\) is defined as the maximum expected return achievable by following any strategy after a state \(s\) is reached and an action \(a\) is taken:  
\[Q^{*}(s,a)=\underset{\pi}{max}\mathbb{E}[R_{t}\mid s_{t}=s,\;a_{t}=a,\pi], \tag{4}\]  
where \(\pi\) is a function that maps policies to emergency response actions; emergency response action \(a\in A\), and \(A\) is a list of possible marine oil spill emergency response actions decision makers can take for the current spill scenario. Equation (5) shows that the optimal value function \(Q^{*}(s,a)\) gives the maximum emergency response action value for spill scenario \(s\) and action \(a\) achievable by policy \(\pi\):  
\[Q^{*}(s,a)=\sum_{s^{\prime}\in S}P^{a}_{s\to s^{\prime}}\bigg{(}R^{a}_{s \to s^{\prime}}+\gamma\underset{a^{\prime}\in A}{max}Q^{*}(s^{\prime},a^{ \prime})\bigg{)}, \tag{5}\]  
where \(P^{a}_{s\to s^{\prime}}\) is the transition probability and \(R^{a}_{s\to s^{\prime}}\) is the reward at state \(s\) translated to \(s^{\prime}\). This equation is in agreement with the following intuition: the optimal strategy involves selecting the emergency response action \(a^{\prime}\) that maximizes the expected value, which is a \(\gamma\)-related cumulative reward function when the optimal value \(Q^{*}(s^{\prime},a^{\prime})\) of the sequence spill scenario \(s^{\prime}\) at the next time step is known for all possible emergency response actions \(a^{\prime}\). The optimal action value function obeys an important identity known as the Bellman optimization equation, which can also be used as an iterative updating formula with a learning rate parameter \(\alpha\):  
\[Q(s,a)\leftarrow(1-\alpha)Q(s,a)+\alpha\Big{(}R_{s\to s^{\prime}}^{a}+ \gamma Q(s^{\prime},a^{\prime})\Big{)}. \tag{6}\]  
The Q-network is a neural network with a weight of \(\theta\) as a function approximator to estimate the action value function \(Q(s,a;\theta)\approx Q^{*}(s,a)\). A Q-network can be trained by minimizing a sequence of the loss function \(L(\theta)\) that changes at each iteration \(t\), where \(y_{t}\) is the target for iteration \(t\) and \(\rho(. )\) is a probability distribution over a sequence of oil spill scenarios \(s\) and emergency response actions \(a\). \[L_{t}(\theta_{t})=\mathbb{E}_{s,a\sim p(.)}[y_{t}-Q(s,a;\theta_{t})], \tag{7}\]  
In this paper, the model is trained with an actor-critic strategy [23]. The actor selects a behavior based on probability, and the critic estimates performance based on the actor. The critic is trained at every step, and the actor synchronizes with the parameters of the critic model after specific steps. The neural network of actors and critics consists of a nine-layer convolution neural network for the state function approximator. The input to the neural work is a vector of the oil spill scenario instance.