## 2 Materials and Methods
### The Reinforcement Learning Background  
Reinforcement learning encompasses methods that enable an agent to develop a policy \(\pi\), guiding its future behavior through interactions with the environment. If a particular action \(a\) leads to a better performance (i.e., higher accumulated rewards), the algorithm reinforces the agent to execute this action more frequently. RL can be viewed as unsupervised learning: the agent gathers scalar feedback (rewards) based on its interactions with the environment. The agent's performance is measured by analyzing these collected rewards, which in turn allow it to optimize its actions to maximize them. At each time step \(t\), the agent, in state \(s\), performs an action \(a\). As a result, it transitions to a new state \(s^{\prime}\) and receives a reward \(r\). Note that receiving a reward \(r\) at every transition is not mandatory in all environments; it is specific to so-called _dense_ reward settings, as studied in this work. Given that the subsequent state \(s^{\prime}\) depends only on the current state \(s\) and action \(a\), rather than on any prior states or actions, the Markov Decision Process (MDP) framework is used to describe the system dynamics. This framework helps the agent learn the optimal policy by evaluating the outcomes of its actions and updating its value estimates for states or actions accordingly. Through the MDP, the agent can consider actions with long-term effects by taking into account both the immediate and future rewards, ultimately learning over extended interactions with the environment to choose actions that maximize the cumulative expected reward rather than focusing solely on short-term gains. The expected cumulative return from an episode, representing the total rewards that the agent may gather starting from time \(t\) to until the episode ends, is given by the following:  
\[G_{t}=r_{t+1}+\gamma r_{t+2}+\gamma^{2}r_{t+3}+\ldots\ =\ \sum_{k=0}^{\infty} \gamma^{k}r_{t+k+1}, \tag{12}\]  
where  
* \(r_{t+1},r_{t+2},\ldots\) represent the sequence of rewards the agent receives at successive time steps following time \(t\);
* \(\gamma\) (the discount factor), with \(0\leq\gamma\leq 1\), specifies the weight the agent assigns to future rewards relative to immediate ones. The state-value function \(V(s)\) and the action-value function \(Q(s,a)\) are fundamental concepts in RL because they allow the agent to anticipate the expected future rewards. The state-value function \(V^{\pi}(s)\) defines the expected cumulative reward when starting in state \(s\) and following policy \(\pi\):  
\[V^{\pi}(s)=\mathbb{E}_{\pi}\big{[}G_{t}\bigm{|}S_{t}=s\big{]}. \tag{13}\]  
The action-value function \(Q^{\pi}(s,a)\) denotes the expected cumulative reward when the agent takes action \(a\) in state \(s\) and subsequently follows policy \(\pi\):  
\[Q^{\pi}(s,a)=\mathbb{E}_{\pi}\big{[}G_{t}\bigm{|}S_{t}=s,\ A_{t}=a\big{]}. \tag{14}\]  
The agent's objective is to find an optimal policy \(\pi^{*}\) that maximizes the value function. The Bellman equation for the optimal state-value function is \[V^{*}(s)\ =\ \max_{a}Q^{*}(s,a), \tag{15}\]  
while for the optimal action-value function, this is given by  
\[Q^{*}(s,a)\ =\ \mathbb{E}\big{[}\,r_{t+1}\ +\ \gamma\,\max_{a^{\prime}}Q^{*}(s^{ \prime},a^{\prime})\ \big{|}\ \ S_{t}=s,\ A_{t}=a\big{]}. \tag{16}\]  
In practice, functions \(V(s)\) and \(Q(s,a)\) are often approximated using parametric forms, such as neural networks. Algorithms like Soft Actor-Critic (SAC), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Truncated Quantile Critics (TQCs) implement the actor-critic architecture, where the actor represents the policy and the critic estimates the action-value function. In Soft Actor-Critic (SAC), the policy \(\pi_{\phi}(a\ |\ s)\) is stochastic and parameterized by a neural network with the parameters \(\phi\). The goal is to maximize both the cumulative reward and the entropy of the policy, the latter of which encourages exploration.