## Appendix A RL Agents
### Ddpg  
DDPG [25] is an actor-critic, model-free reinforcement learning algorithm designed for continuous action spaces. It extends the DQN's [24] capabilities to continuous actions, which are typical in real-world systems. Unlike the DQN, which is value-based and suitable for discrete actions, DDPG employs a deterministic policy gradient approach.  
The algorithm uses two networks: an actor, which learns a policy that maps states to actions, and a critic, which evaluates the action-value function \(Q(s,a)\). The critic's objective is to maximize the expected return, while the actor optimizes the policy based on feedback from the critic. The _critic_ learns the action-value function \(Q(s,a)\) using the Bellman equation:  
\[Q^{\pi}(s_{t},a_{t})=\mathbb{E}[r_{t}+\gamma Q^{\pi}(s_{t+1},\mu(s_{t+1}))] \tag{10}\]  
The _actor_ learns a deterministic policy \(\mu(s)\), which directly maps states to actions: \(\mu(s|\theta^{\mu})=a\). The actor is trained by following the policy gradient:  
\[\
abla_{\theta^{\mu}}J\approx\mathbb{E}\Big{[}\
abla_{a}Q(s,a|\theta^{Q}) \
abla_{\theta^{\mu}}\mu(s|\theta^{\mu})\Big{]} \tag{11}\]  
Similar to the DQN, DDPG employs a _replay buffer_ to break the correlation between consecutive transitions, enhancing training stability and efficiency. Both the actor and critic networks have corresponding target networks, \(Q^{\prime}\) and \(\mu^{\prime}\), which are slowly updated to provide stable target values during training:  
\[\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime} \tag{12}\]  
Since the actor outputs deterministic actions, exploration is induced by adding noise to the action selection process. DDPG typically uses an _Ornstein-Uhlenbeck_ noise process for temporally correlated exploration; this noise helps in exploring continuous action spaces more effectively than simple uncorrelated Gaussian noise.  
\[a_{t}=\mu(s_{t}|\theta^{\mu})+\mathcal{N}_{t} \tag{13}\]  
DDPG's key advantage is its efficiency in continuous action spaces, directly outputting continuous actions without requiring discretization. This makes it ideal for tasks requiring precise control, such as real-time portfolio weight adjustments. Additionally, DDPG incorporates target networks for both the actor and critic, which significantly improves the stability of learning, especially in environments with high variance. Another strength of DDPG is that it is an off-policy algorithm, meaning it can learn from past experiences stored in the replay buffer, thus making better use of data and reducing the variance in policy updates.  
However, DDPG also presents some disadvantages, especially in terms of exploration. Since it outputs deterministic actions, DDPG relies heavily on adding noise to ensure sufficient exploration. If the exploration is not well tuned, this can lead to suboptimal policies, particularly in highly stochastic environments such as financial markets with unpredictable price movements. Moreover, DDPG is sensitive to hyperparameter settings, such as the learning rates for the actor and critic, the type of noise process used, and the size of the replay buffer. Poor tuning of these parameters can result in slow learning or, in some cases, divergence of the model.