## Appendix A RL Agents
### Sac  
SAC [27] is an off-policy, actor-critic algorithm based on the maximum entropy reinforcement learning framework. SAC aims to maximize both the expected return and the entropy of the policy. The inclusion of entropy in the objective encourages exploration and prevents premature convergence to suboptimal policies by promoting stochasticity in action selection. SAC is particularly well suited for continuous control tasks due to its ability to handle high-dimensional action spaces efficiently. It combines stable, sample-efficient learning with strong exploration capabilities, making it a robust choice for real-world problems like portfolio optimization, where market conditions can be highly dynamic and uncertain.  
The core of SAC revolves around maximizing a modified objective function that balances reward maximization and policy entropy. The objective is defined as  
\[J(\pi)=\sum_{t=0}^{T}\mathbb{E}_{(s_{t},a_{t})\sim\rho_{\pi}}[r(s_{t},a_{t})+ \alpha\mathcal{H}(\pi(\cdot|s_{t}))] \tag{11}\]  
where \(\mathcal{H}(\pi(\cdot|s_{t}))\) is the entropy of the policy at state \(s_{t}\), and \(\alpha\) is a temperature parameter that controls the trade-off between reward maximization and entropy maximization. SAC maintains two Q-value functions \(Q(s,a)\), a value function \(V(s)\), and a policy \(\pi(a|s)\). The update for each function is based on the following.  
The Q-function is updated using soft Bellman backups:  
\[Q_{\theta}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\sim p}[V_{ \psi}(s_{t+1})] \tag{12}\]The value function update is  
\[V_{\psi}(s_{t})=\mathbb{E}_{a_{t}\sim\pi}[Q_{\theta}(s_{t},a_{t})-\log\pi(a_{t}|s _{t})] \tag{11}\]  
And, lastly, the policy is updated by minimizing the KL divergence between the policy and a Boltzmann distribution of the Q-function:  
\[J_{\pi}(\phi)=\mathbb{E}_{s_{t}\sim D}\Big{[}\mathbb{E}_{a_{t}\sim\pi_{\phi}}[ \alpha\log\pi_{\phi}(a_{t}|s_{t})-Q_{\theta}(s_{t},a_{t})]\Big{]} \tag{12}\]  
SAC's most significant advantage is its ability to balance exploration and exploitation through the entropy term. By promoting exploration, SAC avoids the risk of prematurely converging to suboptimal policies, a common issue in deterministic approaches like DDPG. SAC is also off-policy, meaning it can reuse past experiences stored in a replay buffer, making it sample-efficient compared to on-policy methods. This efficiency is critical in data-scarce environments like financial markets. Furthermore, SAC leverages a stochastic policy, which allows for better exploration and smoother learning compared to algorithms that rely on deterministic policies.  
SAC's reliance on entropy maximization introduces a trade-off: while promoting exploration, the algorithm can occasionally over-explore, leading to slower convergence, especially if the temperature parameter \(\alpha\) is not well tuned. Additionally, SAC can be more computationally intensive than simpler policy gradient methods due to the need for maintaining and updating multiple value functions and a policy network. The performance of SAC is also sensitive to the choice of hyperparameters, particularly the temperature \(\alpha\), which must be carefully adjusted for each environment to achieve an appropriate balance between exploration and exploitation.