## 2 Materials and Methods
### Problem Formulation  
In this paper, the portfolio optimization problem is framed as an RL task, where the objective is to find an optimal policy that maximizes the cumulative returns of a portfolio over time. The RL environment simulates the financial market and provides observations in the form of environmental signals, which include either OHLC prices or technical indicators, as described in Section 2.4. Depending on the chosen agent configuration, the observations are processed as either vectors or 3-dimensional tensors, as detailed in Section 2.5. This flexibility allows the RL agent to handle different types of input data effectively, depending on the chosen feature extractor (MLP or CNN). The agent consists of two core components: a feature extractor (e.g., MLP or CNN) to process data and a decision-making algorithm that determines the portfolio weights based on extracted features. The output of the agent is a vector of dimension \(1\times n_{\text{instruments}}\), including the risk-free instrument. For continuous-action agents (SAC, DDPG), the weights are normalized to sum to 1, representing a proportional allocation to each instrument. In contrast, for discrete-action agents (DQN, PPO), the vector contains a single component equal to 1, representing a full allocation to one instrument, while the remaining components are set to 0. The precise architecture and functioning of the agent are further detailed in Appendix A. In each rebalancing period \(t+f\), with \(f\) being the rebalancing frequency in trading periods, the agent proposes a new set of portfolio weights, which are combined with the previous portfolio weights decided at time \(t\) according to Section 2.2 to compute the return over the rebalancing period. The calculation accounts for slippage and transaction costs, modeled based on weight changes, which decrease the return of bought or sold positions as a percentage. One of the assumptions of our paper is that instruments are liquid and fractional assets, as opposed to discrete units (shares) priced at \(P_{i}\). Specifically, the agent's proposal results in three subsets:  
* H (held positions): Assets where the portfolio weight remains unchanged \(w_{i,t}=w_{i,t+f}\). * B (bought positions): Assets where the portfolio weight has increased, requiring the purchase of additional units, \(w_{i,t}<w_{i,t+f}\). * S (sold positions): Assets where the portfolio weight has decreased, requiring the sale of some units, \(w_{i,t}>w_{i,t+f}\). For the first trading period, the return is computed by adjusting the portfolio based on these three subsets, taking into account the impact of market fluctuations during the initial transition period. Using a weight decomposition, we can consider the portfolio weights as the evolution from time \(t-f\) to time \(t\), where the weights are defined as follows:  
* \(\vec{w}_{h}\) corresponds to the portfolio weights of the held positions between \(t\) and \(t+f\) and corresponds, for each instrument, to \(min(w_{i,t},w_{i,t+f})\). * \(\vec{w}_{b}\) corresponds to the positive weight deltas \(\delta_{w_{i}^{+}}=w_{i,t+f}-w_{i,t}\), where \(w_{i,t+f}-w_{i,t}>0\), and corresponds to increased positions on an instrument (buys). * \(\vec{w}_{s}\) corresponds to the negative weight deltas \(\delta_{w_{i}^{-}}=w_{i,t+f}-w_{i,t}\), where \(w_{i,t+f}-w_{i,t}<0\), and corresponds to decreased positions on an instrument (sales).