## 2 Materials and Methods
### Experimental Design
#### 2.7.2 Simulation Environment  
The simulation environment employed in this study is a custom implementation developed in accordance with the _Gym_([PERSON] et al., 2017) and _Gymnasium_([PERSON] et al., 2017) standards. It is specifically designed to train reinforcement learning agents from the _StableBaselines_3([PERSON] et al., 2017) library, as described in Section 2.3, and is tailored to the requirements of portfolio optimization. This environment ensures compatibility with various RL agents while providing realistic financial simulations for effective agent training.  
In each rebalancing period, the environment generates an observation consisting of time-series data of selected technical indicators, as described in Section 2.4. The observation covers the preceding _lookback_steps_, providing the RL agents with the necessary context to make informed portfolio allocation decisions. The way the observations are structured depends on the feature extractor network used:  
* MLP Configuration: The observation is flattened into a 1-D vector of size \(1\times X\), where \(X\) is determined by the product of the number of instruments, the lookback periods, and the number of technical indicators (i.e., \(X=n_{\text{instruments}}\times\text{lookback\_periods}\times\text{technical\_indicators}\)). Each instrument's data over the lookback window are concatenated sequentially across this vector.
* CNN Configuration: The observation is organized as a 3-D tensor with dimensions lookback_periods \(\times n_{\text{instruments}}\times\text{technical\_indicators}\). This format preserves the temporal and spatial relationships between the data points, enabling the CNN to identify both time-series trends and correlations across instruments.  
Figure 3 provides an illustration of an observation with \(lookback=3\) for the same technical indicator across four different investment instruments, shown as both a 1-D vector for the MLP feature extractor and a tensor with dimensions \(lookback\times indicators\times instruments\) for the CNN. Additional indicators will be concatenated along the same axis for the 1-D vector (MLP) and stacked along the Y-axis for the 3-D tensor (CNN).  
The output of the feature extractor is designed to encapsulate the context provided by the recent history of technical indicators, creating a latent representation of the market state. This representation is then passed to the agent network, where it is used to either map directly to an action in continuous-action agents or to estimate the Q-value for discrete-action agents based on the current latent state.  
The action selected by the agent varies according to the type of RL algorithm employed:  
* For discrete-action agents such as the DQN, the action corresponds to a full allocation to a single instrument, with the agent selecting the action via an argmax operation rather than normalizing Q-values to generate continuous portfolio weights.  
* For continuous-action agents, the action represents the percentage allocation across instruments, ensuring that \(\forall_{i}:w_{i}\geq 0\) and \(\sum_{i}w_{i}=1\), which guarantees long-only positions and a fully invested portfolio. The reward function in the environment is computed as the log-return between consecutive rebalancing periods. This is calculated by multiplying the portfolio weights by the prices of each instrument, as outlined in Section 2.6. This framework enables the agent to optimize cumulative log-returns over the duration of the episode.