## 2 Materials and Methods
### Reward Function  
In this study, the reward function is based on the cumulative log-returns of the agent over the course of each episode. A _return_, in financial terms, represents the profit or loss generated by an asset over a specific time interval. While both arithmetic and log-returns are commonly used, log-returns are generally preferred in this context for several reasons. One of the key advantages of log-returns is their additive property over multiple periods, which facilitates the calculation of cumulative returns in reinforcement learning (RL) settings. This is particularly important when summing rewards over an entire episode. However, there is some debate in the literature regarding the appropriateness of log-returns, as certain studies suggest they may underestimate extreme price fluctuations compared to arithmetic returns [6; 41]. The log-return for a given time period is computed as \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Agent** & **Component** & **Layer** & **Neurons** & **Activation fn** \\ \hline \multirow{2}{*}{DQN} & \multirow{2}{*}{Agent} & 1 & 64 & ReLU \\  & & 2 & 64 & ReLU \\ \hline \multirow{4}{*}{DDPG} & Actor & 1 & 400 & ReLU \\  & & 2 & 300 & ReLU \\ \cline{2-5}  & \multirow{2}{*}{Critic} & 1 & 400 & ReLU \\  & & 2 & 300 & ReLU \\ \hline \multirow{4}{*}{PPO} & Policy net & 1 & 64 & ReLU \\  & & 2 & 64 & ReLU \\ \cline{2-5}  & \multirow{2}{*}{Value net} & 1 & 64 & ReLU \\  & & 2 & 64 & ReLU \\ \hline \multirow{4}{*}{SAC} & Actor & 1 & 256 & ReLU \\  & & 2 & 256 & ReLU \\ \cline{2-5}  & \multirow{2}{*}{Critic} & 1 & 256 & ReLU \\  & & 2 & 256 & ReLU \\ \hline \hline \end{tabular} The _input_size_ of the first layers is equal to a 1-D vector of dimension 24 \(\times\)_loobluck_ in the experiments with only OHLC prices and 72 \(\times\)_loobluck_ in the experiments with technical indicators from Section 2.4.  
\end{table} 
Table 2: Feature extractor architectures for the MLP for each agent type. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Convolutional Layer** & **Input Channels** & **Output Channels** & **Kernel Size** & **Stride** & **Padding** & **Activation fn** \\ \hline
2D Convolution & input\_size & 32 & 2 & 1 & 0 & ReLU \\
2D Convolution & 32 & 64 & 2 & 1 & 0 & ReLU \\ \hline
**Fully Connected Layer** & **Input Size** & **Output Size** & & & **Activation fn** \\ \hline Flatten & (64, 32, 2, 2) & 6656 & - & - & - & - \\ Linear & 6656 & 128 & - & - & - & ReLU \\ \hline \hline \end{tabular} The _input_size_ of the first layers is equal to a 3-D tensor of dimension _loobluck_\(\times\) 6 \(\times\) 4 in the experiments with only OHLC prices and _loobluck_\(\times\) 6 \(\times\) 12 in the experiments with technical indicators from Section 2.4.  
\end{table} 
Table 3: The architecture of the CNN feature extractor. \[r_{t}=log(\frac{P_{\text{close}}}{P_{\text{open}}}) \tag{11}\]  
where \(P_{\text{close}}\) is the closing price, and \(P_{\text{open}}\) is the opening price for the time period \(t\). The reward function is designed to align with the agent's objective of maximizing cumulative log-returns over an entire episode. The log transformation simplifies the reward computation, allowing the total reward \(R\) for an episode to be expressed as the sum of the log-returns between rebalancing periods. For each rebalancing interval, the log-return is calculated based on the portfolio value \(V\) at the beginning and end of the interval. The portfolio value is derived as the dot product of the asset weights \(w\) and the corresponding asset prices \(P\) at each time step:  
\[R=\sum_{t=1}^{T}\log\biggl{(}\frac{V_{t}}{V_{t-1}}\biggr{)} \tag{12}\]  
where \(V_{t}\) represents the portfolio value at time \(t\), which is given by  
\[V_{t}=\vec{w}_{t}\cdot\vec{P}_{t} \tag{13}\]  
with \(w_{t}\) being the vector of portfolio weights and \(P_{t}\) the asset prices at time \(t\). Cumulative returns are a well-established metric within the financial industry, frequently used to evaluate and compare the performance of various investment strategies.