Bilevel Reinforcement Learning for Stock Data with A Conservative TD Ensemble

Authors: Haochen Yuan, Minting Pan, Yunbo Wang, Xiaokang Yang
Primary area: reinforcement learning
Keywords: Reinforcement learning, stock markets, portfolio optimization

Abstract:
Reinforcement learning (RL) has shown significant promise in stock trading. A typical solution involves optimizing cumulative returns using historical offline data. However, it may produce less generalizable policies that merely "memorize" optimal buying and selling actions from the offline data while neglecting the non-stationary nature of the financial market. We frame stock trading as a specific type of offline RL problem. Our method, MetaTrader, presents two key contributions. First, it introduces a novel bilevel actor-critic method that spans both the original stock data and its transformations. The fundamental idea is that an effective policy should be generalizable across out-of-distribution data. Second, we propose a novel variant of conservative TD learning, utilizing an ensemble-based TD target to mitigate value overestimation, particularly in scenarios with limited offline data. Our empirical findings across two publicly available datasets demonstrate the superior performance of MetaTrader over existing methods, including both RL-based approaches and stock prediction models.

Decision: reject
Reviewer scores: 3, 3, 8, 5

Reviews and discussion:
<thread>
creation date:
1729501908701

thread_type:
official_review

<thread_object>
creator: reviewer_jzah

<content>
summary:
This paper seeks to enhance classical reinforcement learning through the introduction of two techniques: bilevel learning and conservative TD ensemble. The effectiveness of these improvements is evaluated using experiments on two stock datasets.

strengths:
The paper is well-structured, and the ablation studies are conducted thoroughly.

weaknesses:
### Major Issues:

1. **Lack of Discussion on Non-Stationarity**: There is insufficient discussion on how the paper addresses the issue of non-stationarity. As I understand it, the paper attempts to mitigate non-stationarity through specific data transformations. However, the rationale behind the choice of these three particular transformations is unclear. For instance, in F1, why is only the top 10% considered and not the bottom 10% as well? The action space allows for both long and short positions. Regarding F2, how does disrupting the order of observations in time help overcome non-stationarity? If time observations are treated as completely independent, the sequence would be unpredictable. For F3, as explained in Appendix A, future data is compressed into the present to create a non-empty input. Applying this approach in the out-of-sample leads to the future information issue. These questions raise concerns about the experimental validity.
  
2. **Input Construction and Covariance Matrix**: A critical component of the model input is the sample covariance matrix between stocks. How exactly is this matrix computed? According to random matrix theory, when the number of assets is close to the number of time observations, the sample covariance matrix becomes singular and provides a biased estimate of the true covariance matrix. Although this method may still be applicable to the two datasets in the paper (with 300 and 100 stocks, respectively), it raises concerns about the scalability of this approach to larger datasets.
  
3. **Experimental Shortcomings**:
  
  - First, lines 401-412 involve the calculation of true values, and there appear to be issues with this in the paper. To compute the true value theoretically, a globally optimal policy is required. I may have missed some details, but I hope the authors can clarify this in their rebuttal.
  - Second, regarding the performance metrics, the portfolio return aligns with the annual return. It would strengthen the results if the maximum drawdown (MDD) were also reported alongside the annual return.
  - Finally, the stock prediction strategy—buying only the single stock with the highest predicted value—is problematic, as it introduces excessive randomness into the strategy. A more convincing approach would be to purchase the top 10% of stocks based on cross-sectional prediction values, with equal weighting across them.

### Minor Issues:

1. Some of the terminology used in the paper is unconventional. For instance, "portfolio return" in Table 1 is more commonly referred to as "cumulative return," and "annual return" should be labeled as "annualized return." Additionally, the risk-free rate used in the analysis is not specified.
  
2. Several references on ensembled Q-learning have been published, but this paper lacks a rigorous discussion of these existing works.

questions:
1. Please carefully reconsider how you structure your network inputs.
2. Please ensure that the comparison between the proposed method and classical methods is conducted fairly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729545659574

thread_type:
official_review

<thread_object>
creator: reviewer_q9cs

<content>
summary:
This paper studies a way to train an RL policy for stock trading using past transaction data. To deal with the non-stationary and out-of-distribution data in the deployment phase, the paper proposes to perform (i) data transformation (augmentation), (ii) updating the Q function via pessimistic ensemble among multiple subsets of (augmented) data, and (iii) fine-tuning the data collected in the most recent time period. The experiment results demonstrate that the proposed method works better than existing online and offline RL approaches in the real stock transaction data.

strengths:
- **Data transformation (augmentation) makes most of the application-specific features of finance.**

As a key component of the proposed approach, the paper proposed to perform data augmentation to enhance the generalizability of the model to out-of-distribution (OOD) data. In many RL settings, this is usually not a promising approach due to the difficulty of predicting/obtaining rewards for the OOD (state, action) pairs. 

However, in the finance application, this is not the case because we can assume individual actions do not make a huge impact on the market dynamics, and what depends on the actions (selling or buying) is only the individual reward they get from these transactions. This paper nicely exploits this structure and does data augmentation.

- **Approaches, including bi-level optimization and fine-tuning, sounds reasonable and the components are harmoniously combined.**

The paper combines the aforementioned components in a sound way, and there are no components that seem redundant or unnecessary. The motivations behind why the proposed method needs each component are also well explained. While each component itself (e.g., fine-tuning and conservative learning) is not very novel, yet, I think this idea is worth sharing. 

- **Well done ablations and experiments.**

The baselines are picked from representative algorithms of approaches in predictions, online and offline RL. Also, ablations on w/ and w/o data transformation, w/ and w/o fine-tuning, and runtime comparison are all informative for readers.

- **Clarity of the paper.**

Overall, the paper is clearly written and easy to follow.

weaknesses:
To be honest, there is not much to point out for weakness. However, the following might have room for improvement.

- **Related work on offline RL and offline-to-online RL**

The current related work focuses on the existing approaches for learning a stock-trading policy. Adding discussion on why the conventional offline RL does not work and why the proposed method works might be helpful for readers who are not very familiar with offline RL. Also, there is some work on offline-to-online RL, which does online fine-tuning after performing offline RL. This approach is relevant to the proposed method, and can be worth mentioning in the related work section.

- **Description about the proposed method in Introduction**

While the paper was in general very well written, the description of the proposed method in the introduction may have some room for improvement. It was difficult for me to imagine what the proposed method looks like only from the introduction (in particular, it was not clear to me what "in-domain profits" refer to and what the outer-loop optimization does), while It became clear after reading Section 3.

questions:
- How does the proposed method (and other baseline) determine the trajectory length? Are there any guidelines to pick a reasonable trajectory length if there are some tradeoffs between short and long trajectories?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730637236688

thread_type:
official_review

<thread_object>
creator: reviewer_hygz

<content>
summary:
This paper applies RL to portfolio management, focusing on generalization in dynamic financial markets. Traditional RL models often overfit offline data, leading to rigid policies. To address this, the authors propose MetaTrader, introducing a bilevel actor-critic approach and a conservative TD learning variant with an ensemble-based target to reduce value overestimation. Experiments on two public datasets indicate that MetaTrader achieves improvements over existing RL and stock prediction models, suggesting enhanced adaptability in changing markets.

strengths:
•	OOD (out-of-distribution) issues are prevalent in the stock market. This paper focuses on augmenting offline data to train a policy with stronger generalization capabilities to handle distribution shifts in online stock data. This is a crucial and interesting topic.

•	The writing is logically structured and easy to follow, making the content clear and accessible to readers. The arguments are presented in a straightforward manner, allowing readers to grasp the key points without difficulty.

•	The paper includes appropriate experiments and ablation studies to support its findings. These experiments provide evidence for the proposed method’s effectiveness, though certain aspects of the experimental setup could be further improved for a more comprehensive evaluation.

weaknesses:
•	Relying on training the policy with real and augmented data, followed by fine-tuning on real data to address the OOD problem in the stock market, is not an ideal approach. This can be observed in the following two aspects:

1) The augmentation methods have limitations. For instance, do the F2 and F3 methods mentioned in the paper potentially introduce data leakage? Additionally, is the choice of the top 10% in F1 (why specifically 10%?) reasonable?

2) The effectiveness of fine-tuning is questionable. Based on findings in the application of RL to finance, the performance of fine-tuning RL can be similar to or even weaker than training RL from scratch. The experiments in the paper do not include a comparison between fine-tuning RL and training RL from scratch to validate its effectiveness.

Note: To clarify, “training from scratch” here refers to directly using the original data for portfolio management without involving data transformations in training.

•	The experimental setup has some flaws, detailed as follows:

1) Data Validity: The experimental data only goes up to 2022, without including more recent data (e.g., up to 2024), which makes the findings less convincing. Websites like Yahoo Finance (https://finnhub.io/), Alpaca (https://docs.alpaca.markets/), and FMP (https://site.financialmodelingprep.com/developer/docs#chart-intraday) provide easy access to the latest data, which would strengthen the study’s confidence.

2) Stock Selection and Market Representation: While the CSI300 index theoretically consists of 300 stocks, only 88 were selected in this study. Additionally, both chosen stock markets consist of only around 80 stocks, without distinguishing between different market scales. This limited selection makes the results less persuasive, and it would be beneficial to include stock markets of varying scales for a more comprehensive analysis.

3) Evaluation Metrics: The choice of metrics is somewhat limited. The study only focuses on return-related metrics (even though the Sharpe Ratio accounts for risk-adjusted returns, it still primarily measures profitability). Including risk metrics such as VOL and MDD would provide a more balanced evaluation of performance.

Note: I noticed that this paper follows the experimental setup of StockFormer. However, using data from two years ago is not ideal and updating the data could significantly improve the credibility of the experiments. Furthermore, StockFormer included MDD as a risk metric, so it’s unclear why this study chose not to incorporate it in its evaluation criteria.

•	Benchmark Selection**.** Many financial companies still use machine learning and deep learning methods based on prediction, such as LightGBM, LSTM, and Transformer models. Including these in the experiments would provide a more convincing comparison. Qlib (https://github.com/microsoft/qlib) can be a helpful reference for implementing these methods.

questions:
•	As described in the weaknesses section, is there a potential issue of data leakage with the three types of data transformations mentioned? Additionally, why is it set to the top 10%? It would be helpful if the authors could clarify these points.

•	In Figure 3, the augmented data is labeled from 1 to M, while the finetuning data is labeled as M+1 to M'. If I understand correctly, should it actually be M+1 to M+M'? If this is not a typo, please disregard this comment. If this is indeed a writing error, it would be helpful for the authors to adjust any other relevant parts of the paper accordingly.

•	Lines 87–94 mention that the state data consists of action-free and action-dependent components. I would like to know more about the data processing methods used. For example, variables like price, cash holdings, and position can exhibit significant differences over extended trading periods. For instance, Apple’s (AAPL) price on the NASDAQ was around $12 in 2011, but by 2022, it had exceeded $130. This shift in stock prices can lead to substantial differences in cash, position, and price scales. Without normalization, how is RL training stability maintained? It would be helpful if the authors could explain this in detail.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666192744

thread_type:
official_review

<thread_object>
creator: reviewer_muzc

<content>
summary:
The paper introduces "MetaTrader," a reinforcement learning (RL) approach for stock trading that aims to overcome challenges of overfitting to offline data and lacking generalizability in non-stationary financial markets. MetaTrader trains policies to perform well on transformed out-of-distribution (OOD) data, with a conservative temporal difference (TD) ensemble, designed to mitigate value overestimation in offline RL. Through bilevel actor-critic training on both original and transformed stock data, MetaTrader builds robustness to OOD scenarios. Empirical results on two stock datasets indicate MetaTrader's superiority over existing RL and stock prediction methods, with improved performance in portfolio returns and risk-adjusted metrics like Sharpe ratio, demonstrating its potential for robust and adaptable trading strategies.

strengths:
This paper addresses an interesting and important problem: stock trading in the offline RL setting. The bilevel optimization approach combined with conservative TD learning directly addresses practical challenges in offline RL for trading, particularly the out-of-distribution (OOD) generalization problem.

The paper effectively positions stock trading as a unique offline RL problem, underscoring the potential for dataset expansion through their decoupled MDP framework.

 The empirical results demonstrate a comparison with other existing models, including SARL, FinRL-SAC, FinRL-DDPG, and StockFormer. Real-world data sets, specifically the CSI-300 Index and NASDAQ-100, are used.

weaknesses:
The notations are not clearly explained and are sometimes used inconsistently, which makes them hard to follow. Some points related to clarity are outlined below.

1. Observation space on page 2: “$K$ technical indicators that reflect the temporal trends of stock prices”. There is no explanation of what technical indicators are or how they reflect the temporal trends.

2. State space on page 2: “The action-free market state $h_t$ is composed three types of latent states $s_t^{\text relat}$, $s_t^{\text long}$, $s_t^{\text short}$ generated from the observation $o_t^{\text price}$, $o_t^{\text stat}$, $o_t^{\text conv}$”. There is no explanation about how $s_t$ is generated by $o_t$. Instead, the authors refer to Eq (1) on page 3, however, Eq (1) does not include $s_t$. 

3. Dimension of $h_t$ on page 3: The dimension of $h_t$ is stated as the number of stocks times $D$, but there is no explanation of what the notation $D$ represents.

4. Daily prices in Appendix A: The input data for daily prices are denoted by $p_t$. This should be written as $o_t^{\text close}$. The notations are used inconsistently.

5. Metrics in Appendix C: The evaluation metrics for the experiments are defined in Equations (4), (5), and (6). The definitions are very vague and the notations are not clearly explained, lacking connection to $o_t$, $h_t$, $z_t$, $a_t$ defined in Section 2. 


The authors manipulate the input data using the three transformation methods. While the paper notes that data transformations are used to create diverse subsets, it does not provide a clear rationale for the chosen transformations. These transformations can substantially impact the underlying factors driving stock prices and alter correlations between stocks, so an explanation of their selection and intended effects would strengthen the work. Appendix A explains their methods; however, there is no justification or theoretical background provided. Although the empirical studies demonstrate that the proposed algorithm outperforms the existing methods using two real-world data sets, the benefits of using the proposed algorithm are not clear to me, because it functions like a black-box solution.


Using an ensemble-based conservative TD target to mitigate overestimation in offline RL, though effective, lacks novelty, as this technique has been explored extensively in previous research.


Since this work primarily emphasizes empirical results over theoretical contributions, the experimental section should be more robust. For instance, the significance of the findings in Tables 3, 4, and 5 cannot be interpreted without standard deviations. To provide a clearer picture of the proposed algorithm’s stability, the authors could use a rolling training/test set and report both the mean and standard deviation of performance metrics. This would offer a more comprehensive view of the algorithm’s consistency and reliability.

questions:
Please see the weaknesses listed above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732546357371

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Uploaded

comment:
We thank all reviewers for the constructive comments and have updated our paper accordingly. Please check out the new version!

Important changes and new results include:
1. Clarify the technical novelty of this work (Section 1).
2. Explain the key inspirations behind the proposed data transformation methods (Section 3.2).
3. Compare our method with existing ensemble-based Q-learning methods (Section 3.3 and Appendix G).
4. Add the standard deviations of experiments with multiple random seeds (Tables 3, 4, and 5).
5. Add discussions and experiments on a larger market scale (Section 4.4 and Appendix F.1).
6. Add risk evaluation by Maximum Drawdown (Appendix F.2).
7. Add evaluation on more recent data (Appendix F.3).
8. Analyze the effectiveness of model finetuning for the RL-for-Finance methods (Appendix F.4).

Minor changes:
1. Add a footnote to provide a detailed explanation of the meaning of "in-domain profit" in Section 1.
2. Correct the notation of the latent states in Section 2.
3. Add a description of notation $D$ in Section 3.1.
4. Modify the superscript of the data range $M+M^{\prime}$ in Figure 3.
5. Modify terminologies in Section 4: Portfolio Return -> Cumulative Return, Annual Return -> Annualized Return.
6. Include prior work for offline-to-online RL in Section 5.
7. Modify the notations in Appendix A to align with the notations in the main text.
8. Clarify the data normalization method used in our model (Appendix B.3).
9. Polish the calculation formula for the evaluation metrics (Appendix C).
10. Introduce details of the technical indicators (Appendix E).

We appreciate the great efforts by the AC and reviewers. We are very confident that our paper is qualified for ICLR in both novelty and experimental results. Please do not hesitate to let us know for any additional comments on the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732546564767

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (Part 1)

comment:
> 1. To Reviewer MUZC, hygz, jZAh: Why are these data transformation methods chosen?

**(1) Why do we use data transformation? How do we handle non-stationarity?**

One primary contribution of our work is to enhance the generalization capability of the policy. In practice, we approach this from both the algorithmic and data perspectives, which are closely intertwined. 
- From the algorithmic perspective, we adopt the bilevel optimization algorithm to prevent overfitting during offline learning, in the sense that keeping the model from simply memorizing the optimal policy based on specific patterns in the training set.
- From data perspective, our data transformation methods fundamentally aim to generate meaningful augmented data that aligns well with the bilevel optimization algorithm. 

Specificially, we employ data transformations to simulate the extreme changes of the highly complex and stochastic stock market dynamics.

**(2) Why are these transformation methods chosen?**

We acknowledge that an unsuitable data transformation method may introduce extra noise and negatively impact the model's performance. In our approach, we treat stock market data as multivariate time series, those dynamic patterns can be typically viewed as a combination of three components:
- Short-term randomness,
- Long-term trends,
- Multi-scale seasonal patterns.

Accordingly, the three data transformation methods employed in our work are designed to simulate out-of-distribution (OOD) yet plausible market changes that have not been included in the training set, with each method focusing on one of the three dynamic components:
- $F_1$ **for OOD short-term randomness**: This method attempts to answer a key question: *How would MetaTrader respond if the top 10% most profitable stocks (a common threshold in stock trading to balance profit and risk) suddenly collapsed the following day? Can our policy effectively adapt to such a scenario?* $F_1$ is specifically designed for this purpose. It modifies the prices of the top 10% stocks, simulating the effects of unexpected events on individual stocks. Based on this, the bilevel learning scheme prevents the policy from overfitting to the stocks that perform exceptionally well during the training periods.
- $F_2$ **for OOD long-term trends**: This transformation is driven by such a question: How would our policy perform when faced with changes in the overall temporal trends of the stocks? By simulating varying market conditions influenced by long-term disruptions, it evaluates the policy's robustness in navigating such scenarios.
- $F_3$ **for OOD multi-scale correlations**: This transformation scales the seasonal patterns of the market by downsampling the training data, enabling the model to capture multi-scale correlations between the stock changes. 
 
**NOTE: No test data leakage occurs with these transformation methods!** We apologize for the misleading description of $F_3$ in our original submission (Line 185): "*We downsample the original time series by four and concatenate the squeezed sequences with subsequent data.*" This was an incorrect description of our actual implementation (see Appendix A). We have rephrased this in the revision. For all data transformation methods, we carefully divided the training and test sets based on dates, ensuring that all transformations were applied exclusively to the training set. This guarantees no data leakage and ensures a fair comparison among all methods.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a reinforcement learning (RL) approach for stock trading. It uses data transformation techniques and a conservative TD ensemble to improve generalization in non-stationary markets. The methodology is well-structured, and the experiments show promising results, with improvements over existing models.

However, there are key concerns. The rationale behind the chosen data transformations, such as focusing only on the top 10% of stocks, is unclear and may limit the model's adaptability. Additionally, using small observation windows for covariance matrix estimation may introduce bias, and an ablation study to assess this would strengthen the paper. The comparison with ensemble Q-learning methods is also missing, which would provide further context for the approach.

In conclusion, the paper offers valuable insights but requires further clarification and experiments to address these issues.