LOB-Bench: Benchmarking Generative AI for Finance - with an Application to Limit Order Book Markets

Authors: Peer Nagy, Sascha Yves Frey, Kang Li, Svitlana Vyetrenko, Stefan Zohren, Ani Calinescu, Jakob Nicolaus Foerster
Primary area: datasets and benchmarks
Keywords: finance, generative models, time series, state-space models, benchmark

Abstract:
We present **LOB-Bench**, a benchmark designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB). We enable a rigorous and comprehensive model comparison by providing both a theoretical framework and an open-source Python package. Addressing the lack of consensus on evaluation paradigms in the literature, where qualitative comparison of stylized facts is prevalent, our work offers a crucial building block for advancing generative AI for financial data. LOB-Bench provides a standardized method to numerically assess the quality of various model classes that generate limit order book data in the widely used LOBSTER format. It provides a range of quantitative characteristics and includes a simple parametric benchmark model as a baseline. Our framework measures distributional differences in conditional and unconditional statistics between generated and real LOB data, supporting a flexible multivariate statistical evaluation across different model classes. The benchmark features commonly used LOB statistics such as spread, order book volumes, order imbalance, and message inter-arrival times, along with adversarial scores derived from a neural network trained to differentiate between real and generated data. Additionally, LOB-Bench evaluates "market impact metrics" by computing cross-correlations and price response functions for specific events in the data. We present empirical benchmark results for a generative autoregressive state-space model, for a (C)GAN, and parametric LOB model. We find that the autoregressive GenAI approach beats traditional model classes.

Decision: reject
Reviewer scores: 6, 1, 3

Reviews and discussion:
<thread>
creation date:
1730215137127

thread_type:
official_review

<thread_object>
creator: reviewer_zxuj

<content>
summary:
The gist of the paper is to evaluate the realism of generated LOB orders. In more detail, the authors want to numerically assess the quality of the generated data instead of just using stylized facts, classic LOB evaluation format, nor only cross-entropy, usually used in generative approaches. The authors want to evalute pre-trained models in *sampling regime*.

strengths:
The paper presents some interesting ideas about evaluating autoregressive models by binning the generated order into some aggregation function $\Phi$ to avoid accumulating errors in time. The quantitative evaluation metric is the L1 distance between $p(x)$ and $\hat{p}(x)$. What this means is still a mystery to me and I'd appreciate to have more clarification from the authors here.

weaknesses:
### Discussion over the meaning of L1 as a qualitative evaluation metric
I appreciate the authors taking the effort of running all stylized facts experiments and stress-testing 2 SoTA and 1 baseline method. However, the authors claimed in the beginning that just having stylized facts, already thoroughly explored by Vitrienko et al. [5], isn't sufficient. While they support this claim by introducing the L1 as a qualitative metric, it lacks substance and overpresenting these stylized facts overshadows the "novelty" of the benchmark. Anyhow, to the best of my understanding, the L1 should quantitatively say how good a LOB generation model performs. Nevertheless, I argue that it'd be simpler and more than enough to train a model -- even a simple one -- on a financial downstream task -- say mid-price prediction -- on "fake" generated orders and measure its MSE. Then, at test time, one would measure the model's MSE on real orders and see how well the model trained on generated orders generalizes over the real data. This would be a sufficient indicator to assess the goodness and realism of the generated data. Furthermore, you wouldn't even need to bin your data and go through unnecessary hoops just for the sake of the benchmark. So, the question raised here is *Why is the literature in LOB generation "dying" to have your contribution? What are you bringing to the table that future works are going to benefit from?*

### Shortsightness and lack of discussion within the benchmark
As a benchmark paper, I would expect the authors to give insights and directions for future autoregressive/generative LOB models. This is lacking which undermines its usefulness. Again *What is it that future researchers would benefit from this benchmark?*

### Weaknesses
1. **Why did you share your GitHub repository not anonymized? Now, I know your identity and have reported it to the ACs**.
2. Lines 69-84 represent the authors' contribution and the writing is sloppy and all over the place. So are the authors trying to evaluate how good SoTA methods predict the mean-price on the newly generated orders? Doesn't this just cover the cross-entropy aspect that GenAI literature uses for evaluation? Where's the realism here?
3. It is unclear from lines 86-89 why model derailment is due to true data seeding. Care to discuss?
4. There's missing information about *coletta* and limited results for GOOG (lines 390-394). What missing info?
5. In Figure 6, it is evident that the trend of MO$_0$, and, especially, MO$_1$ aren't captured by *lobs5*. How do you justify this?

Minors:
1. There might be a missing reference in line 76 where you stat "Our aggregator functions are closely inspired by metrics used in literature,
such as spread etc.". How are they inspired by the literature? Which work specifically are you inspired from?
2. Who said that GANs are worst-case aggregators? Why is that?
3. It would be better to introduce already the models you're using in the benchmark in line 86. Why only 2 models if you're proposing a benchmark? There are plenty of other generic time series generation models that you can try to adapt for LOB [1-4]. 
4. Reference sections/equations correctly:
* e.g., line 265 $\rightarrow$ Equation~\eqref
* e.g., line 307 $\rightarrow$ Section~\ref
5. Lines 97-98 and 98-99 are the same sentence. This is where the authors also reveal their identity.
6. Nobody calls cross-entropy *xent*. Please amend this.


[1] Lim H, Kim M, Park S, Park N. Regular time-series generation using SGM. arXiv preprint arXiv:2301.08518. 2023 Jan 20.

[2] Li J, Wang X, Lin Y, Sinha A, Wellman M. Generating realistic stock market order streams. In Proceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3 (Vol. 34, No. 01, pp. 727-734).

[3] Shi Z, Cartlidge J. Neural Stochastic Agent-Based Limit Order Book Simulation: A Hybrid Methodology. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems 2023 May 30 (pp. 2481-2483).

[4] Hultin H, Hult H, Proutiere A, Samama S, Tarighati A. A generative model of a limit order book using recurrent neural networks. Quantitative Finance. 2023 Jun 3;23(6):931-58.

[5] Vyetrenko S, Byrd D, Petosa N, Mahfouz M, Dervovic D, Veloso M, Balch T. Get real: Realism metrics for robust limit order book market simulations. In Proceedings of the First ACM International Conference on AI in Finance 2020 Oct 15 (pp. 1-8).

## **Even in case the authors didn't reveal their identity, I would still have rejected this paper**

questions:
See weaknesses, and the following:

1. Have you tried the KL-divergence for $\mathbb{D}$ in Eq. 1? Why is L1 more suitable? Can you run some experiments please to see this effect?
2. Does Figure 2 measure the L1 loss on all these dimensions?
3. Fig 15 $\rightarrow$ So, you trian a GAN on real and generated data from *lobs5* and you achieve an AUCROC of 82\%? This means that the generated orders are easily identified as fake, undermining what the authors claim in their original works. How come? What's the AUCROC for *coletta* and the *baseline*?
4. I might have missed it but what's the *baseline* model here?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730564872508

thread_type:
official_review

<thread_object>
creator: reviewer_trtg

<content>
summary:
This paper proposes a unified benchmark framework, LOB-Bench, designed to quantitatively evaluate generative AI models with a focus on limit order book (LOB) data. Unlike prevailing qualitative measurements, this framework provides numerical results to analyze the accuracy of how the LOB generative models resemble the real data. Compared to current quantitative metrics in the literature, such as cross-entropy, LOB-Bench is claimed to better deal with issues like distribution shifts and autoregressive gaps within data. LOB-Bench is applied to evaluate a range of generative models. The experiment section presents some interesting findings. Specifically, the results indicate that autoregressive GenAI’s excel in learning LOB data more effectively than traditional methods.

strengths:
1. This paper highlights the essential demands for quantitative metrics to evaluate generative AI for finance and addresses this critical problem by introducing several innovative quantitative metrics, which are well-motivated and useful for the further development of generative AI in finance.
2. This paper effectively employs visualizations to show the quantitative results, making them intuitive and comprehensive for a broad audience.

weaknesses:
1. This paper provides a non-anonymous code link on line 097, which violates the double-blind protocol of the conference. The authors should take care to remove or anonymize the link for the review process. 
2. This paper displays several signs of incomplete editing that require further proofreading. Specific issues include a non-functional GitHub link on line 033, and repeated sentences between lines 097-099. Before submitting, the authors should conduct a thorough proofreading pass, paying particular attention to consistency in links and removing any redundant text. 
3. Although the paper introduces a new benchmark for evaluating generative AI models, it does not adequately compare this benchmark with existing alternatives. Furthermore, the experiments fail to convincingly demonstrate the benchmark's claimed ability to effectively address distribution shifts or autoregressive gaps within the generated data. What are some existing benchmarks against which the proposed method can be compared against? Specific experiments that clearly demonstrate superiority in dealing with distribution shift or autoregressive gaps in the generated data would make the method more convincing.

questions:
1. As mentioned above, this paper does not demonstrate that LOB-Bench addresses autoregressive gaps and distribution shifts effectively. Can you provide a detailed analysis or case studies that illustrate the benchmark's performance in managing these specific challenges?
2. Could you demonstrate how LOB-Bench differentiates between generated and real LOB data in certain scenarios where traditional methods such as cross-entropy fail to?
3. How does the proposed benchmark influence the development and refinement of generative AI models? Are there examples where insights derived from LOB-Bench can directly lead to improvements in model architecture or training methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730761801783

thread_type:
official_review

<thread_object>
creator: reviewer_guks

<content>
summary:
The paper presents the LOB-Bench, a benchmark designed to evaluate the quality and realism of generative models for Limit Oder Book(LOB) data. It provides a theoretical framework and Python package that systematically compares generative models like autoregressive models, (C)gAN, and agent-based models. It uses a range of metrics like spread, order book volumes, order imbalance, and adversarial scores to asses the distributional differences between generated and real LOB data. Empirical results demonstrate that the autoregressive generative approach outperforms traditional models, offering a standardized evaluation for financial data generation.

strengths:
1) The paper proposes unconditional and conditional evaluations, allowing for a nuanced comparison of generative models. It accounts for model drift and evaluates market impact metrics, making the framework robust.

2) The use of widely adopted LOBSTER data for empirical evaluations makes the benchmark relevant for practitioners in the financial domain.

3) The open-source code enhances the reproducibility and practical unity of the work.

weaknesses:
1) The paper explores generative models; some more, like advanced deep learning architectures, could be explored.
2)  The evaluation highlights that errors accumulate over longer sequences. The benchmark does not fully address how to mitigate this. 
3) Certain model results with rate events like market orders have data sparsity. and generating additional data might not be practical.

questions:
1) The framework does not account for hidden liquidity, which is critical for LOB modeling. The model can be improved in this area. 
2) Why did you focus on autoregressive models, (C)GANs, and agent-based models? Have you considered other deep learning architectures for LOB data generation, such as transformers or diffusion models?
3) Could you clarify why you selected the L1 and Wasserstein-1 distances as the primary metrics for distributional comparison? How do these metrics handle the complexities of financial time series data, such as fat-tailed distributions or volatility clustering?
4) The benchmark evaluates market impact metrics, but how do you account for outliers or extreme events in real-world LOB data? Are these adequately captured in your current framework?
5)  The conditional evaluation approach looks at conditional distributions of specific LOB statistics. How would this framework perform when dealing with very high-dimensional data, and does it scale well for more complex scenarios?
6) The paper uses the LOBSTER dataset for evaluation. Given its limitations (e.g., limited asset coverage, certain missing features), do you think the results would generalize to other datasets or more diverse market conditions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732297503271

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal Remarks

comment:
First of all, we must apologise for the clerical error of including a non-anonymised Github link, which was left in the manuscript by mistake at the time of submission. We have fixed this oversight in the updated manuscript and would now kindly ask the reviewers to judge the paper on its merits and help us improve the paper as much as possible and leave the decision regarding the Github link to the AC.

We thank reviewers for some detailed constructive criticism. We hope that we were able to address most concerns satisfactorily and made corresponding changes to the manuscript. We hope that reviewers are now in a position to increase their scores of the paper as we have cleared up some confusion and have aimed to address points of criticism, or to otherwise provide directions for how to further improve the paper.


**Positives**

We thank the reviewers for positive feedback, which we would summarize as follows.

LOB-Bench is judged to offer a nuanced and robust framework for comparison of generative models including *unconditional*, *conditional* and divergence-based evaluation, going beyond just reporting cross-entropy and reproducing stylized facts. The paper also offers the interesting idea of evaluating autoregressive models by binning generated data. Further positives are the inclusion of market impact metrics, well-moticated metrics, and an intuitive and comprehensive visualization of quantitative results. Furthermore, the open-source code enhances reproducibility and practical utility.
It was also lauded that the paper provides the interesting new substantive result, that autoregressive GenAI outperforms for LOB data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732698693994

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Engagement

comment:
This is a friendly reminder for reviewers **GuKS** and **trTG** to please also *engage* with the rebuttals as the rebuttal process is drawing to a close.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper develops a benchmark to evaluate generative models for limit order books. They also evaluate several models classes including a GAN, parametric model, and an autoregressive model. Two reviewers raise several concerns including the value of the benchmark for future development, the comparison to other benchmarks, and the readability. The question around the utility of the benchmark centered on the question of what the benchmark was powered to detect. Quoting one of the reviewers "Specific experiments that clearly demonstrate superiority in dealing with distribution shift or autoregressive gaps in the generated data would make the method more convincing." These questions were partially addressed in the reply.