LOB-Bench: Benchmarking Generative AI for Finance - an Application to Limit Order Book Data
Authors: Peer Nagy
Sascha Yves Frey
Kang Li
Bidipta Sarkar
Svitlana Vyetrenko
Stefan Zohren
Ani Calinescu
Jakob Nicolaus Foerster
Venue: ICML 2025 poster (2025)
Keywords: finance
generative models
time series
state-space models
benchmark
Primary area: applications->time_series
TL;DR: LOB-Bench offers a rigorous framework and open-source Python package for standardized evaluation of generative limit order book data models, addressing evaluation gaps and enhancing model comparisons with quantitative metrics.

Abstract: While financial data presents one of the most challenging and interesting sequence modelling tasks due to high noise, heavy tails, and strategic interactions, progress in this area has been hindered by the lack of consensus on quantitative evaluation paradigms. 
To address this, we present **LOB-Bench**, a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER format. 
Our framework  measures distributional differences in conditional and unconditional statistics between generated and real LOB data, supporting flexible multivariate statistical evaluation. 
The benchmark also includes features commonly used LOB statistics such as spread, order book volumes, order imbalance, and message inter-arrival times, along with scores from a trained discriminator network. Lastly, LOB-Bench contains "market impact metrics", i.e. the cross-correlations and price response functions for specific events in the data. 
We benchmark generative autoregressive state-space models, a (C)GAN, as well as a parametric LOB model and find that the autoregressive GenAI approach beats traditional model classes.

LOB-Bench:
Benchmarking Generative AI for Finance - -an Application to Limit Order Book Data
Peer Nagy" 12
Sascha Frey *3 Kang Li Bidipta Sarkar? Svitlana Vyetrenko 5 Stefan Zohren'
Anisoara Calinescu Jakob Foerster?
Abstract
frequency domain due to high noise, heavy tails, and multi- agent interactions. Generative Al (GenAl) currently revo- lutionizing different fields, ranging from natural language processing image generation and real world applications. Perhaps surprisingly, the backbone of these methods is sim- ply self--uper pre-training on large datasets using a next-token prediction loss on autoregressive sequence mod- els (Nie et al., 2024; Dubey & et. al., 2024; Liu et al., 2024).
While financial data presents one of the most challenging and interesting sequence modelling tasks due high noise, heavy tails, and strate- gic interactions, progress in this area has been hindered by the lack of consensus quantita- tive evaluation paradigms. To address this, we present LOB-Bench, a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER for- mat. Our framework measures distributional dif- ferences in conditional and unconditional statis- tics between generated and real LOB data, sup- porting flexible multivariate statistical evaluation. The benchmark also includes commonly used LOB statisties such as spread, order book vol- umes, order imbalance, and message inter-arrival times, along with scores from a trained discrimi- nator network. Lastly, LOB Bench contains "mar- ket impact metrics", i.e. the cross-correlations and price response functions for specific events in the data. We benchmark generative autoregres- sive state-space models, a (C)GAN, as well as a parametric LOB model, and find that the autore- gressive GenAl approach beats traditional model classes. Code and generated data are available at: https://lobbenchgithbbbb github.oo/.
Recently, Nagy et al. (2023) applied this paradigm limit order books (LOB), the mechanism through which stock markets keep track of buy and sell orders to determine any-time prices. Specifically, in contrast to prior works, which model only high level features (Cont et al., 2010; Coletta et al., 2022; Byrd et al., 2020), this approach learns a token-level distribution over messages in the LOBSTER dataset (Huang Polak, 2011).
An accurate, low level generative model of the financial sys- tem is extremely valuable from societal and commercial point of view. For example, it could unlock better mecha- nism design, stability analysis, or learned-algoritee (e.g. order execution (Frey et al., 2023)) by providing counterfac- tuals.
A key question then is how determine the realism and trustworthiness of GenAI, and of other generative finan- cial models. On the one hand, for high-level approaches and "old school" agent-based modelling (Byrd et al., 2020; Chiarella & lori, 2002; Paulin, 2019; Llacay & Peffer, 2018) the evaluation is usually based on a qualitative analysis of
1. Introduction
Practitioners have long been interested high-quality syn- thetic financial data, which is especially difficult in the high-
data popiie
generated
(6:0 cki)
- elewe (#60 cl.)
"Equal contribution 'Oxford-Man Institute of Quantitative Finance, University of Oxford FFoesseer Lab for Al Research, University of Oxford 'Department of Computer Science, Uni- versity of Oxford "Department of Statistics, University of Ox- ford J.P.. Morgan AI Research. Correspondence to: Peer Nagy <peernagy @ eng.ox.ac.uk>.
Diy, [k]
7 p. [8.]
cond #core
Σ, D/3,,
[A]]
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Figure 1: Schematic LOB-Bench methodology for con- ditional distributional evaluation

---

LOB-Bench: Benchmarking Generative AT for Finance
new sell order
For conditional distributional evaluation, we first apply an aggregator function and group these results into "buckets" based on the conditioning variable. We then score each of the resulting conditional distributions using the process de- scribed earlier. This approach enables, for example, assess- ing whether the distribution of bid-ask spreads, conditioned on the time of day, aligns with the corresponding condi- tional distribution in real data. To derive a single metrie, we compute the average loss across the conditioning buckets, weighted by the probability of each bucket. Furthermore, we can also use this to evaluate model-drift by aggregating on the sampling step and comparing to the unconditional data, which is a good proxy for model-derailmmnt in open-loop sampling. See Figure 1 for a process schematic.
Ask aide (leel)
price
mid price
Biid side (buy)
size
Figure 2: Schematic of the LOB. Not immediately ex- ecutable orders are placed in First-In, First-Out (FIFO) queues at the specified limit price. Sell limit orders are placed on the Ask side (red) and Buy limit orders on the Bid side (blue).
We test our evaluation framework on five different gener- ative models: four modern GenAl models (Coletta et al., 2022; Nagy et al., 2023; Peng et al., 2023; 2024) and a widely-used classic model as a baseline (Cont et al., 2010). All models are tested on data of Alphabet Inc (GOOG) and Intel Corporation (INTC) stock. We don't present detailed results for the Coletta model trained on INTC, because the architecture was developed only for small-tick stocks and therefore fails on INTC data (Coletta al., 2022). We find evidence of "model derailment," since the distance scores increase for longer unrolls (Figure 5). We also find that the LOBS5 model is best able to reproduce the standard price-impact curves that are well-known in the economics and finance literature (Eisler et al., 2012).
whether the model reproduces known high-level patterns "stylized facts" from the literature, "impact" or the famous "square-root law" (Tóth et al., 2016; Brokmann et al., 2015; Almgren et al., 2005b). However, most of these metrics are unquantifiable and may be disconnected from ground truth data.
On the other hand, for GenAl the standard evaluation for pre- training is simply cross-entropy, i.c. how closely the model is able to predict the next token on held-out data. Unfortu- nately, this does not capture how the model performs under autoregressive sampling, when generating sequences of data one token at a time, where error accumulation can cause distribution shifts. In many applications of GenAI this is not a problem, since the pre-trained models are merely used as starting points for task specific finetuning (e.g. RLHF), rather than in their "bare" form. In contrast, we want to evaluate the pre-trained models in the sampling regime to unlock the mentioned use-cases.
Our contributions are summarized as follows:
A novel LOB benchmark for distributional evaluation: the first LOB benchmark focused on full distributional quan- tification of model performance. This addresses limitations of prior work, which relied on qualitative comparisons of stylized facts, making rigorous model comparisons infeasi- ble and hindering research progress.
Interpretable scoring functions for targeted improve- ments: using intuitive scoring functions enables targeted model development and refinement.
To address this, we propose a general framework for eval- uating the similarity between the distribution induced by generative LOB models and the ground-truth data. At a high level, our unconditional evaluation consists of three steps. We first introduce set of aggregator functions, Φ which map from high-dimensional time series LOB data into a set of d subspaces. Secondly, we compute histograms to estimate distributions for the ground-truth and generated data in these subspaces and, finally, use a distance metric, e.g. L1, compare differences in these estimates. Some of the aggregator functions chosen closely inspired by met- rics used in literature (Vyetrenko et al., 2021; Paulin, 2019; Chiarella & Iori, 2002; Cont, 2001). They also directly re- late to generative adversarial networks (Goodfellow et al., 2014), where the discriminator network is equivalent to a worst-case aggregator function for given generator.
Difficult challenge of discriminator scores: discriminator- based scoring sets a high bar for future generative models, even when most other statistics are closely aligned.
Identification of a common failure mode: divergence met- rics, computed as distributional errors as a function of unroll step, highlight a prevalent failure mode to guide research.
Ease of use and accessibility: open-source, straightfor- ward apply benchmark which only requires data in the LOBSTER format.
Extensibility to additional scoring functions.
Transferability to other domains: the theoretical frame- work is adaptable to other high- nigh-dimensional generative time series tasks beyond LOB data.
2

---

LOB-Bench:
Benchmarking Generative AI for Finance
We hope our benchmark will provide a much-needed start- ing point for evaluating GenAl models in finance and allow more machine learning scientists to develop new sequence models for this important and challenging do- main. Our code and additional resources are available at: http:///obbench.iituub...
phenomena, which emerge as a consequence of multiple participant interactions, which are difficult to model other- wise. However, they are notoriously challenging to calibrate (Vyetrenko et al., 2021; Paulin, 2019). To circumvent cali- bration, conditional generative adversarial networks were used to learn simulators from historical LOB data, that are both realistic and responsive (Coletta et al., 2023). Most recently, an end-to-end autoregressive generative model that produces tokenized LOB messages in the spirit of generative AI was shown to achieve a high degree of realism (Nagy et al., 2023).
2. Background
2.1. Limit Order Book (LOB)
Later sections of this paper rely the reader's understand- ing of the mechanisms of electronic markets, so we briefly review them here. Public exchanges such as NASDAQ and NYSE facilitate the buying and selling of assets by accept- ing and satisfying buy and sell orders from multiple market participants. The exchange maintains an order book data structure for each asset traded. The LOB represents a snap- shot of the supply and demand for the asset at given time. It is an electronic record of all the outstanding buy and sell limit orders organized by price levels. A matching engine, commonly using a price-time priority mechanism, is used to pair incoming buy and sell order interest as mentioned in Bouchaud et al. (2018). Order types are further distin- guished between limit orders and market orders. A limit order (Figure 2) specifies price that should not be exceeded in the case of buy order (bid). or should not be gone below in the case of sell order (ask). A limit order queues a resting order in the LOB at the corresponding side of the book. A market order indicates that the trader is willing to accept the best price available immediately.
2.3. Autoregressive LOB models
In machine learning, autoregressive modelling is key com- ponent of language models like GPT. By learning the prob- ability distribution of the next token given the previous tokens, autoregressive language models can generate co- herent text (Radford et al., 2019). Cross-entropy is a loss function commonly used to train classification models in deep learning. It measures the dissimilarity between the predicted class probabilities and the true class labels (Good- fellow et al., 2016). Cross-entropy loss is the negative log likelihood of the true class labels under the predicted dis- tribution. Minimizing the cross-entropy is equivalent to maximizing the likelihood of the data (Murphy, 2012). The cross-entropy loss over a sample of size N, with V classes can be expressed as:
L
yi (v)
log n
In real-time trading, injecting orders into the market induces other market participant activity that typically drives prices away from the agent. This activity is known market impact (Almgren & Chriss, 1999; Almgren et al., 2005a). Presence of market impact in real time implies that a realistic trading strategy simulation should include deviation from historical data. Therefore, realistic market impact emulation an important consideration in limit order book modelling.
where (*) equals if true class is v and 0 otherwise, and of")) is the predicted probability for class v. Cross-entrops loss heavily penalizes confident misclassifications and in- centivizes the model to output calibrated probabilities that match the empirical distribution of the classes. Although it is different from the KL divergence, cross-entropy can be expressed as the sum of the entropy of the true distribu- tion and the KL divergence between the true and predicted distributions (Cover & Thomas, 1999).
2.2. LOB Models
LOB simulation is an important technique for evaluating trading strategies and testing counterfactual market scenar- ios. The extent to which results from such simulations can be trusted depends on how accurately they emulate real world environments. Traditionally, it is common to use historical market data to train and backtest a trading strat- egy, thereby making the assumption of negligible market impact. This is based on the premise of small agent orders and a sufficient time between consecutive trades (Spooner et al., 2018). However, the "no market impact" assumption is not valid for larger order sizes or high frequency of orders. Agent- methods naturally allow to study such
3. Related Literature
The LOB plays a crucial role in modern financial markets. With the FI-2010 dataset, Ntakaris et al. (2018) released the first publicly available high-frequency LOB dataset for benchmarking mid-price prediction models. This pre- processed dataset containst orders for five stocks the Nasdaq Nordic market for ten consecutive days. Although useful and effective for preliminary tests and comparisons of LOB algorithms, F1-2010 does not allow a comprehensive evaluation of robustness and generalisation ability (Zhang et al., 2019). similar benchmark for average price and
3

---

LOB-Bench: Benchmarking Generative AI for Finance
volume prediction in Chinese stock markets is provided by Huang al. (2021). As with other currently available benchmarks, this work falls short of evaluating GenAl mod- els with a fully distributional lens. Cao et al. (2022) propose a benchmark dataset, which plays a complementary role to LOB-Benche With DSLOB, they provide a synthetic LOB dataset, generated by a multi-agent simulation with shocks, which generates labelled in- and out-of-distributinns sam- ples. In contrast, LOB-Ba MeB-Bench does not require training specific dataset, and instead focuses general-purpose model evaluation and comparison.
Evaluating generative models in any domain is fundamen-
tally matter of comparing distributions. Our benchmark performs exactly this task. It reduces a high-dimensional distribution of sequences of order book states b E B and . to scalars by using scoring func- tions 2; : (M x B) + R, i E N. One-dimensional score distributions can then be compared between real and model- generated data using various norms or divergences D. By estimating the difference between the unconditional real data distribution p(((d)} and the data distribution under the model ((((d)) i.e. II] [p ( } á (d)) different generative models can be ranked on their ability to match features of the data.
message events m E
To evaluate the performance of generative models in the LOB enviromment, several studies have proposed relevant metrics. Coletta al. (2023) investigated the interpretability, challenges, and robustness of conditional generative models. They grouped LOB states based certain attributes and statistics and then performed conditional generation on these groups. Vyetrenko et al. (2021) proposed several statistics to assess the realism of LOB simulators, such as order arrival rate, order distance distribution, and price volatility, whilst Paulin (2019) further considers lagged autocorrelations, and liquidity of trades.
To evaluate the magnitude of the "autoregressive trap" the benchmark evaluates error divergence of distri- butions, conditional on the interval of the forecast- ing step 1. E N. for interval limits a, E N: D [P { ()((de((os)) || á (ddrEj.,)))) This allows quanti- fying distribution shift during inference.
Our framework uses both the L1 norm and the Wasserstein- distance as loss metrics. To estimate the L1 norm, we first bin the data. As robust binning algorithm, we use the Freedman-Diaconis rule (Freedman & Diaconis, 1981), which computes the bin width as 2 21QR, where n is the combined sample size and IQR the inter-quartile range of the real and generated data. The [0, 1]-scaled Ln norm, also called the total variation distance, can then be estimated as:
In summary, although some studies have addressed the eval- uation of generative models for LOBs, unified bench- marking framework is still lacking. Existing research often uses qualitative methods to compare statistical regularities of generated data with real data, lacking quantitative eval- uation metrics. Therefore, establishing comprehensive benchmarking framework for evaluating LOB generative models is essential for advancing the field.
2 lp - Pll
1
Σ bgbins
1 P
Deount bwiithh
P
Deount
burkdth
(1) While the I,1 measure has the benefit of being bounded in the interval [o, 1]. the Wasserstein-1 distance, or earth mover's distance, as proposed by Rubner et al. (2000), has the advantage of being sensitive to the distance between the scores. To make losses between different scoring functions comparable, we vean-variance normalize the data before calculating the Wasserstein-1 I distance.
4. Evaluation Framework
As the success of LLMs has shown, generative models can already achieve impressive performance by autoregressive training, or "next-token prediction" alone. However, not all model classes are auto-regressive or allow the explicit computation of conditional "next-token probabilities." pro- hibiting cross-entropy based evaluation or calculating model perplexity (Chen et al., 1998). However, there is still need to evaluate such model classes, where we can merely sample data. Another reason why single- token cross-entrope loss insufficient is the so-called "autoregressive trap" (Zhang et al., 2024). Even small errors in next-token prediction task can accumulate over long sequences, moving away from the training distribution. Out-of-distributie forecasts then become increasingly worse until the generating dis- tribution completely derails or collapses. This emphasizes the need to evaluate statistics over entire sequences, rather than focusing solely on cross-entropy. A benchmark frame- work should therefore also measure how fast such errors accumulate by evaluating distributions conditional the forecasting horizon.
For equal sample sizes we can compute the Wasserstein-1 1 distance as follows. Let (ddeat)) be the i-th order statistic of a score computed from a real data sample drawn from p and tb (dgen)()) the i-th order statistic using generated data drawn from p. Then we have:
W (p.
Σ (dreatt))) - (dgen)
(2)
To evaluate a generative model's ability to adapt to different contexts, we also estimate differences between conditional score distributions
(3) In this case, (2 is binned into 10 data deciles by of the pooled real and generated data, Distance estimates of these
D [p (4h (d) (d)) || A ( (d) (c)))
4

---

LOB-Bench Benchmarking Generative AI for Finance
10 conditional distributions are then weighted according to the mean of the estimated density of both distributions. Letting X = & (d) and Y = P2(d), conditional metric
this context, executions are considered market orders.
dir
event type is MO or LO;
E
(5)
can be evaluated as
dir if event type is CA.
D D |Y e b,)] P( + bb2)
E
(4)
This allows calculation of the response function - equation (6). This is calculated empirically using the time average
2
(( )T) of the change in the sign-adjusted mid-price Pt = following a given event, for different timetimes L. The event lag times are chosen to be distributed uniformly on a logarithmic scale between and 200 ticks. The prices are normalized by tick size to enable a comparison between various stocks.
This approach addresses a specific type of distribution shift: the variation of scores, L1. across the distribution of an- other score, Φ2. For instance, if the conditioning function $ represents the mean time of messages within data sequence, this framework allows us to analyze how distribu- tion shifts affect any score of interest, Φ1, and assess the generative model's ability to replicate this dynamic behavior accurately.
R (1) ((pr4 - -pt)ee T)r
(6)
Eisler et al. (2012) identify averaged response functions for 14 random stocks over period of 53 trading days. Whilst such analysis gives good baseline to which we can com- pare our results, for model evaluation we instead directly compare the functions between model-generated and real sequences (matched based on the seeded starting point or the time of day) for individual stocks. Once the response functions are calculated, we create measure of comparison to obtain score of dissimilarity:
Our methodology formalizes and naturally extends common evaluation practices for synthetic mie-dimesioonl time se- ries, such as financial returns, which typically emphasize distributional similarity. Our framework enables a quanti- tative assessment of distributional properties in structured high-dimensional time series. By adapting the scoring func- tions, our approach could also be applied to financial transac- tions, payment data, streamed price quotes in forex markets, multi-asset limit order books, or decentralized crypto market protocols.
L. AR,
Reedl (1) - (2)
L
1=1
4.1. Impact Response Functions
which is aggregated across all event types by taking the mean AR = |m Exen AR
A primary difficulty with historical LOB data is that coun- terfactual scenarios are impossible to evaluate, as the data do not respond to additional injected orders. Generative models are unique opportunity to generate response to counterfactual scenarios as they address this limitation.
4.2. Adversarial Measurement
The concept of adversarial measurement involved develop- ing a pre-trained discriminator capable of effectively dis- tinguishing between true and generated trajectories. This discriminator is a binary classifier, generating a probability estimate of a trajectory being real. We only use the order- book states as input, ignoring the message sequences. The discriminator is trained using two batches of data, each of dimension (S x T D). S denotes the number of sequence samples within the batch, T the length of the orderbook sequences, and D is the dimension of the orderbook state representation.
It is therefore crucial that such models be evaluated on their ability to provide a realistic response to different events. As an underlying methodology, the seminal work by Eisler et al. (2012) is used as a basis to compare the impact of different event types. This methodology focuses only on the impact of events, which change the price or quantity of the best bid and ask orders (also known as touch orders), which also constitutes a key limitation of the method.
All events which affect the best prices are classified into one of six order types π E II: market orders (MO), limit orders (LO) and cancellations (CA), which are further sub- divided into those which affect the mid-price, indicated with subscript 1, and those who do not, with subscript O: II = (MO,, MO,, LO,, LO,, Co, CAI).
The discriminator aims to find the "worst-case" function d- that maximally separates the real and generated distribu- tions by choosing 4 such that it maximizes the divergence between them, i.e., x- = arg max4 D[p( p (( (d)))) This h³, which can be interpreted as a dimensionality reduc- tion operation on the sequence of order book states b e B to a scalar s, gh (M x B) + IR, can be considered to be an adversarial scoring function. The discriminator attempts to identify the most glaring flaws and differences between
Using the convention in LOBSTER data, we define the direc- tion (dir) as 1 for events on the bid side and -1 on the ask side. The events are given an e value based on the expected direction of impact on the mid-price they will provoke. In
5

---

LOB-Bench: Benchmarking Generative AI for Finance
the real and generated samples, distilling these into a single dimension.
5. L.B-Beench Package
L1 Loss (GOOG)
Given the sparsity of changes between successive order- book states, we devised an encoding scheme to optimize the discriminator's performance. An orderbook state com- prises the price and quantity from the top n price levels on both the bid and ask sides. In our experiments, resulting in dimension D= 40. Changes in the orderbook state typically triggered by events that affect single price-quantity pair. To achieve a more concise, yet infor- mative, representation of the discriminator network, we chose to represent the orderbook based on these changes. Thus the book states E B and message events m E M map to dimensoamnli vectors through E N functions v : (M x B) + R². These changes encompass each change in the mid-price the relative price level where the change occurs, and the corresponding change in quantity. Our discriminator utilizes ID convolutional neural net- work (Conv ID) (Lecun & Bengio, 1995; Kiranyaz et al., 2019) as a feature extractor, followed by an attention mech- anism (Vaswani et al., 2017) to capture long-term depen- dencies across the time steps. Empirical results show that this model, trained and tested on GOOG data from 2023, achieves a Receiver Operating Characteristic (ROC) score of 0.83, indicating that the generated data can be discrimi- nated reasonably accurately. However, the baseline model's performance for GOOG and INTC was poor, with a discrim- inator ROC score of around 1, indicating significant room for future model improvement. High discriminability may result from model errors, indicated by imperfect model scores (see Section 6 and Figures 13ff). A distributional mismatch in single scoring function can be sufficient to make fake data identifiable. To mitigate this issue, future research could evaluate adversarial performance by training a discriminator on perturbed data and reporting scores con- ditioned on the noise level. particularly as models improve on this benchmark.
ask volume touch log time to cancel
bid volume touch ask volume
log Inter arrival time
bid volume
10
orderbook imbelance
limit ask order depth
spread
limit bid order depth
-0.
ofi
ask cancellation depth
vol per min
bid canceliation depth
bid cancellation levels ask cancellation levels
limit ask order levels limit bid order levels
lobs5
baseline
rwkv4
rwkv6
coletta
Figure 3: Model comparison spider plot: the LOBS5 model beats the baseline and coletta model on almost all scores. Note: the radial axis is inverted by plotting the negative loss (larger better).
Based on the evaluation framework outlined in section 4, we develop Python benchmark package, allowing for a convenient and comprehensive evaluation of generated LOB data. The benchmark is highly customizable, scoring functions & can easily be added, removed, or modified, and provides standardized model comparison using the default scoring functions provided. The benchmark reports aggregate model scores by computing the mean, median, and inter-quartile mean (IQM') across all conditional and unconditional scoring functions, along with bootstrapped confidence intervals.
The benchmark performs both unconditional and conditional evaluation of generated data, by computing distributions of statistics of interest conditionally on the value of another statistic. To evaluate the magnitude of the effect of error
divergence or "snowballing errors," distributions are also evaluated conditional the prediction horizon. Distribu- tional accuracy is measured by computing the L -nomr and Wasserstein- I distance between the real and generated dis- tributions. Specific supported examples of more complex conditional distributions are the response functions, describ- ing the distribution of events conditional on other events having occurred at a certain prior lag. As these distributions usually have high variance, and to be consistent with the extant literature, we instead measure mean absolute differ- ences in their means for range of lags to evaluate market impact curves.
4.3. Mid-price Prediction
To evaluate the potential value added from additional gener- ated synthetic data, measure the impact of the additional training data on prediction quality for a simple downstream task. Concretely, we adapt the LOBCAST (Prata et al., 2024) implementation to train an MLP with one million parameters to classify the mid-price movement for different prediction horizons. The MLP is trained to predict three classes: up, down, and stationary. Following Prata et al. (2024), classes are defined based on the movement of the mid-price over threshold value. For each generative model evaluated, two separate MLPs are trained, one on only real data, and another on real & generated data. Fl-scores calcu- lated on held-out real data are then evaluated as a measure of the impact of using the generated data in training.
We include multiple conditional scoring functions from the finance literature, for example, ask volume conditional on the spread, the spread conditional on the hour of the day, and the spread conditional on the volatility of 10ms returns.
'mean all values between the 25. and 75. percentile
6

---

LOB-Bench: Benchmarking Generative AI for Finance
Statistic
Description
Bid-Ask Spread
Difference between the highest price buyer is willing to pay (the bid) and the lowest price a seller is willing to accept (the ask)
Order Book Imbalance
Imbalance for the best prices is computed as (bid size - ask size)/ (bid size + ask size) Time between successive order book events (on log-scale due to a long right tail) Time between submission and first (partial) cancellation for cancelled limit orders, measured log-scale.
Message Inter-Arrival Time
Time-to-Cancel
Bid/Ask Volume
The volume of all orders on the bid, respectively ask, side of the LOB. We also evaluate the volume only at the best price levels.
Limit & Cancellation Depths
Absolute distance of new limit orders or cancellations from the mid-price The price levels at which events occur E N
Limit & Cancellation Levels
Volume per Minute
Traded volume in one-second intervals, scaled to a minute.
Order Flow Imbalance (OFI)
Metric from (Cont al., 2012) considering the imbalance in submitted orders for a rolling window of messages.
OFI (Up/Stay/Down)
OFI (see above), conditional on the subsequent message's mid-price move: Up/Static/Down
lobs5 baseline
rwkv4
X
coletta
L1
Wasserstein
IOM
IQM
INTC
INTC
median
mean
x
x
mean
X
X
GOOG
median
I0M
GOOG
median
10M
X
mean
X
x
*
0.1
0.2
0.3
0.5
0.6
0.7
0.8
0.2
0.4
0.6
0.8
1.0
Figure 4: Model score summaries (lower is better). The LOBS5 model achieves the lowest overall scores. Coletta beats the buseline the Wasserstein metric, but not for L,. Error bars are bootstrapped 99% Cls.
The benchmark also evaluates model response functions (6) in aggregate. Individual L, distances A R are calculated for each lag time and averaged to produce aggregate impact scores.
LOBSTER data interface to facilitate the conversion of data formats. For the RWKV models, we apply autoregressive next-token prediction, but on a larger model (170 million parameters), without any data pre-processing and using an off-the-shelf byte-pair tokenizer (Sennrich et al., 2016), as is used for LLMs. These models are trained solely on message data, without any order books, which do not require propa- gation of a calculated orderbook state, unlike in Nagy et al. (2023). The S5, RWKV, and baseline results presented here are computed, following Nagy et al. (2023). for Alphabet (GOOG) and Intel (INTC) stock on a sub-sample of the test data from Jamuary 2023. The Coletta model is trained on three days from January 2019 and tested on three subse- quent days, following the procedure in Coletta et al. (2022), which is necessary due to the high computational cost for training and inference with Coletta. Comparing all models, we observe that the LOBS5 model provides state-of-the-an performance on the benchmark task.
6. Results
As first test case for our benchmark, we adapt the autore- gressive state-space model using $5 layers (Gu et al., 2021) from Nagy et al. (2023) (LOBS5). Particularly, we scale up the model size to 35 million parameters and more than double the training period to the entire year of 2022.
We also evaluate data generated by the models from Cont al. (2010) (baseline), Coletta et al. (2022) (Coletta) and models based on Peng et al. (2023; 2024) (RWKV 4 and 6). The baseline model, which employs parametric arrival pro- cesses, is adapted to generalize across both small and large tick limit order book (LOB) dynamics by utilizing estimated empirical arrival rates directly, rather than fitting a power law. Additionally, we infer data features present in LOB- STER, such individual message IDs, which are not gen- erated by Cont et al. (2010). This inference is particularly important for capturing order cancellations, as we uniformly sample target limit orders from the available orders at the specified price level. For the Coletta model, we implement a
Figure 3 presents a key benchmark feature to compare mul- tiple models across multiple score dimensions, allowing an examination of individual strengths and weaknesses. To pro- vide summary scores per model, Figure 4 reports the mean, median, and inter-quartile mean for the LI and Wasserstein- metrics for all models 2 Error bars demarcate the 99%
"The Coletta model (Coletta et al., 2022) was trained on both
7

---

LOB- LOB-Bench: Benchmarking Generative AI for Finance
GOOG lobs5
L1 Divergence GOOG lobs5
Spread
Orderbook imbalance
spread
orderbook imbelance
0.004
0.003
1.-
0.002
0.10
0.20
0.001
0.000
100
150
200
250
300
150
400
0.0
-0.5
0.0
0.5
0-100
Prediction Hortzon [messages]
100 200 300 000000000000000000000
0-100 100-200200 300 30000004000000000 -000000-000
Prediction Horizon [messages]
Log inter arrival time
Lag time cancel
log |nter_arrivaal time
log time to cancel
0.4
0.3
Densty
L1 score
0.085
0.23
0.2
0.1
0.080
0.22
0.0
-10
-5
9.00
-10
-5
0
0-100 100 200200--00000 300 100--0000000000000000000000
0-100 200-200200- 300 J00-40p400-300 Prediction Horlzon (messages)
Prediction Hortzon Imessages)
Figure 5: LOBS5 results (left): histogram matching of unconditional score distributions for real and generated data. (right): error accumulation - the further out the prediction horizon, the worse is the model performance an important model characteristic to measure.
bootstrapped confidence intervals. Metrics for individual scoring functions are shown in Figure 12 (Appendix E).
L1 Loss (GOOG)
0.8-
The benchmark also measures error divergence by compar- ing distributions of scoring functions, conditional the inference time step. These demonstrate the rate at which dis- tributions diverge from real data. Results show increasing errors across all models with the fastest divergence exhib- ited by the RWKV models across most scores. For the S5 model in particular, scoring functions with a dependence on features of the book states, which only gradually change, such book volume, are expected to diverge, the initial real data seed decays. However, the rate of decay can still be compared between models. See Figure 18 in appendix E for LI divergence curves and appendix D for a discussion of and an ablation experiment on the effect of bin sizes on divergence scores.
0.5
0.4
0.2
00
spe
Log toginaarttn am
Aut tim can
Bid Bndunkto
Ast vel pat
Aun aonssee and seep Lonn bod and dtop pok and
cin divo dop Lan cosltev croitey
Lan Ase can dev Biat can
sol-peccon mon
Di
otssta
dow
Figure 6: L1 distance between real and generated data histograms (incl. 99% Cls). baseline performs well on LOB depth and level-related scores, and much worse on time and volume metrics. LOBS5 dominates I.1 loss for GOOG.
The response functions for Alphabet (GOOG) are shown in Figure 7 for the Baseline and LOBSS models. The LOBS5 model generally reproduces curves similar to real data but does so better for small-tick stock GOOG. In contrast, the baseline model (Cont et al., 2010) cannot faithfully reproduce impact curves. We do not post the impact curves for the Coletta and RWKV models, as they quickly diverge due error accumulation in inference. The average L1 distance between real and generated impact curves (see equation (7)) is AR 2.45 for LOBS5, and AR = 126 for RWKV-6. For the LOBS5 model, we observe differences mainly in the MO orders at short lags. This is due to the JAX-LOB simulator (Frey et al., 2023), which used by the LOBS5 model during inference. JAX-LOB splits limit orders, which can only be partially filled, into
execution messages and additional resting limit orders, thereby merging a multilevel midprice change into a single order book update.
F1-scores for mid-price trend forecasting using a simple MLP, following the LOBCAST implementation (Prata et al., 2024), are illustrated for Alphabet (GOOG) in Figure 8. We observe that, for the Coletta model in particular, including generated synthetic data markedly reduces the resulting prediction precision and recall. This in line with model rankings based on the distributional distances shown in Figure 5. Generally, the same pattern also present across the other models, particularly for short prediction horizons, although to less extreme extent. Mixing generated data from either the baseline or LOB-S5 models into the training set has no significant effect for longer prediction horizons. Although it is desirable for models to generate data which increase prediction performance, not acting detrimentally
GOOG and INTC data, but failed produce reasonable results for INTC, which expected since the model was designed for small-tick stocks, whereas INTC is not.
8

---

LOB-Bench: Benchmarking Generative AI for Finance
microscopic responze
stock ticker: GOOG
while also making it easier for finance practitioners to use best-practice tools.
Data Sequences
Generated from Deseline Generated from
175
NELE
130
One of the interesting aspects of generative Al models for microstructure data is the ability to model counterfactuals, which is closely related to the notion of price impact in finan- cial modelling. Factoring in the reactions of other market participants to one's actions with conventional approaches is very challenging, but our benchmark suite for generative LOB models provides extensive tests to evaluate whether generated data reproduces the expected response functions at a larger scale. Future research could involve measuring the extent to which generative models match the market impact laws in the literature, such as the "square root law" (SRL. (Tóth et al., 2016). We hope that LOB-Bench opens the door to many new studies, including the training of re- inforcement learning algorithms and multi-agent models for tasks such as trade execution, with the ability to model realistic reactions of different market participants.
125
1.00
of
Eownita
Events lag 0)
Evenbs lag
Figure 7: Comparison of tick-normalized mid-price impact response functions for different event types between real and generated data-sets. Shaded regions are 99% confidence intervals. Compared are the LOBSS and stochastic baseline models. In contrast to the baseline, LOBSS reproduces most features of the expected impact response functions.
Comparison mid-price prediction for GOOG
0.45
Acknowledgements
lobs5 (real only
0.35
(real and generated)
baseline (real ony)
We gratefully acknowledge the Oxford-Man Institute of Quantitative Finance and the Foersler Lab for Al Research for providing access their GPU compute servers. These computational resources were instrumental in carrying out the experiments and analyses presented in this work. AC acknowledges funding from a UKRI AI World Leading Re- searcher Fellowship (grant EP/W002949/1. AC and JF ac- knowledge funding from JPMC Faculty Research Award. For the purpose of Open Access, the authors have applied CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.
beseline (real and genersted)
coletta (real
0.25
coletta (wall and generated)
0.26
40
Prediction Hloricon
Figure 8: Comparison of F1-scores for the MLP classifi- cation models trained on historical (real) data only (solid lines), and trained on both real data and data generated (dashed lines) for subset of the generated models. The error bars depict the 95% confidence intervals bootstrapped from five different training seeds.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
is a minimum expectation. The results show that current generative models are not yet able to generate data aiding this mid-price trend prediction task. Beyond this conclusion, further quality differentiation between the models based on this task is limited. In contrast, the distributional LOBB Bench evaluation results provide a clearer picture of model strengths and potential for future improvement.
7. Conclusions
We introduce LOB-Bench, an evaluation framework for generative Al models for order-book modelling. Crucially, framework contains analysis tools that make it easy for users across the machine learning and finance domains to benchmark their message-level order-flow models.
We believe that B0B-Bench will greatly facilitate core ML research working on sequence modelling to apply their inno- vations to this challenging and relevant real-world problem
9

---

LOB-Bench: Benchmarking Generative AI for Finance
References
Cont, R. Empirical properties of asset returns:
stylized facts and statistical issues. Quantitative Finance,
Almgren, R. and Chriss, N. Optimal execution of portfolio transactions. Journal of Risk, 1999.
1(2):223-236, February 2001.
ISSN 1469-7688,
1469-7696. doi: 10. 10.10807713666600.
URL
http: hett://www.tart wwt.w/...aooo...ommoomoo inee cww/coo////oooooo / 10. N0.1080/713665670.
Almgren, R., Thum, C., Hauptmann, E.,, and Li, H. Direct estimation of equity market impact. RISK, 18, 04 2005a.
Cont, R., Stoikov, S., and Talreja, R. A stochastic model for order book dynamics. Operations research, 58(3): 549-563, 2010.
Almgren, R., Thum, C., Hauptmann, E., and Li, H. Direct Estimation of Equity Market Impact. Risk, 18(7):58-66 2005b.
Cont, R., Kukanov, A., and Stoikov, S. The price impact of order book events. JOURNAL OF FINANCIAL ECONO- METRICS (Winter 2014), 12(1):47-88, 2012.
Bouchaud, J.-P., Bonart, J., Donier, J., and Gould, M. Trades, quotes and prices: financial markets under the micro- scope. Cambridge University Press, Cambridge, 2018.
Cover, T. M. and Thomas, J. A. Elements of information theory. John Wiley & Sons, 1999.
Brokmann, X. Sérié, E., Kockelkoren, J., and Bouchaud, J.-P. Slow Decay of Impact in Equity Markets. Market Microstructure and Liquidity, 0550077 December 2015. ISSN 2382- URL com//di//
DeepMind, Babuschkin, I., Baumli, K., Bell, A., Bhupati- raju, S., Bruce, J., Buchlovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Dedieu, A., Fantacci, C., Godwin, J., Jones, C., Hemsley, R., Hennigan, T., Hes- sel, M., Hou, S., Kapturowski, S., Keck, T., Kemaev, I., King, M., Kunesch, M., Martens, L., Merzic, H., Mikulik, V., Norman, T., Papamakarios, G., Quan, J., Ring, R. Ruiz, F., Sanchez, A., Sartran, L.., Schnei- der, R., Sezener, E., Spencer, S., Srinivasan, S., Stano- jević, M., Stokowiec, W., Wang, L., Zhou, G., and Vi- ola, F. The DeepMind JAX Ecosystem, 2020. URL http: / com google-deepmind
6266. doi: 10.1142/8332222061550077. wwwp:./wlldsciieeoooo
https: /www.
abss/0..142//22.......... World Scientific Publishing Co.
Publisher:
Byrd, D., Hybinette, M., and Balch, T. H. Abides: Towards high-fidelit multi-agent market simulation. In Proceed- ings of the 2020 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, pp. 11-22, 2020.
Cao, D., El-Laham, Y., Trinh, L.., Vyetrenko, S., and Liu, Y. Dslob: a synthetic limit order book dataset for bench- marking forecasting algorithms under distributional shift. arXiv preprint ar arXiv:2211.11533 2022.
Defazio, A. and Mishchenko, K. Leamning-rat--ree learning by d-adaptation. The 40th International Conference on Machine Learning (ICMI. 2023), 2023.
Dubey, A. and et. al. The Llama 3 Herd of Models, July 2024. URL AtR.sttp::/ http:s:///xxvvvvovv www.iiiioorraooooooo absp/bb//24000 21783v2.
Chen, S. F., Beeferman, D., and Rosenfeld, R. Evaluation metrics for language models. 1998.
Chiarella, C. and lori, G. A simulation analysis of the microstructure of double auction m Quanti- tative Finance, 2(5):346-353, October 2002. ISSN 1469-7688, 1469-7696.
Eisler. Z., Bouchaud, J.-P., and Kockelkoren, J. The price impact of order book events: market orders, limit orders and cancellations. Quantitative Finance, 12(9):1395- 1419, 2012.
doi: 10.1088/1469-7688222
5/303. URL httt:://www.taaddfee doi htt//ab//100 /10 108////99-97788///////////
com/
Freedman, D. and Diaconis, P.
On the histogram a density estimator: L 2 theory. Zeitschrift für Wahnscheinlichkkitss theorie und verwandte Gebiete, 57(4):453-476, 1981.
Coletta, A., Moulin, A., Vyetrenko, S., and Balch, T. Learn ing to simulate realistic limit order book markets from data as a world agent. In Proceedings of the Third ACM International Conference on Al in Finance, pp. 428-436, 2022.
Frey, S., Li, K., Nagy, P., Sapora, S., Lu, C., Zohren, S., Foerster, J., and Calinescu, A. Jax-lob: A gpu- apu-accelerated limit order book simulator to unlock large-scale reinforce- ment learning for trading. In Proceedings of the Fourth ACM International Conference on Al in Finance, 2023.
Coletta, A., Jerome, J., Savani, R., and Vyetrenko, S. Con- ditional generators for limit order book environments: Explainability, challenges, and robustness. In Proceed- ings of the Fourth ACM International Conference on Al in Finance, pp. 27-35, 2023.
Goodfellow, I., Pouget-Abadic, J., Mirza, M. Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
10

---

LOB-Bench: Benchmarking Generative AI for Finance
Goodfellow, I., Bengio, Y., and Courville, A. Deep learning. MIT press, 2016.
Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of
training recurrent neural networks, 2013. URL https:
Gu, A., Goel, K., and Ré. C. Efficiently modeling long sequences with structured state spaces. In International Conference Learning Representations, 2021.
Paulin, J. Understanding flash crash contagion and systemic risk: calibrated agent-based approach. University of Oxford, January 2019. https://ora.. wwporaoo....oo..oo www.aaac.kkobjccteep.. /etijeesss//iid 4wwf--eef-----------------------
http:/puullrg//c//mmmmtpp/////
Huang, C., Ge, W., Chou, H., and Du, X. Benchmark dataset for short-term market prediction of limit order book in china markets. The Journal of Financial Data Science, 3 (4):171-183, 2021.
URL
Huang, R. and Polak, T. Lobster: Limit order book recon- struction system. Available SSRN 1977207, 2011. 1. doi: https://doi.or//1000 1ttp//0...11//.mm....m 111.9///m..997207.
Peng. B., Alcaide, E., Anthony, Q.. Albalak, A., Areadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. Rwkv: Reinventing mns for the transformer era.
arXiv preprint arXiv:2305.13044 2023.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Peng, B., Goldstein, D., Anthony, Q. Albalak, A., Al- caide. E., Biderman, S., Cheah, E., Du, X., Ferdinan, T., Hou, H., et al. Eagle and finch: Rwkv with matrix- valued states and dynamic recurrence. arXiv preprint MrXiv:240400889 2024.
Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M., and Inman, D. J. 1D Convolutional Neural Networks and Applications: A Survey, May 2019, URL http:// arxiv. org/ abs / 119/999551000 03554.
Prata, M., Masi, G., Berti, I.., Arrigoni, V., Coletta, A., Cannistraci, I., Vyetrenko, S., Velardi, and Bartolini. N. Lob-based deep learning models for stock price trend prediction: benchmark study. Artificial Intelligence Review, 57(5):116 16, 2024.
Lecun, Y. and Bengio, Y. Convolutional networks for im- ages, speech, and time-series. In The handbook of brain theory and neural networks. MIT Press, 1995.
Liu, Y., Qin, G., Huang, X., Wang, J., and Long, M. Au- toTiime: Autoregressive Time Series Forecasters via Large Language Models, February 2024. URL https: / //arxiv... org/ abs / 24022 02370v2.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Llacay. B. and Peffer, G. Using realistic trading strate- gies in an agent-based stock market model. Computa- tional and Mathematical Organization Theory, 24(3): 308-350 September 2018. ISSN 1572-9346. doi: 10. 10.110077/1155888-17-92988 org/10. 10.100000001 2ttg:/0..000//11000--11-----1110
Rubner, Y., Tomasi, C., and Guibas, L. J. The earth mover's distance as metric for image retrieval. Int. J. Comput. Vis., 40(2):99-121, 2000. doi: 10.1023/ A:1026543900054, URL httt::://doi.rgo// 0T23/A:10226539900550
URL attps://dol
Murphy, K. Machine learning: a probabilisttic perspectvve.. MIT press, 2012.
Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Compu- tational Linguistics. doi: 10.18653//1/P11-1116 URL https aaa/aaaaaaaoooooooooooaoo/oooooa ogg wttpg//16-1122. 6-1162/.
Nagy, P., Frey, S., Sapora, S., Li, K., Calinescu, A., Zohren, S., and Foerster, Generative ai for end-to-end limit order book modelling: A token-level autoregressive gen- erative model of message flow using deep state space network, 2023.
Nie, Y., Kong, Y., Dong, X., Mulvey, J. M., Poor, H. V., Wen, Q. and Zohren, S. A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges, June 2024. URL https: org/abs/ 22.. 11903v1.
Spooner, T., Fearnley, J., Savani, R., and Koukorinis, A. Market making via reinforcement learning. In Proceed- ings of the 17th International Conference Autonomous Agents and Multiagent Systems, pp. 434-442, Stockholm, Sweden, 2018
Ntakaris, A., Magris, M., Kanniainen, J., Gabbouj, M., and Iosifidis, A. Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods. Journal of Forecasting, 37(8):852 866, 2018.
Tóth, B., Eisler, Z., and Bouchaud, J.-P. The square-root impact law also holds for option markets. Wilmott, 2016 (85):70-73, 2016.

---

LOB-Bench:
Benchmarking Generative AI for Finance
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, I.,, and Polosukhin, I. Attention Is All You Need, August 2017. URL http:///a /tttp://axxiii
hts//bb/11000 03762.
Vyetrenko, S., Byrd, D., Petosa, N., Mahfouz, M., Dervovic, D., Veloso, M., and Balch, T. Get real: realism metrics for robust limit order book market simulations. In Pro- ceedings of the First ACM International Conference on Al in Finance, pp. 1-8, 2021.
Zhang, Z., Zohren, S., and Roberts, S. Deeplob: Deep con- volutional neural networks for limit order books. IEEE Transactions on Signal Processing, 67(11):0011-3012, 2019.
Zhang, Z., Zhang, Q., and Foerster, J. Parden, can you repeat that? defending against jailbreaks via repetition. arXiv preprint arXi::2405.07932, 2024.
12

---

LOB-Bench: Benchmarking Generative AI for Finance
A. Benchmark Code
The benchmark code can be found on GitHub at https//gtthhbb.com//eeemg//bbbb bench, additional resources can be found on the project website http:://obbench.iihuu github.io/.
The benchmark suite provides convenient API functionality evaluate model data for range of scoring functions and metrics. A specification of such functions and loss metrics can be defined in configuration dictionary, which can then be passed to a function performing the unconditional and conditional model evaluation. Similarly, the benchmark provides functions to compute the market impact curves, along with a mean LI score. A default configuration dictionary, specifying the scoring functions reported here, evaluated using LI and Wasserstein-1 loss, is similarly provided for easy reproducibility.
To run the benchmark, real and generated data sequences must stored in LOBSTER format as esv files. Files must be separated by real data, generated data, and (real) data used condition the generation. A more detailed description can be found GitHub.
B. LOBS5 Training Details
Test Loss
5002
CP3HC
30
Stop
20 40
60
80
100
Figure 9: Test set (2023 data) loss curves for the LOBS5 model, measuring the mean per-message negative log- log-likelihood for INTC (green) and GOOG (red) throughout 100 training epochs. Message cross-entropy after 100 epochs 19.14 for INTC and 17.80 for GOOG.
Starting from the model introduced by Nagy et al. (2023), we have scaled up the model size by adding additional S5 layers, with resulting parameter count of approximately 35M (compared originally 6.3M). The training consisted of 100 epochs of shuffled data sequences from the entire year of 2022, training with a total training budget of 30.4 L.40 days (3.8 days across 8 GPUs). Adam (Kingma & Ba, 2014) was used as an optimizer with cosine learning rate schedule. Losses on the test set over the course of training are displayed in Figure 9.
With this larger model, we also successfully removed the explicit error correction mechanism, which originally rejected semantically incorrectly generated messages, as error rates could be sufficiently reduced by scaling the model.
13

---

LOB-Bench: Benchmarking Generative AI for Finance
C. RWKV Training Details
Training Loss over time for DO0G
Training Loss over Time for INCC
>857 7uk 4
HerSy narsy
100
300-
ALL
Message
20
20
100
10P
10F
107
107
Namber oagen
NurTber Massages
Figure 10: Training loss curves for RWKV model training. The y-axis represents the average negative log likelihood of the messages being trained on, calculated as the sum of negative log likelihoods of each token in the message. The bold lines represent the exponential moving average of the true curve (presented with lower opacity) with an a of 0.1 to better highlight the trend.
We trained all RWKV models using an autoregressive training scheme with only message data (i.e., without any orderbook state information), and apply the same message filtering protocol as Nagy et al. (2023). We first tokenized each LOBSTER dataset using byte pair encoder (Sennrich et al., 2016) trained on GOOG 2017 messages, resulting in datasets of 5.5 billion tokens for INTC 2022 (corresponding 276 million messages) and 7.5 billion tokens for GOOG 2022 (corresponding to 380 million messages). We divide each dataset into chunks of 16384 tokens, and randomly shuffle these chunks for training.
For training, we initialize the parameters from the open source base pretrained RWKV models, each consisting of 170 million parameters, and train them on 8 chunks per optimization step, using the DAdapt-AdamW optimizer (Defazio & Mishchenko, 2023) in Optax (DeepMind et al., 2020), without scaling the learning rate or using any learning rate schedulers. For stability, we clipped the maximum global gradient norm to 1.0 (Pascanu et al., 2013). In total, training all 4 of our RWKV models (2 model architectures, each over datasets) took 10 L40S days. We present loss curves in Figure 10.
D. Sensitivity of Divergence Metrics to the Bin Size
We adopt a dynamic bin size determined by the Freedman-Diaconis (FD) rule. which is specifically designed to adapt to the underlying data distribution. As a result, we do not anticipate significant sensitivity to the choice of bin size. This choice is further supported by a theoretical convergence property: the FD rule minimizes the integrated mean squared error (IMSE) between the histogram and the true data distribution (Freedman & Diaconis, 1981).
To empirically evaluate the impact of different bin sizes on the reported divergence scores, conduct the following experiments. The same divergence metric computations are repeated with half the regular bin size, as well as with double the regular bin size. To illustrate low bin size sensitivity, we perform this experiment for the LOBS5 model across all evaluated divergence metrics. We note that smaller bin sizes tend to increase both L1 and Wasserstein- errors, as data points, which would have been grouped into a single bin, could then be part of two separate bins. On the contrary, larger bin sizes tend to decrease errors, as data points close to each other are more likely to be grouped together.
To empirically assess the robustness of our results bin size variation, we repeat the divergence metric computations using both half and double the default FD bin size. This analysis is conducted using the LOBS5 model across all evaluated divergence scores. We observe that reducing the bin size generally increases the Lı errors. This is likely because data points that would have previously fallen within the same bin may now fall into separate bins, amplifying divergence estimates. Conversely, increasing the bin size tends to reduce these errors, as nearby data points are more likely to be aggregated into the same bin, leading to smoother approximations.
Figure 11 shows the mean divergence scores for the LOBS5 model, along with 99% confidence intervals. Doubling or
14

---

LOB-Bench: Benchmarking Generative AI for Finance
halving the bin size results in a consistent decrease or increase in error scores, respectively. Notably, the magnitude of these deviations is comparable to the width of the confidence intervals under the default FD bin size. Given that these changes relatively small and systematic, conclude that while the choice of theoretically grounded binning rule is important, model ranking remains stable as long as the same binning strategy is applied uniformly across all evaluated models.
0.15
with
hall bin width
0.11
0.10
0-100
100-200
200-300
300-000
400-500
Prediction Horizon (messagest
Figure 11: L1 Divergence Scores with half and double regular bin size. Large bin size deviations (halfing or doubling) systematically affect the level of scores but not their ranking.
E. Additional Figures
L1 Loss (GOOG)
Wassersteln (GOOG)
1.25
0.6
3.00
0.75
0.4
0.50
0.0
Ord mdd
Log onee metoriio
and cane
Lien and and deop
Ask can dogo
tad can dap
Lim sts and lev
Lim Aakk cwn lev Riit can lov rolperomon
ont
on
Oef up cia or duu
0.00
Ord.oo
Log iniiitaaaci accamm
Lop tim tucan Ase vot tow
Sidvokto tok
Ask. vol
Bisuee
Lin Linusst and sepp
sim Lwmbbi..rr siop Ask can dop
Bit can dee
Lim Lmmrrt ore
Gen wwd.....oo..... ord.ee
Ask coccke lev
Shrt can
Vet pespersnc min
on
on
oruto
ufloww
GOOG-LJ
(b) GOOG- Wasserstein
L1 Loss (INTC)
Wasserstein Loss (INTC)
2.5
Wosserstain Lass
2.0
0.6
2.5
3.0-
0.
0.0
Oud tedi Logg nn acctim
tongin to ceo Ask vel tow
Bid uno
Ask vo!
Birt vet
Lin Minassccom dup
Lim bid mnd dlop
dsk dapo
thd car dip
Lin ase ord lev
Ask ley
But can lew
on Oflip
offitta
Off offe
0.0
Ord km Lop bot scton Log tom to enu ASE vol. too edd
vot
Bid vol
Lóm Aom peeeesoooo cups dap
Lom bett ond seop
Aukk dap dop
Bnd dopp
Lim ask ard low Lin but oest oy
Bid low
on orlup citato
on dow
Lim bid ordt
con
(c) INTC LI
(d) INTC- Wasserstein
Figure 12: LI and Wasserstein-1 I errors of generated unconditional distributions for easy comparison between Alphabet
(GOOG) and Intel (INTC). Error bars show 99% bootstrapped confidence intervals.
15

---

LOB-Bench Benchmarking Generative AI for Finance
lobs5
Spread
Orderbook imbalance
Spread
Orderbook imbalance
0.004
0.4
Denatty
Daniby
Demslyy
0.002
0.5
0.002
0.2
0,000
100
150
250
300
330
400
0.0
0.0
0.000
100
120
140
180
6.0
-0.5
Log inter arrival time
Log time to cancel
inter arrival time
time to cancel
Donstty
0.1
Demilyy
61
Donetty
0.05
Dendity
0.05
0.0
0.0
0.00
0.00
Ask volume touch
Bid volume touch
Ask volume touch
Bid volume touch
0.n10
0.010
Denety
Dondty
0.00010
Danuiit
0.00010
0.005
0.00000
0.000
0.000
0.00000
0.00000
250
500
750
1000
250
500
750
1000
5000
10000
15000
3000
10000
15000
Dangty
Derstt
Deresty
5000
10000
15000
5000
10000
15000
20000
40000
50000
80000
0
20000
40000
60000
80000
Limit ask order depth
Limit order depth
Limit ask order depth
Limit bid order depth
Denth
0.0002
Danoty
0.005
Densty
0.005
0.001
0.001
0.000
250
500
750
1000
1250
0.000
250
500
750
1000
1250
1500
0.000
30
100
150
200
250
300
350
0.000
50
100
150
200
250
300
350
Ask cancellation depth
Bid cancellation depth
Ask cancellation depth
Bid cancellation depth
Deminy
0.007
Dendit
0.002
Damibyy
0.0050
0.003
0.001
0.0025
0.000
750
1250
0.000
750
1250
0.000
30
200
300
0.0000
50
150
300
Limit ask order levels
Limit bid order levels
Limit ask order levels
Limit bid order levels
0.2
Density
0.3
Denaity
611
Donsity
0.
Denaty
0.
0.0
0.0
0.
0.0
Ask cancellation levels
Bid cancellation levels
Ask cancellation levels
cancellation levels
Dxcrity
Dorsity
0.1
0.2
0..0
per
Ofi
0.004
Deraity
Dendtt
0.002
20000 40000 50000 50000100000
-20
-10
10
20
L000200000000400000
200
1100
0
100
200
up
stay
Ofi up
stay
Denoly
0.04
20
00.0
Dencky
0.004
Densty
0.004
0.02
0.002
0.002
0.00
20
-10
10
20
0.00
-20
-10
10
20
0.000
-100
100
200
0.000
200
-100
0
100
200
Ofi down
Ofi down
0.04
Denvity
Denvby
0.004
0.00
0.000
-30
-100
100
(a) GOOG
(b) INTC
Figure 13: LOBS5 histograms comparing score distributions for real (blue) and generated (orange) LOB data for Alphabet (GOOG) and Intel (INTC) stocks. Overall, the generative LOBS5 model evaluated here, adapted from Nagy et al. (2023), does a good job in matching data along various dimensions. Bigger errors in matching distributions are visible e.g. spread (GOOG), orderbook imbalance (INTC) and time to cancel (GOOG and INTC).
16

---

LOB-Bench: Benchmarking Generative AI for Finance
baseline
Spread
Orderbook imbalance
Spread
Orderbook imbalance
Degdby
0.004
Dendty
Denily
Demely
0.002
0.002
0.000
300
150
300
-0.5
0.000
100
120
180
0.0
0.5
Log inter arrival time
Log time cancel
Log inter arrival time
Log time to cancel
Deralty
Deretty
Density
Dereity
0.1
-10
0.0
0.0
-15
0.0
Ask volume touch
Bid valume touch
1n.5
Ask volume touch
2e.1
Bid volume touch
Denstty
Dereh
0.02
Dewsty
0.01
600
400
40000 50000
Inwh
Denek
0.00010
Deneyy
0.0001
Densty
Demmty
0.00005
2500
5000
7500
1000012300
2500
5000
7500
10000 12500
25000 50000 75000100000 25000
25000 Sc000 750010000
Limit ask order depth
Limit bid order depth
Limit ask order depth
Limit bid order depth
Densby
0.002
Denuly
Densty
0.00
0.005
0.003
0.001
0.000
0.000
250
500
750
1000 1250
250
500
750
3000 1250
50
100
150
200
250
300
350
50
100
150
200
250
300
350
Ask cancellation depth
Bid cancellation depth
Ask cancellation depth
Bid cancellation depth
Dandit
Denaty
0.0
0.005
Danstt
0.005
8.001
0.001
0.000
6.000
0.000
0.000
250
500
750
1000
1250
250
500
750
1000
1250
50
100
150
200
250
100
150
50
100
200
250
300
350
Limit ask order levels
Limit bid arder levels
Limit ask order levels
Limit bid order levels
Derdtt
Deneky
Derdiy
Deraly
0.1
0.2
0.0
0.
1.5
3.5
1.0
1.5
Ask cancellation levels
Bid cancellation levels
Ask cancellation levels
Bid cancellation levels
0.4
Oonpty
Dandky
0.1
Daaetty
0.2
Densty
0.
0.
1.5
1.5
16-5
Vol per min
per
0.04
Demaiyy
Dendtty
0.2
20000 40000 60000 80000 100000
-20
-10
10
100000
200000
300000
-100
-50
50
100
up
stay
up
stay
0.04
Dundit
0.050
Denaity
0.0050
Denstty
0.2
0.02
0.025
0.0025
-20
-10
10
20
-20
-10
10
20
-100
100
200
-100
-50
50
100
Ofi down
down
Derdty
0.050
Dersty
0.0005
0.025
0.0023
0.000
0.0000
20
-10
0
10
20
200
100
100
(a) GOOG
(b) INTC
Figure 14: baseline histograms comparing score distributions for real (blue) and generated (orange) LOB data for Alphabet (GOOG) and Intel (INTC) stocks. The (Cont et al., 2010) model does a decent job matching some of the scores, particularly discrete ones, such depths and levels. Clear shortcomings are visible scores such as orderbook imbalance or volumes.
17

---

LOB-Bench: Benchmarking Generative AI for Finance
Spread
Orderbook imbalance
Spread
Orderbook Imbalance
1.0
Demsty
0.005
Densty
Danilyy
0.002
0.5
0,000
200
00
-1.0
-0.5
0.000
100
150
250
350
0.0
Log inter arrival time
Log time cancel
Log inter arrival time
Log time cancel
8.
0.10
Density
Denstty
Deruity
0.05
0.3
0.05
0.05
0.00
0.0
0.00
0.00
-7.5
Ask volume touch
Bid volume touch
Ask volume touch
Bid valume touch
Dew meweh
00.030
0,00
Denett
0.004
0.00050
0.005
0.00025
0.000
0.000
0.000
0.00000
500
500
530
1000
1500
3000
Ask volume
0.0002
0.00010
Denaty
Denety
Demely
0.0002
0.00005
5000 10000 15000 20000 25000
0.0000
5000
10000
15000
10000
20000
30000
5000 10000
15000 20000 25000
ask order depth
Limit bid order depth
Limit ask order depth
Limit order depth
0.004
Denshy
0.001
Denelly
Dameit
0.0050
0.000
0.0025
250
500
750
1000
1250 1500
250
500
750
1000
1250
200
400
600
800
1000
200
400
600
B00
Ask concellation depth
Bid cancellation depth
Ask cancellation depth
Bid cancellation depth
Derety
0.004
Dandtt
Density
0.004
Dunstyy
0.0050
8.002
0.02
0.002
0.0255
0.000
0.00
0.000
0.0000
1500
250
750
1000
1250
200
400
100
200
800
8on
Limit ask order levels
Limit bid order levels
Limit ask order levels
Limit bid arder levels
Dersky
0.10
Dondty
0.-
Derdtr
02
2050
0.25
0.00
0.0
0.0
0.00
Ask cancellation levels
Bid cancellation levels
Ask ca cancellation levels
Bid cancellation levels
Dwieky
0.10
0.2
0.3
0.1
0.2
0.30
per
Vol
0.10
Danily
Densty
0.02
0.06
0.003
25000 50000
75000 100000120000
-20
-10
20
100000 200000 300000 400000
-50
-25
25
50
Ofi up
stay
up
stay
0.30
Demady
0.05
Danalty
Dersity
Density
0.06
0.01
-20
-10
10
-20
-10
0
10
20
-0
-20
0
20
40
60
-50
-25
0
25
50
down
Ofi down
Deraty
0.05
Dendty
0.00
20
-10
10
0.0
75
50
-25
25
(a) GOOG
(b) INTC
Figure 15: rwkv4- histograms comparing score distributions for real (blue) and generated (orange) LOB data for Alphabet (GOOG) and Intel (INTC) stocks. The model produces volatile data with larger spreads, missing correct order levels, leading difficulty matching book volumes.
18

---

LOB-Bench: Benchmarking Generative AI for Finance
Spread
Orderbook imbalance
Spread
Orderbook imbalance
Deniiy
0.010
Dendty
Danilty
0.03
Density
0.4
0.005
0.00
200
300
600
700
0.00
100
250
300
350
0.0
Log inter arrival time
Log time cancel
Log inter arrival time
Log time cancel
Densaty
0.10
Persiy
Dapaly
0.30
Densiy
0.10
0.05
0.05
0.05
0.00
0.00
0.00
0.00
Ask volume touch
vol volu tauch
Ask volume touch
Bid volume touch
Dew
0.010
0.005
2.00000
Dansty
0.005
0.00003
0.000
0.000
0.0000
0.0000
500
750
500
5000
20000
5000
10000 12500
Densy
0.0005
Densty
Denpty
5000
10000
15000 20000
0.0000
5000
10000
15000
25000 50000
75000
10000000022000
20000
40000
60000
Limit ask order depth
Limit bid order depth
Limit ask order depth
bid order depth
Denuhy
Densiy
Deneily
0.004
Denstt
0.010
0.002
0.005
0.000
500
1000
1500
2000
250
500
750
1000
1250
100
200
300
400
500
50
100
150
200
250
300
350
Ask cancellation depth
Bid cancellation depth
Ask cancellation depth
Bid cancellation depth
0.02
Densty
Deraity
Denaity
0.04
Densty
0.01
0.09
0.02
0.00
0.000
0.00
0.00
500
1000
1500
2000
250
500
750
1000
1250
100
200
100
400
500
50
100
150
200
250
300
350
Limit ask order levels
Limit bid order levels
Limit ask order levels
Limit bid order levels
Densky
0.10
Dereit
0.2
Derdiyy
0.2
Deralty
0.5
0.05
0.00
0.0
01
1.5
3.5
0.0
1.0
1.5
Ask cancellation levels
Bid cancellation levels
Ask cancellation levels
Bid cancellation levels
Dwreky
Densky
Denetty
0.2
Densty
0.50
0.08
0.3
0.0
0.
1.5
1.5
per
per
0.2
Denalty
Dendtty
3.5
5000000005000000000.
-30
-20
-10
10
20
300009030000000500000
-150
-100
-50
50
300
up
stay
up
stay
0.2
Dariaty
Danaly
Danaity
Dendfty
0.1
0.05
0.02
-20
20
-30
-20
-10
10
20
-100
-50
50
-150
-000
-50
50
300
Ofi down
down
Dersit
0.04
Danaty
0.02
0.00
20
20
0.00
150
-100
-50
0
50
(a) GOOG
(b) INTC
Figure 16: rwkv6- histograms comparing score distributions for real (blue) and generated (orange) LOB data for Alphabet (GOOG) and Intel (INTC) stocks. The model has similar shortcomings to RWKV 4 (wrong price levels, mismatched book volumes etc.) due tokenization raw data and missing order book information.
19

---

LOB-Bench: Benchmarking Generative AI for Finance
GOOG coletta
Spread
Orderbook imbalance
Densty
0.
0.0001
giverstee
Denstyy
0.0000
20000
30000
-05
Log
Log to cancel
Densty
Denstty
0.2
0.0
10.0
-5.0
-25
Ask volume touch
Bid volume touch
0.05
Densty
0.05
50
100
150
200
250
50
100
150
200
250
Ask volume
Bid volume
Densly
Densty
0.01
250
500
750
1000
1250
1500
250
500
750
1000
1250
Limit ask order depth
Limit bid order depth
Density
56002000025000
20000
Ask cancellation depth
Bid cancellation depth
0.0001
Deraity
Densfty
0.0000
Limit ask order levels
Limit order levels
Derstt
Donaty
0.1
0.0
0.0
Ask cancellation levels
Bid cancellation levels
Densev
Donst
0.0
100
10
Vol per min
Ofi
0.0002
Density
5000
10000
15000
-2
Ofi up
Ofi stay
Density
0.1
Density
Ofi down
Denety
0.1
0.0
Figure 17: coletta GOOG histograms comparing score distributions for real (blue) and generated (orange) LOB data for Alphabet (GOOG) and Intel (INTC) stocks.
20

---

LOB-Bench: Benchmarking Generative AI for Finance
Divergence coletta
Divergence INTC rwkv6
99% Drowppee
98% O Ver
spread
orderbook imbalance
spread
orderbook imbalance
0.4
0.50
0.5
0.7
L3
0.55
0.0
0.00
0.0
100- [messages]
Predictior [messages]
Prediction Herizon [messaçes]
Prediction Herizon [messaçes]
log inter arrival, time
time to cancel
log inter arrival time
log time to cancel
soore
0.50
scoe
0.5
0.25
0.0
0.00
0.0
0.0
200-300
100 vou-au0
300-400 400-500
100-200 230 vou-a00 [messages] bid volume touch
Predictior [messages]
Prediction Herizan [messages]
Prediction Herizan [messaçes]
ask volume touch
bid volume touch
ask volume touch
0.50
0.50
05-
0.25
0.25
0.00
0.00
0.0
0.0
100-200
200-300 300-400 400-500
0-100
300-100
0-100 300-100
0-100
100-230 500-100
Predictior Horizon [messages]
Prediction Horizan [messages]
Prediction Hortzan [messages]
Prediction Horazn [messages]
ask volume
bid volume
ask volume
bid volume
1.0
1.0
L1.500
0.50
0.5
0.25
L3
0.5
0.5
0.00
0.0
100-200 200-900
400-500
0-100
200- 100 500-400
6-100
w0-400
0:100
100--20 500 500-400
Predictior Horizon [messages]
Prediction Herizan (mestages]
Prediction Horizan (messages]
Prodiction [messages]
limit, ask order depth
limit, bid order depth
limit ask order depth
limit bid order depth
10
0.50
0.5
0.25
0.5
0.5
0.00
0.0
0-100
100 300-400
100
Predictior Horizon [messages]
Prediction Horizon [messages]
Prediction Horizon [messaçes]
Prodiction Horizon [messages]
ask cancellation depth
bid cancellation depphh
ask cancellation depth
bid cancellation depth
1.0
0.50
0.50
score
0.25
0.25
0.5
0.5
0.00
0.00
100-200
400-500
0-100
200-300 200-400
100 200 300-400
0:100
100-200 300 300-400
Predictior Horizon [messages]
Frediction Horizon
Frediction Horizon
Prediction Horizon [messages]]
limit ask order levels
limit bid order levels
limit ask order levels
limit bid order levels
0.6
1.0
0.2
0.-
0.1
0.0
0:100 100-200 200-300 300-400 400-500
0-100
100-200 200-300 300-400
6-100 100- 200-300 300-400 403-500
0:100 100-230 300-400
Predictior Hortzon [messages)
Prediction Hortzan
Prediction Hortzan [mestages]
Prediction Horaon
ask cancellation levels
bid, cancellation levels
ask cancellation levels
bid cancellation levels
L1 scom
score
0.5
0.0
0:100 100-200 400-500
0-100 100-200 200-300 300-400
6-100 100-200 200-300 300-400 403-500
0:100 100--20
Prodictior Hortzon [messages]
Prediction Hortzan (messages]
Prediction Hortzan (messages)
Prodiction [messages]
vol. per min
vol. per min
1.0
L1 score
0.50
score
0.5
0.25
0:100 100-200 200-300 400-500
0-100 100 200-300 300-400 Prediction Horizon (messages)
Predictior Horizon [messages]
(a) GOOG
(b) INTC
Figure 18: LI error divergence: comparing the LI errors of score distributions of real data with generated data distributions at a specific horizon into the future shows accumulating model errors. This is explainable due to snowballing errors caused by teacher forcing (conditional next token loss). A good model should be able to control errors for sequence lengths as long as possible. To provide significance threshold over pure sampling noise, the dotted lines plot the 99. percentile of L.1 error between bootstrapped samples of only real data.
21

---

LOB-Bench: Benchmarking Generative AI for Finance
(spread volatility) histograms
volatility E [0.00e+00, 1.80e-05] (w=0.10)
0.005
0.000
volatility E [1.80e-05, 2.41e-05] (w=0.10)
0.005
0.000
volatility E [2.41ee05, 2.89e-05] (w=0.10)
0.005
(spread time) histograms
0.000
time E [3.42e+04, 3.76e+04] (w=0.30)
volatility E [2.89e-05, 3.36e-05] (w=0.10)
0.005
0.005
0.000
0.000
time E [3.76e+04, 4.13 (w=0.18)
volatility E [3.36e-05, 3.82e-05] (w=0.10)
0.005
0.005
0.000
0.000
time [4.13e+04, 4.51e+04] (w=0.13)
volatility E [3.82e-05, 4.36e-05] (w=0.10)
0.005
0.005
0.000
0.000
time E [4.51e+04, 4.88 (w=0.11)
volatility E [4.36e-05, 5.10e-05] (w=0. 0.100
0.005
0.005
0.000
0.000
time E [4.88e+04, 5.26e+04] (w=0.12)
volatility E [5.10e-05, 6.12e-05] (w=0.10)
0.005
0.005
0.000
0.000
time E [5.26e+04, 5.63e+04] (w=0.12)
volatility E [6.12e-05, 8.03e-05] (w=0.10)
0.005
0.005
0.000
0.000
time E [5.63e+044 04, 5.76e+04] (w=0.05)
volatility E (8.04e-05, 3.29e-04] (w= 0.10)
0.005
0.005
0.000
0.000
100
200
300
400
500
600
100
200
300
400
500
600
700
800
spread
spread
(a) Bid-ask spread conditional on the hour of the day: spreads higher early in the day, where the generated data also exhibits too narrow spreads.
(b) Spread conditional on volatility: higher volatility corre- sponds higher frequency of higher spreads. The model does not fully capture this change, the higher discrepancy high-volatiiity bins shows.
Figure 19: Histograms of conditional score distributions for real (blue) and generated (orange) data for the Alphabet stock (GOOG). Weights w, expressing the share of data in the bin, measure the impact of the specific conditional distribution (row) on the total metric loss.
22

---

LOB- LOB-Bench: Benchmarking Generative AI for Finance
Receiver Operating Characteristic (ROC)
(book volume spread) histograms
1.0
spread E [1.00e+02, 1.00e+02] (w=0.68)
0.02
True Positive Rate
0.00
0.6
spread [2.00e+02, 2.00e+02] (w=0.27)
0.005
0.000
spread E [3.00e+02, 3.0 (w=0.04)
0.01
0.2
0.00
250
500
750
1000
1250
1500
ROC curve (area 0.83)
book volume
0.0
False Positive
Figure 20: Histograms of total book volume conditional on bid-ask spread for Alphabet stock (GOOG). Weights w, expressing the share of data the bin, measure the impact of the specific conditional distribution (row) on the total metric loss.
Figure 21: LOBS5 ROC curve of the discriminator on test data (GOOG). The discriminator represents a worst- case adversarial score function by learning to effectively differentiate between real and generated sequences of LOB states.
Histogram Logit Scores Generated Sequences Test
0.25
real
generated
0.20
Density
0.1.
0.10
-100 -80
-60
-40
-20
0
20
40
Logit Scores
Figure 22: LOBS5 Histogram logit scores for real and generated sequences on held-out test data (GOOG), Matching this distribution well would indicate high model quality, as even a trained discriminator network would not be able differentiate the distributions.
23

---

:0B-Bench: Benchmarking Generative AI for Finance
Tick-normalised microscopic response functions for stock ticker: INTC Generated from S5
Real Data Sequences
Generated from Baseline
12
MD_0
MO_0
MD_0
MD_1
1.0
*
LO_0
*
LO_0
x LD
L0_1
10_1
L0_1
CA_0
0.8
CA_1
CA
CA
RTt (ticks)
0.6
0.4
0.2
0.0
100%
Events lag
Events lag
Events lag
Figure 23: Comparison of impact response functions for different event types between real and generated data-sets, tick- normalized mid-price response. Shaded regions are 99% confidence intervals. There a comparison between two select models: the LOBS5 and the stochastic baseline. We see that, in contrast to the baseline, the generative model able reproduce much of the expected impact function, though well for GOOG.
24


Reviews:
[{"review_id": "ed2sgj2Sbo", "reviewer": "Reviewer_EUwk", "summary": "This is a great study with multiple important contributions:\n1. The paper introduces a new benchmark for evaluating limit order book (LOB) generated data,  applying aggregator functions to extract LOB-specific statistics and measuring the distribution distance between real and model generated data in both unconditional and conditional settings.\n2. They also evaluate market impact response functions\n3. They very well justify the distance measures used as evaluation score\n4. They demonstrate nicely how their benchmark ranks various models, comparing across multiple scores and also plotting distributions showing a visual comparison.\n5. They show that an autoregressive state-space model (LOBS5) outperforms traditional parametric and GAN-based methods on GOOG and INTC data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer **EUwk** for their thorough review and detailed feedback. We appreciate their recognition of the key strengths of our paper and framework, including the robust justification for our methods and scoring functions, the generality of the methodology, market impact evaluations, fully distributional scoring, and the extensive visualizations provided. We address the queries below, and would welcome any follow-up questions or suggestions. If these have been suitably addressed, we would appreciate an increase in the review score to help build a strong case for acceptance.\n\n## Discriminator Training\nWe do not make any specific claims or recommendations regarding the architecture of the discriminator. Our chosen model, combining Conv1D with attention layers, performs well and consistently distinguishes generated data from real data across all generative models, with a clear gradient in discriminator certainty across models indicating a varying degree of model veracity.\n\nRather than focusing on a particular discriminator design, we emphasize the value of using a discriminator as a learned adversarial scoring function. This approach remains valid as long as the model is sufficiently strong to learn the discrimination task effectively. Consequently, ablations on specific model characteristics are less relevant, as various state-of-the-art models could fulfill this role.\n\nThe key challenge we aim to address is improving generative model performance. In this regard, our chosen discriminator effectively exposes the gap between generated and real data while providing a meaningful learning signal for future progress.\n\n## Sensitivity to Choice of Aggregator Function\nFor a detailed analysis of how benchmark results vary with the choice of aggregator function, we refer to Figure 10 in Appendix D, which presents a bar plot comparing distributional errors across models. To ensure that outliers in scores do not disproportionately impact model rankings, we also report the median and interquartile range (IQR) of scores in the model summary plots (Figure 4).\n\n## Sensitivity of Divergence Metrics to the Binning Strategy\nThank you for suggesting this ablation. We are now evaluating the divergence scores with bins that are half and double their current size. We will report the sensitivity of the results based on the range of resulting scores in the camera-ready version of the paper.\n\nSince we use a dynamic regular bin size determined by the Freedman-Diaconis (FD) rule, which is designed to adapt well to the distribution of the data, we do not expect significant sensitivity to changes in bin size. Furthermore, the following theoretical convergence argument supports this choice: the FD bin size minimizes the integrated mean squared error between the histogram and the theoretical data distribution. Bin size decreases at a rate of n^{1/3}, and the expected number of observations per bin increases without bound. This suggests that, for large sample sizes, our choice of bin size will not introduce substantial inaccuracies in the divergence scores.", "claims_and_evidence": "Authors claim that their benchmark quantitatively assesses the realism of generative models for LOB data. They also claim that based on their findings, the LOBS5 model notably achieves superior performance over competing methods. They provide detailed analysis on their evaluation measures as well as detailed statistical analysis including error divergence curves, bootstrapped confidence intervals, discriminator ROC scores, etc. to support their claims.", "methods_and_evaluation_criteria": "Their proposed framework maps high-dimensional LOB data to 1D scores via aggregator functions, compares histograms of these scores using the L1 norm and Wasserstein-1 distance, for both unconditional and conditional distributions (where the inference timeframe is bounded, such as error divergence over forecast horizons).", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Their empirical results evaluate five different generative models (including traditional parametric methods, GAN-based approaches, and autoregressive models) on LOB data for Alphabet (GOOG) and Intel (INTC). Their analysis is robust: it employs multiple LOB-specific statistics, using a variety of LOB-specific metrics and visually contrasting the distributional differences in synthetic data across several tasks. It also visualizes error accumulation and histogram discrepancies over time, and uses bootstrapped confidence intervals to assess significance, which collectively underscore the models’ varying capabilities in capturing realistic market dynamics. Figures and captions are very clear and informative.", "relation_to_broader_scientific_literature": "This is a strong framework which provides a full distributional evaluation tool tailored to financial data, and is accessible and easily transferable to other domains.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "The key strength of the paper is its novel and detailed evaluation framework that provides clear, quantitative measures for generative realism in LOB data. It would be beneficial to add more ablation experiments for other choices of discriminator network architectures, and other binning strategies.", "other_comments_or_suggestions": "-", "questions_for_authors": "1. How sensitive are your benchmark results to the choice of aggregator functions?\n2. How sensitive are your divergence metrics to the binning strategy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tXBAMucUQr", "reviewer": "Reviewer_KDf5", "summary": "This paper introduces LOB-Bench, a novel benchmark implemented in Python for evaluating the quality and realism of generative AI models applied to Limit Order Book (LOB) data in the LOBSTER format. The benchmark addresses the lack of quantitative evaluation paradigms in financial sequence modeling by providing a comprehensive framework for distributional evaluation. LOB-Bench measures distributional differences between generated and real LOB data, both conditionally and unconditionally, using a suite of relevant LOB statistics (spread, volume, imbalance, inter-arrival times) and scores from a discriminator network. Furthermore, it incorporates \"market impact metrics\" to assess cross-correlations and price response functions for specific events. The authors benchmark several generative models, including autoregressive state-space models, a (C)GAN, and a parametric LOB model, finding that autoregressive GenAI approaches outperform traditional models. The code and generated data are publicly available to facilitate further research and model development.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer KDf5 for their detailed review and thoughtful comments, which not only provide valuable feedback but also highlight key strengths of our paper. In particular, we appreciate the recognition of the important gap our work addresses by introducing a well-founded, fully distributional evaluation of generative LOB data, including assessments of model derailment and price impact functions.  We address the queries and concerns below, and would welcome any follow-up questions or suggestions. If these have been suitably addressed, we would appreciate an increase in the review score to help build a stronger case for acceptance.\n\n## Limited Assets\nThe reviewer raises a concern regarding the limited number of assets evaluated in our study. While we acknowledge that results may vary across different stocks, we selected GOOG and INTC as representative examples because they span a broad spectrum of relevant statistics, such as volatility, relative tick size (tick size relative to stock price), and traded volume.\n\nExpanding the evaluation to additional stocks would require substantial computational resources. Training each considered model class on a new stock dataset takes multiple days on 8 L40S GPUs per stock. Moreover, as is standard practice in the domain, each model is trained on a single stock, since ample training data is usually available, and models are expected to specialize in the characteristics of individual stocks.\n\nSmall-cap and less liquid stocks, while an interesting avenue for future research, present additional challenges beyond the scope of this paper. These stocks provide less training data due to lower trading activity, which complicates deep learning approaches. A promising direction for future work is multi-asset modeling, which could leverage data across multiple stocks to enhance training and capture stock-independent dynamics.\n\nIt is important to emphasize that the benchmark itself is designed to allow researchers to evaluate models on any stock of their choice, and its usefulness is not limited to the two stocks presented in our paper. Our package includes efficient code that enables fitting baseline model parameters to new stock data in seconds to minutes. Furthermore, we plan to expand our research by developing new models that will naturally be evaluated on a broader set of stocks.\n\n\n## Zero-Shot Transfer Evaluation\nAn alternative evaluation could assess zero-shot model transfer—training on one stock (e.g., GOOG) and generating data for another (e.g., WMT). While this reduces computational costs, it introduces challenges.\n\nFirst, seeding mechanisms vary by model class: autoregressive models allow conditioning on history, but the Coletta CGAN model only seeds at the start of the day, and the baseline model does not support seeding. Second, it is unclear whether performance differences in transferred data stem from stock characteristics or model limitations. A more robust approach would involve training on multiple stocks, which is beyond the paper’s scope.\n\nThat said, our benchmark supports such research directions, as it remains agnostic to the model training regime.\n\n\n## Discriminator Scores\nWe have added the following to the paper:\n> “High discriminability may result from model errors, as indicated by imperfect model scores [see …]. A distributional mismatch in a single scoring function can be sufficient to make fake data identifiable. To mitigate this issue, future research could evaluate adversarial performance by training a discriminator on perturbed data and reporting scores conditioned on the noise level, particularly as models improve on this benchmark.”\n\nLow discriminator scores suggest that even state-of-the-art models still struggle to generate perfectly indistinguishable LOB data. Future improvements may follow two paths: (1) training larger supervised models on richer datasets, akin to recent LLM advancements, or (2) explicitly minimizing discriminator scores, as in adversarial frameworks like GANs (Goodfellow et al., 2014) or GAIL (Ho et al., 2016).\n\n\n## Beyond LOB Data\nTo highlight the transferability of our methodology, we added the following to the paper:\n> “Our methodology formalizes and naturally extends common evaluation practices for synthetic one-dimensional time series, such as financial returns, which typically emphasize distributional similarity. Our framework enables a quantitative assessment of distributional properties in structured high-dimensional time series. By adapting the scoring functions, our approach could also be applied to financial transactions, payment data, streamed price quotes in forex markets, multi-asset limit order books, or decentralized crypto market protocols.”\n\n\n## Documentation\nWe provide a Jupyter notebook demonstrating LOB Bench usage and will further improve the package documentation before the paper’s potential publication.", "claims_and_evidence": "Yes, the claims made in the submission are generally well-supported by clear and convincing evidence. The paper claims to introduce a novel benchmark and demonstrate its utility in evaluating generative models for LOB data. This claim is supported by:\n\n* Development of LOB-Bench: The paper clearly describes the components of LOB-Bench, including the various scoring functions, evaluation metrics (L1 norm, Wasserstein-1 distance), and conditional evaluation methodologies. The availability of the code further strengthens this claim.\n\n*Benchmarking Experiments: The authors present experimental results comparing several generative models (LOBS5, baseline, Coletta, RWKV4, RWKV6) on GOOG and INTC stock data. Figures 3, 4, and 5 visually and quantitatively demonstrate the performance differences across models and metrics.\n\n* Identification of Model Derailment: Figure 5 and related discussion provide evidence for \"model derailment\" in longer unrolls, a crucial insight for generative sequence models.\n\n* Reproducibility: The paper explicitly mentions the availability of code and generated data, enhancing the reproducibility and verifiability of their findings.\n\nThe evidence presented, particularly the comparative benchmarking results and the analysis of model derailment, effectively supports the paper's claims about the value and utility of LOB-Bench.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are sensible and well-justified for the problem of benchmarking generative models for LOB data. Given that this is primarily a benchmark paper, the focus is appropriately placed on the design of comprehensive and relevant evaluation criteria rather than introducing novel modeling methods.\n\nThe evaluation criteria are well-chosen because:\n\n* Distributional Evaluation: Shifting from qualitative analysis of \"stylized facts\" to quantitative distributional evaluation addresses a crucial gap in the field and provides a more rigorous approach to model comparison.\n\n* Relevant LOB Statistics: The inclusion of commonly used LOB statistics like spread, volume, imbalance, and inter-arrival times ensures that the benchmark is grounded in domain knowledge and evaluates models on features relevant to financial practitioners.\n\n* Discriminator Network: Incorporating a discriminator network provides an adversarial perspective on model realism and captures complex, high-dimensional data characteristics that might be missed by simpler statistical metrics.\n\n* Market Impact Metrics: Including market impact metrics assesses the model's ability to generate realistic responses to counterfactual scenarios, a key aspect for financial applications.\n\n* Conditional Evaluation: Evaluating conditional distributions allows for a more nuanced understanding of model performance under different market conditions and over forecasting horizons, addressing the \"autoregressive trap\" issue.\n\nWhile benchmark datasets are mentioned (LOBSTER, FI-2010), the core contribution lies in the evaluation criteria and the LOB-Bench framework itself, which are well-suited for the task of rigorously assessing generative LOB models. The focus on distributional similarity and market-relevant metrics makes the benchmark highly pertinent to the application domain.", "theoretical_claims": "There are no significant theoretical claims in this paper that require formal proof verification. The paper is primarily empirical and methodological, focused on developing and demonstrating a benchmark rather than establishing new theoretical results. Therefore, this question is not applicable.", "experimental_designs_or_analyses": "The experimental designs and analyses are generally sound and valid for the purpose of demonstrating LOB-Bench. However, a limitation is the scope of assets considered:\n\n* Limited Asset Universe: The experiments are primarily conducted on data for only two assets: Google (GOOG) and Intel (INTC). While these are representative stocks, evaluating the benchmark and the models on a broader range of assets, including stocks with different market capitalizations, liquidity profiles, and industry sectors, would significantly strengthen the generalizability of the findings. Restricting the analysis to just two assets limits the external validity of the conclusions about model performance and benchmark utility.\n\nDespite this limitation, the internal validity of the experiments is well-maintained. The comparisons between models are conducted fairly using consistent evaluation metrics and the LOB-Bench framework. The analysis of model derailment and the breakdown of performance across different scoring functions are insightful and contribute to the understanding of generative LOB models.", "relation_to_broader_scientific_literature": "The key contributions of this paper are directly related to the growing body of literature on generative AI in finance, particularly in the domain of market microstructure modeling and simulation.\n\n* Generative Financial Models: The paper builds upon recent work applying generative AI to financial data, citing papers like Nagy et al. (2023) which pioneered token-level generative modeling of LOB data. It extends this line of research by focusing on rigorous evaluation.\n\n* LOB Simulation: The paper is directly relevant to the literature on LOB simulation, which has traditionally relied on agent-based models or parametric approaches (cited in the introduction). LOB-Bench offers a new paradigm for evaluating the realism of these simulations, particularly those powered by GenAI.\n\n* Benchmark Datasets and Evaluation Metrics in Financial ML: The paper addresses the broader challenge of benchmarking machine learning models in finance, where evaluation has often been ad-hoc and lacking in standardized metrics. It contributes to the emerging field of financial machine learning benchmarks.\n\n* Autoregressive Sequence Models: The paper implicitly connects to the broader literature on autoregressive sequence models, highlighting the \"autoregressive trap\" problem and the importance of evaluating models beyond next-token prediction accuracy, especially relevant in the context of LLMs and generative models.\n\nWhile the application is tightly focused on LOB data, the methodological contribution of a comprehensive distributional benchmark for generative models is more broadly relevant to evaluating sequential generative models in other domains, even if the paper itself doesn't explicitly explore these extensions.", "essential_references_not_discussed": "Based on my familiarity with the literature on generative financial modeling and limit order book research, there do not appear to be essential related works that are critically missing from the paper's citations and discussion.", "other_strengths_and_weaknesses": "Strengths:\n\n* Originality and Significance: LOB-Bench fills a critical gap by providing the first comprehensive, quantitative benchmark for generative AI models applied to LOB data. This is a significant contribution as it enables rigorous model comparison and advancement in this important area.\n\n* Practicality and Accessibility: The benchmark is implemented in Python and is open-source, making it easily accessible and usable by researchers and practitioners. The use of the standard LOBSTER format further enhances its practicality.\n\n* Comprehensive Evaluation Framework: LOB-Bench incorporates a wide range of relevant metrics, including statistical distributions, discriminator scores, and market impact measures, providing a holistic view of model performance.\n\n* Identification of Model Derailment: The paper highlights the important issue of \"model derailment\" in autoregressive models, providing a valuable insight for the community.\n\n* Clarity: The paper is generally well-written and clearly explains the LOB-Bench framework, evaluation metrics, and experimental results.\n\nWeaknesses:\n\n* Limited Asset Scope: As mentioned before, the experiments are somewhat limited by focusing on only two assets (GOOG and INTC). Expanding the asset universe would strengthen the generalizability of the findings.\n\n* Application Specificity: While LOB-Bench is valuable for LOB data, the paper could briefly discuss the potential for generalizing the principles of distributional benchmarking to other types of sequential financial data or time-series domains. While the framework itself is somewhat adaptable, the paper's framing is very LOB-centric.\n\n* Discriminator Score Challenge: The paper notes that discriminator-based scoring sets a high bar. While this is a valuable metric, further discussion on how to interpret and potentially address low discriminator scores in future model development could be beneficial.", "other_comments_or_suggestions": "* Expand Asset Coverage: In future work, consider benchmarking on a more diverse set of assets to demonstrate the robustness and generalizability of LOB-Bench and the evaluated models. Perhaps including stocks from different sectors, market caps, and liquidity levels.\n\n* Explore Domain Adaptation: Briefly discuss the potential for adapting LOB-Bench or its principles to evaluate generative models in other financial domains, such as options markets, FX markets, or even broader time-series data.\n\n* Enhance Code Documentation: While the code is available, ensure comprehensive documentation and potentially example notebooks to facilitate easier adoption and use by the community.\n\n* Investigate \"Discriminator Score Challenge\" Further: Explore strategies for improving discriminator scores for generative LOB models in future research, potentially through adversarial training techniques or modified model architectures.", "questions_for_authors": "* Generalizability across Assets: Given the current experiments focus on GOOG and INTC, how do you anticipate the relative performance of the benchmarked models might change when evaluated on a significantly broader and more diverse set of assets (e.g., including small-cap stocks, less liquid stocks, or stocks from different industry sectors)? Understanding the authors' perspective on asset generalizability would help assess the benchmark's broader applicability. If they anticipate significant changes in model ranking or benchmark sensitivity, it might suggest future directions for improving LOB-Bench's robustness.\n\n* Beyond LOB Data: While LOB-Bench is specifically designed for Limit Order Book data, do you see potential avenues for adapting the core principles of distributional benchmarking and the evaluation metrics used in LOB-Bench to other types of sequential financial data or time-series forecasting tasks? Exploring the potential for broader methodological impact would increase the perceived significance of the work beyond the specific LOB domain. If authors have ideas for broader applications, it would strengthen the paper's contribution.\n\n* Discriminator Score Interpretation: The paper notes the \"difficult challenge of discriminator scores.\" Could you elaborate on how you interpret relatively lower discriminator scores for even state-of-the-art models? Does this primarily indicate limitations in the discriminator's ability to capture subtle differences, or are there fundamental aspects of generated LOB data that remain distinguishable from real data even for the best models? Clarification on the interpretation of discriminator scores would be valuable for guiding future model development and understanding the limitations of current generative LOB models. A nuanced response would demonstrate a deeper understanding of the evaluation methodology.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxBhTCLCjK", "reviewer": "Reviewer_Nn9N", "summary": "The paper introduces LOB-Bench, a benchmark designed to evaluate the quality of generative models for limit order book (LOB) data. The authors propose a quantitative evaluation framework that measures distributional differences between generated and real LOB data. LOB-Bench assesses key LOB metrics such as spread, order book volumes, order imbalance, and market impact using unconditional and conditional statistical comparisons with L1 norm and Wasserstein-1 distance. It also incorporates adversarial evaluation via a discriminator network to distinguish real from synthetic data. The study benchmarks various generative models, finding that the LOBS5 model outperforms traditional approaches. It accurately replicates price impact functions, while classic LOB models fail in this aspect.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer Nn9N for their detailed remarks and for recognizing the potential of this work in establishing a foundation for developing more robust and interpretable generative models in finance. We address the queries and concerns below, and would welcome any follow-up questions or suggestions. If these have been suitably addressed, we would appreciate an increase in the review score to help build a stronger case for acceptance.\n\n## Generalizability to Different Stocks\nOur proposed framework is inherently stock-agnostic. Users of LOB Bench can evaluate their models on any asset within the LOBSTER universe – or even beyond, provided the data is converted into the LOBSTER format. For this study, we selected two stocks that differ along key metrics and are representative of assets that models could be trained on in practice.\n\nWhile we acknowledge that different asset classes, such as ETFs, may yield different results, exploring this was beyond the scope of our work, which focuses on presenting the benchmark itself. We also refer reviewer Nn9N to our response to reviewer KDf5 regarding “Limited Assets” and “Zero-Shot Transfer Evaluation” for additional context on our stock selection.\n\n## Sensitivity to Different Market Conditions\nWe consider the evaluation of different market conditions to be distinct from the evaluation of additional stocks. A single stock can experience varying market conditions over time, including fluctuations in volatility, spread, and trading volume. Our benchmark directly accounts for this by evaluating conditional distributions.\n\nFor instance, the conditional distribution of message inter-arrival times, given the spread, captures how a model reflects variations in inter-arrival times across different market conditions (i.e., spread regimes). This approach ensures that the benchmark effectively assesses a model’s ability to adapt to dynamic market environments.\n\n## Alternative Practical Evaluation: Downstream Tasks & Sim-to-Real Transfer\nWe acknowledge the concerns regarding the reliance on distributional similarity metrics, such as the L1 norm and Wasserstein-1 distance. While these metrics are valuable for comparing real and generated data, they may not fully measure the degree of practical utility of synthetic data in all real-world applications.\n\nTo address this, we propose including a brief evaluation of mid-price trend forecasting—a downstream task relevant to algorithmic trading—in the camera-ready version of the paper. Specifically, we will assess model performance on a held-out set of real data after training on three different datasets: (1) real historical data, (2) a combination of real and generated data, and (3) purely generated data. This allows assessing the sim-to-real transfer gap and thus evaluating the quality of the generated model. It is worth noting, however, that this approach has its shortcomings. \n\nThe implementation will follow the model architectures proposed in Prata et al. (2024), focusing on the best-performing BINCTABL architecture, along with DeepLOB and a simple LSTM. This evaluation will serve as a complementary assessment to the distributional metrics, providing insights into how synthetic data impacts mid-price forecasting accuracy. Usefulness for predicting mid-prices might be a very discrete property of generated data, and a potential failure in this regard might prove uninformative for future model development. \n\n\n## RL Using Generated Data\nTraining reinforcement learning (RL) agents using generated data in a limit order book (LOB) setting is an active area of ongoing and future research. This approach represents another instance of sim-to-real transfer evaluation. Generative models have the potential to overcome the limitations of training policies solely on static historical data by introducing dynamic, action-dependent data trajectories, thereby enriching the training environment.\n\n## Grammar and Spelling\nThank you for highlighting these specific typos. We have now corrected them in the manuscript.", "claims_and_evidence": "The claims made in the submission are largely supported by quantitative evaluation methods, comparative analyses, and empirical results.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in LOB-Bench make sense for evaluating generative AI models in the limit order book (LOB) modeling context. More specifically, the unconditional and conditional distributional comparisons provide a comprehensive statistical framework for evaluating how closely generated LOB data resembles real market data. Metrics like L1 norm and Wasserstein-1 distance effectively measure distributional accuracy across different time horizons and order book features.\n\nHowever, the benchmark is tested on Alphabet (GOOG) and Intel (INTC) stocks, but it is unclear whether the results generalize to other stocks which can be more volatile. Furthermore, the framework focuses on statistical similarity but does not evaluate how well synthetic data supports real-world financial applications like algorithmic trading backtests and market stability simulations.", "theoretical_claims": "The paper does not involve theorems.", "experimental_designs_or_analyses": "The results section evaluates generative models for limit order book (LOB) data using the LOB-Bench framework, comparing LOBS5, baseline, Coletta, and RWKV models. As mentioned before, one specific concern is that there is a lack of robustness across different market conditions. The models are tested on only two stocks (GOOG & INTC), limiting insights into their performance across different asset classes, market regimes, and varying liquidity conditions. Expanding the benchmark to include volatile stocks, ETFs, and multi-market scenarios would improve generalizability.", "relation_to_broader_scientific_literature": "This paper builds on and extends multiple areas of research in financial market simulation and generative AI. It directly connects to the work of Vyetrenko et al. (2019) on realism metrics for limit order book (LOB) market simulations, which established the importance of evaluating synthetic financial data against empirical LOB properties. While Vyetrenko et al. relied on agent-based models (ABMs) to replicate market behavior, LOB-Bench focuses on deep learning-based generative models, benchmarking S5 state-space models, RWKV transformers, and GANs. This work is also closely related to Coletta et al. (2023), which explored conditional generative adversarial networks (CGANs) for LOB environments, emphasizing market reactivity and stylized fact replication. While CGANs aim to simulate realistic order flows, LOB-Bench goes further by introducing quantitative evaluation metrics (L1 norm, Wasserstein-1 distance) and adversarial discrimination to measure the closeness of synthetic and real data systematically. Ultimately, this work has the potential to establish a foundation for developing more robust and interpretable generative models in high-frequency trading.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "The study focuses primarily on distributional similarity metrics (L1 norm, Wasserstein-1) and market impact curves but does not evaluate how well the generated data supports actual trading strategies. Without backtesting in a simulated or real trading environment, it’s unclear if these generative models can improve decision-making for market participants.", "other_comments_or_suggestions": "Minor improvement on grammar: \n1. \"... which is explainable as it was intended for small-tick stocks, which INTC is not.\" -> \"... which is expected since the model was designed for small-tick stocks, whereas INTC is not.\"\n2. \"passed to function\" -> \"passed to a function\"\n3. \"The mode produces...\" -> \"The model produces...\"", "questions_for_authors": "The following questions are listed in descending order of importance:\n\n1. How generalizable is the evaluation system across different stocks and market regimes?\n\n2. The paper introduces distributional evaluation metrics (L1 norm, Wasserstein-1 distance) to compare real and generated LOB data. How sensitive are these metrics to different market conditions (e.g., high volatility vs. low volatility periods)? Have you considered alternative evaluation metrics. For instance, evaluating models trained with synthetic data using the real data?\n\n3. The paper suggests that LOB-Bench-generated data could be useful for reinforcement learning (RL) training. Have you tested any RL-based trading strategies using synthetic data, and how well did they generalize to real market conditions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Congratulations for having your paper accepted!

The reviewers have provided valuable feedback to the authors for clarifying out certain issues that will improve the manuscript in the final form.
I would advise the authors take the time till the camera ready version to incorporate the feedback.

Once again, congratulations.
Decision: Accept (poster)