SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, to demonstrate the realism of AMLworld, we discuss several statistics about the LI-Large dataset. LI-Large has 176M transactions spanning a little over 3 months. We focus on LI-Large, but apart from their size and laundering rates, the other five datasets have similar characteristics. Several figures in the Appendix provide grounding about the realism of our data. Figure 7 histograms annualized transactions per account, and its distribution roughly aligns with U.S. Federal Reserve data [51]. Table 5 shows the distribution of transaction formats like ACH, wire, and cheque. Again numbers correspond roughly with Federal Reserve statistics [41].
Overall 1 in 1,750 transactions is laundering in LI-Large. For comparison, Starnini et al. [56] report on real data with 180M transactions over 6 months from a major Italian bank. They find 855 laundering motifs (equivalent to our Gather-Scatter and Scatter-Gather patterns), each with "less than 20 [nodes] total". This amounts to around 1 laundering transaction per 21,000. By comparison Appendix Table 7 indicates LI-Large has 8,212 combined Gather-Scatter and Scatter-Gather transactions - yielding a very similar 1 laundering transaction per 21,440 transactions from these patterns.
4 Performance Evaluation of Machine Learning Models
In this section, we provide a preliminary evaluation of the suitability of our synthetic datasets for creating effective machine learning models. A deeper treatment and evaluation is beyond the scope of this "datasets and benchmarks" submission, but can be found in the work of Egressy et. al. [20]. In a key way, these measurements are even better than using real data: the ground truth labels are complete, whilst many laundering transactions in real data are never detected.
We train Gradient Boosted Tree (GBT) and popular message-passing Graph Neural Network (GNN) models on the datasets introduced in Section 3.5, using the tabular- and graph-based data representations, respectively. Parameter tuning for GBTs and GNNs is described in Section C of the appendix.
Source and destination account IDs of the transactions are not used as features in our experiments, which prevents our models from recognizing laundering transactions simply by learning account IDs.
GBT Baselines We use LightGBM [29] and XGBoost [16] as our GBT baselines, which are widely used machine learning models for tabular data. GBT baseline experiments were performed using version 3.1.1 of LightGBM and version 1.7.6 of XGBoost. To improve the performance of these GBT models, we use Graph Feature Preprocessor (GFP) [48, 49], which creates additional features for the datasets. To achieve this, GFP interprets the input data as a graph and extracts various graph-based features from this graph, such as vertex statistics and number of simple cycles [13, 12]. As a result, GBT models can exploit the underlying graph structure of the datasets. Section D of the appendix provides details on how GFP was configured for the experimental evaluation.
GNN Baselines GIN (Graph Isomorphism Network) with edge features [65, 25] and PNA (Principal Neighbourhood Aggregation) [60, 18] are used as GNN baselines. Since AML is a transaction classification problem, we also include a baseline using edge updates (GIN+EU) [11]. This approach is similar to the architecture used by Cardoso et al. [15], which has recently been used in self-supervised money laundering detection. Note that the GNN results for the Large datasets are not available because training models for these datasets require significantly more compute resources.
Because the vertices are accounts and the edges are transactions in our financial transaction graph representation, detection of illicit transactions is an edge classification problem. All our GNNs use a final edge readout layer that performs classification using the edge embedding and the respective endpoint node embeddings as input. To reduce the complexity, we use neighborhood sampling [23] when training and testing our GNN models. We sample 100 one-hop and 100 two-hop neighbors.
Data Split We use a 60-20-20 temporal train-validation-test split, i.e., we split the transaction indices after ordering them by their timestamps. The data split is defined by two timestamps,  $t_1$  and  $t_2$ . Train indices correspond to transactions before time  $t_1$ , validation indices to transactions between times  $t_1$  and  $t_2$ , and test indices to transactions after  $t_2$ . However, in the GNN case, the validation and test set transactions need access to the previous transactions to identify patterns. So we construct train, validation, and test graphs. This corresponds to considering the financial transaction graph as a dynamic graph and taking three snapshots at times  $t_1$ ,  $t_2$ , and  $t_3 = t_{max}$ . The train graph contains only the training transactions (and corresponding nodes). The validation graph contains the training and validation transactions, but only the validation indices are used for evaluation. The test graph contains all the transactions, but only the test indices are used for evaluation. This is the most likely scenario faced by banks and financial authorities when classifying unseen batches of transactions.
Evaluation of GBT and GNN Baselines As our proposed datasets are naturally very imbalanced, traditional metrics like accuracy do not provide a reliable measure of model performance. In this context, we have chosen to emphasize the F1 score for the minority class. Detailed precision and recall scores for the GNN baselines can be found in appendix Section E. Table 2 showcases the inference performance for the range of models evaluated. These results show that message-passing GNNs that exploit the connectivity between accounts are effective in capturing laundering transactions. Advanced GNN architectures such as PNA and  $\mathrm{GIN + EU}$ , which respectively implement extended message aggregation and edge update mechanisms, significantly enhance the GNN performance. Furthermore, the combination of GFP with gradient boosting techniques, specifically LightGBM and XGBoost, also demonstrates strong performance across different datasets. An interesting observation is the near-par performance of PNA with GBT baselines, even without relying on handcrafted features.
The LI datasets present more significant predictive challenges compared to the HI datasets. This is primarily attributed to their lower illicit ratios as well as the longer time span of their laundering patterns. The primary difference between LI and HI datasets is that criminals launder less frequently in LI, which makes it harder to discover the laundering patterns. To address this, our models incorporate the measure of weighting the predictions of the minority class higher in the loss function. However, considering the intricacies of dealing with the LI datasets, exploring other strategies, such as the Pick and Chose method [36], which employs a label-balanced sampler, or GraphSMOTE [66], which introduces synthetic minority class samples in the embedding space, may be imperative.
The prediction accuracy on the more challenging LI datasets can be improved by reusing or fine-tuning models pretrained on the HI datasets. Table 3 shows that reusing the pretrained LightGBM and PNA models on the HI-Medium and -Large datasets enables higher F1 scores than those of XGBoost models trained on the respective LI datasets (see Table 2). Furthermore, the classification
Table 2: Minority class F1 scores (%). HI indicates a higher illicit ratio. LI indicates a lower ratio.
Model	HI-Small	LI-Small	HI-Medium	LI-Medium	HI-Large	LI-Large
GIN [65, 25]	28.70 ± 1.13	7.90 ± 2.78	42.30 ± 0.44	3.86 ± 3.62	NA	NA
GIN + EU [11, 15]	47.73 ± 7.86	20.62 ± 2.41	49.26 ± 4.02	6.19 ± 8.32	NA	NA
PNA [60]	56.77 ± 2.41	16.45 ± 1.46	59.71 ± 1.91	27.73 ± 1.65	NA	NA
GFP [48, 49] + LightGBM [29]	62.86 ± 0.25	20.83 ± 1.50	59.48 ± 0.15	20.85 ± 0.38	48.67 ± 0.24	17.09 ± 0.46
GFP [48, 49] + XGBoost [16]	63.23 ± 0.17	27.30 ± 0.33	65.70 ± 0.26	28.16 ± 0.14	42.68 ± 12.93	24.23 ± 0.12
Table 3: Minority class F1 scores (%) of models trained on HI datasets and evaluated on LI datasets.
Figure 6: Effect of data and model sharing across banks for (a) HI-Medium and (b) HI-Large datasets.
performance on LI data can be improved further for the LI-Small dataset, where directly using the models trained on HI datasets is ineffective, by fine-tuning the HI PNA models using the LI dataset.
In Figure 6, we report results on HI-Medium and HI-Large datasets for individual banks using GFP to create features and LightGBM to build models. Related experiments performed on the LI-Medium and LI-Large datasets can be found in Section F of the appendix. For brevity, we focus on the top-30 banks that participate in the largest number of transactions in each dataset. We evaluate three setups:
- In the shared graph, shared model setup, banks share all their transactions with each other to create a shared financial transaction graph. The GFP is applied to this shared graph to extract graph features, which are also shared. A shared LightGBM model is built using this shared data, which the banks use to score their own transactions. Note that the results given in Table 2 show the aggregate F1 score for all the banks produced using the same setup.
- The private graph, private model setup computes the F1 scores using the LightGBM models trained separately by each bank on its private data. In this setup, a bank sees only the transactions performed by its own account holders. A private graph is constructed using these transactions, and the GFP is executed on this private graph to extract features. In addition, each bank trains its own model on its own data, so there is no model sharing either.
- In the private graph, shared model setup, each bank creates a private transaction graph and extracts graph features from it using GFP as in the previous case. The banks do not share the source and destination account ids with each other, but share the remaining transaction features, including the graph features. A shared LightGBM model is trained using this data.
Figure 6 shows that, in general, financial institutions can achieve higher F1 scores by building shared machine learning models. However, building shared models on top of shared financial transaction graphs leads to even more significant improvements. To achieve such accuracy improvements in a privacy-preserving manner, financial institutions can use differentially-private model and topology sharing techniques. Even though evaluation of such techniques is outside the scope of our work, we believe that our contributed datasets offer a valuable testing ground for building such approaches.
5 Future Work and Research Avenues
The focus of this paper is the generation of high-quality datasets and their use to build effective anti-money laundering models. Here are some additional research opportunities that follow naturally:
1. Historically, AML models have focused on transactions at one bank – because that is the only data available to individual banks that review transactions for laundering. Our synthetic
data enables analysis across multiple banks as shown in Figure 6. With that comes a need to build privacy-preserving machine learning models and graph topology-sharing techniques.
2. Whether considering transactions at one bank or across banks, there is a need for machine learning and deep learning models that can detect complex AML patterns in graph datasets.
3. Building on points (1) and (2), using deep neural networks, it may be possible to use our synthetic cross-bank data for pre-training, before fine-tuning with real-world data. Our initial experiments given in Table 3 demonstrate that transfer learning is a viable approach.
4. As new models and algorithms are developed, it is important that they run efficiently in terms of time, memory consumption, CPU and GPU use, etc. Techniques to achieve such efficiency when analyzing financial transactions differ from those needed in other domains.
5. Others may find ways to improve our data generation method. For example, with our agent-based approach, there can be challenges around the stability of the data generation: minor parameter tweaks can sometimes yield significant changes in aggregate results, such as the laundering rate. Generative models for graphs could be a promising research avenue.
We have outlined AMLworld, a detailed, multi-agent, virtual world approach to generating synthetic financial data labeled for money laundering. We have established that the generated data matches real-world data in key regards. AMLworld data also has perfect tagging for laundering activity – something essentially impossible with real data, where tags typically miss a great deal of laundering activity. By providing realistic tagged data, AMLworld facilitates the development of money laundering detection models, and our experiments provide baseline scores, demonstrating the utility and efficacy of machine learning models. Lastly, AMLworld enables the generation of models to detect what-if laundering patterns not yet observed in real data. Six AMLworld datasets are now publicly available on Kaggle [50]. Our initial evaluation on these datasets shows that GNNs and GBTs can be effective solutions for identifying laundering transactions. While GBTs require some feature engineering to capture complex laundering patterns, GNNs often produce competitive results without any feature engineering. Further research is needed to enable collaboration and cooperation between financial institutions through the use of differentially-private graph topology and model sharing techniques.
7 Ethics
We believe that our release of this synthetic laundering data helps in the AML fight.
As noted in the Introduction, most laundering activity currently goes undetected. This suggests the malefactors have the upper hand – as opposed to domains like credit card fraud, where the large majority of fraud is detected and quickly. One key goal in releasing this data is to facilitate improvements in laundering detection to enable results closer to credit card fraud detection.
A key advantage for credit card fraud detection is that a close ground truth is well understood – via customer feedback about unpurchased items. Unfortunately, no good ground truth exists for laundering activities in real financial data – leading to high error rates and much missed fraud. By contrast, our synthetic data has this ground truth that has been so helpful against credit card fraud.
Beyond helping the good side, we believe our data has limited potential to help the bad side. An obvious way we might do that is by publishing the algorithms used by bank X to detect laundering. Launderers could then change their behavior so as to avoid being flagged by the algorithms. With only a static set of data, it is much harder for launderers to change their behavior to avoid detection.
To this point, we emphasize that what we have open-sourced is a set of data – not the code to generate the data. If launderers had our source code, it would be easier for them to tweak the data produced and check the results versus whatever detection algorithms they could get their hands on.
We offer a few additional observations on ethics in Appendix B.
The support of the Swiss National Science Foundation (project numbers: 172610 and 212158) for this work is gratefully acknowledged.
[1] [n.d.]. Money Laundering. https://en.wikipedia.org/wiki/Danske_Bank-money_ laundering_scandal
[2] [n.d.]. Money laundering. https://en.wikipedia.org/wiki/Money_laundering
[3] 2015. Measuring the Economy: A Primer on GDP and the National Income and Product Accounts. , 2 pages. https://www.bea.gov/sites/default/files/methodologies/nipa_primer.pdf
[4] 2017. Blind Spot in the AML Regime. https://www.finextra.com/blog/posting/14298/online-payments-the-blind-spot-in-the-aml-regime
[5] 2017. From Suspicion to Action: Converting financial intelligence into greater operational impact. , 4 pages. https://www.europol.europa.eu/cms/sites/default/files/documents/ql-01-17-932-en-c_pf_final.pdf
[6] 2022. Money Laundering. https://www.unodc.org/unodc/en/money-laundrying/overview.html
[7] 2022. National Money Laundering Risk Assessment. , 21 pages. https://home.treasury.gov/system/files/136/2022-National-Money-Laundering-Risk-Assessment.pdf
[8] Erik Altman. 2021. Synthesizing Credit Card Transactions. In Proceedings of the Second ACM International Conference on AI in Finance (ICAIF'21), Anisoara Calinescu and Lukasz Szpruch (Ed.). ACM, Virtual Event. https://doi.org/10.1145/3490354.3494378
[9] Ansh Ankul. 2021. IBM AMLSim Example Dataset. (2021). https://www.kaggle.com/datasets/anshankul.ibm-amlsim-example-dataset.
[10] V K Balakrishnan. 1997. Graph Theory. McGraw-Hill Professional, New York, NY.
[11] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 (2018).
[12] Jovan Blanuša, Paolo Ienne, and Kubilay Atasu. 2022. Scalable Fine-Grained Parallel Cycle Enumeration Algorithms. In Proceedings of the 34th ACM Symposium on Parallelism in Algorithms and Architectures (SPAA). ACM, Philadelphia PA USA, 247–258. https://doi.org/10.1145/3490148.3538585
[13] Jovan Blanuša, Kubilay Atasu, and Paolo Ienne. 2023. Fast Parallel Algorithms for Enumeration of Simple, Temporal, and Hop-constrained Cycles. ACM Trans. Parallel Comput. 10, 3 (Sept. 2023), 1-35. https://doi.org/10.1145/3611642
[14] Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M. Bronstein. 2023. Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting. IEEE Trans. Pattern Anal. Mach. Intell. 45, 1 (Jan. 2023), 657-668. https://doi.org/10.1109/TPAMI.2022.3154319
[15] Mário Cardoso, Pedro Saleiro, and Pedro Bizarro. 2022. LaundroGraph: Self-Supervised Graph Representation Learning for Anti-Money Laundering. In Proceedings of the Third ACM International Conference on AI in Finance. 130-138.
[16] Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD '16). ACM, New York, NY, USA, 785-794. https://doi.org/10.1145/2939672.2939785
[65] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018).
[66] Tianxiang Zhao, Xiang Zhang, and Suhang Wang. 2021. Graphsmote: Imbalanced node classification on graphs with graph neural networks. In Proceedings of the 14th ACM international conference on web search and data mining. 833-841.
A Realism of Data Generation 16
B Ethical Use of the Data 18
C Hyperparameter Tuning of Machine Learning Models 18
D Graph Feature Preprocessor Configuration 19
E Additional GNN Experiments 20
F Additional GBT Experiments 20
G Datasheet 23
G.1 Motivation 23
G.2 Composition 23
G.3 Collection Process, Uses, Distribution and Maintenance 24
A Realism of Data Generation
As described in Section 3.5, we used AMLworld to create several synthetic AML datasets. The datasets are published on Kaggle [50] under a Community Data License Agreement. As noted previously, data is divided into two high-level groups: HI and LI. The HI datasets have slightly higher illicit ratios (more laundering) than LI. Both HI and LI are further divided into small, medium, and large datasets, with large having  $175M - 180M$  transactions. Table 4 provides a number of basic statistics about all six Kaggle datasets. We include more detailed analyses below.
Table 4: Public Synthetic Data Statistics.  $\mathbf{{HI}} =$  Higher Illicit (more laundering).  $\mathbf{{LI}} =$  Lower Illicit.
Small	Medium	Large
Statistic	HI	LI	HI	LI	HI	LI
# of Days Spanned	10	10	16	16	97	97
# of Bank Accounts	515K	705K	2077K	2028K	2116K	2064K
# of Transactions	5M	7M	32M	31M	180M	176M
# of Laundering Transactions	3.6K	4.0K	35K	16K	223K	100K
Laundering Rate (1 per N Trans)	981	1942	905	1948	807	1750
Figure 7 histograms annualized transactions per account. It gives a micro view of how different entities drive the overall transaction counts found in Table 4. Figure 7 also provides a touch point to real data: numbers roughly align with U.S. Federal Reserve data [51]. Table 5 shows the distribution of transaction formats used in the dataset, such as ACH, wire, and cheque. Again numbers correspond roughly with Federal Reserve statistics [41].
Table 8 breaks down laundering transactions between (a) transactions following one of 8 standard patterns discussed in Section 3.2 – in particular Figure 2; and (b) the integration transactions disguised as other activities (e.g. employee payroll or company supplies).
As with other data, we base salary and pension amounts on real data, in this case from the US Internal Revenue Service [54]. Figure 8 shows the number of returns filed for the 2018 tax year for each amount of salary income and pension income. Having proper distribution of salaries and pensions in turn helps drive accurate modeling and statistics for transaction sizes and frequency, as just discussed. The $0 top bin in Figure 8 reflects the fact that some people have no salary income or no pension income. Beyond salary and pension people can have income from other sources such as interest or dividends.
Table 5: Distribution of transaction counts by format in LI-Large.
Format	# Transactions	Format	# Transactions
Cheque	69,720,485	Reinvestment	7,258,251
Credit Card	49,923,366	Wire	6,346,402
ACH	21,650,558	Bitcoin	3,601,817
Cash	18,069,965
Table 6: Histogram of # of nodes (accounts) in LI-Large laundering patterns. Gather Scatter has 2 counts: (a) # of nodes from which initial funds come; (b) # of nodes to which funds ultimately go.
# of Nodes	Fan-out	Fan-in	Cycle	Random	Bipartite	Stack	Scatter-Gather	Gather-Scatter
Min	Max	(1)	(2)
1	2	50	47	72	70	72	87	54	67	67
2	4	46	54	50	44	30	31	47	43	38
4	8	53	54	84	79	62	56	64	61	55
8	12	57	62	76	62	51	48	48	42	52
12	18	71	62	16	23	62	37	63	71	72
18	∞	0	0	0	0	0	0	0	0	0
Table 7: Occurrence count for each laundering pattern in LI-Large.
Pattern	Pattern Count	# Trans in Pattern	Pattern	Pattern Count	# Trans in Pattern
Fan-out	277	2,014	Stack	259	3,239
Fan-in	279	2,003	Random	278	1,831
Cycle	298	2,326	Scatter-gather	276	4,202
Bipartite	277	1,858	Gather-scatter	284	4,010
Table 8: Laundering rates in LI-Large. Ratio is the total transaction count divided by the laundering transaction count.
Description	# of Trans	Ratio
Laundering Trans. - Patterns	19,461	9,047
Laundering Trans. - Other	81,143	2,170
Laundering Trans. - Total	100,604	1,750
Figure 8 indicates that there are about  $3.4 \times$  as many returns showing salary income as pension income, although returns can show both<sup>3</sup>. We assume about  $62.5\%$  of people in our models have salaries – matching the value from the U.S. Department of Labor for labor force participation of the adult workforce [42]. Following the IRS ratios, about  $18.3\%$  of the population in our datasets has pension income, with around half of those pensioners also receiving a salary.
Figure 7: Annualized transaction rate across accounts in LI-Large.
3For income tax purposes in the U.S., salary income also includes total wages for hourly workers.
Figure 8: Distribution of U.S. tax returns for 2018 by salary income and pension income.
Table 9: Successive halving parameter configurations used for hyperparameter tuning of shared models trained using all banks of a dataset (Multi-bank models) and private models trained using the data of a single bank (Single-bank models).
Multi-bank models	Single-bank models
Datasets	Small	Medium	Large	Medium	Large
x0	1000	100	16	1000	100
η	2	2	2	1.5	2
r0	0.1	0.1	0.2	0.3	0.1
Additional statistics about our virtual world were provided in Section 3.5.
B Ethical Use of the Data
We addressed a number of ethics issues in Section 7. We offer a few additional observations here.