# Representation learning with a transformer by contrastive learning for money laundering detection

Harold Guéneau, Alain Celisse, Pascal Delange

<figure><img src="image_1.png" /></figure>

## To cite this version:

Harold Guéneau, Alain Celisse, Pascal Delange. Representation learning with a transformer by contrastive learning for money laundering detection. 2025. hal-05140202

HAL Id: hal-05140202

https://hal.science/hal-05140202v1

Preprint submitted on 4 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Representation learning with a transformer
by contrastive learning
for money laundering detection

Harold Guéneau *¹, ²*, Alain Celisse¹, and Pascal Delange³

¹Laboratoire SAMM, Université Paris 1 Panthéon-Sorbonne, Paris, France

²Shine

³Marble

July 4, 2025

Abstract

The present work tackles the money laundering detection problem. A new procedure is intro-
duced which exploits structured time series of both qualitative and quantitative data by means
of a transformer neural network. The first step of this procedure aims at learning representations
of time series through contrastive learning (without any labels). The second step leverages these
representations to generate a money laundering scoring of all observations. A two-thresholds ap-
proach is then introduced, which ensures a controlled false-positive rate by means of the Benjamini-
Hochberg (BH) procedure. Experiments confirm that the transformer is able to produce general
representations that succeed in exploiting money laundering patterns with minimal supervision
from domain experts. It also illustrates the higher ability of the new procedure for detecting non-
fraudsters as well as fraudsters, while keeping the false positive rate under control. This greatly
 contrasts with rule-based procedures or the ones based on LSTM architectures.

# 1 Introduction

Money laundering represents a significant global concern with an annual estimated amount from 2 to 5% of the world GDP [LP21]. It strongly impacts the ability of governments to collect taxes and fight crime, damaging financial institutions' stability and affecting the economic development. Therefore, addressing money laundering has become an important challenge for political and financial institutions to prevent these substantial financial losses. The United Nations (UNODC) defines money laundering as the processing of criminal proceeds to disguise their illegal origin [Nat], making its detection particularly challenging. For instance perpetrators mimic legal financial behaviors by potentially keeping accounts dormant for years with an licit activity before using them fraudulently. Another difficulty results from frequent changes of money laundering patterns along the time as public policies and other environmental factors make new fraud patterns more lucrative [LVF24, SBG23]. This evolving nature of money laundering makes their detection even more challenging and constantly requires new detection strategies since traditional ones can quickly become obsolete.

Financial institutions operate under strict regulatory requirements for money laundering detection and suspicious activity reporting. A non-compliance can result in a revocation of the financial license or large fines for any customer that would be proved as negligent. This falls under the catch-all concept of “anti-money laundering” framework (AML) [Eur24]. Financial institutions therefore employ dedicated teams of financial analysts in order to comply with these regulations and to avoid financial losses. However due to the high volume of customers and transactions in modern financial systems, these investigations cannot be entirely hand-made. In order to let the financial analysts handle only the last steps of investigation, financial institutions have implemented so-called automated ”rule-based

<footnote>*harold.gueneau@proton.me</footnote>
systems” [HZSB16, BLS20]. These are typically based on expert knowledge of past fraud patterns. For instance the trigger event can be a fraudulent payment, a specific customer activity, a periodic review of all accounts. Rule-based systems provide explicit audits and have some explanatory capabilities, which is an asset in a regulatory context (banks are expected to provide explanations about each suspicious case, which can help the analysts during the investigation). While rule-based systems provide a framework with inherent explainability, their efficiency suffers severe limitations in money laundering detection. Such rule-based detections often exhibit a very low precision with from 95% to 98% of false positives [LP21]. Such a deficiency of rule-based systems results from their high sensitive to hard-coded thresholds, which are difficult to update in real time. Frequent modifications of these rules would be necessary as fraud patterns evolve, which turns out to be difficult due to the inertia of large financial institutions.

By contrast, machine learning approaches [KPSA21] can address a lot of ongoing challenges, and complement existing rule-based frameworks, for instance by reducing the dependency on such scenarios or by uncovering new patterns associated with money laundering. The most straightforward approach to design such a machine learning approach is to consider tabular data in input made of macroscopic features describing the customer and its associated activity (transaction aggregates, or graph descriptors for example) [EBA+22, KPSA21, SNY+10, TSOD20].

However a lot of information is lost when designing these aggregated features to reshape the available knowledge into tabular data. Moreover, designing meaningful aggregates also requires some expert knowledge, while being limited in the detection of new patterns. By contrast, the present work addresses this question by considering the whole set of raw transactions (not only aggregates or other summary metrics) collected along the time ([TPD<sup>+</sup>23] for example). To be more specific, a transformer is trained from raw transaction time series in the same spirit as what is made in HAMLET [TPD<sup>+</sup>23], with some differences. However only one transformer is exploited in the present context for encoding the whole time series, unlike HAMLET which rather relies on two transformers (one for encoding the transactions and the other for the time series). Transformer neural networks were initially developed in the context of Natural Language Processing. Since language can be interpreted as a "time series of words", a transformer can capture temporal dependencies across more general types of time series [RPC<sup>+</sup>22, DTGMR<sup>+</sup>24, KG22, WZZ<sup>+</sup>23]. In the present money laundering context, the transformer is learning representations from raw transaction streams, where each event is composed of both quantitative and categorical attributes. This creates a far larger "vocabulary" of possible feature combinations that represents diverse types of financial events.

Another important difference between Hamlet ([TPD<sup>+</sup>23]) and the present work owes to the use of contrastive learning as a means for (pre-)training the transformer without any labeled data, whereas HAMLET does exploit the available labels along the training process. Indeed beyond input data structure, accurately detecting money laundering is challenging because of the lack of correctly labeled data. Fraudulent customers remain hard to identify, with incomplete descriptors, and the expected overall proportion of fraudulent transactions is (fortunately) low compared to legal transactions, which makes the learning problem highly imbalanced [KPSA21]. Contrastive learning ([HWZ<sup>+</sup>24]) and more generally self-supervised learning ([GCZ<sup>+</sup>24]) have proven to be of great help in such situations where truly labeled data are not available. For instance *Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels* [ZBM<sup>+</sup>22] demonstrates the advantage in considering a pre-training step in presence of noisy labels. Moreover, Self-Supervised Learning (SSL) methodologies (including contrastive learning) maintain performance under severe class imbalance [LHGM22]. Contrastive learning has been already used with time series. For instance, Contrastive Predictive Coding [vdOLV19] has been successfully applied to a variety of time series. However capturing the temporal relationships in the context of financial time series is of different nature compared to classical use-cases where Contrastive Predictive Coding has been used up to now (more similar to NLP problems).

The main contributions of the present work are three-fold: As illustrated by Figure 1, a first contribution consists in feeding a transformer architecture with raw time series of transactions instead of aggregates. This reduces the usual reliance on expert knowledge for designing such aggregates summarizing the valuable information. One specificity of the present context is that each transaction is described by a vector of both qualitative and quantitative descriptors, which makes the time series a complex structured object. A second contribution of the present work stems from the use of contrastive learning for (pre-)training the transformer without labeled data. By contrast with the supervised contrastive learning detailed in [KTW+21], contrastive learning is performed with no label.
<figure><img src="image_3.png" /><figcaption>Figure 1: Summary of the proposed process</figcaption></figure>

It rather leverages a similarity criterion which allows for deciding if two observations are similar or not. This refines the selection of positive and negative examples with more nuances, potentially capturing relationships that might be missed if relying only on class labels. Our third main contribution to the specific field of money laundering detection relies on introducing a two-threshold classification procedure (see Figure 1) for distinguishing non-fraudulent accounts from fraudulent ones. Unlike most of ongoing rule-based strategies, these two thresholds are calibrated by means of the Benjamini-Hochberg multiple testing procedure, which allows for controlling the False Discovery rate (FDR) criterion and therefore avoids too many false positives. All of this results in a considerably reduced amount of false negatives compared to state-of-the-art rule-based strategies.

The remaining part of the paper is structured as follows. Section 2 details the specificities of the money laundering detection, and also describes how to learn the representation of our complex time series. The contrastive learning step (for the transformer pre-training) is exposed in Section 2.3, while the sampling of positive and negative examples is explained along Section 2.4. The application of the representation learning strategy to the money laundering problem is then explained in Section 3. The two-thresholds strategy used for detecting fraudulent accounts and non-fraudulent ones is exposed in Section 3.2, while the calibration of these two thresholds by means of the BH procedure is detailed in Section 3.3. Section 4 finally illustrates the general behavior of the full procedure on simulation experiments carried out from a (subset of a) real-life anonymized dataset.

# 2 Representation learning of financial time series

The present section describes the contrastive pre-training step, which corresponds to the blue box (first step) of Figure 1. The goal of this step to produce representations of time series of financial events (qualitative and quantitative descriptors). Such a representation embeds the financial time series into a finite-dimensional vector also named as an *embedding*. Sections 2.1 and 2.2 introduce the structure of the dataset and the main elements of the pre-training step. Then Section 2.3 details the contrastive learning procedure used to train the transformer neural networks (seen as an encoder). As a key ingredient of the contrastive learning strategy, the sampling of negative and positive examples is detailed along Section 2.4. Finally, Section 2.5 discusses related works in representation learning.

## 2.1 Data description and notation

Let us consider a dataset $\mathcal{D} = \{x_0, x_1, x_2, \dots, x_{N-1}\}$ of $N$ observations. For each $1 \le i \le N$, it is important to figure out that $x_i \in \mathbb{R}^{T \times d_{input}}$ denotes a time series of length $T$. Each time series
$x_i = (x_{i,1}, \dots, x_{i,T})$ is made of $T$ vectors of dimension $d_{input}$ describing $T$ successive financial events (transactions). Each vector $x_{i,t}$ has both quantitative and qualitative features describing the event (see Figure 2).

<figure><img src="image_4.png" /><figcaption>Figure 2: Illustration of a time series in the dataset $\mathcal{D}$</figcaption></figure>

These features can be the direction of money transfer, the type of transaction (e.g. card payment or bank transfer), the amount of transferred money, or the date for instance. Let us mention that qualitative data can be encoded through a categorical encoder such as the One-hot encoding [ZQF24] or the target encoding [Mic01] for example.

For motivating the use of a transformer [VSP+17] to encode such a time series of events, let us mention that each of these financial events is very similar to a word within a sentence (with its specific ordering). Therefore similarly to what is usually made in NLP [RG19], a time series of such events can be embedded using a transformer deep neural network that will learn the financial semantic without using any label.

In addition to this dataset $\mathcal{D}$, each individual also comes with complementary descriptors so that one can define a new set $\mathcal{D}' = \{x'_0, x'_1, x'_2, \dots, x'_{N-1}\}$, where every $x'_i \in \mathbb{R}^{d_{comp}}$ is a vector of $d_{comp}$ numeric descriptors associated with the time series $x_i$.

<figure><img src="image_5.png" /><figcaption>Figure 3: Illustration of the dataset $\mathcal{D}'$</figcaption></figure>

As illustrated by Figure 3, these features can be both categorical and numerical. The categorical ones include characteristics such as the legal form of a customer or the industry sector classification. The numerical features are mainly simple aggregated metrics derived from the time series in $\mathcal{D}$, it can be the mean or the sum of the averaged payout value over a period of time. Let us emphasize that the vectors within $\mathcal{D}'$ lack expressiveness to achieve optimal performance across a broad range of setups particularly in the context of money laundering detection. This claim is supported by the simulation experiments results described in Section 4.4 where tabular data (aggregates) cannot achieve the same statistical performance as the raw time series. However, the information in $\mathcal{D}'$ can still be leveraged to design a similarity criterion and therefore decide which observations are similar, and which are not. This similarity criterion is a corner stone of the (unsupervised) contrastive learning strategy (see Section 2.3).

In the money laundering detection context, the data are classically highly imbalanced since most of accounts are non-fraudulent ones, as described in Section 4.1 where at most 5% of the test set data are fraudsters. The proposed two-steps methodology as well as some aspects of the pre-training made by contrastive learning help in mitigating the strong imbalance in the data.
## 2.2 General pipeline for representation learning: Pre-training

The general pipeline for representation learning relies on several blocs that are combined as illustrated by Figure 4, which summarizes the whole pre-training process.

**Transformer bloc** The dataset $\mathcal{D}$ is used as an input of the transformer neural network (orange bloc). The main merit of the transformer is to output an embedding matrix, which encodes the initial dataset $\mathcal{D}$ as $N$ row-vectors in a latent space of finite dimension. The transformer neural network architecture [VSP$^{+}$17] is a powerful tool for processing time series [WZZ$^{+}$23]. It has proven great efficiency in a variety of fields, particularly in NLP, but also in biology [DTGMR$^{+}$24] or in finance [KG22].

<figure><img src="image_6.png" /><figcaption>Figure 4: Contrastive pre-training of the transformer encoder</figcaption></figure>

**Contrastive learning bloc** The embedding output by the transformer is then sent in input of the contrastive learning step (blue bloc), the goal of which is the transformer pre-training. Contrastive learning is a robust and flexible methodology which leverages the auxiliary dataset $\mathcal{D}'$ and enables us overcoming the lack of truly labeled data. The auxiliary dataset $\mathcal{D}'$ serves for designing a similarity measure used along the contrastive learning process following the principle: "similar individuals should have close representations". This is achieved by optimizing a loss with respect to the weights of both the encoder (transformer) and the so-called *projection head*, which is a Multi-Layer perceptron (MLP) Deep Neural Network (DNN) with the output layer of low dimension (for reducing the computational costs) [CKNH20].

Let us emphasize that a so-called *memory bank* [HFW$^{+}$20] has been also introduced for saving the computational resources. As illustrated by the bottom-right bloc within Figure 4, the tabular data from $\mathcal{D}'$ are stored in a "memory bank" with a limited size, which is a means for only storing relevant observations (therefore limiting the memory costs).

In the following, Algorithm 1 provides a high-level description (in pseudo-code) of the main steps of the contrastive learning process that will be further explained along the next sections. The input of the contrastive learning are the weights of the transformer (encoder) denoted by the function $f_\theta$, and the ones of the projection head (dimension reduction) denoted by $h_\gamma$. The contrastive learning aims at optimizing all these weights simultaneously. The loss function used for this optimization is denoted by $\mathcal{L}_{\text{InfoNCE}}$ [vdOLV19].
**Algorithm 1** Pseudo code to illustrate contrastive pre-training of the transformer encoder

Input: $\mathcal{D}$, $\mathcal{D}'$, number of epochs e, number of batches b

2: Initialization: $f_{\theta}(\cdot)$ (Transformer Encoder), $h_{\gamma}(\cdot)$ (Projection head), $\mathcal{L}_{\text{InfoNCE}}$ (Loss), $\mathcal{M} = \emptyset$ (Memory bank)

for $i \in [1, e] \cap \mathbb{N}$ do

4: Define a collection of b batches of observations from $\mathcal{D}$:

$$ \mathcal{B} = \{B_1, \dots, B_b\} $$

for $j \in [1, b] \cap \mathbb{N}$ do

6: for $x_{ref} \in B_j$ do
Compute

$$ h_{\gamma}(f_{\theta}(x_{ref})) = h_{\gamma}(u_{ref}) = z_{ref} $$

→ Section 2.3

8: Update $\mathcal{M}$:

$$ \mathcal{M} \leftarrow \mathcal{M} \cup \{z_{ref}\} $$

→ Section 2.4

Sample one positive example and $K-1$ negative ones from $\mathcal{M}$:

$$ \mathcal{X} = \{z_+\} \cup \{z_-^1, z_-^2, \dots, z_-^{K-1}\} $$

10: Sample noisy "copies" of examples from $\mathcal{X}$ ($\epsilon_+$, $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$):

$$ \mathcal{X}^{\epsilon} = \{z_{+} + \epsilon_{+}\} \cup \{z_{-}^{1} + \epsilon_{1}, \dots, z_{-}^{K-1} + \epsilon_{K-1}\} $$

→ Section 2.4

Compute the loss $\mathcal{L}_{\text{InfoNCE}}(z_{ref}, \mathcal{X}^{\epsilon})$

→ Section 2.3

12: end for

Optimize $f_{\theta}(\cdot)$ and $h_{\gamma}(\cdot)$ (minimizing $\mathcal{L}_{\text{InfoNCE}}$) with stochastic gradient descent

14: end for
end for

16: Output $f_{\theta}(\cdot)$

At each step of the process, a reference observations $x_{ref}$ is randomly chosen. The encoder function is composed with that of the projection head, which outputs a low-dimensional representation $z_{ref}$ that is sent to the memory bank.

The next step at the core of the contrastive learning process consists in a sampling step: Based on a similarity measure, a similar observation to reference one is chosen $z_+$ (called positive example) as well as dissimilar ones (called the negative examples) denoted $\{z_-^i\}_{1 \le i \le K}$. As already suggested, the weights of the transformer combined with the projection head are optimized for maximizing the closeness between the representations of the reference observation and that of the positive examples while minimizing the distance between the representations of the reference example and the ones of the negative examples.

## 2.3 Transformer and contrastive learning

In order to produce the required representation of complex time series, it is necessary to consider an encoder function $f_{\theta}$ (the transformer here) which maps the input (time series) $x_i \in \mathbb{R}^{T \times d_{input}}$, for $i \in [0, N-1]$ onto a finite-dimensional output vector $u_i$ (called the representation or the embedding) in a latent space $\mathcal{U}$ of dimension $d_{latent}$. Along the contrastive learning (pre-training process), this representation in the latent space is "projected" onto a low-dimensional vector $z_i$ of dimension $d$ by means of the projection head denoted by $h_{\gamma}$, defined from $\mathcal{U}$ to the $d$-dimensional space $\mathcal{Z}$, with $d \ll d_{latent}$ [GAHG22].

These two important steps are introduced and detailed in what follows.
**Transformer** The transformer architecture [VSP<sup>+</sup>17] excels at capturing long-range dependencies, which can be crucial for highlighting long-range money laundering patterns. Such patterns can be operated over several months to be as discreet as possible. One important feature of the transformer is the *self-attention* mechanism [VSP<sup>+</sup>17] it incorporates. This self-attention mechanism enables to focus on relevant long-range patterns while building the representations, especially with long sequences. Moreover compared to traditional time series encoders such as RNNs [Elm90, RJW89] and LSTMs [SH97], the transformer one has a lower computational complexity and is easier to parallelize (helping in processing large datasets) [VSP<sup>+</sup>17].

It is important to mention that the self-attention mechanism the transformer relies on is invariant by permutations. Without any positional information hard coded in the input observation, the transformer would not suitably account for the temporal structure of time series. Therefore, the transformer architectures require temporal (or positional) information to effectively process sequential data [DSS21]. Unlike the original implementation by [VSP<sup>+</sup>17] in the context of natural language processing (NLP), another specificity of the present work is to handle the sequential structure of transactions by learning the best positional embedding along the training process ([WC20]).

The transformer (illustrated in Figure 5) performance is governed by a large set of hyperparameters such as the number of encoder layers, the latent space dimensionality, and the number of attention heads ([VSP<sup>+</sup>17]).

<figure><img src="image_7.png" /><figcaption>Figure 5: Illustration of the transformer encoder</figcaption></figure>

A few optimization parameters such as the learning rate and the dropout rate impact the convergence stability as well as the generalization performance. All the hyperparameters used along the experiments reported in Section 4 are detailed in Appendix A.2.

**Motivating contrastive learning for money laundering detection** One main idea of the present work is to train the transformer encoder by means of contrastive learning [LKHS20, CKNH20, HFW<sup>+</sup>20]. The contrastive learning is a self-supervised learning (SSL) approach which further exploits (if available) the intrinsic data structure to train a model without exploiting any label. In the present context this is made possible by designing a similarity criterion which allows for learning the semantics in the data [ZBM<sup>+</sup>22]. This similarity can rely on the domain knowledge or on structural properties of the data. This methodology is highly relevant in the present context of money laundering detection, since labeled data are scarce or often unreliable.

**Transformer and Projection head for contrastive learning** The contrastive learning process integrates a projection head after the transformer encoder in order to improve the performance and the generalization power of the encoder [XGN<sup>+</sup>24] [GAvdHG22]. This has been implemented in SimCLR
for example [CKNH20]. Along the contrastive learning process, the transformer encoder $f_\theta$ is composed with the projection head function $h_\gamma$ to output a final projected representation $z$ of an observation $x$. This architecture improves the model generalization using intermediate layers to extract more high-level and general features in $u$, while maintaining a task-specific optimization on another representation $z$, processed further in the neural network. It is related to the logic studied in transfer learning (as described in [KCLS+22] for example), and to the idea of pre-training a large language model before fine-tuning it for a specific task in natural language processing [HR18].

More precisely, the encoding process starts with the transformer encoder $f_\theta(\cdot)$ with parameters $\theta$ (the transformer weights), generating a general-purpose representation $u \in \mathcal{U}$, with dimension $d_{\text{latent}}$ in a latent space. The purpose of this representation is to encode some valuable information that is hidden in the input space, which would improve the generalization performance. This representation $u$ is then mapped into a smaller and specialized space through the projection head $h_\gamma(\cdot)$, with parameters $\gamma$ (the projection head weights). This projection head is classically an MLP neural network [GAHG22] parametrized by the number of hidden layers, the hidden-layers dimensionality, the activation functions, a normalization strategy and a dropout rate. The choice of the hyperparameter values used in our experiments is detailed in Appendix A.2. The projection head map $h_\gamma(\cdot) : \mathcal{U} \to \mathcal{Z}$ produces a specific (projected) representation $z \in \mathcal{Z}$ such that $h_\gamma(u) = z$, with low dimension $d \ll d_{\text{latent}}$. Let us also mention that the projection head reduces the optimization time complexity by reducing the dimension of the general representation $u$ of into a smaller space. The InfoNCE is indeed computed from $z$, which has a smaller dimension than $u$.

**Sampling similar and dissimilar observations** The present work combines time-series observations in $\mathcal{D}$ with tabular information in $\mathcal{D}'$ used to design the similarity criterion (which is a key ingredient in contrastive learning). For a so-called reference observation, the tabular data from $\mathcal{D}'$ are used to select positive and negative examples (related to the reference observation). Indeed, the set $\mathcal{D}'$ contains aggregated metrics and descriptors used to measure similarity between candidate observations and the reference ones. Given a reference observation, the similarity criterion defined on $\mathcal{D}'$ leads to distinguish between observations similar to the reference one (the so-called positive examples) and dissimilar ones (the negative examples).

Let $x_{ref}$ be the reference observation from which an observation must be computed, and let $\mathcal{X}$ denote a set of $K$ example observations, with $x_{ref} \notin \mathcal{X}$, one positive example $x_+$ (expected to be similar to the reference observation $x_{ref}$), and $K-1$ negative observations (expected to be dissimilar to $x_{ref}$). These observations are all processed through an encoder (composition of a transformer $f_\theta$ with a so-called projection head $h_\gamma$) to create an embedding of each observation. Let $z$ be such an embedding of an observation $x$, $z_{ref}$ the one of a reference observation $x_{ref}$, and $z_+$ the one of a positive example $x_+$. Then, one can define the set of positive examples $\mathcal{Z}_+$ and the one of negative examples $\mathcal{Z}_-$ by

$$ 
\begin{aligned}
\mathcal{Z}_+ &= \mathcal{Z}_+(z_{ref}) = \{z \mid z \text{ is a positive example with respect to } z_{ref}\} \\
z_+ &= z_+(z_{ref}) \in \mathcal{Z}_+ \\
\mathcal{Z}_- &= \mathcal{Z}_-(z_{ref}) = \{z \mid z \text{ is a negative example with respect to } z_{ref}\} \\
z_- &= z_-(z_{ref}) \in \mathcal{Z}_- \\
\mathcal{X} &= \{z_+\} \cup \{z_-^1, z_-^2, \dots, z_-^{K-1}\}, \text{ with } z_+ \in \mathcal{Z}_+, \text{ and } \{z_-^1, z_-^2, \dots, z_-^{K-1}\} \in \mathcal{Z}_-
\end{aligned}
$$

Notice that the notation $z_+ = z_+(z_{ref}e)$ for instance reflects the dependence of positive examples with respect to the reference observation that has been previously selected. The same remark also holds for negative examples, which therefore implies the same dependence for the set $\mathcal{X}$. Let us recall that this set is used along the minimization of the InfoNCE loss.

**Increasing the diversity: Data augmentation by adding a Gaussian noise** According to Algorithm 1, each positive and negative example is perturbed by adding a Gaussian noise $N(0, \sigma^2)$, with $\sigma^2 > 0$. This mechanism helps in creating "new observations" from the original ones by Gaussian perturbation [DT17], which yields realistic observations provided $\sigma^2$ remains small enough compared to the local structure of the data. In our experiments from Section 4, the Gaussian noise variance $\sigma^2$ is a hyperparameter the value of which is calibrated through a grid search strategy (see Appendix A.2). Let us also emphasize that this data augmentation mechanism is applied at the level of the projection space of dimension $d$ that is, on finite-dimensional vectors $z_i = h_\gamma \circ f_\theta(x_i)$, for $1 \le i \le N$. This contrasts
with data augmentation strategies classically involved in contrastive learning where the observations are perturbed at the level of the original space (here the time series). Actually there already exist meaningful strategies for perturbing images in the original space [CKNH20]. However to the best of our knowledge, practical analogues for time series do not exist yet, one possible explanation being the difficulty of preserving complex dependence structures along the time.

**The InfoNCE loss** The pre-training of the transformer (made by contrastive learning) relies on the optimization of the InfoNCE loss [vdOLV19]. This loss relies on a similarity measure $\rho(\cdot, \cdot) : \mathcal{Z} \times \mathcal{Z} \to \mathbb{R}_+$. The choice made here for $\rho$ is the classical cosine similarity [Sin01] given by

$$ \rho(z, z') = \frac{\langle z, z' \rangle}{\|z\| \cdot \|z'\|}, $$

where $\langle \cdot, \cdot \rangle$ (respectively $\| \cdot \|$) denotes the Euclidean scalar product (resp. the Euclidean norm) in $\mathcal{Z}$.

For a selected reference observation $x_{ref}$ (mapped through the projection head composed with the encoder) onto $z_{ref} = h_{\gamma} \circ f_{\theta}(x_{ref})$, positive and negative examples are randomly chosen to build the set $\mathcal{X}$ of cardinality $K$ as explained above. The InfoNCE loss is then given by

$$ \mathcal{L}_{\text{InfoNCE}}(\eta) = \mathbb{E} \left[ -\log \left( \frac{\exp(\rho(z_+, z_{ref})/\mathcal{T})}{\sum_{z \in \mathcal{X}, z \neq z_+} \exp(\rho(z, z_{ref})/\mathcal{T})} \right) \right], \quad (1) $$

where $\mathcal{T} > 0$ denotes the so-called temperature parameter, and the expectation is computed over all possible choices of reference observation and related positive and negative examples. Let us also emphasize that $\eta = (\theta, \gamma)$ is a parameter concatenating both $\theta$ and $\gamma$. The dependence of the InfoNCE loss with respect to $\eta$ reflects the fact that the reference vector $z_{ref}$ as well as all positive and negative examples within $\mathcal{X}$ depend on the weights of the transformer (denoted by $\theta$) and the projection head (denoted by $\gamma$). Therefore modifying any of these weights would result in a new final projection of these observations and then would vary the InfoNCE value. For a reference observation, the weights $\theta$ of the transformer $f_{\theta}$ and $\gamma$ of the projection head $h_{\gamma}$ are optimized so that the computed representation of positive examples remains close to that of the reference observation, whereas the representation of negative examples is far away from that of the reference observation at the level of the projected space $\mathcal{Z}$ of dimension $d$.

The InfoNCE can be interpreted as a softmax on the similarity between the (projected) reference observation and the (projected) examples. The temperature $\mathcal{T}$ is an essential hyperparameter that drives the minimization. It aims at scaling the output of the softmax. Lowering the value of $\mathcal{T}$ results in requiring closer (in terms of similarity) representations for similar observations. Otherwise increasing the temperature allows for spreading the projected representation of similar observations. In the experiments reported in Section 4, the temperature value is calibrated via a grid search strategy as explained in Appendix A.2.

## 2.4 Positive and negative examples selection

As explained along previous Section 2.3, the contrastive learning uses randomly selected observations as references and for each of them, then samples one positive example and $K-1$ negative examples relatively to this reference observation.

The key ingredient behind this process is a similarity criterion defined from the complementary descriptors that have been gathered in the dataset $\mathcal{D}'$. In our context $\mathcal{D}'$ contains a lot of information about the considered accounts: the average payout value over a period of time, and categorical features such as the legal form of the customer or for example the industry sector it belongs to. Therefore an observation $x_i$ is described within $\mathcal{D}'$ in terms of descriptors (aggregates) collected within a vector $x'_i \in \mathbb{R}^{d'}$. For computing how similar two time series $x_i$ and $x_j$ are, the present approach introduces a similarity between vectors $x'_i$ and $x'_j$. In a nutshell, the respective representations of $x_i$ and $x_j$ must become closer as $x'_i$ and $x'_j$ are more similar to each other.

One idea presented in this work is therefore to sample positive and negative examples in $\mathcal{D}$, based on the similarity measured between elements in $\mathcal{D}'$. From an implementation perspective, a memory bank manages the sampling of these examples by storing the representations produced by the encoder $h_{\gamma} \circ f_{\theta}(\cdot)$, providing several key advantages as described below.
**Memory bank** This work implements a memory bank $\mathcal{M}$ that stores the embeddings produced during the training procedure. The memory bank is a tool already used in contrastive learning (see MoCo [HFW+20] for instance).

Let us define $\mathcal{M}_b$ the memory bank at the iteration $b$ (i.e. after $b$ batches using the notations of Algorithm 1). After first initializing the memory bank with $\mathcal{M}_0 = \emptyset$, the memory bank is then updated as follows from each new batch of (projected) representations $\{z_1, \dots, z_{\mathbf{b}_{size}}\}$:

$$ \left\{ \begin{array}{ll} \mathcal{M}_{t+1} = \mathcal{M}_t \cup \{z_1, \dots, z_{\mathbf{b}_{size}}\}, & \text{if } |\mathcal{M}_t| + \mathbf{b}_{size} < M \\ \mathcal{M}_{t+1} = (\mathcal{M}_t \setminus \mathcal{O}_t) \cup \{z_1, \dots, z_{\mathbf{b}_{size}}\}, & \text{otherwise,} \end{array} \right. \qquad (2) $$

where $\mathcal{O}_t$ denotes the set of the $|\mathcal{M}_t| + \mathbf{b}_{size} - M$ oldest elements within $\mathcal{M}_t$, with $M$ denoting the maximum cardinality of the memory bank, and $\mathbf{b}_{size}$ being the current batch cardinality. In other words, the memory bank is iteratively updated by including the most recent elements, which results in removing the oldest elements from the memory bank each time the maximum cardinality $M$ has been achieved for the memory bank.

A memory bank also improves upon the selection of positive and negative examples by allowing more diversity in the sampling. Indeed, it stores representations over several batches ($\mathbf{b}_{size} < M$), which helps for reducing the dependency on a specific batch (especially when the batch size is small). Furthermore along the contrastive learning process, the memory bank stores the (projected) representations produced by different versions of the encoder combined with the projection head (the weights of which evolve along the iterations). As a matter of fact, this also helps stabilizing the training. On the one hand, it introduces a temporal ensembling [LA17] by forcing the encoder to learn representations that remain consistent over the time (similarly to a momentum). On the other hand, the outdated embeddings produced by previous versions act as regularizers, adding some noise and then preventing from some overfitting.

**Positive example sampling** For each reference observation $x_{ref} \in \mathcal{D}$ sampled from a batch, one positive example $x_+$ (with the corresponding representation $z_+ \in \mathcal{Z}_+$ associated) is required in order to compute the InfoNCE loss $\mathcal{L}_{InfoNCE}(\eta)$ (see Eq. 1). This positive example has to be highly similar to the reference observation.

The memory bank $\mathcal{M}$ stores all the embeddings already processed among which one can choose a positive example. The present proposal builds positive examples by randomly sampling an observation from the top $\kappa$-closest neighbors in $\mathcal{D}'$ ($\kappa \in \mathbb{N}^*$) to the reference observation, where the $\kappa$ neighbors have been ranked according to the similarity criterion over $\mathcal{D}'$. In the present context, the similarity is the Euclidean norm in $\mathcal{D}'$. $\kappa$ is a hyperparameter and needs to be calibrated for the considered dataset. In the experiments, $\kappa$ is set to 50 (the 50 closest companies with respect to the reference observation, as explained in Appendix A.2). The implementation relies on the approximate nearest-neighbor search [AMN+98] for saving the time and memory resources.

All of this is illustrated by Algorithm 2 in pseudo-code, which outputs an instance of a positive example given a memory bank $\mathcal{M}$ and a reference observation $x_{ref}$ (and its counterpart $x'_{ref}$ within $\mathcal{D}'$).

<caption>**Algorithm 2** Pseudo-code for positive example sampling</caption>

**Input:** Memory bank $\mathcal{M}$, dataset $\mathcal{D}'$, reference observation $x'_{ref} \in \mathcal{D}'$, Number of neighbors $\kappa > 0$
2: Compute the distance $r_\kappa$ between $x'_{ref}$ and its $\kappa$th neighbor in $\mathcal{D}' \cap \mathcal{M}$
Set of $\kappa$-closest neighbors in $\mathcal{M}$:
$$ \mathcal{V}_\kappa(x'_{ref}) = \{1 \le i \le N \mid z_i \in \mathcal{M}, \text{ and } \|x'_i - x'_{ref}\| \le r_\kappa\} $$
4: Sample uniformly at random one index $i_+$ among $\mathcal{V}_\kappa(x'_{ref})$
Positive example: $z_+ \leftarrow z_{i_+} \in \mathcal{M}$
6: **Output:** $z_+$

The algorithm starts by computing the distance (Euclidean norm in $\mathcal{D}'$) between $x'_{ref}$ and its $\kappa$th neighbor in the memory bank. Let us mention that there is a slight abuse of notation here since the
elements of the memory bank are the (projected) representations of the $x_i$ in $\mathcal{Z}$. Then $\mathcal{D}'$ cannot formally intersect $\mathcal{M}$. Of course when writing this, one identify any vector $x'_j$ with its representation $z_j$ in $\mathcal{Z}$ since the underlying observation $j$ is the same. As long as the distance $r_\kappa > 0$ to the kappath neighbor is known, then the set $\mathcal{V}_\kappa(x'_{ref})$ of the $\kappa$-closest neighbors can be (uniquely) computed (if one moreover assumes that there is no tie). The final positive example is nothing bu any of the elements within $\mathcal{V}_\kappa(x'_{ref})$ that has been chosen uniformly at random. Let us emphasize that any alternative strategy for choosing the positive example would be possible. For instance any importance sampling strategy taking into account the distance to $x'_{ref}$ would make sense as well, although this is not the scheme that has been considered here.

**Negative example sampling** Similarly, for each reference observation $x_{ref} \in \mathcal{D}$, a set $\mathcal{Z}_- \subset \mathcal{Z}$ of negative examples $z_-$ is designed (by considering the (projected) representations of the original observations $x$).

One known issue with negative examples is the risk of sampling among observations that are similar to the reference one [HKW$^{+22}$. Committing such a mistake would mean that the optimization step would push away representations of similar observations, which is precisely what one tries to avoid.

An important contribution of the present work consists in a new strategy for selecting the negative examples based on a preliminary clustering of the observations within $\mathcal{D}'$. Indeed as long as $K \ge 1$ clusters have been computed from the similarity criterion over $\mathcal{D}'$, then $x'_{ref}$ does belong to a unique cluster. Therefore choosing any negative example among the observations belonging to the other $K - 1$ remaining clusters is a means for avoiding the selection of observations that are too similar to the reference one. The present proposal suggests to define the clustering by means of the Lloyd's algorithm for solving the K-means problem [IEA$^{+23}$. Here the number of clusters is chosen from the data by maximizing the silhouette score [Rou87].

In what follows, Algorithm 3 details the main steps of the strategy introduced above. From a clustering algorithm $\Omega_K(\cdot) : \mathcal{D}' \to \{1, \dots, K\}$ that outputs the cluster label $\ell(x')$ of an observation $x' \in \mathcal{D}'$, Algorithm 3 starts by computing the label $\ell_{ref} = \ell(x'_{ref})$ of the reference observation $x'_{ref}$.

**Algorithm 3 Pseudo-code for negative example sampling**

**Input:** Memory bank $\mathcal{M}$, dataset $\mathcal{D}'$, reference observation $x'_{ref} \in \mathcal{D}'$, number of clusters $K \ge 1$, clustering algorithm $\Omega_K(\cdot)$ on $\mathcal{D}'$.

$$2: \ell_{ref} = \ell(x'_{ref}) = \Omega_K(x'_{ref})$$

Compute the indices of candidate negative observations:

$$\mathcal{I}_- = \{1 \le i \le N \mid \Omega_K(x'_i) \ne \ell_{ref}, \text{ and } z_i \in \mathcal{M}\}$$

4: Sample randomly without replacement $K - 1$ elements among $\mathcal{I}_-$:

$$\mathcal{S}_- = \{i_1, \dots, i_{K-1}\}$$

Define the $K - 1$ negative examples:

$$\mathcal{Z}_- = \{z_1^-, z_2^-, \dots, z_{K-1}^-\} = \{z_j \in \mathcal{M} \mid j \in \mathcal{S}_-\}$$

6: **Output:** $\mathcal{Z}_-$

All indices of the observations that are not assigned to cluster $\ell_{ref}$ are collected within the set of candidate negative observations $\mathcal{I}_-$. After randomly sampling $K - 1$ distinct indices from $\mathcal{I}_-$, the set of negative examples is computed. Any (refined) sampling scheme could be exchanged with the uniform without replacement one that is used along Algorithm 3. For instance, keeping track of the distance of the candidate negative observations to $x'_{ref}$ could be explored as well.

**Gaussian noise injection** Coming back to Algorithm 1, the sampling of positive and negative examples is modified by adding some Gaussian noise, which results in substituting $\mathcal{X}$ with $\mathcal{X}^\epsilon$.
Adding such a Gaussian noise can be motivated by the need for exploring new situations (not covered within the dataset). Indeed covering all possible transaction events and patterns is not possible with a limited amount of data, which would limit the learning potential of the transformer. This challenge is particularly relevant in the money laundering context where the labels are highly imbalanced with fraudulent patterns that are severely underrepresented. It makes the task of learning the fraudsters' distribution a very challenging one. In this respect, adding some Gaussian noise to the positive and negative examples allows for exploring new patterns.

Let us also mention that one specificity of the present proposal is to perturb the observations at the level of latent (projection) space $\mathcal{Z}$. This originality results from the difficulty of defining meaningful perturbations of time series which would respect the dependence structure along the time. Notice also that in the present context, the variance $\sigma^2$ of the Gaussian noise is a means for controlling the level of perturbations added to the original observations. Another possible interpretation of the role of adding this Gaussian noise is to perform some regularization at the level of the latent space. This prevents the pre-training (learning the representation) to overfit the limited number of available observations. In the context of classification for example [Bis95, NYMH17] reported that this helps in building smoother classification frontiers.

Presently Figure 6 illustrates the idea underlying the addition of a Gaussian noise to the original data at the level of the latent space.

<figure><img src="image_8.png" /><figcaption>Figure 6: Illustration of how gaussian noise behave for positive example sampling</figcaption></figure>

For a reference observation (blue cross), the positive examples (green crosses) are similar to the reference observation. Therefore they should be close to the latter. According to our strategy (based on clustering) for defining negative examples (red crosses), these ones must to be "far away" from the reference observation (and at least further than positive examples). On the illustration, the dashed balls centered at the positive examples suggest the possible locations for Gaussian perturbed representation of any positive example $z_+ + \epsilon$. In particular, increasing $\sigma^2$ would result in increasing the radius of the balls. All of this results in replacing the set $\mathcal{X}$ with $\mathcal{X}^\epsilon$ defined by

$$ \mathcal{X}^{\epsilon} = \{z_{+} + \epsilon \cup \{z_{-}^{1} + \epsilon_{1}, z_{-}^{2} + \epsilon_{2}, \dots, z_{-}^{K-1} + \epsilon_{K-1}\}\}, \quad (3) $$

where $\epsilon_1, \dots, \epsilon_{K-1} \sim \mathcal{N}_d(0, \sigma^2 I_d)$ are independent and identically distributed Gaussian random vectors. Let us emphasize that such a perturbation strategy in the latent space can also serve to remedy the strong imbalance in the dataset between fraudsters and non-fraudsters (fraudsters being highly under-represented here). A similar idea has been already exploited in [KCC21] with a different noise distribution.
## 2.5 Representation learning: Some related works

In Computer Vision (CV) for example, a classic approach of self-supervised learning could be to consider a convolutional autoencoder as explained by [DMSY17]. The model is trained to compress and then reconstruct an image from an intermediate representation $z \in \mathbb{R}^d$ in a latent space. A similar strategy could be obtained by applying a LSTM autoencoder to time series [SMS16]. As above, this model is composed of a LSTM encoder compressing the time series in the latent space, and then of a second LSTM (decoder) that tries to reconstruct the original time series. This possible alternative is also explored in the present work as a benchmark compared to the use of a transformer (see Section 4.3 for more details).

Regarding the self-supervised learning process, alternatives have already been discussed in the literature. For instance, Natural Language Processing (NLP) can be seen as time-series modeling. Therefore it is common to approach text data with a transformer [VSP+17]. This transformer is pre-trained with self-supervised learning to capture the semantic of a language, mainly by addressing two main tasks: One task consists in predicting the next word in a sentence (a GPT-like approach [RNSS18]). The other task rather randomly masks a word in a sentence and tries to retrieve it afterwards (a BERT-like approach [DCLT19]). These two strategies of self-supervised learning are suitable for language processing since they exploit the well-known sequential arrangement of the words. If one changes the order of some words in a sentence, the meaning can radically change: "The cat chased the mouse" and "The mouse chased the cat" share the same words while having two different meanings for example. The words order in a sentence is a highly essential structure that has to be accounted for in NLP, which is somewhat different from what is considered in money laundering detection. When dealing with transactions time series, the precise location of most of transactions will not impact the "meaning" of the resulting time series as much as it does in NLP. As a consequence, the rationale underlying a sequence of transactions is somewhat different from what is observed in NLP. This makes the use of the same semi-supervised learning approach (for instance trying to predict the next transaction) challenging in the context of money laundering. In particular when considering a sequence of transactions, it is more relevant to tackle blocks of financial transactions in self-supervised learning.

# 3 Money laundering detection

The main purpose here is to detail the process allowing to exploit the representations for money laundering detection designed from contrastive learning (see details in Section 2). These representations are expected to be generic, and are meant to capture the semantic of financial transactions from the dataset $\mathcal{D}$.

Section 3.1 introduces the downstream task of money laundering detection, and its connection with the learned representations. As a by-product of these representations, a classification score can be computed that allows to classify any account as a fraudster or non-fraudster (binary classification). Section 3.2 then describes a two-thresholds approach based on this score. This approach can help for investigating money laundering by removing the less suspicious observations, while having a focus on the most suspicious ones. Finally, Section 3.3 presents a procedure for calibrating the two thresholds to efficiently control the false positive rate.

## 3.1 From representation learning to money laundering detection

The approach described along Section 2 mainly focuses on the representation learning that is, the pre-training of the transformer by contrastive learning. The transformer (acting as an encoder) is pre-trained to produce a generic representation of the observations that extracts as much relevant information from the data (complex time series) as possible. An important feature is that *this pre-training task does not require any labeled data*. Once such a generic representation has been learned, it is then exploited to solve our downstream task, which is money laundering detection.

By contrast with the pre-training task from Section 2, the specific money laundering detection comes with a few (uncertain) labeled data. These labels result from a careful hand-made analysis of the accounts that have been flagged as "potential anomalies" among which a few are labeled as fraudsters and the others are not. Let us also mention that the accounts that have not been raised as "potential anomalies" are considered as non-fraudsters by default.
From now on, let $\mathcal{Y} = \{0, 1\}$ denote the set of possible labels. In the context of money laundering $y_i = 0$ means the account $1 \le i \le N$ is a "non-fraudster", whereas $y_i = 1$ means the account $i$ is a fraudster. Let us further assume that the labels $y_i$ are given with the original descriptors $x_i$ so that one observes $N$ independent and identically distributed couples $(x_i, y_i) \in \mathcal{D} \times \mathcal{Y}$, for $1 \le i \le N$. For addressing this new (classification) task, a classifier $g_\mu(\cdot) : \mathcal{U} \to \mathcal{Y}$ is now combined with the representation output by the transformer $f_\theta(\cdot)$. This combination is illustrated by Figure 7, while their classification performance at a point $(x, y)$ is measured by the classification loss $\ell_c(\cdot, \cdot) : \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}_+$. In what follows, $\ell_c(\cdot, \cdot)$ is chosen to be the cross-entropy [MMZ23].

<figure><img src="image_9.png" /><figcaption>Figure 7: Classification process after contrastive pre-training</figcaption></figure>

At this stage, the training can be performed in two ways:

1. The first one consists in considering the weights of the transformer as fixed (after the pre-training), and only optimize the classification loss with respect to the parameters $\mu$ of the classifier $g_\mu(\cdot)$. Therefore the training is that of the classifier given the representation, denoted by CR in what follows. This leads to solve

$$ \mu_{CR}^* \in \operatorname{Argmin}_{\mu} \{\mathcal{L}_{\text{classif-CR}}(\mu)\}, \quad (4) $$

where $\mathcal{L}_{\text{classif-CR}}(\mu) = \mathbb{E}_{(x,y)}[\ell_c(y, g_\mu(f_\theta(x)))]$. In practice since $\mathcal{L}_{\text{classif-CR}}(\mu)$ depends on the unknown probability distribution of $(x, y)$, it is approximated using stochastic gradient descent (SGD) [Bot10] with mini-batch (of cardinality 512) as suggested by Algorithm 1. This outputs an estimator $\hat{\mu}_{CR}$ of the optimal classifier parameters (see Appendix A.2 for details about the choice of the hyperparameter values).

2. The second one consists in considering the weights $\theta$ of the transformer (output by the pre-training pipeline) as an "informed" initialization that can be further optimized. Then the alternative strategy jointly optimizes the classification loss with respect to both $\theta$ and $\mu$. This is called the transformer fine tuning, denoted by FT in Eq. (5). the resulting criterion to be minimized is

$$ \mathcal{L}_{\text{classif-FT}}(\theta, \mu) = \mathbb{E}_{(x,y)}[\ell_c(y, g_\mu(f_\theta(x)))] \quad (5) $$

where the loss function is minimized with respect to $(\theta, \mu)$. Following the same reasoning as above, the optimization is made by SGD, which results in an estimator $(\hat{\theta}_{FT}, \hat{\mu}_{FT})$ of the optimal weights.

These two strategies are explored along Section 4.3 where their practical performances are illustrated on simulation experiments from real data.
In what follows, $g_\mu$ has been chosen to be the logistic regression (binary) classifier [vHIC88, TSOD20]. It is important to notice that, for each point $(x_i, y_i)$ ($1 \le i \le n$), logistic regression also outputs a score $s_i$ that is an estimator of the *a posteriori* probability for the individual $i$ to be-long to class 1 given its location $x_i$. In Section 3.2, this score is exploited as an indicator of being a fraudster. Moreover for improving the practical performance of the classifier, some $\ell_2$ regularization has been added on the unknown parameters for making the optimization problem strongly concave [BV04].

## 3.2 A two-thresholds approach

Our money laundering detection approach can be conceptualized as a binary classification problem where a score between 0 and 1 is associated with each bank account. This score can be interpreted as the probability of being a fraudster given the observed location $x_i$. As already mentioned, the prevalence of false positives in classical rule-based systems ([LP21]) generates substantial operational costs and also limits the identification of high-risk money laundering cases. In this respect Machine Learning can help in controlling this error rate at a prescribed level for avoiding any false positive inflation.

The classical money laundering context comes with a huge imbalance between the two classes to be recovered: the dataset only contains a very small proportion of fraudsters. This strong imbalance suggests to use a two-thresholds strategy for tackling differently non-fraudsters and fraudsters. A first threshold is settled for removing the less suspicious observations from the list of observations triggered by alerts, and a second (larger) threshold is also designed for targeting the more suspicious observations for further investigations.

As a consequence, analysts can save time by only focusing on observations that are most likely fraudsters. On the other side, a lot of observations can now be removed from the set of suspicious observations, avoiding any waste of time by further investigating on them.

**Remark 1** *At this stage, it should be clear that the above scores associated with all observations could serve for ranking the observations from the most suspicious one to the one with the lowest score. The analyst could then focus on the top-k scores for checking which ones are true positives among them. k here is chosen according to the available time to spend on this step. However the limitation of this approach is that there is no theoretical guarantee on the number of false positives induced by this naive strategy. By contrast one originality of the present paper suggests two data-driven thresholds for detecting non-fraudsters and fraudsters while controlling relevant error rates at a prescribed level.*

**Designing a low-threshold for discarding non-fraudsters** The aim is to detect a maximum number of non-fraudulent accounts while controlling the proportion of false positives. This first step allows for removing these accounts from the suspicious observations. This is achieved by setting a threshold $T_l$, called "low threshold", such that scores below this threshold are declared non-fraudulent. This low threshold $T_l$ is automatically chosen by controlling the proportion of mistakes among the list of suspicious accounts declared as non-fraudsters. The proportion of mistakes is quantified by means of the False Discovery Rate (FDR) criterion. [BH95].

**Designing a high-threshold for detecting fraudsters** The second step consists in focusing on the more risky observations which are likely to be fraudsters. Based on the intuition that the score should take large values for fraudsters, another threshold $T_h$, called "high threshold", is designed. The procedure declares as fraudsters all accounts with a score larger than this threshold. Based on a similar reasoning as the one exposed for the low threshold, the high threshold $T_h$ is automatically computed by maximizing the number of detections (accounts flagged as fraudsters by the procedure) while controlling FDR at a prescribed level. This control avoids any false positives inflation unlike what usually happens with classical rule-based strategies ([LP21]).
<figure><img src="image_10.png" /><figcaption>Figure 8: Illustration of the expected normalized distributions of the scores, conditionally of being non-fraudster (in blue) and of being fraudster (in red)</figcaption></figure>

As illustrated in the Figure 8, the fraudulent scores in output of the model can be studied conditionally to the label (fraudulent accounts vs non-fraudulent accounts). Fraudulent observations are expected to have scores on the far right side of the distributions supports, while the non-fraudulent observations should have scores on the far left side. The two thresholds are helping in the analysts investigation by highlighting the more risky accounts and the less risky ones, at a prescribed FDR level. Let us insist on the fact that the FDR control is another originality of the present work in the context of money laundering detection compared to existing approaches in the domain.

## 3.3 Threshold calibration via Benjamini-Hochberg procedure

The two-threshold approach briefly discussed in Section 3.2 leads to considering two distinct classification problems: either rejecting accounts the scores of which are larger than $T_h$ (detected as fraudsters), or rejecting accounts with scores lower than $T_l$ (detected as non-fraudsters). For both problems a decision is taken for each account, while the proportion of wrong detections should be kept under control. Therefore the statistical performance is quantified by means of the False Discovery Rate (FDR) [BH95] especially designed to tackle such problems.

**Remark 2** Let us notice that FDR is related to the so-called precision criterion in such a way that keeping FDR below a level $0 < \alpha < 1$ is equivalent to keeping the precision value larger than $1 - \alpha$. However compared to other possible metrics such as the recall or F1-score, it turns out that FDR is the only one for which a data-driven procedure does exist ([BH95]) which reaches the desired FDR control at a prescribed level $\alpha$.

**FDR performance measure** For each account $0 \le i \le N - 1$, a score $s_i$ has been previously computed, as described in Section 3.1. Based on this score $s_i$, a decision is formally taken regarding respectively the threshold $T_h$ (respectively the threshold $T_l$): The account is declared as fraudster if $s_i \ge T_h$ (resp. as non-fraudster if $s_i \le T_l$). In the present context (unlike the scenario described in the Section 2), the training dataset is made of labeled data $(x_i, y_i)$ where $y_i \in \{0, 1\}$ encoding the true status of the account $i$. For detecting potential fraudsters (by computing $T_h$), a false positive is therefore an account $i$ with label $y_i = 0$ and score $s_i \ge T_h$. This leads to define the FDR value for threshold $T_h$, denoted by $FDR_h$, as

$$FDR_h = FDR(T_h) = \mathbb{E} \left[ \frac{\sum_{i=0}^{N-1} (1 - y_i) \mathbf{1}_{(s_i \ge T_h)}}{\max \left( 1, \sum_{i=0}^{N-1} \mathbf{1}_{(s_i \ge T_h)} \right)} \right], \quad (6)$$
where $1_A$ denotes the indicator function of the event A. The interested reader is referred to the seminal paper [BH95] where FDR is detailed and analyzed. By analogy with Eq. (6), the FDR value for the $T_l$ threshold, denoted by $FDR_l$, is given by

$$FDR_l = FDR(T_l) = \mathbb{E} \left[ \frac{\sum_{i=0}^{N-1} y_i 1_{(s_i \le T_l)}}{\max \left( 1, \sum_{i=0}^{N-1} 1_{(s_i \le T_l)} \right)} \right]. \quad (7)$$

**Benjamini-Hochberg (BH) procedure** In the present work, the calibration of the thresholds $T_l$ and $T_h$ is made by using a classical multiple testing procedure called the Benjamini-Hochberg (BH) procedure ([BH95, BY01]). Focusing on the high threshold $T_h$, the BH-procedure outputs a data-driven value for $T_h$ such that the resulting strategy – detecting as fraudsters all accounts with a score larger than $T_h$ – enjoys an FDR value controlled at level $\alpha \in [0, 1]$.

The BH-procedure relies on a p-value $p_i^h$ computed for an account i and quantifying how much likely this account is a fraudster compared to non-fraudster accounts. Mathematically, $p_i^h$ is given by

$$p_i^h = \mathbb{P}_{s' \sim P_0} (s' > s_i),$$

where $P_0$ denotes the probability distribution of the scores of non-fraudster accounts. As emphasized above, such a p-value depends on the *unknown* distribution $P_0$. But fortunately the present supervised context allows for estimating this p-value by

$$\hat{p}_i^h = \frac{\sum_{j \neq i} (1 - y_j) 1_{(s_j > s_i)}}{\sum_{\ell \neq i} (1 - y_\ell)}, \quad 0 \le i \le N - 1.$$

Let us recall that $\sum_{\ell \neq i} (1 - y_\ell)$ equals the number of scores among the training set data that are non-fraudsters (since $T_h$ is presently considered). A similar definition of the estimated p-values can consistently be formulated for $T_l$ as follows

$$\hat{p}_i^l = \frac{\sum_{j \neq i} y_j 1_{(s_j < s_i)}}{\sum_{\ell \neq i} y_\ell}, \quad 0 \le i \le N - 1,$$

where $\sum_{\ell \neq i} y_\ell$ denotes the total number of fraudsters within the training set.

Once the p-values have been estimated according to the methodology detailed above, the BH procedure is applied to compute the two required thresholds $T_h$ and respectively $T_l$. Focusing on $T_h$ the BH-procedure starts by ranking the estimated p-values $\hat{p}_{(1)}^h \le \dots \le \hat{p}_{(N)}^h$, where the notation $\hat{p}_{(i)}^h$ refers to the $i$th smallest p-value within the set $\{\hat{p}_{(i)}^h, \dots, \hat{p}_{(N)}^h\}$. Then for a prescribed level $\alpha \in (0, 1)$, the number of rejections is given by

$$i_{BH,\alpha}^h = i_{BH}^h(\alpha) = \max \left\{ 1 \le i \le N \mid \hat{p}_{(i)}^h \le i\alpha/N \right\}. \quad (8)$$

Next Figure 10 illustrates how the BH-procedure works on an example. The right-most location where the straight line crosses the p-values curve indicates the number of rejections at level $\alpha$ ($i_{BH,\alpha}^h$), while the corresponding value on the y-axis denotes the value of $\hat{p}_{(i_{BH,\alpha}^h)}^h$.
<figure><img src="image_11.png" /><figcaption>Figure 9: BH procedure and ranked p-values</figcaption></figure>

The threshold $T_h$ is then given by plugging the BH index $i_{BH,\alpha}^h$ in the ordered score that is,

$$T_{h,\alpha} = T_h(\alpha) = s_{(i_{BH,\alpha}^h)}, \qquad (9)$$

where $s_{(1)} \ge \dots \ge s_{(N)}$ denote the non-increasing observed scores. Importantly [BH95] established that, with independent p-values following continuous probability distributions, rejecting all the accounts with a score larger than or equal to $T_{h,\alpha}$ results in

$$FDR_{h,\alpha} = FDR_{T_{h,\alpha}} = \frac{\sum_{i=0}^{N-1} (1 - y_i)}{N} \alpha \le \alpha. \qquad (10)$$

Let us emphasize that the actual FDR level is $\frac{\sum_{i=0}^{N-1} (1-y_i)}{N} \alpha$, which can be strongly lower than $\alpha$ in cases where $\frac{\sum_{i=0}^{N-1} (1-y_i)}{N} \ll 1$. As long as one considers the high threshold $T_h$, the latter ratio equals the proportion of non-fraudsters among the observations. In the money laundering context, only a small number of fraudsters is actually expected. Therefore $\frac{\sum_{i=0}^{N-1} (1-y_i)}{N} \approx 1$ and there should not be any large difference between $\alpha$ and the actual FDR control as long as one considers $T_h$.

However the situation is completely reversed when considering $T_l$ since in this case, the actual FDR control is $\frac{\sum_{i=0}^{N-1} y_i}{N} \alpha$, where $\frac{\sum_{i=0}^{N-1} y_i}{N}$ denotes the proportion of fraudsters. As long as one considers the low threshold $T_l$, $\frac{\sum_{i=0}^{N-1} y_i}{N} \ll 1$ and one needs to take this phenomenon into account for avoiding a too conservative decision [Sto02, CR10]. This can be overcome by adjusting the targeted control level $\alpha'$ by choosing $\alpha' = N/\sum_{i=0}^{N-1} y_i \alpha$ in the case of the low threshold. Indeed plugging in this value of $\alpha'$ within the BH-procedure leads to the desired level since

$$FDR_{l,\alpha'} = \frac{\sum_{i=0}^{N-1} y_i}{N} \alpha' = \alpha. \qquad (11)$$

As long as supervised observations are available, the above correction is obvious. However without supervision, there still exist approaches for estimating the proportion of fraudsters based on the known probability distribution of p-values under the null hypothesis as explained for instance in [Sto02, CR10].

**Remark 3** Let us emphasize that the BH-procedure relies on the last crossing-point between the straight line with slope $\alpha'/N$ and the p-values curve. For defining a threshold $T_h$ such that the corresponding FDR value is equal to the prescribed level, it is then necessary that such a crossing-point does exist. In situations where $\alpha'$ is too small, then it can happen that no such crossing-point does exist. Then it results in no rejections, meaning that the desired level is so small that rejecting no ones is the only (degenerate) solution to this problem.

Algorithm 4 describes in pseudo-code the successive steps corresponding to the application of the BH-procedure.
<caption>**Algorithm 4** Pseudo code of the BH-procedure for $T_h$</caption>

**Input:** List of p-values: $\hat{p}_0^h, \dots, \hat{p}_{N-1}^h$, level $\alpha \in (0, 1)$, labels: $(y_0, \dots, y_{N-1}) \in \{0, 1\}^N$.
Compute $\alpha' = \frac{N}{N_t}\alpha$, with $N_t = \sum_{i=0}^{N-1} y_i$.
Sort the p-values: $\hat{p}_{(1)}^h \le \dots \le \hat{p}_{(N)}^h$.
Define the list $pValueSorted = (\hat{p}_{(1)}^h, \dots, \hat{p}_{(N)}^h)$.
Define the list $CriticalValues = (\alpha'/N, 2\alpha'/N, \dots, \alpha')$.
Define $\mathcal{R} = \emptyset$.
2: **for** $i \in [1, N] \cap \mathbb{N}$ **do**
    3: **if** $p_{(i)}^h \le \frac{i^{\alpha'}}{N}$ **then**
    4: $\mathcal{R} \leftarrow i$
    5: **end if**
6: **end for**

    Compute $i_{BH,\alpha}^h = \max(\mathcal{R})$.

**Output:** $i_{BH,\alpha}^h$

# 4 Simulation experiments

The present section is devoted to the empirical assessment of the approach detailed along this work. Section 4.1 describes all the features of the real datasets used to make the assessment. Then the ability of our approach for designing a valuable representation in a latent space is studied. In the context of money laundering, a relevant representation of the accounts (in the latent space) should allow for easily distinguishing between fraudsters and non-fraudsters. Therefore Section 4.2 first quantifies this performance in terms of the quality of the representations' visualization, at the latent space level. In a second step, Section 4.3 rather focuses the classification problem where accounts are to be classified as fraudsters or non-fraudsters by means of a (basic) logistic regression classifier. Lastly Section 4.4 illustrates the improvement achieved by our procedure in terms of power (number of detections): For a given FDR level, our approach leads to more detections compared to other competitors.

## 4.1 Real-life dataset description

**Complex (real-life) time series** For the purpose of the present assessment, a dataset $\mathcal{D} = \{x_1, x_1, x_2, \dots, x_N\}$ has been created based on real-life data (see also Appendix A.1). Each observation $x_i = (x_{i,1}, \dots, x_{i,T})$ is a bank account described by a time series of length 3 months collecting all financial transactions $x_{i,t}$ along this period. It is worth emphasizing that the resulting time series of each account is highly complex. Indeed at each time where a transaction occurs, the latter is described by both quantitative and qualitative descriptors, making the encoding of such an object highly challenging. More precisely every transaction is described by both quantitative and qualitative descriptors such as:

* Value associated with the transaction (quantitative feature).

* Direction of the transaction (payin or payout, qualitative feature)

* Type of payment (a card payment or a transfer for example, qualitative feature)

* Country associated with the counterparty (qualitative feature)

* Some descriptors indicating if there are some keywords explaining the transaction (in a description field for example, qualitative feature)

* ...

In the present context, money laundering detection is made at the account/customer level that is, an account is declared or not as fraudster based on a set of suspicious transactions with the attributes described above along a given period of time. In particular on the one hand, the present procedure does not aim at detecting specific transactions as fraud events, which can be the topic of a future work (but is out of the scope of the present one). On the other hand, it seems important to keep in
mind that a given account can exhibit a fraudulent behavior on a given period of time, but can have
a normal behavior outside of this period.

**Remark 4** Let us finally emphasize that to the best of our knowledge, such a real-life dataset does not already exist in the literature. The dataset has been made freely available on Kaggle dataset: https://www.kaggle.com/datasets/haroldg/time-series-of-transactions-money-laundering/data

**Confidentiality** In order to protect the confidentiality of users, the data have been anonymized using differential privacy techniques combined with a PCA transformation [IS16, HBMAL19]. For this procedure, the privacy parameter $\epsilon$ is equal to 1. After such an anonymization process, the data only reflect the generic financial activity of a set of accounts corresponding to companies from various industrial domains over a period of 3 months.

**Train/test sets** The dataset has been split into two subsets corresponding to a train and a test set. Appendix A.1 describes with more details the construction of the train and the test sets from the dataset $\mathcal{D}$. Assuming that the relevant descriptors of normal time series (as opposed to time series of abnormal accounts) should not vary along the time, the training and test sets have been created by considering two disjoint time periods. The model is trained at a given reference date based on the elapsed period until that date. The predictions are made on time series of 3 months of transactions, from a period of 6 months following the reference date as illustrated by Figure 10 below.

<figure><img src="image_12.png" /><figcaption>Figure 10: Train set and test set</figcaption></figure>

**Strongly imbalanced data** Here the goal is to detect accounts (companies) involved in money laundering. One important specificity of our dataset is its strong imbalance regarding the (expected) very low proportion of fraudsters among all observations. For instance the training set contains 27% of fraudsters (which is not so bad), while the test set only contains 5% of fraudulent companies (which is clearly worse). Let us recall that each account is labeled as non-fraudster or fraudster based on a running detection process during each considered period. An account can be labeled as non-fraudster in the training set and can be flagged as fraudster in the test set. In other words the occurrence of fraudulent events vary along the time, which explains the variation of the proportion of fraudsters in the test set compared to the training set.

Let us also mention that due to confidentiality requirements, the cardinality of the designed dataset is small compared to all the data available. Another specificity of this dataset owes to the sampling rate of the fraudsters. Actually the proportion of fraudsters is higher in the current dataset than the one of the full dataset. We recall that the current dataset is only a small fraction of the full real-life dataset. Detecting fraudsters would be almost impossible if the fraudsters proportion is too low as well (at least as low as in the full dataset). The main reason for increasing the proportion of fraudsters in the current dataset is to help the model comparison. Obviously if the proportion of fraudsters is so low that all models perform equally poorly, drawing any meaningful conclusion would be almost impossible.

**Noisy labels** As mentioned before, the datasets are real-life ones. The current labeling protocols define the "non-fraudulent" observations as observations that are not yet detected. But, it is possible to have "non-fraudulent" observations that are "fraudulent" in reality (until a potential future detection). This ambiguity necessitates detection frameworks robust to label noise (possible label mistakes).

**Using aggregates** From the full dataset, another dataset denoted by $\mathcal{D}'$ is designed as well. It contains the same number of observations/accounts as $\mathcal{D}$, but each account is now described by more
classical aggregates globally reflecting the full sequence of transactions along the considered time period [TSOD20]. Instances of these aggregates are the summation, the maximum or the minimum of the payouts along the past months to name but a few. It should be clear that such aggregates describe less precisely the account behaviors than the time series used in $\mathcal{D}$. One reason is that they do not incorporate any information regarding the sequential structure of transaction events, which can be highly informative in practice for money laundering detection.

Also in addition to these aggregates, each account $x'_i$ in $\mathcal{D}'$ has some high-level descriptors of the customer such its legal form, its activity category (NAF code in France for exemple).

## 4.2 Representation learning visualization

From a candidate representation – called an embedding – of all observations in the latent space, the true labels can be used for measuring the "homogeneity" of the classes once displayed at the latent space level. Figure 11 illustrates the transformer's embedding locations in a (projected) latent space, revealing a cluster of fraudsters in the top-right corner (within the red circle). Multiple training runs with different random seeds systematically highlight one or two such compact fraudster clusters over small regions in the latent space. This visualization represents a typical outcome from contrastive pre-training, showing the transformer's ability to capture specific features among the descriptors that are related to suspicious patterns.

<figure><img src="image_13.png" /><figcaption>Figure 11: Embeddings of the transformer projected with t-SNE - Comparison between money laundering labels (class 0: Non-Fraudster and class 1: Fraudsters)</figcaption></figure>

In contrast, Figure 12 displays representations output by the LSTM Autoencoder [SMS16] (see Section 2.5).
<figure><img src="image_14.png" /><figcaption>Figure 12: Embeddings of the LSTM Autoencoder projected with t-SNE - Comparison between money laundering labels (class 0: Non-Fraudster and class 1: Fraudsters)</figcaption></figure>

Whereas it is possible to identify a cluster of fraudsters at the bottom of the picture, fewer fraudsters belong to this region compared to Figure 11. In addition, representations exhibit a greater variability across the successive runs (for different seeds) compared to the transformer one.

Appendix A.3 provides a set of visualizations across different random seeds, highlighting the transformer improvement in terms of stability of the results. This is consistent with the results presented in Sections 4.3 and 4.4, where the transformer achieves higher detection rates.

## 4.3 Distributions of fraudsters and non-fraudsters

One of the main goals of the present work is to distinguish the sub-populations of fraudsters and non-fraudsters from the initial observations. After a representation (equivalently an embedding) has been learned in a latent space from Section 3.1, a classification rule (logistic regression) is fed up with this representation for classifying the observations as fraudsters or non-fraudsters. The following reviews the performance of the embeddings output by the LSTM Autoencoder [SMS16] and the transformer combined with the logistic regression classifier [vHIC88, TSOD20] used for answering the money laundering detection task. The tabular dataset *D'* is also considered as an input for the logistic regression and an XGBoost [CG16], allowing for a comparison with domain-informed features. To go further, as described along Section 3.1, the transformer is also fine-tuned on the labels.

A specific difficulty raised by the money laundering detection is the strong imbalance between classes: fraudsters are at most 27% of the observations in the training set, and only 5% in the test set (see Section 4.1). This leads us to replace the usual classification metrics by rather assessing how much the scores output by the logistic regression exhibit a different behavior depending on the true labels. A first illustration of the two distributions output by logistic regression is given by Figure 13 (Left panel), where only tabular data have been used to learn the two distributions. Two histograms are displayed: red for the distribution of (the scores of) fraudsters, and blue for non-fraudsters. The continuous curves on the picture are the kernel density estimators respectively of the fraudsters distribution (red) and non-fraudsters distribution (blue) learned with a Gaussian kernel. An effective approach should lead to a probability distribution of fraudsters (red) concentrated on the right (corresponding to high scores), whereas the distribution of non-fraudsters (blue) should be shifted to the left. Intuitively the smaller the overlap between the two distributions, the more effective the underlying methodology (cf. Figure 8).
<figure><img src="image_15.png" /><figcaption>Figure 13: Normalized and non-normalized classification score distribution - Logistic Regression on tabular data only</figcaption></figure>

<figure><img src="image_16.png" /></figure>

Figures 13 and 14 demonstrate the performance of logistic regression and of XGBoost on the tabular dataset $\mathcal{D}'$. From the left panel of Figure 13, it appears that tabular data from $\mathcal{D}'$ only lead to a poor embedding. Indeed the non-fraudsters distribution truly exhibits a bump on the left. However the fraudsters distribution does not clearly exhibit any similar bump on the right. Moreover the supports of the two probability distribution are almost coinciding. As a consequence any detection rule which would consist in detecting as fraudsters all observations with a score larger than a threshold would unavoidably lead to a large number of false positives. This conclusion is also supported by the right panel of Figure 13 where the counts of each class are displayed. On the right tail (in [0.85; 1]) of the bar plot, non-fraudsters and fraudsters are mixed with an almost equal cardinality. The weak ability of the procedure for distinguishing the two populations actually reflects the poor quality of the aggregates from $\mathcal{D}'$ (see Section 4.1) that have lead to these embeddings. Therefore such aggregates are not informative enough to summarize complex time series such as the ones considered in $\mathcal{D}$. Nevertheless such a rough representation could be seen as a first improvement compared to ongoing rule-based strategies for raising alerts in money laundering detection.

<figure><img src="image_17.png" /><figcaption>Figure 14: Normalized and non-normalized classification score distribution - XGBoost on tabular data only</figcaption></figure>

<figure><img src="image_18.png" /></figure>

A similar analysis can be performed on the Figure 14, displaying the results of an XGBoost classifier on the dataset $\mathcal{D}'$. From the left panel, it appears that the model is able to exhibit two bumps for the fraudster and non-fraudster distributions. Especially, the bump associated to the fraudsters on the left is more pronounced than the one seen with logistic regression in Figure 13. But on the right panel, the results are worse, leading to a higher false positive rate (which is confirmed in the results later displayed in Section 4.4). Again, the supports of the two probability distributions are almost coinciding (similarly to the logistic regression, but even more). As a consequence, any detection rule which would consist in detecting as fraudsters all observations with a score larger than a threshold would unavoidably lead to a large number of false positives.
<figure><img src="image_19.png" /><figcaption>Figure 15: Normalized and non-normalized classification score distribution - Logistic Regression on LSTM autoencoder representations</figcaption></figure>

By contrast, Figure 15 displays the results obtained from an LSTM autoencoder [SMS16], where the complex time series from $\mathcal{D}$ are used instead of their proxies in $\mathcal{D}'$. The LSTM autoencoder parameters have been learned from $\mathcal{D}$ and hyper-parameters have been tuned with a grid search strategy. By comparison between the left panels of Figures 13, 14, and 15, one observes similar (and still relatively poor) results for the LSTM autoencoder, demonstrating the complexity of handling money laundering detection based on the complex dataset $\mathcal{D}$. Structured time series are more informative regarding the problem at hand. However, the model is not able to leverage efficiently the time series to extract money laundering patterns. The LSTM results are included in the comparison to serve as a baseline when introducing our approach based on a transformer learned by contrastive learning and applied to the dataset $\mathcal{D}$.

<figure><img src="image_20.png" /><figcaption>Figure 16: Normalized and non-normalized classification score distribution - Logistic Regression on contrastive learned representations</figcaption></figure>

The performance of the transformer-based approach are displayed in Figure 16. The transformer is optimized for learning an embedding of the observations in $\mathcal{D}$. This learning step is made using contrastive learning and data in $\mathcal{D}'$ since no label from $\mathcal{D}$ is used at the representation learning step (See Section 2 and Section 3.1 for more details). The hyperparameters from Eq. 4 such as the learning rate and temperature are calibrated using a grid search strategy.

The left panel of Figure 16 shows two main improvements compared to the previous graphs. In the left tail of the non-fraudsters distribution (blue), the width of the non-overlapping part is visually much bigger than the previous approaches, with no overlapping before a score of 0.4 and a lot of observations in [0.0; 0.4]. This is also visible from the corresponding right panel of Figure 16 where no fraudsters lie in this range. The second visible improvement is the higher concentration of fraudsters in a close neighborhood to 1 (higher bump), with a very low density for non-fraudsters in this area. Combining these two features makes the two classes more distinguishable. It therefore confirms that the transformer trained with contrastive learning is more powerful than the LSTM autoencoder for designing an effective embedding of the complex financial time series in the context of money laundering detection.
For the sake of completeness, the fine tuning of the transformer has been also included in the comparison. It only differs from the previous approach by a new optimization step of the transformer parameters when the latter is combined with a logistic regression classifier. The former parameters learned from the previous step now serve as an (informative) initialization of the parameters (for hopefully avoiding bad local minima). The optimization procedure has now been adapted for the classification task. Here the purpose is to check whether there is (or not) something to gain in re-optimizing the transformer parameters compared to the results previously discussed from Figure 16.

<figure><img src="image_21.png" /><figcaption>Figure 17: Normalized and non-normalized classification score distribution - Transformer fine-tuned after contrastive pre-training</figcaption></figure>

From the left panel of Figure 17 one clearly observe a strong change regarding the non-fraudsters probability distribution shape. Compared with the one of Figure 16, it is now more peaky in a close neighborhood of 0. Moving to the right panel, it appears that only a small fraction of the non-fraudsters population has a score above 0.4, which deeply contrasts with the right panel of Figure 16. However the improvement for the fraudsters distribution is unfortunately not as clear. This may result from the strong imbalance in our data between the class of non-fraudsters (at least 75%) and the one of fraudsters (at most 25% in the training set).

## 4.4 Calibrating classification thresholds

Illustrations such as Figure 16 from previous Section 4.3 highlight that defining a unique threshold for distinguishing between fraudsters and non-fraudsters would necessary lead to an overly large number of mistakes in both classes. This results from the overlapping supports of these two probability distributions. As a consequence, Section 3.2 describes a detection procedure relying on two thresholds respectively denoted by $T_h$ and $T_l$ respectively for high-threshold (in the right tail) and the low-threshold (left tail). These thresholds are calibrated by means of the BH-procedure consistently with the material exposed along Section 3.3.

Our present purpose is to illustrate how this BH-procedure actually behaves in practice in terms of the number of output detections (as fraudsters for $T_h$ or as non-fraudsters for $T_l$). This illustration is carried out with the four strategies reviewed along previous Section 4.3 that is, Logistic regression with Tabular Data (LTD), XGBoost with Tabular Data (XGBTD), LSTM and logistic regression (LSTM), Transformer and logistic regression (Transf), and Fine Tuning (FT). The results are reported in Table 1 for $T_h$ and Table 2 for $T_l$ from applying the above five strategies on the test set data (with detailed specifications in Section 4.1).

In most settings, the BH-procedure achieves the expected result. However, there are a few exceptions where the result cannot be achieved since the required control level $\alpha$ is too low (too conservative control). In such a case the results are reported as "NA" in Tables 1 and 2.

**Behavior of the High-threshold $T_h$** In Table 1, the column entitled $FDR_h$ displays the required FDR level. Results are provided for four values of $FDR_h$, namely from 0.7 to 0.1. For each FDR value, the corresponding number of fraudsters is much larger for Transf and FT than for other competitors. Depending on $FDR_h$, it is twice larger than the number of detected fraudster of LSTM (and even
more for lower $FDR_h$ values). For a prescribed proportion of false detections, Transf and FT uniformly identify more fraudsters than XGBTD or TD.

<table><caption>Table 1: Performance summary on the high threshold</caption><thead><tr><th>Model</th><th>FDR<sub>h</sub></th><th>Frauds detected</th><th>% total frauds</th><th>F1-score</th></tr></thead><tbody><tr><td>Logistic Regression - tabular data</td><td>0.40</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>XGBoost - tabular data</td><td>0.40</td><td>1.0 ± 0.0</td><td>0.86</td><td>0.02</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.40</td><td>8.5 ± 6.2</td><td>7.33</td><td>0.13</td></tr><tr><td>Logistic Regression - transformer</td><td>0.40</td><td>23.6 ± 12.8</td><td>20.34</td><td>0.30</td></tr><tr><td><strong>Transformer fine-tuning</strong></td><td><strong>0.40</strong></td><td><strong>26.7 ± 14.5</strong></td><td><strong>23.02</strong></td><td><strong>0.33</strong></td></tr><tr><td>Logistic Regression - tabular data</td><td>0.30</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>XGBoost - tabular data</td><td>0.30</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.30</td><td>5.5 ± 4.0</td><td>4.74</td><td>0.089</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.30</strong></td><td><strong>12.3 ± 11.6</strong></td><td><strong>10.60</strong></td><td><strong>0.18</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.30</td><td>10.2 ± 9.2</td><td>8.79</td><td>0.15</td></tr><tr><td>Logistic Regression - tabular data</td><td>0.20</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>XGBoost - tabular data</td><td>0.20</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.20</td><td>2.6 ± 1.9</td><td>2.41</td><td>0.04</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.20</strong></td><td><strong>4.0 ± 5.0</strong></td><td><strong>3.45</strong></td><td><strong>0.06</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.20</td><td>0.8 ± 2.5</td><td>0.69</td><td>0.01</td></tr><tr><td>Logistic Regression - tabular data</td><td>0.10</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>XGBoost - tabular data</td><td>0.10</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.10</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.10</strong></td><td><strong>0.9 ± 2.8</strong></td><td><strong>0.78</strong></td><td><strong>0.01</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.10</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr></tbody></table>

Let us emphasize that the large $FDR_h$ values considered in Table 1 reflect the fact that the distributions of fraudsters and non-fraudsters are strongly overlapping in the right tail of Figure 16 and fraudsters are only a small fraction of the full observations (5% in the test set). Since the supports are not disjoint close to 1, detecting each true fraudster can only be made at the price of including quite a few non-fraudsters. Nevertheless it is important to keep in mind that this is already by far better than the state-of-the-art rule-based approaches currently used to answer this question.

**Behavior of the Low-threshold $T_l$** Results for $T_l$ are collected in Table 2. By contrast with the (large) FDR values from Table 1, the four values considered here 0, 1%, 2%, and 3% are usual ones. Let us also notice that the largest possible value of $FDR_l$ is smaller or equal to 5%, which results from the strong imbalance between classes: Rejecting all observations as non-fraudsters would result in at most 5% of false positives (that is, the proportion of fraudsters among the observations in the test set).
<table><caption>Table 2: Performance summary on the low threshold</caption><thead><tr><th>Model</th><th>FDR<sub>l</sub></th><th>Non-frauds detected</th><th>% non-frauds</th><th>F1-score</th></tr></thead><tbody><tr><td>Logistic Regression - tabular data</td><td>0.03</td><td>1807.3 ± 3.6</td><td>82.56</td><td>0.89</td></tr><tr><td>XGBoost - tabular data</td><td>0.03</td><td>1766.0 ± 0.0</td><td>80.68</td><td>0.88</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.03</td><td>1858.0 ± 138.0</td><td>84.88</td><td>0.91</td></tr><tr><td>Logistic Regression - transformer</td><td>0.03</td><td>2118.3 ± 11.0</td><td>96.77</td><td>0.97</td></tr><tr><td><strong>Transformer fine-tuning</strong></td><td><strong>0.03</strong></td><td><strong>2125.8 ± 10.5</strong></td><td><strong>97.11</strong></td><td><strong>0.97</strong></td></tr><tr><td>Logistic Regression - tabular data</td><td>0.02</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>XGBoost - tabular data</td><td>0.02</td><td>104.0 ± 0.0</td><td>4.8</td><td>0.09</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.02</td><td>871.1 ± 476.4</td><td>39.79</td><td>0.57</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.02</strong></td><td><strong>1932.8 ± 35.9</strong></td><td><strong>88.30</strong></td><td><strong>0.93</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.02</td><td>1903.8 ± 64.4</td><td>86.97</td><td>0.92</td></tr><tr><td>Logistic Regression - tabular data</td><td>0.01</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>XGBoost - tabular data</td><td>0.01</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.01</td><td>74.0 ± 88.5</td><td>3.38</td><td>0.07</td></tr><tr><td>Logistic Regression - transformer</td><td>0.01</td><td>1188.3 ± 208.4</td><td>54.29</td><td>0.70</td></tr><tr><td><strong>Transformer fine-tuning</strong></td><td><strong>0.01</strong></td><td><strong>1292.5 ± 144.3</strong></td><td><strong>59.05</strong></td><td><strong>0.74</strong></td></tr><tr><td>Logistic Regression - tabular data</td><td>0.0</td><td>17.0 ± 0.0</td><td>0.02</td><td>0.015</td></tr><tr><td>XGBoost - tabular data</td><td>0.0</td><td>19.0 ± 0.0</td><td>0.87</td><td>0.02</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.0</td><td>35.1 ± 30.8</td><td>1.60</td><td>0.03</td></tr><tr><td>Logistic Regression - transformer</td><td>0.0</td><td>237.5 ± 144.2</td><td>10.85</td><td>0.20</td></tr><tr><td><strong>Transformer fine-tuning</strong></td><td><strong>0.0</strong></td><td><strong>419.4 ± 149.5</strong></td><td><strong>19.16</strong></td><td><strong>0.32</strong></td></tr></tbody></table>

Another important feature of the results is that, for a given FDR level, the transformer uniformly achieves the best performance (in terms of the number of detected true non-fraudsters). This does not only illustrate the gain in considering the Transformer instead of an LSTM autoencoder, but it also reflects that detecting non-fraudsters is easier than detecting fraudsters. Nevertheless let us emphasize that detecting as non-fraudsters some accounts allows for saving time and computational resources in practice since it avoids considering a large number of false alarms with a very small probability of mistakes.

# 5 Conclusion

This research addresses the critical challenge of money laundering detection in financial system with a two-steps methodology, combining a transformer that processes transactions time series and an adaptive strategy based on two thresholds that control the false positive rate. This strongly contrasts with traditional rule-based systems, which allow for auditability but definitely struggle to maintain a reasonable false positive rate and cannot adapt to evolving patterns.

Our methodology mitigates these limitations through contrastive learning. Encapsulated in a pre-training step, it enables learning representations of financial time series, reducing the dependency on available labels. This self-supervised step captures the relevant behaviors in the time series, enabling transferability to a downstream task. The second step, with the two-thresholds approach optimizes the investigation time: It rejects most risky observations while simultaneously discarding those that are considered as not risky. The purpose of this is to reduce the number of unnecessary investigations while keeping the false discovery rate under control. Experimental results on real-life data demonstrate the potential enhancement allowed by representation learning in this context. It raises opportunities to outperform traditional rule-based systems.

The proposed framework exhibits some sensitivity to the hyper-parameters tuning (temperature, learning rate, noise intensity, etc.). This requires an extensive calibration step of these hyper-parameters to find an optimal combination, which can reveal time consuming. However this calibration step has to be carried out once and can further be updated from time to time for adapting to an evolving context. Moreover, the empirical experiments have been only carried out from a subset of the full available dataset. Despite this limitation, the reported empirical results demonstrate a consistent behavior, reducing a lot the number of false positives compared to rule-based strategies. All of this even suggests better results on larger datasets.
References

[AMN+98] Sunil Arya, David M. Mount, Nathan S. Netanyahu, Ruth Silverman, and Angela Y. Wu. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. *J. ACM*, 45(6):891–923, November 1998.

[BH95] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. *Journal of the Royal Statistical Society: Series B (Methodological)*, 57(1):289–300, 1995.

[Bis95] Christopher M. Bishop. Training with noise is equivalent to Tikhonov regularization. *Neural Computation*, 7(1):108–116, 1995.

[BLS20] Luigi Bellomarini, Eleonora Laurenza, and Emanuel Sallinger. Rule-based anti-money laundering in financial intelligence units: Experience and vision. In Sotiris Moschoyiannis, Paul Fodor, Jan Vanthienen, Daniela Inclezan, Nikolay Nikolov, Francisco Martín-Recuerda, and Ioan Toma, editors, *Proceedings of the 14th International Rule Challenge, 4th Doctoral Consortium, and 6th Industry Track @ RuleML+RR 2020 co-located with 16th Reasoning Web Summer School (RW 2020) 12th DecisionCAMP 2020 as part of Declarative AI 2020, Oslo, Norway (virtual due to Covid-19 pandemic)*, 29 June - 1 July, 2020, volume 2644 of CEUR Workshop Proceedings, pages 133–144. CEUR-WS.org, 2020.

[Bot10] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In *Proceedings of COMPSTAT'2010: 19th International Conference on Computational Statistics* Paris France, August 22-27, 2010 *Keynote, Invited and Contributed Papers*, pages 177–186. Springer, 2010.

[BV04] Stephen Boyd and Lieven Vandenberghe. *Convex Optimization*. Cambridge University Press, Cambridge, 2004.

[BY01] Yoav Benjamini and Daniel Yekutieli. The control of the false discovery rate in multiple testing under dependency. *The Annals of Statistics*, 29(4):1165–1188, 2001.

[CG16] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In *Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, KDD '16, page 785–794. ACM, August 2016.

[CKNH20] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.

[CR10] Alain Celisse and Stéphane Robin. A cross-validation based estimation of the proportion of true null hypotheses. *Journal of Statistical Planning and Inference*, 140(11):3132–3147, 2010.

[DCLT19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. *Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, 1:4171–4186, 2019.

[DMSY17] Jianfeng Dong, Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. Learning deep representations using convolutional auto-encoders with symmetric skip connections, 2017.

[DSS21] Philipp Dufter, Martin Schmitt, and Hinrich Schütze. Position information in transformers: An overview, 2021.

[DT17] Terrance DeVries and Graham W. Taylor. Dataset augmentation in feature space, 2017.

[DTGMR+24] Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P. de Almeida, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. Nucleotide transformer: building and evaluating robust foundation models for human genomics. *Nature Methods*, 2024.
[EBA<sup>+</sup>22] Ahmad Naser Eddin, Jacopo Bono, David Aparício, David Polido, João Tiago Ascensão, Pedro Bizarro, and Pedro Ribeiro. Anti-money laundering alert optimization using machine learning with graphs, 2022.

[Elm90] Jeffrey L. Elman. Finding structure in time. *Cognitive Science*, 14(2):179-211, 1990.

[Eur24] European Parliament. Proposal for a directive on the mechanisms to be put in place by the member states for the prevention of the use of the financial system for the purposes of money laundering or terrorist financing, 2024.

[GAHG22] Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Understanding and improving the role of projection head in self-supervised learning. *arXiv preprint arXiv:2212.11491*, 2022.

[GAvdHG22] Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Understanding and improving the role of projection head in self-supervised learning, 2022.

[GBNL23] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank, 2023.

[GCZ<sup>+</sup>24] Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2024.

[HBMAL19] Naoise Holohan, Stefano Braghin, Pól Mac Aonghusa, and Killian Levacher. Diffprivlib: the IBM differential privacy library. *ArXiv e-prints*, 1907.02444 [cs.CR], July 2019.

[HFW<sup>+</sup>20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020.

[HKW<sup>+</sup>22] Tri Huynh, Simon Kornblith, Matthew R. Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation, 2022.

[HR18] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification, 2018.

[HWZ<sup>+</sup>24] Haigen Hu, Xiaoyuan Wang, Yan Zhang, Qi Chen, and Qiu Guan. A comprehensive survey on contrastive learning. *Neurocomputing*, page 128645, 2024.

[HZSB16] Tamer Hossam, Mohamed Zaki, Tarek Sobh, and Khaled Badran. Design of a monitor for detecting money laundering and terrorist financing. *Journal of Theoretical and Applied Information Technology*, 85:425-436, 03 2016.

[IEA<sup>+</sup>23] Abiodun M. Ikotun, Absalom E. Ezugwu, Laith Abualigah, Belal Abuhaija, and Jia Heming. K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data. *Information Sciences*, 622:178-210, 2023.

[IS16] Hafiz Hasnain Imtiaz and Anand D. Sarwate. Symmetric matrix perturbation for differentially-private principal component analysis. *2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 2339-2343, 2016.

[KCC21] Beomsoo Kim, Jang-Ho Choi, and Jaegul Choo. Augmenting imbalanced time-series data via adversarial perturbation in latent space. In Vineeth N. Balasubramanian and Ivor Tsang, editors, *Proceedings of The 13th Asian Conference on Machine Learning*, volume 157 of *Proceedings of Machine Learning Research*, pages 1633-1644. PMLR, 17-19 Nov 2021.

[KCLS<sup>+</sup>22] Hee E. Kim, Alejandro Cosa-Linan, Nandhini Santhanam, Mahboubeh Jannesari, Mate E. Maros, and Thomas Ganslandt. Transfer learning for medical image classification: a literature review. *BMC Medical Imaging*, 22(1):69, 2022.
[KG22] Damian Kisiel and Denise Gorse. Portfolio transformer for attention-based asset allocation, 2022.

[KPSA21] Dattatray Vishnu Kute, Biswajeet Pradhan, Nagesh Shukla, and Abdullah Alamri. Deep learning and explainable artificial intelligence techniques applied for detecting money laundering-a critical review. *IEEE Access*, 9:82300-82317, 2021.

[KTW+21] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, 2021.

[LA17] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, 2017.

[LHGM22] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance, 2022.

[LKHS20] Phuc H. Le-Khac, Graham Healy, and Alan F. Smeaton. Contrastive representation learning: A framework and review. *IEEE Access*, 8:193907-193934, 2020.

[LP21] Karel Lannoo and Richard Parlour. Anti-money laundering in the eu: Time to get serious. ceps task force report 28 jan 2021., January 2021.

[LVF24] Serhiy Lyeonov, Tetiana Vaslieva, and Hanna Filatova. *Financial Fraud and Cybercrime in Wartime: An Overview of the Scientific Landscape and Insights from Countries Engaged in Military Conflict*. Academic Research and Publishing UG (i. G.), Hamburg, 2024.

[Mic01] Daniele Micci-Barreca. A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. *SIGKDD Explorations Newsletter*, 3(1):27-32, jul 2001.

[MMZ23] Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications, 2023.

[Nat] United Nation. Money laundering, proceeds of crime and the financing of terrorism.

[NYMH17] Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017.

[RG19] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.

[RJW89] David Zipser Ronald J. Williams. A learning algorithm for continually running fully recurrent neural networks. *Neural Computation*, 1:270-280., 1989.

[RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical Report TR-2018-001, OpenAI, 2018.

[Rou87] Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. *Journal of Computational and Applied Mathematics*, 20:53-65, 1987.

[RPC+22] Javier Fernández Rodríguez, Michele Papale, Michele Carminati, Stefano Zanero, et al. A natural language processing approach for financial fraud detection. In *CEUR workshop proceedings*, volume 3260, pages 135-149. CEUR-WS. org, 2022.

[SBG23] Roy Majed Sinno, G. Yarrow Baldock, and Kimberly Gleason. The evolution of trade-based money laundering schemes: a regulatory dialectic perspective. *Journal of Financial Crime*, 2023.
[SH97] Jürgen Schmidhuber Sepp Hochreiter. Long short-term memory. Neural Computation, 9:270-280., 1997.

[Sin01] Amit Singhal. Modern information retrieval: A brief overview. IEEE Data Eng. Bull., 24:35-43, 2001.

[SMS16] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using lstms, 2016.

[SNY+10] Agus Sudjianto, Sheela Nair, Ming Yuan, Aijun Zhang, Daniel Kern, and Fernando Cela-Díaz. Statistical methods for fighting financial crimes. Technometrics, 52(1):5-19, 2010.

[Sto02] John D Storey. A direct approach to false discovery rates. Journal of the Royal Statistical Society Series B: Statistical Methodology, 64(3):479-498, 2002.

[TPD+23] Maria Paola Tatulli, Tommaso Paladini, Mario D'Onghia, Michele Carminati, and Stefano Zanero. Hamlet: A transformer based approach for money laundering detection. In Shlomi Dolev, Ehud Gudes, and Pascal Paillier, editors, Cyber Security, Cryptology, and Machine Learning, pages 234-250, Cham, 2023. Springer Nature Switzerland.

[TSOD20] Pavlo Tertychnyi, Ivan Slobozhan, Madis Ollikainen, and Marlon Dumas. Scalable and imbalance-resistant machine learning models for anti-money laundering: A two-layered approach. In Enterprise Applications and Services in the Finance Industry, 2020.

[vdOLV19] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.

[vHlC88] J.C. van Houwelingen and S. le Cessie. Logistic regression, a review. Statistica Neerlandica, 42(4):215-232, 1988.

[VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30:5998-6008, 2017.

[WC20] Yu-An Wang and Yun-Nung Chen. What do position embeddings learn? an empirical study of pre-trained language model positional encoding, 2020.

[WZZ+23] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. In International Joint Conference on Artificial Intelligence (IJCAI), 2023.

[XGN+24] Yihao Xue, Eric Gan, Jiayi Ni, Siddharth Joshi, and Baharan Mirzasoleiman. Investigating the benefits of projection head for representation learning, 2024.

[ZBM+22] Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M. Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), page 387-397. IEEE, January 2022.

[ZQF24] Wenbin Zhu, Runwen Qiu, and Ying Fu. Comparative study on the performance of categorical variable encoders in classification and regression tasks, 2024.
# A Appendix

## A.1 dataset details

In the following we provide details about the datasets used along the present work.

<table><caption>Table 3: Datasets details</caption><thead><tr><th></th><th>Value</th></tr></thead><tbody><tr><td>Number of transaction in <i>D</i> - train set</td><td>2 343 143</td></tr><tr><td>Number of transaction in <i>D</i> - test set</td><td>150 689</td></tr><tr><td>Number of features (after PCA) in <i>D</i></td><td>43</td></tr><tr><td>Number of companies in <i>D'</i> - train set</td><td>5428</td></tr><tr><td>Number of companies in <i>D'</i> - test set</td><td>2305</td></tr><tr><td>Number of features (after PCA) in <i>D'</i></td><td>6</td></tr><tr><td>% of fraudsters - train set</td><td>26.76%</td></tr><tr><td>% of fraudsters - test set</td><td>5.03%</td></tr></tbody></table>

**Train/test sets** As described in Section 4.1, the train and test sets are respectively created from disjoint time periods as illustrated by Figure 10.

**Training and testing label imbalances** The training and testing sets have different label imbalances. The fraudsters in the test set are the one detected during a short period of time (few months). On the contrary, the train set has a "stock" of fraudsters during a long period of time (several years). These two amounts are then compared to the total number of observations at the reference date (for the train set) and 6 months later (for the test set). These two amounts are on the same scale, while the amount of fraudsters is very different in the train set and in the test set, explaining the difference of imbalances.

**Each firm has several time series** Figure 18 emphasizes that every customer is associated with several time series corresponding to successive sliding windows.

<figure><img src="image_22.png" /><figcaption>Figure 18: Transaction windows used in creating the dataset *D*.</figcaption></figure>

## A.2 Hyperparameters and training details

We optimized the key hyperparameters — temperature, learning rate, number of layers, number of neurons, dropout rate, noise intensity, and memory-bank size — via a grid search guided by the RankMe metric [GBNL23]. These hyperparameters strongly influence the representation quality and the results reported in Section 4.

**Transformer pre-training with contrastive learning** Please find below a table describing the details about the main hyperparameters used during the contrastive pre-training of the transformer.
<table><caption>Table 4: Transformer hyperparameters - Contrastive pre-training</caption><thead><tr><td>Hyperparameter</td><td>Value</td></tr></thead><tbody><tr><td>Projection head Dimension</td><td>16</td></tr><tr><td>Projection head Size</td><td>2</td></tr><tr><td>Memory bank size</td><td>4000</td></tr><tr><td>Max length</td><td>256</td></tr><tr><td>Number of heads</td><td>10</td></tr><tr><td>Number of layers</td><td>5</td></tr><tr><td>Number of neurons</td><td>64</td></tr><tr><td>Dropout rate</td><td>0.1</td></tr><tr><td>Number of negative examples</td><td>128</td></tr><tr><td>Number of positive neighbors</td><td>50</td></tr><tr><td>Number of epochs</td><td>100</td></tr><tr><td>Batch size</td><td>512</td></tr><tr><td>Gradient accumulation step</td><td>8</td></tr><tr><td>Learning rate</td><td>0.0005</td></tr><tr><td>Temperature</td><td>0.2</td></tr><tr><td>Weight decay</td><td>0.01</td></tr><tr><td>Optimizer</td><td>AdamW</td></tr><tr><td>Noise intensity</td><td>0.05</td></tr><tr><td>Gradient clipping max norm</td><td>5.0</td></tr><tr><td>Output representation dimension</td><td>64</td></tr></tbody></table>

**Transformer fine-tuning** Please find below a table describing the details about the main hyper-parameters used during the fine tuning of the transformer, after the contrastive pre-training.

<table><caption>Table 5: Fine-tuning hyperparameters - Contrastive pre-training</caption><thead><tr><td>Hyperparameter</td><td>Value</td></tr></thead><tbody><tr><td>Number of epochs</td><td>10</td></tr><tr><td>Batch size</td><td>1024</td></tr><tr><td>Learning rate</td><td>0.001</td></tr><tr><td>Gradient clipping max norm</td><td>5.0</td></tr><tr><td>Optimizer</td><td>AdamW</td></tr></tbody></table>

**LSTM autoencoder** Please find below the main hyperparameters of the LSTM autoencoder, used as a benchmark in the experiments.

<table><caption>Table 6: LSTM hyperparameters - Contrastive pre-training</caption><thead><tr><td>Hyperparameter</td><td>Value</td></tr></thead><tbody><tr><td>Number of layers</td><td>3</td></tr><tr><td>Hidden size</td><td>128</td></tr><tr><td>Embedding size</td><td>64</td></tr><tr><td>Dropout rate</td><td>0.25</td></tr><tr><td>Max length</td><td>256</td></tr><tr><td>Batch size</td><td>512</td></tr><tr><td>Optimizer</td><td>AdamW</td></tr><tr><td>Learning rate</td><td>0.001</td></tr><tr><td>Weight decay</td><td>0.01</td></tr><tr><td>Number of epochs</td><td>100</td></tr></tbody></table>
## A.3 Representation learning visualization with different seeds

### A.3.1 Transformer with contrastive learning

<figure><img src="image_23.png" /><figcaption>Figure 19: Embeddings of the transformer projected with t-SNE - Comparison between money laundering labels (class 0: Non-Fraudster and class 1: Fraudsters)</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>Figure 20: Embeddings of the transformer projected with t-SNE - Comparison between money laundering labels (class 0: Non-Fraudster and class 1: Fraudsters)</figcaption></figure>
<figure><img src="image_25.png" /><figcaption>Figure 21: Embeddings of the transformer projected with t-SNE - Comparison between money laundering labels (class 0: Non-Fraudster and class 1: Fraudsters)</figcaption></figure>

### A.3.2 LSTM Autoencoder

<figure><img src="image_26.png" /><figcaption>Figure 22: Embeddings of the LSTM Autoencoder projected with t-SNE - Comparison between money laundering labels (class 0: Non-Fraudster and class 1: Fraudsters)</figcaption></figure>
<figure><img src="image_27.png" /><figcaption>Figure 23: Embeddings of the LSTM Autoencoder projected with t-SNE - Comparison between money laundering labels (class 0: Non-Fraudster and class 1: Fraudsters)</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Figure 24: Embeddings of the LSTM Autoencoder projected with t-SNE - Comparison between money laundering labels (class 0: Non-Fraudster and class 1: Fraudsters)</figcaption></figure>
A.4 High thresholds - full tables

<table><caption>Table 7: Performance summary on the high threshold - Full table</caption><thead><tr><th>Model</th><th>FDR<sub>h</sub></th><th>Frauds detected</th><th>% total frauds</th><th>F1-score</th></tr></thead><tbody><tr><td>Logistic Regression - tabular data</td><td>0.80</td><td>48 ± 0.0</td><td>0.41</td><td>0.27</td></tr><tr><td>XGBoost - tabular data</td><td>0.80</td><td>43 ± 0.0</td><td>37.06</td><td>0.26</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.80</td><td>49.7 ± 8.8</td><td>42.80</td><td>0.27</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.80</strong></td><td><strong>83.50 ± 2.4</strong></td><td><strong>72.0</strong></td><td><strong>0.31</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.80</td><td>79.7 ± 3.4</td><td>68.70</td><td>0.30</td></tr><tr><td>Logistic Regression - tabular data</td><td>0.70</td><td>37.0 ± 0.0</td><td>31.90</td><td>0.31</td></tr><tr><td>XGBoost - tabular data</td><td>0.70</td><td>22.0 ± 0.0</td><td>18.97</td><td>0.23</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.70</td><td>37.7 ± 9.1</td><td>32.50</td><td>0.31</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.70</strong></td><td><strong>67.9 ± 5.5</strong></td><td><strong>58.53</strong></td><td><strong>0.39</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.70</td><td>67.7 ± 4.9</td><td>58.36</td><td>0.39</td></tr><tr><td>Logistic Regression - tabular data</td><td>0.60</td><td>33.0 ± 0.0</td><td>28.45</td><td>0.33</td></tr><tr><td>XGBoost - tabular data</td><td>0.60</td><td>10.0 ± 0.0</td><td>8.62</td><td>0.14</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.60</td><td>25.3 ± 11.4</td><td>21.81</td><td>0.28</td></tr><tr><td>Logistic Regression - transformer</td><td>0.60</td><td>54.8 ± 3.6</td><td>47.24</td><td>0.43</td></tr><tr><td><strong>Transformer fine-tuning</strong></td><td><strong>0.60</strong></td><td><strong>55.7 ± 3.4</strong></td><td><strong>48.02</strong></td><td><strong>0.43</strong></td></tr><tr><td>Logistic Regression - tabular data</td><td>0.50</td><td>16.0 ± 0.0</td><td>13.79</td><td>0.22</td></tr><tr><td>XGBoost - tabular data</td><td>0.50</td><td>1.0 ± 0.0</td><td>0.86</td><td>0.02</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.50</td><td>16.4 ± 11.8</td><td>14.15</td><td>0.22</td></tr><tr><td>Logistic Regression - transformer</td><td>0.50</td><td>42.1 ± 5.4</td><td>36.29</td><td>0.42</td></tr><tr><td><strong>Transformer fine-tuning</strong></td><td><strong>0.50</strong></td><td><strong>44.6 ± 6.2</strong></td><td><strong>38.45</strong></td><td><strong>0.43</strong></td></tr><tr><td>Logistic Regression - tabular data</td><td>0.40</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>XGBoost - tabular data</td><td>0.40</td><td>1.0 ± 0.0</td><td>0.86</td><td>0.02</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.40</td><td>8.5 ± 6.2</td><td>7.33</td><td>0.13</td></tr><tr><td>Logistic Regression - transformer</td><td>0.40</td><td>23.6 ± 12.8</td><td>20.34</td><td>0.30</td></tr><tr><td><strong>Transformer fine-tuning</strong></td><td><strong>0.40</strong></td><td><strong>26.7 ± 14.5</strong></td><td><strong>23.02</strong></td><td><strong>0.33</strong></td></tr><tr><td>Logistic Regression - tabular data</td><td>0.30</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>XGBoost - tabular data</td><td>0.30</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.30</td><td>5.5 ± 4.0</td><td>4.74</td><td>0.089</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.30</strong></td><td><strong>12.3 ± 11.6</strong></td><td><strong>10.60</strong></td><td><strong>0.18</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.30</td><td>10.2 ± 9.2</td><td>8.79</td><td>0.15</td></tr><tr><td>Logistic Regression - tabular data</td><td>0.20</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>XGBoost - tabular data</td><td>0.20</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.20</td><td>2.6 ± 1.9</td><td>2.41</td><td>0.04</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.20</strong></td><td><strong>4.0 ± 5.0</strong></td><td><strong>3.45</strong></td><td><strong>0.06</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.20</td><td>0.8 ± 2.5</td><td>0.69</td><td>0.01</td></tr><tr><td>Logistic Regression - tabular data</td><td>0.10</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>XGBoost - tabular data</td><td>0.10</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Logistic Regression - LSTM</td><td>0.10</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr><tr><td><strong>Logistic Regression - transformer</strong></td><td><strong>0.10</strong></td><td><strong>0.9 ± 2.8</strong></td><td><strong>0.78</strong></td><td><strong>0.01</strong></td></tr><tr><td>Transformer fine-tuning</td><td>0.10</td><td>0.0 ± 0.0</td><td>0.0</td><td>0.0</td></tr></tbody></table>