<figure><img src="image_1.png" /></figure>

HAL
open science

# Self-Attention Mechanism in Multimodal Context for Banking Transaction Flow

Cyrile Delestre, Yoann Sola

<figure><img src="image_2.png" /></figure>

## To cite this version:

Cyrile Delestre, Yoann Sola. Self-Attention Mechanism in Multimodal Context for Banking Transaction Flow. 2024. hal-04584517v2

HAL Id: hal-04584517

https://hal.science/hal-04584517v2

Preprint submitted on 13 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Self-Attention Mechanism in Multimodal Context for Banking Transaction Flow

Cyrile Delestre

Crédit Mutuel Arkéa

Le Relecq-Kerhuon, France

cyrile.delestre@arkea.com

Yoann Sola

Crédit Mutuel Arkéa

Le Relecq-Kerhuon, France

yoann.sola@arkea.com

*Abstract*—Banking Transaction Flow (BTF) is a sequential data found in a number of banking activities such as marketing, credit risk or banking fraud. It is a multimodal data composed of three modalities: a date, a numerical value and a wording. In this work, we propose a self-attention-based approach for modeling BTFs. We trained two general models on a large amount of BTFs in a self-supervised way: one RNN-based model and one Transformer-based model. We proposed a specific tokenization in order to be able to process BTFs. The performance of these two models was evaluated on two banking downstream tasks: a transaction categorization task and a credit risk task. The results show that fine-tuning these two pre-trained models allowed to perform better than the state-of-the-art approaches for both tasks.

*Index Terms*—Self-Attention Mechanism, Banking Transaction Flow, Transformer, RNN, Multimodal, Credit risk

## I. INTRODUCTION

The use of machine learning (ML) in banking has grown significantly in recent years, particularly in areas involving customer behavior and risk assessment.

In a marketing context, the information we want to extract from banking transactions is the type of spending habits or the household income. This information allows us to advise or categorize consumers according to a commercial or customer knowledge objective. Nowadays, most banks offer Personal Financial Management (PFM), which can help the clients to improve financial management, through a personalized view of finances and advice, without having any knowledge.

Since the last global financial crisis, risk management in banking has become highly regulated to limit systemic risk and preserve the integrity of the financial system [5] [8]. Among these risks, credit risk is closely monitored by regulatory authorities and is central to bank solvency [28]. At loan origination, banks compute a default score to balance commercial objectives with regulatory requirements, typically applying an acceptance threshold to this score.

Among the most informative data sources for such applications are Banking Transaction Flows (BTFs), which provide structured, sequential traces of financial activity. Due to their sensitivity and granularity, BTFs are a reliable and difficult-to-manipulate source of customer information.

In this work, we focus on the challenge of modeling BTFs in their full complexity. A BTF is inherently multimodal: each transaction consists of a date, a numerical amount, and a free-text wording. Traditional approaches often simplify BTFs by

applying handcrafted feature extraction and discarding either the temporal structure or some modalities, leading to a loss of information and suboptimal performance.

We propose a general-purpose, pre-trained representation for BTFs that maintains their sequential and multimodal nature. Our contributions are threefold:

* We introduce a novel tokenization strategy that jointly encodes the three modalities of a transaction (date, amount, wording), designed to be compatible with PSD2 standards.

* We develop and compare two architectures for pretraining on large volumes of unlabeled BTF data: a bidirectional RNN with attention and a Transformer encoder, both trained with multimodal self-supervised objectives.

* We evaluate these models on two downstream tasks representative of real-world banking applications: transaction categorization and credit risk scoring. In both cases, fine-tuning the pretrained models outperforms strong baselines.

This work is motivated by the emergence of Open Banking [9] and the increasing standardization of financial data under PSD2 (Payment Services Directive 2) [14] regulation. We aim to build generic, reusable representations of BTFs that can benefit a wide range of financial institutions and services.

## II. RELATED WORK

We found several application of machine learning to banking activities in the literature, such as marketing segmentation [38] [40], general banking risk management [28] and credit risk [16] [6] [18]. Operational risks were also dealt by several works, e.g. the detection of fraudulent transactions [43] [34] or money laundering [26].

Some works successfully tried to use machine learning in temporal point processing [44] [39], allowing to model event sequences in continuous time space.

We also found interesting research about deep learning applied to BTF modelling: [2] used contrastive learning inside a self-supervised learning process, [3] applied RNN for a credit loan use case, and [15] made adversarial attacks on deep learning models of transaction records. These works are not based on the same framework as our work (the PSD2 framework), and often include more features than our BTF definition, e.g. the Merchant Category Code (MCC).
The self-attention mechanism first appeared in the Natural Language Processing (NLP) field [4]: the words are decomposed in tokens (e.g. subwords) and the attention allows to indicate the semantic links between all the tokens from a given sequence (a sentence or a paragraph). Each token is processed with respect to the context around it (i.e. all the tokens before and after). The attention mechanism quantifies the relationship between events within a given sequence (the model is then called an encoder) or between two sequences (a cross-encoder). In the literature, it can be found associated with RNN [30] or inside the Transformer architecture [42]. The attention mechanism also appeared in several other fields such as image processing [13] or audio processing [17]. It also started to be used in banking use cases: in credit card fraud detection [7], in credit risk [29], in stock price prediction [10], as well as for general representation of BTF [32].

All these works are promising and shows a real interest of the deep learning community in the banking areas. However, we did not find deep learning modelling approaches based on the PSD2 definition of BTF. We will see that the use of self-attention mechanism can fulfill these need, allowing to create a useful generic model in the context of open banking.

### III. BANKING TRANSACTION FLOW

BTF represents all the events of banking transactions and a transaction is an event carried out on a bank account of a natural person (as opposed to a legal person). This transaction represents a bank transfer, a withdrawal from an Automated Teller Machine (ATM), a check issue or remittance, a purchase from a Point of Sale (POS), etc. The scope defined in the PSD2 framework is the events set that occur on the current account (also called checking account). An event is represented by three modalities:

1. The transaction processing date is the date the transaction was taken into account and is officially reflected in the customer's account maintenance. The date is only accurate to the day and, depending on the channel through which the event transited, may have a one or two day delay between the action taken by the customer and the official presence on his account. This date represents the first modality and offers information on the chronology of events on a monthly scale (the year scale will not be taken into account in this paper). These are therefore macro-ordered events but disordered in the daily temporality;

2. The second modality is the amount associated to the transaction, which represents the transaction value. This is a real number and the sign indicates the transaction direction (debit or credit). Hereafter, this value is in euros, but it can be in any other unit;

3. Finally, the third and last modality is the wording that accompanies the transaction. This is rich information that indicates the transaction channel (ATM, check, etc.) but also includes information that may be of personal origin (wording instructed by the client for debit transfers, for example) or organizational (wording instructed

by a third party for credit transfers or purchases via a POS, for example).

In the following section we will present the preprocessing and transformations done to convert a multi-modals events serie into a sequence compatible with and processable by our two models.

The figure 1 shows a summary of the whole approach: the tokenization, the models and the sub-tasks of the pre-training process. It should clarify the explanations made throughout the following sections.

#### A. Preprocessing

Preprocessing is a very important step, allowing to normalize data. Particular care will be taken for the textual modality contained in the wordings of the BTFs: it is an unstructured and nonstandard field. The variability encountered in the wordings can be non-informational, adding noise to the input, such as check or ATM withdrawal numbers, or even irrelevant information, such as dates, information already carried by the transaction date modality. These parts will be replaced by tags to greatly reduce the non-informational wordings diversity. All the characters will also be put in lower case and special characters and accents will be removed, in order to not be case sensitive. Table I shows some real examples of what can be found as wordings and their associated normalizations.

#### B. Tokenization

A key phase of the modeling is to build the morphosyntax of BTF, i.e. building a syntax and a dictionary relevant to the sequence of events (banking transactions) found in our data. The adopted strategy is first of all a daily ordering of the events by ascending amounts. In this way, the amounts embedding representation will guarantee the intra-daily position encoding and the amount information.

Regarding the wording modality, a special case appears with the use of the “space” character. When two events are juxtaposed together (in cross-encoding tasks for example), two separation types can be found: the extra-wording spaces symbolizing the separation between two events, and the intra-wording spaces symbolizing the words separation within a given event wording. To account for this specificity, two different encodings are needed for these two separation types. Thus, if we denote the extra-wording separator by □ and the intra-wording separator by ◦, and taking the examples from table I and assuming that it is already correctly ordered, the wordings sequence becomes:

“□chq<digits>□ret< dab<digits>plancoet
□vir>pole>emploi>bretagne<date>
□carte<date>leclerc>brest<empty>”;

In this work, we will use the following encodings: □ = U+2581 and ◦ = Ø. The decision not to encode the intra-word separator character was made to allow for more efficient tokenization of multi-word expressions. The aim is that when a specific multi-word sequence appears frequently in the data —
<figure><img src="image_4.png" /></figure>

<table><caption>Fig. 1. Global models diagram and their pre-training heads.</caption><thead><tr><th>Type</th><th>Raw data</th><th>Preprocessed data</th></tr></thead><tbody><tr><td>Digits</td><td>CHQ 2141367<br>RET DAB 351267 PLANCOET</td><td>chq &lt;digits&gt;<br>ret dab &lt;digits&gt; plancoet</td></tr><tr><td>Date</td><td>VIR POLE EMPLOI BRETAGNE 08/21<br>CARTE 08/10 LECLERC BREST</td><td>vir pole emploi bretagne &lt;date&gt;<br>carte &lt;date&gt; leclerc brest</td></tr><tr><td>Other</td><td>@!_</td><td>&lt;empty&gt;</td></tr></tbody></table>

<caption>TABLE I
PATTERN DETECTION AND WORDING NORMALIZATION.</caption>

such as company names or other recurring phrases — a unique token representing the entire sequence can be created in the dictionary. If a separator were retained between the words, it would appear in every instance and eventually be assigned its own token. This would prevent the formation of a single token for the full sequence, as it would be split into multiple tokens. The SentencePiece Unigram algorithm was chosen [25], [24] for the creation of the dictionary $\mathcal{X}$ of BTF wording. Although T. Kudo and J. Richardson [24] did not notice any significant performance difference between a Byte-Pair-Encoding (BPE) [37] and the Unigram, recent work [45] revealed that a Unigram tokenizer has a better behavior on corpora not dealing with the same information, showing a more generalizing aspect of the created dictionary. This behavior seems to be interesting in the case where the bank flow comes from another organization than the one in which the modeling was not trained. The chosen dictionary size is 7k words and the dictionary has been trained on 1 million sequences composed of 1 month of banking operations.

The wording encoding will drive the other tokenizers. For that, we introduce 3 control tokens that will be used later: [BOS] to mark the beginning of a sequence, [EOS] marking the end of a sequence and [SEP] to mark the separation between two sequences. The latter will be used afterwards to contextualize a Natural Language Inference (NLI) problem as a separation marker between a “premise” sequence and a “hypothesis” sequence. Thus, we have two cases, mono-sequence and bi-

sequence, where here are the possible schemes:

$$ \text{monoSeq} = [\text{BOS}] \text{Seq} [\text{EOS}] \qquad (1) $$

$$ \text{biSeq} = [\text{BOS}] \text{Seq}_1 [\text{SEP}] \text{Seq}_2 [\text{EOS}] \qquad (2) $$

The amounts tokenizer is composed of a dictionary $\mathcal{A}$ of 2.5k elements. This tokenizer is composed of 3 quantifiers divided into three quantization zones: a linear zone and two exponential zones. In order to create these quantizers and to make them representative of a certain ground reality, 1 million operations amounts were taken randomly to define the different zones boundaries. The linear quantizer extrema were chosen at the first amounts quantile for the low boundary and at the 99th quantile for the high boundary. These limits represent - 1750 and +2760 euros and are quantized on 1250 quantization steps. In order to limit the tokenizer saturation, two quantizers exponential composed of 625 steps have been placed at the extremities of the linear quantizer going to the extrema values of saturation chosen at $\pm 100k$ euros. Figure 2 illustrates the amounts tokenizer composition. In order to remain coherent with the wordings tokenizer and that the amounts information is reflected on the entire wordings, the amount associated to a transaction will be repeated throughout the decomposition in sub-elements of the wordings tokenization.

For the date modality, a normalization of the day in the month is performed, which allows to bypass the heterogeneity of the month lengths. Then, a linear quantizer allows to quantize the day in the month on 30 quantization steps representing a dictionary size (named $\mathcal{D}$) of 30 elements. Finally, as for

<footnote>* - RNN: $e_{NSP} = e_{BOS} + e_{EOS}$ - Transformer: $e_{NSP} = e_{BOS}$</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 2. Scheme of the three quantizers composing the tokenizer of the amounts. Transaction amount is represented as a function of ids tokens or steps.</figcaption></figure>

the amount tokenizer, the elements will be repeated along the decomposition of the wordings into sub-wordings.

Finally, a last tokenizer is used to mark the identity of the sequence in an NLI context. Indeed, if the sequence is the premise then it will be marked by a token identity of 0 and, if it represents the hypothesis it will be marked by the identity 1. In the mono-sequence case (eq.1) all the elements will have an identity of 0.

To conclude this part, a representation of all these preprocessing and tokenization elements are represented in figure 1 in the part upper labelled “pre-processing”. In the next section we will discuss in more detail the two models used in this article.

## IV. MODELIZATION

In this part we will discuss the technical aspect of the two models. Firstly, the embedding part allowing to encode the tokens is identical for both models. The goal of this step is to create a latent representation of tokens. The input of the embedding step is a token and the output is a vector with $d$ dimensions. The dimension of the embedding representation is the same for all the modalities. In the following, we will consider that a sequence is composed of $N$ events. We note respectively $\mathbf{X}$, $\mathbf{A}$, $\mathbf{D}$ and $\mathbf{T}$ all in $\mathbb{R}^{d \times N}$, the latent representation of the tokens sequence composing the wordings, the amounts representation, the temporal modality (representing days) and the identity.

At each sequence event, only the wordings tokens are different. In order to carry the information in a uniform way from each modality to each event, the final embedding representation will be the sum of each modality contributions. Thus, for a wording decomposed into sub-wordings, the other modalities will add a common bias to all events composing it. We note this representation as follows:

$$ \mathbf{E}_0 = \mathbf{X} + \mathbf{A} + \mathbf{D} + \mathbf{T} \quad (3) $$

$$ \mathbf{E}_0 = [\mathbf{e}_{0,0}, \dots, \mathbf{e}_{0,n}, \dots, \mathbf{e}_{0,N-1}] \in \mathbb{R}^{d \times N} \quad (4) $$

Finally, the parameters number in the modeling embedding part can be summarized as follows:

$$ p_{\text{emb}} = d \times (|\mathcal{X}| + |\mathcal{A}| + |\mathcal{D}| + 2) \quad (5) $$

### A. Recurrent Neural Network

The first model is an implementation of the historical architecture for this type of problem, i.e. RNN. It is built

around a bidirectional recurrent network architecture [36] and the final embedded representation is inspired by the ELMo modeling [33]. The bidirectional layers allows to get rid of the events causality, so the $n$-th event will be influenced by the events preceding and following it.

The RNN structure used is a Long Short Term Memory (LSTM) [21] with a projection allowing to have a recurrent network with an internal latency representation larger than the output one. This strategy has already shown some effectiveness in some applications such as speech recognition [35]. The internal dimension of the recurrent model is denoted $h$ and the external one is our event representation dimension $d$ with $h > d$.

Both directions of the LSTM are composed of $L$ layers and a layer normalization [1] is applied to each layer. The representations computed by the two directions are then given to an attention layer. The goal of this layer is to compute the relations between each token representation, using the self-attention mechanism [4].

This approach introduced by ELMo [33] has two advantages. Firstly, it limits the vanishing gradient effect through the network layers and secondly, it allows to have a model which will make each embedding representation of the network directly contributing to the output. And since these contributions are made of trainable parameters, the model will adapt to the downstream task and choose the optimal abstraction level of representation for this task. Indeed, the lower the layer level, the less the interactions between the different events are taken into account and vice versa. In the field of NLP it has already been shown that, for some tasks, the abstraction level can play an important role in the task performance [23]. Finally, the parameters number of the RNN modeling part can be written as:

$$ p_{\text{rnn}} = 2L \times (9dh + 8h + 2d) + L + 1 \quad (6) $$

### B. Transformer

The Transformer architecture introduced in [42] made an original use of the attention mechanism, allowing to remove several limitations. Until this architecture, the events ordering was preserved and was primordial for the RNN-based architectures. The information of the ordering is no more necessary for the Transformer model as this neural network architecture is articulated around a functional memory (based on the attention mechanism) that will essentially react according to the presence or absence of events. The ordering becomes secondary and relations between distant events are easier on this type of structure.

We used the classical Transformer architecture, with a number $L$ of layers. Each layers is composed of two sublayers:

* A multi-head attention layers allowing to compute the relations between the tokens representations in different
sub-space: a different vector space is used for each head, learnt during the training process. This sub-layer is composed of *J* heads.

* A feed-forward neural network composed of one hidden layer with a dimension *h*, with *h* > *d*. Contrary to the original paper, the activation function is a GELU [19], more efficient during the learning phase than a ReLU activation. This activation function is relatively common for this type of modeling, [12], [31]. A layer normalization [1] is also applied to this neural network.

The output of both sub-layers is sequence of vectors of dimension *d*.

Finally, as for the embedding and RNN parts, here are the parameters number of the Transformer modeling:

$$p_{tf} = L \times (4d^2 + 2dh + 9d + h) \quad (7)$$

## V. PRE-TRAINING

In this section we will discuss the architecture parameters of the two models, the cost function used for training, and the training parameters.

In table II are specified the models parameters that remained free until then and that allow to define the final topology. The layers number *L* of the RNN network is the one used in the ELMo architecture [33], the other parameters are inspired by BERT architecture [12]. Thus, using equations 5, 6 and 7, it is possible to determine the parameters number for each of these two networks. It is interesting to note that the parameters number is very close between the two models, representing an equivalent complexity level. Thus the differences in performance measurements cannot be attributed to one model being more complex than the other.

### A. Dataset Description

The pre-training dataset consists of 2 million anonymized customer BTF sequences, each spanning two consecutive months of banking activity. This results in a training corpus of approximately 1.29 billion tokens and 70 GB of processed transactional data. Each customer contributes a premise (month 1) and a hypothesis (month 2) sequence, enabling the Next Sequence Prediction (NSP) task by pairing either a real continuation or a randomly sampled sequence from another customer. This setup allows the generation of up to 4 trillion distinct sequence pairs for training.

On average, a customer has 55±31 transactions per month, corresponding to a mean of 214±135 tokens after tokenization. The high variability in transaction counts justifies the use of attention-based models that are robust to sequence length.

Importantly, the dataset provides a favorable tokens-to-parameters ratio of approximately 15:1, with 1.29B tokens used to train models of 92M parameters. This aligns well with the Chinchilla scaling law [22], which recommends such ratios for optimal model efficiency and generalization.

<table><caption>TABLE II
PARAMETERS FOR BOTH STRUCTURES.</caption><thead><tr><th>ATTRIBUTE</th><th>RNN</th><th>TRANSFORMER</th></tr></thead><tbody><tr><td><i>d</i></td><td></td><td>768</td></tr><tr><td><i>h</i></td><td></td><td>3072</td></tr><tr><td><i>J</i></td><td>Ø</td><td>12</td></tr><tr><td><i>L</i></td><td>2</td><td>12</td></tr><tr><td>LAYER NORM. EPS.</td><td></td><td>1 · 10<sup>-5</sup></td></tr><tr><td>PARAM. NB. EQ.(5, 6, 7)</td><td></td><td>92M</td></tr></tbody></table>

### B. Pre-training Tasks

The pre-training strategy consists in 3 subtasks that we will detail:

**Masked Wording Model (MWM):** is similar to BERT's Masked Language Model (MLM) but focused on transaction wordings. It consists in training the model to estimate the dynamically masked wording during the training phase. The probability $p_{MWM}$ represents the proportion of hidden words in the training sequences. Masking is done at the wording level and not at the subwordings level in the tokenization output.

**Masked Amount Model (MAM):** which consists in estimating the dynamically masked amounts. The probability $p_{MAM}$ symbolizes the proportion of the masked amounts and the amounts estimation is done at the wording level. Thus, if an amount is associated to a wording is masked, the mask is repeated as many times as the wording is broken down by the tokenization.

**Next Sequence Prediction (NSP):** this NLI task is similar to the Next Sentence Prediction (NSP) task in BERT. This task defines the sequences strategy (e.g. eq.2) as input to the modeling for pre-training. Thus the first sequence will be one transaction month and the second sequence the continuation of the second month of an individual, or not. This sequence continuity probability will be modeled by $p_{NSP}$.

The loss pre-training function is modeled by the sum of the three subtasks cross-entropies:

$$\mathcal{L}(p, y) = \mathrm{CE}_{\mathrm{MWM}}(p, y) + \mathrm{CE}_{\mathrm{MAM}}(p, y) + \mathrm{CE}_{\mathrm{NSP}}(p, y) \quad (8)$$

This is a hard label type of modeling ($y \in \{0, 1\}$). Finally, the parameters used for learning are shown in table III. The learning rate strategy improves the training performance on complex networks [41].

## VI. PERFORMANCES AND DOWNSTREAM TASKS

In this experimentation part, we will discuss the models performance in terms of execution time and RAM consumption. We will finish with two downstream tasks on two very different subjects allowing to notice the good behavior of the modeling on very diverse tasks.

### A. Hardware Performance

The two models have a very different topological nature and therefore different behaviors with respect to parallelization and RAM consumption. Knowing these characteristics allows
<table><caption>TABLE III
HARDWARE USED, PRE-TRAINING PARAMETERS AND PRE-TRAINING TIMES.</caption><thead><tr><th>ATTRIBUTE</th><th>RNN</th><th>TRANSFORMER</th></tr></thead><tbody><tr><td>HARDWARE</td><td>8×GPU NVIDIA A100 40GB</td><td></td></tr><tr><td>SET OBS./TOK. NB.</td><td>3M/1.29G ~ 70GB</td><td></td></tr><tr><td>MAX. SEQ. TOK. NB.</td><td>1500</td><td>800</td></tr><tr><td>EPOCH NB.</td><td></td><td>50</td></tr><tr><td>MINI-BATCH SIZE</td><td></td><td>32</td></tr><tr><td>GRADIENT ACCU.</td><td></td><td>32</td></tr><tr><td>p<sub>dropout</sub></td><td></td><td>0.1</td></tr><tr><td>p<sub>MWM</sub></td><td></td><td>0.05</td></tr><tr><td>p<sub>MAM</sub></td><td></td><td>0.05</td></tr><tr><td>p<sub>NSP</sub></td><td></td><td>0.5</td></tr><tr><td>WEIGHT DECAY</td><td></td><td>0.01</td></tr><tr><td>LEARNING RATE</td><td>6D 3H37MIN</td><td>5D 13H18MIN</td></tr><tr><td>TRAINING TIME</td><td></td><td></td></tr></tbody></table>

<figure><img src="image_6.png" /></figure>

to better choose the architecture according to the needs and constraints (real time, batch computing, RAM limit, computing power limit, etc.).

Table IV summarizes the execution performances of the two models as a function of the computing cores number and on GPU. We can see that with a low parallelization, the RNN architecture is more efficient than Transformer. This may be due to the fact that, in a mini-batch context, padding is calculated for the Transformer architecture whereas in the RNN architecture it is not. As the number of transactions variability between sequences can be very large this may explain the advantage of the RNN structure. However, the Transformer structure much better supports parallelization with an inverse linear relationship between the cores number and the execution time. Therefore, on infrastructures with a lot of computing cores, the Transformer structure will be preferred.

Though, if the task requires the use of longer sequences (several months for example), the limit in RAM memory may come into account. The RNN structure being recurrent on the events it is thus very little consuming in RAM memory. It nevertheless requires all intermediate layers to calculate the final embedded representation which represents a memory consumption of:

$$RNN \sim dN \times (2L + 1) \sim o(N)$$

As for the modeling based on Transformers structures, only the last layer is necessary for the sequence embedding representation. However, internally the attention head operator is much more consuming in RAM, in particular due to the matrix product $Q^T K \in \mathbb{R}^{N \times N}$. So we could summarize the consumption of this structure by:

$$Transformer \sim dN + N^2 \sim o(N^2)$$

We can see that for small sequences the RNN structure will consume more RAM. However the relation between the

sequence size and the consumption remains linear whereas for the Transformer structure it is in power 2. So we have two possible strategies: either for tasks requiring long sequences we will prefer the RNN structure or we will choose a sequence truncation appropriate to the amount of RAM memory.

## B. Downstream Tasks

<figure><img src="image_7.png" /><figcaption>Fig. 3. Structure type with an encoder. The green (resp. blue) boxes represent the first (resp. second) sequence, the ovals the attention process and the red boxes the output of the models.</figcaption></figure>

Figure 3 summarizes the set of processes that can be performed with an encoder. Each of these categories represents one type of downstream task discussed in the introduction. For example, in the encoder case `seq2vec`, we find the credit risk scoring task; in the `seq2seq` setting, we address transaction categorization.

Although cross-encoders are not covered in this section, it is worth noting that the NSP pre-training task introduced earlier is a cross-encoder `seq2vec` task. It involves comparing two sequences jointly (a premise and a hypothesis) to predict their semantic continuity. This type of task can be useful in applications such as identity theft detection, where the model learns to detect whether a sequence of transactions is consistent with a user's historical behavior or not.

Finally, for the two downstream tasks studied in this article, the model input sequence will be composed of 2 months of BTF history. Indeed, in the PSD2 framework, no history depth is imposed to the banks. We have therefore chosen the minimum, allowing us to extract recurrent information between months.

**Transaction categorization:** in this task, we evaluated the ability of the models to categorize each individual bank transaction found inside BTFs, using the ground truth made by the internal PFM. Used for account management purposes, this transaction categorization system represents an imposing IT architecture (large databases, powerful calculation servers), reducing the portability of the solution. According to the need, models of different sizes could be used. We will compare our models to a Doc2Vec type approach [27] pre-trained on 200k bank wordings after normalization (Sec.III-A). Doc2Vec allows us to have an embedded representation of each bank wording. In order to integrate the other modalities we add the amount and the month day normalized as input to a Gradient
<table><caption>TABLE IV</caption><thead><tr><th>MODEL / (ms ± ms)</th><th>1 CORE</th><th>4 CORES</th><th>8 CORES</th><th>16 CORES</th><th>GPU</th></tr></thead><tbody><tr><td>RNN</td><td><b>897 ± 126</b></td><td><b>343 ± 37</b></td><td>273 ± 30</td><td>274 ± 45</td><td>30 ± 9</td></tr><tr><td>TRANSFORMER</td><td>1254 ± 360</td><td>397 ± 97</td><td><b>226 ± 53</b></td><td><b>149 ± 20</b></td><td><b>17 ± 11</b></td></tr></tbody></table>

AVERAGE COMPUTATION TIME IN MILESECONDS ON INTEL XEON CPUs @ 2.2GHz AND A NVIDIA A100 40GB GPU. THE CALCULATION WAS PERFORMED ON A SEQUENCE OF 1 MONTH OF BANK OPERATIONS OVER 500 OBSERVATIONS WITH A MINI-BATCH SIZE OF 25 (THE CALCULATION TIME IS THEREFORE DIVIDED BY 25 TO RETURN TO A SEQUENCE LEVEL).

<table><caption>TABLE V
PERFORMANCES ON OPERATION CATEGORIZATION TASK.</caption><thead><tr><th>MODEL / (%)</th><th>ACCURACY</th><th>RECALL</th><th>F1-SCORE</th></tr></thead><tbody><tr><td>Doc2Vec</td><td>62.5 ± 0.9</td><td>62.3 ± 0.9</td><td>62.3</td></tr><tr><td>RNN</td><td>62.0 ± 1.0</td><td>62.1 ± 1.0</td><td>61.9</td></tr><tr><td>TRANSFORMER</td><td>76.0 ± 0.8</td><td>76.1 ± 0.8</td><td>75.8</td></tr><tr><td>RNN F.T.</td><td><b>89.5 ± 0.6</b></td><td><b>89.5 ± 0.6</b></td><td><b>89.3</b></td></tr><tr><td>TRANSFORMER F.T.</td><td><b>90.4 ± 0.5</b></td><td><b>90.4 ± 0.5</b></td><td><b>90.2</b></td></tr></tbody></table>

<table><caption>TABLE VI
PERFORMANCES ON CREDIT RISK TASK.</caption><thead><tr><th>MODEL / (%)</th><th>ROC-AUC</th><th>ACCURACY</th><th>F1-SCORE</th></tr></thead><tbody><tr><td>GBDT</td><td>73.8 ± 2.2</td><td>67.9 ± 2.1</td><td>67.8</td></tr><tr><td>DL</td><td>68.4 ± 2.6</td><td>63.4 ± 2.4</td><td>63.3</td></tr><tr><td>RNN</td><td>80.2 ± 2.1</td><td>73.1 ± 2.2</td><td>73.1</td></tr><tr><td>TRANSFORMER</td><td><b>81.8 ± 2.1</b></td><td><b>73.7 ± 2.2</b></td><td><b>73.4</b></td></tr><tr><td>RNN F.T.</td><td><b>83.4 ± 2.0</b></td><td><b>76.5 ± 2.1</b></td><td><b>76.2</b></td></tr><tr><td>TRANSFORMER F.T.</td><td><b>84.4 ± 1.9</b></td><td><b>77.1 ± 2.1</b></td><td><b>77.0</b></td></tr></tbody></table>

Boosting Decision Tree (GBDT). Finally our models will be tested in the case where only the head layer is trained (not changing the network parameters) and in a case where the whole network is finetuned (F.T.). The training set was composed of 98 586 BTFs (corresponding to 12 048 822 operations), while 1 000 BTFs were kept for the test set (128 526 operations). Performances are summarized in table V. We can see that, from scratch, the Transformer model has a much better generalization power than the RNN model by clearly distinguishing itself from the reference Doc2Vec model by an accuracy gain of 10 points. But the important and somewhat unexpected result is that, after a finetuning phase, the two models offer almost identical performances with a gain of 28 points on the accuracy compared to the reference model.

**Credit risk:** consumer credit risk is a two-class classification task that involves determining a score that represents the default risk, calculated on the BTF as a whole. In order to keep a consistent methodology throughout this work, we will only used BTF data for this task. Other data sources such as socio-demographics, balance amounts, etc. will not be used, despite the fact that it generally increases the performance. The usual approach in this kind of problem is to have a first step of feature engineering in order to extract the maximum amount of information encapsulated in the BTFs (e.g. estimation of credit and debit recurrences, savings estimation, fragility detection, etc.), before passing these features through a model. Here, the GBDT is well suited for this task [18]. We also added a naive deep learning model attempting to jointly exploit the three modalities of the data to illustrate the task difficulty. The training set was composed of 6 361 BTFs, while 1 000 BTFs were kept for the test set (each BTF corresponds to a period of 2 months). Performances are summarized in table VI. First, we notice that the reference model (GBDT) have quite good performances and similar to the state-of-the-art [6], [18]. The naive model (DL) shows the difficulty to fully exploit such a data directly. The set of models presented in this paper shows real gains and we can draw the same conclusions as for the previous task.

## VII. CONCLUSIONS

In this work, we introduced a multimodal, self-supervised modeling approach for Banking Transaction Flows (BTFs), based on the self-attention mechanism. Our method preserves the sequential and multimodal structure of BTFs—combining dates, amounts, and textual wordings—via a novel tokenization strategy aligned with PSD2 standards. We developed two pretrained models (RNN-based and Transformer-based) trained on millions of transactions using three self-supervised objectives. Both models were evaluated on real-world downstream tasks: transaction categorization and credit risk scoring. In both cases, fine-tuning the pretrained models yielded strong improvements over standard approaches, demonstrating their generalization power. Notably, the Transformer architecture showed better zero-shot performance, while both architectures achieved similar results after fine-tuning. This suggests that the self-supervised representations are both robust and adaptable.

This work fits naturally into the Open Banking paradigm, offering reusable representations of BTFs that can be deployed across organizations. In future work, we plan to reduce model size and computational cost using knowledge distillation [20] and model quantization [11], enabling lightweight deployment in production environments. We also intend to explore interpretability methods to better understand the contribution of each modality and to make the decision process more transparent for banking practitioners.

## REFERENCES

[1] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, 2016.

[2] Dmitrii Babaev, Nikita Ovsov, Ivan Kireev, Maria Ivanova, Gleb Gusev, Ivan Nazarov, and Alexander Tuzhilin. Coles: Contrastive learning for event sequences with self-supervision. In *Proceedings of the 2022 International Conference on Management of Data*, SIGMOD '22, page 1190–1199, New York, NY, USA, 2022. Association for Computing Machinery.

[3] Dmitrii Babaev, Maxim Savchenko, Alexander Tuzhilin, and Dmitrii Umerenkov. E.t.-rnn: Applying deep learning to credit loan applications. In *Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, KDD '19, page 2183–2190, New York, NY, USA, 2019. Association for Computing Machinery.

[4] Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In *3rd International Conference on Learning Representations, ICLR 2015*, pages
01-15, Schloss Dagstuhl, Leibniz-Zentrum für Informatik, Oktavie-Allee,66687 Wadern, Germany, 2015. dblp.

[5] BCBS. Basel ii: International convergence of capital measurement and capital standards: A revised framework-comprehensive version. Technical report, Basel Committee on Banking Supervision, 2006.

[6] T. Bellotti and J. Crook. Support vector machines for credit scoring and discovery of significant features. *Expert Systems with Applications*, 36(2):3302–3308, 2009.

[7] Ibtissam Benchaji, Samira Douzi, Bouabid El Ouahidi, and Jaafar Jafari. Enhanced credit card fraud detection based on attention mechanism and lstm deep model. *Journal of Big Data*, 8:1-21, 2021.

[8] M. Billio, M. Getmansky, A. W. Lo, and L. Pelizzon. Econometric measures of connectedness and systemic risk in the finance and insurance sectors. *Journal of Financial Economics*, 104(3):535-559, 2012.

[9] Laura Brodsky and Liz Oakes. Data sharing and open banking. *McKinsey & Company*, 1105:01-08, 2017.

[10] Shun Chen and Lei Ge. Exploring the attention mechanism in lstm-based hong kong stock price movement prediction. *Quantitative Finance*, 19(9):1507-1515, 2019.

[11] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 () : 8-bit matrix multiplication for transformers at scale. corr abs/2208.07339 (2022), 2022.

[12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of NAACL-HLT*, pages 4171–4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.

[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weisenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929:01-22, 2020.

[14] European Commission, Council of the European Union, and European Parliament. Regulation (eu) 2015/2365 of the european parliament and of the council on transparency of securities financing transactions and of reuse and amending regulation. Technical Report L 337, vol 58, Official Journal of the European Union, December 2015.

[15] Ivan Fursov, Matvey Morozov, Nina Kaploukhaya, Elizaveta Kovtun, Rodrigo Rivera-Castro, Gleb Gusev, Dmitry Babaev, Ivan Kireev, Alexey Zaytsev, and Evgeny Burnaev. Adversarial attacks on deep models for financial transaction records. In *Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining*, KDD '21, page 2868-2878, New York, NY, USA, 2021. Association for Computing Machinery.

[16] J. Galindo and P. Tamayo. Credit risk assessment using statistical and machine learning: Basic methodology and risk modeling applications. *Computational Economics*, 15(1):107-143, 2000.

[17] Yuan Gong, Yu-An Chung, and James R. Glass. Ast: Audio spectrogram transformer. ArXiv, abs/2104.01778:01-05, 2021.

[18] S. Hamori, M. Kawai, T. Kume, Y. Murakami, and C. Watanabe. Ensemble learning or deep learning? application to default risk analysis. *Journal of Risk and Financial Management*, 11(1):12, 2018.

[19] D. Hendrycks and K. Gimpel. Gaussian error linear units (GELUs), 2016.

[20] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network, 2015.

[21] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.

[22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.

[23] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In *Annual Meeting of the Association for Computational Linguistics*, pages 3651–3657, Florence, Italy, 2019. Association for Computational Linguistics.

[24] T. Kudo and J. Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, 2018.

[25] Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. ArXiv, abs/1804.10959:01-10, 2018.

[26] D. V. Kute, B. Pradhan, N. Shukla, and A. Alamri. Deep learning and explainable artificial intelligence techniques applied for detecting money laundering-a critical review. *IEEE Access*, 9:82300-82317, 2021.

[27] Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In Eric P. Xing and Tony Jebara, editors, *Proceedings of the 31st International Conference on Machine Learning*, Proceedings of Machine Learning Research, pages 1188-1196, Beijing, China, 22-24 Jun 2014. PMLR.

[28] M. Leo, S. Sharma, and K. Maddulety. Machine learning in banking risk management: A literature review. *Risks*, 7(1):29, 2019.

[29] Jingyuan Li, Caosen Xu, Bing Feng, and Hanyu Zhao. Credit risk prediction model for listed companies based on cnn-lstm and attention mechanism. *Electronics*, 12(7):01-18, 2023.

[30] Wenbin Liu, Bojian Wen, Shang Gao, Jiesheng Zheng, and Yinlong Zheng. A multi-label text classification model based on elmo and attention. In *MATEC Web of Conferences*, volume 309, page 03015, Online, 2020. EDP Sciences.

[31] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019.

[32] Naoto Minakawa, Kiyoshi Izumi, Hiroki Sakaji, and Hitomi Sano. Graph representation learning of banking transaction network with edge weight-enhanced attention and textual information. In *Companion Proceedings of the Web Conference 2022, WWW '22*, page 630-637, New York, NY, USA, 2022. Association for Computing Machinery.

[33] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. ArXiv, abs/1802.05365:01-15, 2018.

[34] J. Pun and Y. Lawryshyn. Improving credit card fraud detection using a meta-classification strategy. *International Journal of Computer Applications*, 56(10):41-46, 2012.

[35] H. Sak, A. Senior, and F. Beaufays. Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition, 2014.

[36] M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. *IEEE Transactions on Signal Processing*, 45(11):2673-2681, 1997.

[37] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith, editors, *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics.

[38] M. J. Shaw, C. Subramaniam, G. W. Tan, and M. E. Welge. Knowledge management and data mining for marketing. *Decision Support Systems*, 31(1):127-137, 2001.

[39] Oleksandr Shchur, Ali Caner Türkmen, Tim Januschowski, and Stephan Günnemann. Neural temporal point processes: A review. ArXiv, abs/2104.03528:01-09, 2021.

[40] I. Smeureanu, G. Ruxanda, and L. M. Badea. Customer segmentation in private banking sector using machine learning techniques. *Journal of Business Economics and Management*, 14(5):923-939, 2013.

[41] Leslie N. Smith and Nicholay Topin. Super-convergence: very fast training of neural networks using large learning rates. In Tien Pham, editor, *Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications*, volume 11006, pages 369-386, Baltimore, MD, United States, 2019. International Society for Optics and Photonics, SPIE.

[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, *Advances in Neural Information Processing Systems*, volume 30, pages 01-11, Long Beach, CA, USA, 2017. Curran Associates, Inc.

[43] B. Wiese and C. Omlin. Credit card transactions, fraud detection, and machine learning: Modelling time with lstm recurrent neural networks. In *Innovations in neural information paradigms and applications*, pages 231-268. Springer, Berlin, Germany, 2009.

[44] Junchi Yan. Recent advance in temporal point process: from machine learning perspective. *SITU Technical Report*, Thinklab:01-07, 1997.

[45] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: pre-training with extracted gap-sentences for abstractive summarization. In *Proceedings of the 37th International Conference on Machine Learning*, ICML'20, Online, 2020. JMLR.org.