Multimodal Banking Dataset: Understanding Client Needs through Event Sequences

Authors: Dzhambulat Mollaev, Alexander Kostin, Postnova Maria, Ivan Karpukhin, Ivan A Kireev, Gleb Gennadjevich Gusev, Andrey Savchenko
Primary area: datasets and benchmarks
Keywords: multimodal, multi-temporal, event sequence

Abstract:
Financial organizations collect a huge amount of data about clients that typi-
cally has a temporal (sequential) structure and is collected from multiple sources
(modalities). However, despite the urgent practical need, developing deep learn-
ing techniques suitable to handle such data is limited by the absence of large open-
source multi-source real-world datasets of event sequences. To fill this gap mainly
caused by security reasons, we present the industrial-scale publicly available mul-
timodal banking dataset, MBD, that contains more than 2M corporate clients with
several data sources: 950M bank transactions, 1B geo position events, 5M em-
beddings of dialogues with technical support and monthly aggregated purchases
of four bank’s products. All entries are properly anonymized from real proprietary
bank data. Moreover, we introduce a novel multimodal benchmark incorporating
our MBD and two open-source financial datasets. We provide numerical results
demonstrating the superiority of fusion baselines over single-modal techniques
for each task. Moreover, our anonymization techniques still save all significant
information for introduced downstream tasks.

Decision: reject
Reviewer scores: 6, 8, 3, 5

Reviews and discussion:
<thread>
creation date:
1730140681802

thread_type:
official_review

<thread_object>
creator: reviewer_pgwk

<content>
summary:
The paper introduces a new multimodal dataset, MBD, from over 2 million corporate clients, including bank transactions, geo-locations, technical support dialogues, and monthly aggregated purchases of four banking products. The authors also present a benchmark for evaluating models on this dataset and two other financial datasets, focusing on tasks like purchase prediction and multimodal matching.

strengths:
- Large-scale dataset: The MBD dataset is the largest of its kind, offering a significant amount of data for research purposes.   

- Multimodality: The dataset incorporates various data modalities, providing a more comprehensive view of client behavior.   

- Practical tasks: The benchmark focuses on practically relevant tasks, such as purchase prediction, which can be useful for real-world applications.   

- Anonymization: The authors have taken steps to anonymize the data, protecting client privacy.

weaknesses:
- Lack of novelty: The paper primarily focuses on introducing a dataset and benchmark. For ICLR submissions, I'd see more emphasis on novel methods or algorithms. In addition, a careful evaluation of previous models (for example, the MMBench https://arxiv.org/abs/2307.06281) will bring more novelty to the community. 

- Missing comparison with LLMs: The paper lacks a comparison with more recent and powerful language models like GPT-4 or BloombergGPT, which have shown strong performance in various financial NLP tasks. For example, can we use prompt engineering to guide the GPT4o to process the transactions and geo-locations. 

- Unclear practical impact of the proposed metric: While the paper mentions that the benchmark can lead to financial benefits, I am curious if real world users care recalls/AUCs, or there are better metrics that map to financial success.

questions:
-  The paper could be strengthened by exploring more advanced multimodal fusion techniques beyond late fusion.
-  A more detailed analysis of the anonymization process and its potential impact on model performance would be beneficial.
-  The authors could consider expanding the benchmark to include other relevant tasks, such as risk assessment or fraud detection.

Overall, while the MBD dataset and benchmark are valuable contributions, the paper needs significant revisions to address the lack of novelty and provide a more convincing argument for the practical impact of their work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730507522577

thread_type:
official_review

<thread_object>
creator: reviewer_hmld

<content>
summary:
This paper introduces a large-scale publicly available multimodal banking dataset (Multimodal Banking Dataset, MBD). The MBD contains data from over 2 million banking customers, covering four different modalities: 9.5 million banking transaction records, 1 billion geographic location data points, 5 million customer interactions with technical support embedded dialogues, and banking product purchase activity over a 12-month period. This dataset provides researchers with a rich resource for analyzing customer behavior dynamics and contributes to the development of large-scale multimodal event sequence algorithms in the future.

strengths:
1.A large-scale multimodal banking dataset, MBD, is provided. This dataset contains anonymized banking transactions, geographic locations, and technical support dialogues, which contributes to the development of large-scale sequential event tasks in the future.
2.The dataset addresses privacy concerns through effective data anonymization, ensuring that the algorithm's performance is not significantly compromised.
3.The dataset and experimental code are publicly available, promoting transparency and reproducibility in research.

weaknesses:
1.The dataset's multimodal data includes banking transaction records, geographic locations, dialogue embeddings, and banking product purchase history. However, it appears that many of these modalities are essentially text-based. This differs from typical multimodal datasets, which include modalities such as video, audio, and text. 

2.The main contribution of the paper lies in the introduction of a large-scale dataset, but it lacks innovative methods for addressing related tasks. Additionally, the experimental section presents insufficient comparisons with current state-of-the-art methods. Overall, the paper's innovativeness needs improvement. I suggest that the authors include more experiments involving fully supervised methods [1][2]. Additionally, it would be beneficial to propose a simple and practical innovative approach based on their dataset.
[1]Learning Deep Time-index Models for Time Series Forecasting. [2]Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting

3.This paper contains numerous grammatical and tense issues. Additionally, the expression in the paper is not sufficiently clear, making it difficult for readers to accurately understand certain points and arguments. For example, on page three, phrases like “we selected” and “the dataset was collected”; on page five, “the coordinates were”; line 225 includes “we concentrated,” “whether it was,” and “random noise was added.” Additionally, on line 295, “we propose a downstream task — multimodal matching” (Zong et al., 2023).

questions:
1. Do you have any specific method to model the different text modalities?
2.The paper contains numerous tense and grammatical errors, which do not meet the standards expected for ICLR.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655510060

thread_type:
official_review

<thread_object>
creator: reviewer_e7p1

<content>
summary:
This is not a research contribution per se but it is a dataset paper.

The paper presents the first large-scale multimodal banking dataset for the user community. It offers a new dataset that has millions of users and millions of transactions that have been suitably anonymized. Moreover, the authors have provided baselines for a few standard tasks. The dataset will be released in order to spur research in the use of machine learning for banking applications.

strengths:
(1) This will be first and the largest multimodal banking dataset that will be released. This can potentially be tremendously useful to the research community.

(2) The baseline methods and benchmark data for a few problems outlined will also be immensely useful to the research community.

weaknesses:
(1) The details of the data are somewhat sparse. More details of each type of data will be useful to the reader. Perhaps this article may be useful for improving this aspect of the exposition in the paper: 

https://cacm.acm.org/research/datasheets-for-datasets/

(2)  Can some details of the anonymization be provided without compromising on the privacy of the customers? That can help estimate the errors of any model developed using this data.

(3) The data has been collected during the pandemic period? Will it have any effect on any of the conclusions drawn using the data? For examples, could this lead to systemic under-estimation or over-estimation of any phenomenon? Ideally, it would be useful to have another data-set which is outside of the pandemic period - perhaps the second version of this data set? This can serve as a basis for many natural experiments.

questions:
(1) Will the dataset be openly downloadable or will the authors be controlling the access? Openly downloadable option is obviously preferable.
(2) Will the source code of the bench-marking studies be openly available? 
(3) it will be useful to the community if the set of relevant real-world problems could be articulated that can inspire researchers to work on this dataset. Especially, in order to attract young researchers into the field.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730669726154

thread_type:
official_review

<thread_object>
creator: reviewer_ytbu

<content>
summary:
1. The paper introduces a new dataset, Multimodal Banking Dataset which integrates multiple modalities for over 2 million corporate clients.
2. The authors highlight the potential applications of this dataset for campaign planning and client behavior analysis, using multimodal benchmarks to demonstrate its value along with baseline model implementation.

strengths:
1. The work releases a first large scale banking dataset for public availability for financial applications. 
2. The authors present a good benchmark comparing unimodal and multimodal methods across various predictive tasks. The experimental protocol and metrics are clearly laid out as well.

weaknesses:
1. The authors do not explore or discuss advanced multimodal sequence models or advanced fusion techniques' cross-attention mechanisms as they can better capture interactions across modalities. They mention it at the end as a scope for future work.
2. Though authors discuss using AUC ROC as their metric for mitigating label imbalance issues for example in their campaigning downstream task, they do not discuss or incorporate any additional techniques for handling the label imbalance.
3. Details about anonymization techniques applied are mentioned in the paper but it lacks quantitative evaluation of the impact of these techniques on temporal dependencies within the data.

questions:
1. Is there a plan to expand the set of downstream tasks in future work? Highlighting a larger application list can increase the dataset's appeal across different financial research areas.

Please look into the weakness section for other questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732210272452

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you very much for your comments, which allowed us to address the shortcomings and refine the presentation of the paper.  Below we provide additional experimental results to be included in the paper. Some experiments are still in progress and will be finalized soon. We believe these updates significantly strengthen our work. The key updates are summarized below:

**1. LLM-based methods for our dataset.**

Initially, we experimented with several methods for integrating transactional data into LLMs, but their performance lagged significantly behind specialized baselines, so we did not plan to include these results in the paper. Using a relevant article from NeurIPS’23 (https://openreview.net/forum?id=md68e8iZK1), we conducted preliminary experiments with one of the leading LLMs, which is known to be one of the best models for the language of our dataset. We focused on transactional data from our MBD dataset. Transactions were transformed into text format, processed by the LLM to generate embeddings, and used as input for a gradient-boosting model. 
To speed up the evaluation during the rebuttal period, we considered predictions for the last month only. We examined two kinds of implementation: 1) the pre-trained LLM; and 2) a fine-tuned version of this LLM (setting similar to BloombergGPT). Zero-shot text-pretrained LLM showed considerably lower quality compared to CoLES, while domain fine-tuning improved performance of LLM, though it still did not surpass CoLES. An alternative method, ESQA (https://arxiv.org/abs/2407.12833), which employs a specialized encoder to structure transactional data, also failed to outperform baseline.

|                | target_1      | target_2      | target_3      | target_4      |
|----------------|---------------|---------------|---------------|---------------|
| CoLES + LLM FT | 0.708 ± 0.008 | 0.799 ± 0.030 | 0.777 ± 0.010 | 0.881 ± 0.004 |
| CoLES          | 0.690 ± 0.012 | 0.794 ± 0.036 | 0.756 ± 0.015 | 0.878 ± 0.004 |
| LLM FT         | 0.601 ± 0.023 | 0.676 ± 0.032 | 0.706 ± 0.017 | 0.819 ± 0.007 |
| LLM zero-shot  | 0.552 ± 0.019 | 0.642 ± 0.010 | 0.699 ± 0.012 | 0.770 ± 0.013 |
| ESQA           | 0.592 ± 0.018 | 0.704 ± 0.028 | 0.558 ± 0.013 | 0.777 ± 0.014 |

Moreover, we discovered that LLM methods can extract new valuable features from transactional data, as confirmed by the improved downstream task performance (up to 2% comparing to CoLES) when LLM embeddings were concatenated with CoLES embeddings (“CoLES + LLM FT” in this Table). This is a promising result, indicating the potential of LLMs for transactional data analysis. We will include these new results in the paper after recalculating them for the whole validation set described in the article, what requires additional time. Our preliminary estimates show that we need 2 weeks to reproduce the complete validation pipeline using LLMs from this Table. 

**2. Advanced multimodal fusion techniques.**

To demonstrate the possibility of using more advanced multimodal fusion techniques, we conducted additional experiments of early fusion methods using the cross-attention mechanism, one of the foundations of modern multimodal models mentioned by the reviewers ([1] https://arxiv.org/pdf/2206.06488) and **Crossformer** ([2]  https://openreview.net/pdf?id=vSVLM2j9eie ).

In the following table, we compared the performance of **CrossTransformer** (early fusion, supervised, transformer) with the results presented in our paper for the **Supervised (RNN)** (late fusion, supervised, RNN). 

| Method                            | Modalities     |  ROC-AUC |
|----------------------------- |--------------- |--------------|
| **Supervised (RNN)**       | Trx + Dialog  |  0.821        |
| **CrossTransformer**       | Trx + Dialog  |  0.821        |
| **Supervised (RNN)**       | Trx + Geo     |  0.819        |
| **CrossTransformer**       | Trx + Geo     |  0.815        |

The analysis revealed that **CrossTransformer** performs similarly to or slightly worse than the **Supervised (RNN)** method. To examine the impact of Transformer architectures on the performance of multimodal data processing, we are conducting additional experiments with the **Supervised (Transformer)** model (late fusion, supervised, transformer), built on the same principles as **Supervised (RNN)**. We will include the results of a comprehensive comparison of different groups of methods (late fusion and early fusion) into the paper to provide a complete and thorough comparison of multimodal methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732210654772

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**3. Anonymization.**

Any anonymization (including ours) may remove some specific patterns in data and, hence, impact performance on some downstream tasks, but our experiment proves that our anonymization techniques still save all significant information for introduced downstream tasks. In Subsection 3.2 of our paper, we provided all permissible details of the anonymization process applied to our data. Additionally, Subsection 5.2.1 discusses the impact of anonymization by comparing model performance on the public dataset to that on the private (non-anonymized) dataset version. Figure 2 demonstrates a strong correlation between the results, indicating that the anonymization process has minimal impact on model performance for the downstream task of campaigning. This consistency underscores the reliability of our dataset for benchmarking and advancing research in multimodal event sequence analysis. Additional experimental comparison of results on our MDB and private datasets are available in Appendix B (Tables 6-9). The ranking of methods remains consistent despite differences in absolute metric values. As shown in Table 7, TrxTabGPT (ROC-AUC 0.796) and TrxAggregation (ROC-AUC 0.780) achieve the best performance on the private dataset. Similarly, TrxTabGPT leads on the public dataset with a ROC-AUC of 0.802 (Table 9). Incorporating geolocation and dialogue modalities further improves results, with DialogLast+TrxTabGPT+GeoTabGPT attaining the highest ROC-AUC on both datasets—0.802 on the private dataset (Table 7) and 0.808 on the public dataset (Table 9). Overall, multimodal approaches utilizing TabGPT or Aggregation demonstrate superior performance, with Late Fusion consistently outperforming Blending across private and public datasets (compare results in Table 7 and Table 9 to those in Table 6 and Table 8).

**4. New downstream tasks.**

In addition to the downstream task outlined in Section 7: Conclusion and Future Work, we plan to further develop the multimodal dataset and release an updated version in 2025. The planned improvements include:

1. Extend the time range of the dataset to include data from 2023 and 2024, enabling longitudinal studies and trend analysis.
2. Incorporate new data sources, which will predominantly be utilized as additional input modalities to enhance the dataset's multimodal capabilities.
These will include:
  - Expanded communication data, such as new communication channels and enriched dialogue features (e.g., discussed topics), enabling research into the impact of communication on sales and financial activity.
  - Clickstream data from an internal site, covering visited pages, operations performed, and viewed advertisements, providing insights into customer online behavior.
3. Expanding Prediction Targets, which can serve as both input modalities and prediction targets. These will include:
  - Additional financial products (up to approximately 20), broadening the scope of target variables for prediction.
  - Customer Behavior Insights: customer churn, fraud detection, credit risk assessment 
  - Segmentation of customer life cycles.

These updates will enhance the dataset's versatility for various tasks, enabling more comprehensive evaluations of representations. With a multi-task framework, we aim to facilitate innovative training methods, such as zero-shot learning, and support research in financial applications and customer behavior.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732210762630

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**5. Lack of novelty.**

We agree that our paper mostly concentrates on a new dataset and corresponding benchmark rather than developing innovative models. Our submission form chose the “datasets and benchmarks” as a Primary Area. As mentioned in Section 1. Introduction, we believe that papers in this area should mainly open new research questions and provide opportunities for future studies to develop new models rather than provide a strong innovative model and close the possibilities for significant improvements. As an example, we used an article about the HINT dataset (https://openreview.net/forum?id=kIPyTuEZuAK) that was accepted as a notable-top-25% paper at ICLR 2023.

We should emphasize that our dataset forces development of new techniques rather than simply applying existing models. Indeed, as detailed in Section 3.1 (Modalities), the event sequences in our dataset differ significantly from traditional time series due to their irregular temporal patterns and the inclusion of categorical features. For example, the methods suggested in the review ([1] Learning Deep Time-index Models for Time Series Forecasting; [2] Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting) are primarily designed for predicting future values of a time series, which is fundamentally different from our downstream tasks, and heavily rely on temporal continuity. So they would require substantial adaptation to effectively handle the multiple data sources and asynchronous nature of our data. 

Thus, we believe this contribution will drive meaningful advancements and help bridge a critical gap in research of multimodal sequential data analysis for real industrial use-cases.

We once again thank all the reviewers for assessing our work and for their valuable comments and advice, which helped us improve its presentation. We are happy to answer any additional questions you may have.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732807120083

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for their valuable comments and suggestions. Your insightful feedback has been instrumental in refining our work. We have revised the paper thoroughly to address your comments and improve its overall quality and clarity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
# Summary and Recommendation for Rejection

---

## Strengths:
1. **Dataset Scale and Scope**:  
   - The authors present the first large-scale, multimodal banking dataset (MBD) covering over 2 million corporate clients.  
   - Data sources include bank transactions, geolocation data, customer support dialogues, and aggregated purchases.  

2. **Benchmarking Contribution**:  
   - Includes baseline evaluations comparing unimodal and multimodal methods, showcasing the advantages of multimodal approaches for financial tasks.

3. **Public Availability and Reproducibility**:  
   - The dataset and experimental code are promised to be made publicly available, promoting transparency and fostering further research.

4. **Privacy Measures**:  
   - Anonymization techniques are applied to protect user data while retaining significant information for downstream tasks.

---

## Weaknesses:
1. **Lack of Novelty**:  
   - The paper primarily focuses on dataset creation and lacks significant methodological innovation.  
   - For ICLR, novel methods tailored to the dataset are expected but are absent here.

2. **Sparse Details on Anonymization**:  
   - While anonymization is discussed, there is insufficient quantitative evaluation of its impact on preserving temporal dependencies.  
   - This raises concerns about data reliability, especially for fine-grained tasks like fraud detection.

3. **Limited Methodological Exploration**:  
   - Advanced multimodal fusion techniques, such as cross-attention mechanisms, are not thoroughly explored.  
   - Comparisons with state-of-the-art methods, including large language models (LLMs), are limited.

4. **Repetitive Modality Representations**:  
   - The modalities are primarily tabular or text-based, limiting the diversity seen in standard multimodal datasets (e.g., including audio or video).  

5. **Presentation Issues**:  
   - The manuscript contains grammatical errors, unclear expressions, and inconsistent tense usage, detracting from readability.

---

## Authors' Mitigation:
1. **Advanced Fusion Techniques**:  
   - The authors added preliminary results using early fusion techniques (e.g., CrossTransformer), but these showed no significant performance improvement.

2. **Expanded Benchmarking**:  
   - Efforts to address label imbalance (e.g., stratified batching, oversampling) were included, but metric improvements were marginal.

3. **Anonymization Impact**:  
   - Claims of minimal performance differences between anonymized and original datasets are made, but quantitative evaluations for complex tasks are missing.

4. **Future Dataset Enhancements**:  
   - The authors promise to expand the dataset with new modalities and tasks in future versions but do not address current limitations.

---

## Remaining Weaknesses:
1. **Insufficient Novelty**:  
   - The authors fail to provide novel frameworks or techniques tailored to the dataset.

2. **Unsubstantiated Claims**:  
   - Assertions regarding anonymization and dataset reliability are not adequately supported by experiments or results.

3. **Limited Practicality**:  
   - Metrics like AUC are used as proxies for campaign success, but their real-world implications for financial decision-making are not validated.

4. **Incomplete Benchmarks**:  
   - Comparisons with modern multimodal learning approaches (e.g., MMBench) and integration with LLMs (e.g., GPT-4 or BloombergGPT) are insufficient.

---

## Decision Justification:
While the dataset's scale and intent are commendable, the paper falls short in several critical areas:

1. **Scope Misalignment**:  
   - The focus on dataset release, with minimal methodological contributions, does not align with ICLR's emphasis on innovation.

2. **Reliability Concerns**:  
   - The anonymized dataset's adequacy for sensitive tasks like fraud detection remains questionable.

3. **Limited Community Impact**:  
   - Without novel methods or robust comparative analyses, the paper's contribution is constrained to the dataset.

4. **Quality Issues**:  
   - Presentation flaws and lack of clarity weaken the submission.

---

## Recommendation:
**Reject**.   While the dataset provides a strong foundation for future research, significant revisions and expansions are necessary to meet the standards of a premier conference like ICLR.