ESQA: Event Sequences Question Answering

## Abstract

Abstract Event sequences (ESs) arise in many practical domains including finance, retail, social networks, and healthcare. In the context of machine learning, event sequences can be seen as a special type of tabular data with annotated timestamps. Despite the importance of ESs modeling and analysis, little effort was made in adapting large language models (LLMs) to the ESs domain. In this paper, we highlight the common difficulties of ESs processing and propose a novel solution capable of solving multiple downstream tasks with little or no finetuning. In particular, we solve the problem of working with long sequences and improve time and numeric features processing. The resulting method, called ESQA, effectively utilizes the power of LLMs and, according to extensive experiments, achieves state-of-the-art results in the ESs domain.

## 1 Introduction

Temporal data often comes in the form of event sequences, where each event is characterized by the arrival time and additional structured data. This type of data is widely spread in domains like geoscience (Bergen et al., ), healthcare (Esteva et al., ), sociology (Hossain et al., ), industry (Choi et al., ), e-commerce (Ni et al., ) and finance (Babaev et al., ). Event sequences combine properties of time series and tabular data while having major differences. Unlike time series, events can arrive with irregular time steps and can have structured annotations, similar to tabular datasets. Unlike tabular data, events have timestamps and associated order. These differences require special data processing, modeling, and inference approaches.

The new frontier in machine learning, especially in deep learning, focuses on adapting large language models (LLMs) to domains beyond language. The reasons behind this adaptation is that LLMs can use additional information, not found in domain-specific data, can process textual context of the underlying task, generate answers in a free natural form, can argue its decisions and support dialog with the user. The potential benefits of using LLMs include improved modeling quality and generalization. The latter means that the hybrid model can solve new problems with little or no finetuning, that largely increases the applicability of the model and reduces development costs. Successful applications of LLMs were demonstrated in both time series (Cai et al., ) and tabular datasets (Dinh et al., ), but no effort was made to adapt LLMs to event sequences: financial transactions, electronic health records, activity on different devices and so on. These data characterise a human live and are used to personalise many AI services across different domains.

Event sequence processing with LLM encounters several difficulties. First, structured data must be effectively encoded at the LLM’s input. Textual representation considerably increases the sequence length and can’t be effectively processed by modern Transformer models due to the quadratic complexity. Second, the desired method must be capable of processing long input sequences, even when the downstream tasks require historical data analysis. The problem is similar to the first but focuses on the model architecture rather than input processing. Finally, time feature and the order must be properly provided to the model, as they constitute the essence of event sequences and include important information for solving downstream tasks.

In this paper, we propose a new neural architecture, called ESQA, that exploits the power of LLMs to model event sequences and to solve associated practical tasks. In particular, we for the first time develop a question-answering approach with LLM backbone in the event sequences domain. We show the proposed model is capable of solving multiple downstream tasks without finetuning. When finetuned, ESQA outperforms other methods and achieves a new state-of-the-art.

## 2 Background

Event Sequences. We assume that events, denoted as $e_{i}$, are arranged in sequences $S_{n}={\{e_{i}\}}_{i=1}^{I_{n}}$ based on their association with a common entity. Here, ${I_{n}}$ represents the number of events in the sequence $S_{n}$. An entity could represent a bank customer or a web user, while the events within the sequence might include actions like a completed transaction or a series of clicks. These events are connected by a temporal order: $t(e_{i})<t(e_{i+1})$, where $t(.)$ indicates the time at which the event occurs.
Event sequences encompass a diverse range of attributes, with each event, $e_{i}$, characterized by a set of features ${\{c_{j}\}}_{j=1}^{C}$. These features can be depicted as a vector of values with dimension $C$. Additionally, $Y_{m}$ represents the target variable vector for the problem at hand, which may be based on the value of a sequence feature $c_{m}$ or external variables, such as a bank client’s default status. Attributes of events comprise both numeric $c_{j}^{num}$ and categorical features $c_{j}^{cat}$ of various types. Categorical features define attribute values within a finite set of categories $c_{j}^{cat}\in|c_{j}|=\{cat_{j;1},...,cat_{j;K_{j}}\}$, where $K_{j}$ denotes the number of possible values for the feature $c_{j}^{cat}$ (Lane, ). Numerical features $c_{j}^{num}\in\mathbb{R}$ are those represented as numbers, allowing meaningful arithmetic operations to be performed (Lane, ).

LLMs for Tabular Data.
Large Language Models (LLMs) are a family of neural architectures pretrained on a large corpus of texts. LLMs accept inputs in the form of text and generate textual output. In practice, LLM architecture is composed of three main blocks. The first one is an embedding layer, that converts input text to a sequence of numeric vectors known as embeddings. The second block, the backbone, transforms input embeddings to the output embeddings sequence with possibly different length. The final part of the model maps embeddings to the output text.

There are two main approaches for encoding tabular data at the input of LLM. The first one is to provide a description of each table field in the textual form (Dinh et al., ). This approach suffers from little flexibility and extremely long input sequences. The second approach is to replace the embedding layer, with a newly designed module capable of directly encoding table fields to embeddings with the required number of features. The latter approach is also known as embedding injection and usually achieves better results (Koh et al., ; Huang et al., ).

Question Answering with LLMs.
The popular way to solve problems with LLMs is to design a question such that a valid answer to this question solves the problem (Dinh et al., ). The question must include the context, i.e. all necessary data required for reasoning, and the task definition. This way LLM input is usually composed of the context, task, and connecting words indicating the boundaries of each part.

## 3 Event Sequences Question Answering

The general view of the proposed model, called Event Sequences Question Answering (ESQA), is presented in Figure 1. Below we will give a detailed description of the model’s input and the backbone LLM.

### 3.1 Questions and answers construction

The concept behind this method is to frame all tasks involving temporally structured data as natural language questions and answers. Each task from ${\{task_{m}\}}_{m=1}^{M}$ takes the form $\{Q_{m},X_{m},A_{m}\}$, where $Q_{m}$ is the question that defines the problem, $X$ represents the input data, and $A_{m}$ is the answer sought based on the target variable $Y_{m}$.

A question $Q_{m}$ consists of two components: the prefix and the question body. The prefix initiates the query token sequence and is placed before embeddings of other modalities. The question body then describes the task in textual form. For example, the task of determining the most frequent value of feature $c_{m}$ is represented as: “What is the most frequent value of $c_{m}$ in the entire dataset?”.

Given the nature of time-structured data, we classify questions into two types: extractive and predictive. Extractive questions focus on tasks involving existing event sequences, such as computing statistics or identifying trends and characteristics. Predictive questions, on the other hand, pertain to tasks concerning the prediction of future events or attributes based on available data.

Tasks and their corresponding questions can also be categorized based on the type of response sought: binary, multiple choice, or open-ended. Binary questions seek a straightforward answer, either as $A_{m}\in\{0,1\}$ or in the form of “Yes” or “No”. For instance, a question like “Is drinking water the most frequently purchased product?” can be answered with a simple “Yes” or “No”.

In contrast to binary questions, multiple-choice and open-ended questions assume a specific answer corresponding to the required feature, whether numerical $A_{m}\in\mathbb{R}$ or categorical $A_{m}\in|c_{j}|=\{cat_{1},\dots,cat_{K}\}$. Multiple-choice questions provide a list of possible answer choices. For example, one might ask “What is the most frequently purchased product? Options: black tea; bread; drinking water; grapes.”. Open-ended questions, on the other hand, prompt a direct response, such as “What is the name of the most frequently purchased product? Please provide the name in your response.”.

### 3.2 Events embeddings

To address the integration of event sequences into a language model, we propose adapting the method outlined in previous works (Koh et al., ; Huang et al., ). This involves embedding multi-modal information into an LLM, parameterized by ${\theta}$, by directly mapping it into the intrinsic embedding space $E^{\theta}$, bypassing the discrete text token layer. To achieve this, we introduce a trainable mapping $\phi:Z\rightarrow E^{\theta}$, where $Z$ represents the observation space of temporally structured data. This mapping converts the data into a sequence of $f$-dimensional vectors in$E^{\theta}$, which are then integrated into a sequence of text embeddings. This interleaving of modalities creates a multi-modal input for the LLM.

Figure: Figure 1: Model architecture. The components of the approach that do not require training are colored in blue. Components whose weights are optimised during training are colored in orange. The trainable embeddings and associated tokens are colored in red.
Refer to caption: extracted/5741940/figures/ESQA3.png

ESQA represents all event features as trainable embeddings. It is achieved by encoding each value $x_{ij}$ of a categorical or integer numeric feature $c_{j}$ with a sequential index $k_{x_{ij}}$ based on the total number of unique values for that feature $k=[0,\dots,K_{j}]$. This index uniquely identifies the embedding $emb_{k}$ of a feature value in the embedding matrix $W_{e}$. The embedding dimension is selected based on the formula: $dim(e_{k})=\lceil{\lambda\times K_{j}^{\mu}}\rceil$. The coefficients $\lambda=1.6$ and $\mu=0.56$ have been chosen empirically.

Numerical features in the form of real numbers are discretized into non-overlapping intervals: $B_{j}^{1},\dots,B_{j}^{n}$, $B_{j}^{i}=[b_{j}^{i-1},b_{j}^{i})$. The distribution of the feature $c_{j}$ in the training sample is used to determine these intervals. The number of intervals is chosen based on the approach in (Doane, ), using the formula $n=1+\log_{2}(n)+\log_{2}(1+\frac{|g_{1}|}{\sigma_{g_{1}}})$, where $g_{1}$ is the estimated third-moment skewness of the distribution and $\sigma_{g_{1}}=\sqrt{\frac{6(n-2)}{(n+1)(n+3)}}$. This method is particularly suited for distributions of features that deviate significantly from the normal distribution.

Once the intervals have been defined, Eq. 3.2 determines the value $x_{j;disc}^{num}$ of the $j$’th numerical feature:

$$ $x_{j;disc}^{num}=\begin{cases}b_{j}^{0},&x_{ij}<b_{j}^{0},\\ b_{j}^{n},&x_{ij}\geq b_{j}^{n},\\ b_{j}^{i}&b_{j}^{i-1}\leq x_{ij}<b_{j}^{i}.\end{cases}$ $$

The resulting feature embeddings are concatenated into a tensor $e_{i}^{emb}$ of dimension $dim(e_{i}^{emb})=\sum_{j=1}^{C}|c_{j}|$, which describes a single event $e_{i}$ from the sequence. A vector representation of sequence $S_{n}$ is formed by combining vector representations of individual events into a joint tensor $S_{n}^{emb}$ shown on Fig.2a.

Figure: Figure 2: a) Event sequences features encoding; in the example, there are $N$ numerical and $C$ categorical features, which are concatenated into a tensor $e_{i}^{emb}$ of dimension $dim(e_{i}^{emb})$. b) The event sequence encoder model processes the concatenated feature embedding vectors $S_{n}^{emb}$ for all events within a sequence, ultimately producing a comprehensive embedding $\tilde{S_{n}}^{emb}$ for the entire event sequence.
Refer to caption: extracted/5741940/figures/feature-preprocessing+encoder-2.png

### 3.3 Encoder

After the initial layer of input data embeddings, vectorized event sequences are fed into a specialized encoder model Fig. 2b. This module, based on the architecture of the Transformer decoder, processes sequences of events in an autoregressive manner by predicting each subsequent event. For our implementation, we used both Whisper-tiny and Whisper-small models (Radford et al., ), initialized with weights pre-trained on audio data. The input tensor for the encoder comprises concatenated feature embedding vectors for all events $S_{n}^{emb}$ (Section 3.3.1) and has a size of $dim(S_{n}^{emb})=(I_{n},dim(e_{i}^{emb}))$. The encoder processes this tensor autoregressively, similar to the sequence of text token embeddings, resulting in a sequence of vectors $\tilde{S_{n}}^{emb}$ with a size $dim(\tilde{S_{n}}^{emb})=(I_{n},d_{enc})$. Here, $d_{enc}$ represents the output layer dimensionality of the encoder model. To ensure compatibility between the dimensions of the input embeddings of the event sequences $dim(S_{n}^{emb})$ and the embedding layer of the encoder model $d_{enc}$, we used a linear projection layer.

This choice of encoder architecture is motivated both by the temporal nature of the event sequences, which aligns with autoregressive modelling, and by the results of a series of experiments. Appendix A.1 provides a detailed description of the experiments and their results.

### 3.4 Connector

Figure: Figure 3: The Q-Former model’s architecture is designed to extract the most relevant event sequence representations. It produces $q$ query embeddings for each event sequence, which are then linearly projected to the size of the language model embedding and appended to the embedded question tokens. Subsequently, the joint sequence is transmitted to the LLM.
Refer to caption: extracted/5741940/figures/QFormer-upd-1.png

The output representation of the event sequence encoder grows in dimensionality as the number of events in each sequence increases. This size is crucial, as it must fit within a common multimodal embedding sequence, impacting the extension of the language model’s context length. Our goals are to shorten the event sequence length without significant information loss and to adapt each event’s vectorized representation to match the language model’s embedding dimension. To achieve this, we propose an intermediate connection layer between the event sequence encoder and the LLM. We suggest using the Query Transformer model, or Q-Former (Li et al., ), to efficiently extract features from the encoder output.

The Q-Former architecture (Fig. 3) includes two transformer submodules: a novel modality transformer (originally an image transformer) that works with a fixed image encoder for feature extraction, and a text transformer that functions as both an encoder and a decoder. A set of trainable query embeddings $q$ serves as the input for the novel modality transformer. These queries engage in self-attention, interacting with each other and with the fixed modality features through cross-attention layers in every other transformer block.

In our approach, Q-Former produces $q$ query vectors for each event sequence, which are then passed to the LLM. We use a single fully-connected layer to project the output query vectors into the language model’s text embedding dimension. In this study, we initialize Q-Former with the weights from the BLIP-2 approach, derived from BLIP-2 with the FLAN-T5-xl model (Li et al., ). The architecture and initialization of the connection layer were chosen based on a series of experiments detailed in Appendix A.2.

### 3.5 Language Model

As the backbone for the pre-trained LLM, our approach utilizes the FLAN-T5 family of encoder-decoder models (Wei et al., ). Any process of fine-tuning model parameters influences the model’s proficiency in a specific domain but also causes it to "forget" essential general and linguistic knowledge. To preserve this knowledge and save computational resources, we have frozen most of the LLM parameters. Studies (Lu et al., ; Zhou et al., ) indicate that freezing most of the model’s weights often yields better results than fully fine-tuning a pre-trained LLM.

To efficiently select a limited set of trainable parameters, we propose using Parameter-Efficient Fine-Tuning (PEFT) methods. Specifically, we employed the Low-Rank Adaptation (LoRA) approach (Hu et al., ), which keeps most of the model weights frozen while adding trainable rank decomposition matrices to a subset of the parameters.

## 4 Experiments

In this section, we begin by presenting the evaluation details, which include the comparison methods and the datasets used for evaluation. Following this, we conduct a series of systematic experiments to showcase the capability of the developed ESQA approach in addressing a diverse range of problems based on event sequences.

### 4.1 Experimental setup

Datasets. Sequences of events are prevalent across various domains and tasks, with a particularly high demand for analyzing such data in the fintech sector. In this field, transactional activity of individuals serves as the primary source of information. Consequently, we have chosen to utilize a collection of datasets containing customer transactions from banks and marketplaces as examples of event sequences. The sensitivity of the information in these datasets has significantly influenced our choice and the number of datasets used, given the limited availability of public datasets in this area.

We selected five publicly available datasets with event sequences. These include: AlfaBattle2.0 (Evgeny and Max, ), Age Group Prediction Competition (Sirius, ), X5 Retail Hero: Uplift Modelling for Promotional Campaign from (Babaev et al., ), Taiwan Default of Credit Card Customers from (Yeh and hui Lien, ), and Gender Prediction Competition from (Max, ). The data were divided into two subsets, training and validation sets, ensuring no overlap by unique client identifiers. The proportions for partitioning these sets were chosen independently for each dataset. A detailed description of the datasets used to evaluate the approach’s quality is provided in Appendix B.1.

Baselines. We have chosen representative baseline approaches for analysing event sequences, which have demonstrated effectiveness across various benchmarks. For the AlfaBattle and Taiwan Default of Credit Card Clients datasets, we implemented, trained, and fine-tuned the baseline models ourselves to achieve optimal results. For other benchmarks, we relied on the findings from CoLES (Babaev et al., ), which provide the most current and comprehensive empirical studies of event sequences.

For the next event prediction task we also provide calculated statistical baselines for both numerical and categorical target features to compare the quality of prediction tasks in zero-shot setups. These include, for example, the prediction of the mean or median value for numerical target variables and the prediction of the most frequent value for categorical attributes.

A complete list and detailed description of baselines is provided in Appendix B.2.

### 4.2 Experimental results

#### 4.2.1 Main results

Each dataset used for model evaluation corresponds to a specific downstream task. For instance, the AlfaBattle dataset is utilized for predicting a bank customer’s loan default, while the Age dataset is employed for predicting the age group. It is important to highlight that the AlfaBattle dataset is highly imbalanced, with the positive class constituting less than 3%, while the Gender dataset has a slight over-representation of the class denoting male gender. Therefore, we used the ROC-AUC metric for problems with binary target variables and class imbalance. For multiclass classification with balanced classes, we employed Accuracy. A more detailed explanation of the metric calculation methodology and the assessment of response quality for ESQA is provided in Appendix B.0.2. The results of the experiments on the downstream tasks for datasets described in Section 4.1 are summarized in Table 1.

**Table 1: A comparison of ESQA on the downstream tasks of the five event sequence datasets described in Section 4.1 with the baseline approaches presented in Section 4.2.1. The best results are highlighted in bold and the second best results are underlined.**
| Dataset | AlfaBattle | Age | Gender | X5 | Taiwan |
| --- | --- | --- | --- | --- | --- |
| Metric | AUCROC | Accuracy | AUCROC | Accuracy | AUCROC |
| Handcrafted feat. | 0.7792 | 0.629 | 0.877 | 0.547 | 0.784 |
| Randomly init. RNN | 0.6456 | 0.375 | 0.593 | - | 0.722 |
| CPC | 0.7919 | 0.602 | 0.851 | 0.525 | - |
| Barlow Twins | 0.7878 | 0.634 | 0.865 | - | - |
| CoLES | 0.7921 | 0.640 | 0.881 | 0.539 | - |
| NSP | 0.7655 | 0.621 | 0.852 | 0.425 | - |
| RTD | 0.7910 | 0.631 | 0.855 | 0.520 | - |
| SOP | 0.7238 | 0.512 | 0.785 | 0.428 | - |
| MLM NSP | 0.7591 | - | - | - | - |
| TabFormer | 0.7862 | - | - | - | - |
| GPT | 0.7737 | - | - | - | - |
| ESQA (ours) | 0.7568 | 0.699 | 0.850 | 0.598 | 0.793 |

The results indicate that the ESQA approach matches both self-supervised contrastive and supervised methods in quality. Notably, on the Age and X5 datasets, ESQA surpasses the baseline scores. Although specific comparative results for other models are not available for the Taiwan dataset, ESQA’s impressive performance underscores its effectiveness. These outcomes highlight ESQA’s superior capability in handling multi-class classification tasks with balanced classes.

However, for the client default problem on the AlfaBattle dataset and the Gender dataset, the results are less clear-cut. The CoLES contrastive approach achieves the highest quality for these problems. While ESQA slightly lags behind CoLES, it still shows competitive performance, closely following models like Barlow Twins and RTD, and outperforming the SOP approach. It is important to note that both datasets exhibit class imbalance, which is especially pronounced in the AlfaBattle case.

This leads us to conclude that the ESQA approach generally performs classification tasks as well as, or better than, the selected baseline methods. However, it is significantly affected by the imbalance of the target variable. This limitation can be attributed to the nature of LLMs, originally designed to extract common patterns from text data to model complex language structures.

#### 4.2.2 Predictive tasks

The majority of tasks involving event sequences require answering predictive questions about event features. To address such challenges, we propose utilizing the ESQA approach in a multi-task setting, enabling simultaneous predictions of all features of the next event in the sequence. Experimental results for predictive questions against baselines are detailed in Table 2.

**Table 2: Table comparing ESQA with the baseline approaches presented in Section 4.2.1 for predicting attributes of the next transaction on the AlfaBattle dataset. The best results are highlighted in bold and the second best results are underlined.**
| Attribute | MCC code | Amount | Hour diff |
| --- | --- | --- | --- |
| Metric | Acc./F1 | MAE/MSE | MAE/MSE |
| CoLES | 0.440 / 0.351 | 0.197 / 0.082 | 36.05 / 1586.52 |
| CPC | 0.475 / 0.411 | 0.196 / 0.074 | 34.89 / 1508.71 |
| RNN with CoLES | 0.469 /0.411 | 0.184 /0.077 | 32.25 / 1573.02 |
| CatBoost | 0.440 /0.367 | 0.190 /0.090 | 34.40 / 1613.41 |
| GPT with descr. | 0.462 /0.423 | 0.179 / 0.083 | 32.63 / 1726.42 |
| Text LLM | 0.382 / 0.381 | 0.103 / 0.0176 | 116.38 / 62161 |
| ESQA (ours) | 0.546 / 0.546 | 0.191 / 0.1021 | 18.313 / 1033.87 |

On categorical feature prediction tasks, such as MCC code attribute prediction, ESQA achieves the highest performance with an Accuracy/F1 scores, outperforming all other models, with the closest being CPC. This indicates that ESQA is particularly effective in handling categorical prediction tasks within the transaction history context.

In predicting the numerical Amount attribute, while the Text LLM achieves the lowest MAE/MSE, ESQA still performs competitively. Although ESQA is not the top performer here, it maintains reasonable accuracy, demonstrating its versatility across different types of prediction tasks.

For the temporal Hour diff attribute, ESQA significantly outperforms all other models. The next best model, RNN CoLES, has a much higher MAE/MSE, highlighting ESQA’s superior capability in handling temporal prediction tasks effectively.

#### 4.2.3 Generalization abilities

LLMs possess an extraordinary capacity to generalise to novel, previously unseen tasks. Our method maintains the integrity of the language model’s weights, thereby preserving its inherent capabilities. Moreover, by training adaptors within the attention layers, we expand the domain of zero-shot tasks from exclusively text-based tasks to those based on event sequences. We evaluated the ESQA approach’s adaptability to new predictive tasks following comprehensive pre-training on a set of contextual tasks. The model was trained in multi-task setting on all event features of the AlfaBattle dataset and was subsequently tested in a zero-shot setting across various predictive tasks within the same dataset. Table 3 presents a comparison of our experimental results against statistical baselines, a text baseline, and an ESQA model specifically trained on those predictive tasks.

**Table 3: Table comparing the generalisation abilities of the ESQA approach with the statistical baseline approaches presented in Section 4.2.1, and a text-based approach. The ESQA approach trained on predictive tasks in a multitask setting is referred to as ’ESQA m/t’. While ESQA trained on contextual tasks and adapting to new tasks is referred to as ’ESQA z/s’. The best results are highlighted in bold and the second best results are underlined.**
| Attribute | Stat. baseline | Text-only | ESQA m/t | ESQA z/s |
| --- | --- | --- | --- | --- |
| MCC code, acc. | 0.388 | 0.382 | 0.546 | 0.381 |
| MCC category, acc. | 0.437 | 0.402 | 0.588 | 0.435 |
| Amount, MAE/MSE | 0.241 | 0.103/0.018 | 0.191/0.102 | 0.389/0.228 |
| City, acc. | 0.704 | 0.691 | 0.731 | 0.343 |
| Country, acc. | 0.970 | 0.970 | 0.972 | 0.971 |
| Currency, acc. | 0.987 | 0.986 | 0.987 | 0.988 |
| Op. type gr., acc. | 0.766 | 0.733 | 0.840 | 0.781 |
| Op. type, acc. | 0.499 | 0.393 | 0.633 | 0.543 |
| Op. kind, acc. | 0.548 | 0.494 | 0.693 | 0.598 |
| Days before, MAE/MSE | 140.5 / 23823.3 | 10.5 / 657.2 | 6.3/195.9 | 11.394 / 666.2 |
| Hour diff, MAE/MSE | 36.33 | 116.4/62161 | 18.3/1033.9 | 48.85/3980 |

For the MCC code and MCC category attributes, ESQA multi-task outperforms all baselines, indicating its strength in handling categorical predictions. However, in the zero-shot setting, ESQA’s performance is comparable to the statistical baseline, suggesting room for improvement in scenarios without task-specific training. In predicting the Amount attribute, the text-only approach achieves the best MAE/MSE, while ESQA multi-task shows competitive performance, demonstrating its robustness in handling regression tasks despite not being the top performer. However, the regression problem on real numbers with a large number of decimals is still a challenging task for zero-shot ESQA, which performed poorly. For temporal predictions like Days before and Hour diff, ESQA multi-task significantly outperforms other approaches, showcasing its superior capability in modelling temporal patterns. Overall, ESQA zero-shot performance, while not leading, still provides valuable insights into ESQA’s versatility and potential for improvement in less customised settings.

## 5 Related work

Event sequences.
Temporal Point Processes (TPPs) and their marked variants (MTPPs) can be seen as the simplest forms of event sequences. Previous research was focused on accurate next event prediction with or without neural networks (Liniger, ; Mei and Eisner, ; Xue et al., ). Another branch of research addressed event streams of the general form Padhi et al. (); Babaev et al. (); McDermott et al. (). To the best of our knowledge, question answering with LLMs was not previously applied to TPPs nor event sequence modeling.

Structured modeling with neural networks. The problem of modeling multiple heterogeneous features with neural networks was addressed in tabular neural networks (Yin et al., ; Iida et al., ; Padhi et al., ; Hegselmann et al., ; Yang et al., ; Dinh et al., ). We reuse best practices for making embeddings from categorical and numeric features. At the same time, event sequences require analysis of multiple events at once, while tabular datasets can be processed one row at a time. To this end, ESQA applies encoding method and adapt Q-Former, not seen in tabular neural networks.

LLMs for time series. Previous works made use of LLMs in the context of time series analysis (Gruver et al., ; Cai et al., ; Zhang et al., ). Unlike time series models, ESQA implements a novel context encoding and can process complex data structures.

## 6 Conclusion

In this paper, we introduced Event Sequences Question Answering (ESQA), a novel approach for modelling event sequences with LLMs. Our empirical results demonstrate that our approach performs robustly across various datasets. For several downstream problems, ESQA performs at least as well as specialised baselines 1, and for the task of predicting the attributes of the next event, it significantly surpasses baseline methods 2. Furthermore, we have shown that ESQA can handle multiple tasks simultaneously without any special fine-tuning 3, highlighting its remarkable ability to adapt swiftly to new tasks without the need for complex and time-consuming training. These findings position ESQA as an exceptionally promising approach leveraging the strong generalisation capabilities of LLM backbones for the field of event sequences.

Limitations. Our research has certain limitations. In processing numerical features, ESQA employs value discretization, which introduces an inherent discretization error. This error is significantly influenced by the number of discretization buckets and the ranges of the actual feature values. To mitigate this error, we conducted several additional experiments to refine the pre-processing method. Furthermore, handling time features in event sequences requires special attention. We are actively exploring ways to enhance temporal feature processing within ESQA. In future work, we will focus on implementing these improvements and addressing the challenge of dealing with unbalanced classes.

## Appendix A Architecture components selection

### A.1 Event sequence encoder architecture selection

Training a language model to understand a different modality is not a novel challenge and has been addressed for various data types. Therefore, in our experiments on the event sequence encoder architecture, we built upon advancements from other modalities. We focused on established models for three highly developed modalities: text, images, and audio. For text architectures, we examined several models including encoder-only models like BERT (base, large), encoder-decoder models like T5 (small, base, large), and decoder-only models like GPT (base, medium, large). For image architectures, we used ViT (base, large). For audio models, we considered various versions of Whisper (tiny, small, medium), utilizing only the decoder part of the Whisper architecture.

The models were compared based on their ability to predict the default of a bank client in the AlfaBattle 2.0 dataset, a binary problem where the task is to determine if a bank client will repay a loan based on their transaction history over two years. We used AUC as the metric for comparison. All models were trained from scratch.

We maintained a consistent training scheme across all experiments, employing Adam as the optimizer with a learning rate of 1e-4. A linear warm-up of the learning rate was applied for the first epoch, followed by a linear decay to zero after 10 epochs. To ensure compatibility between the dimensions of transaction embeddings and the dimensions of pretrained model embeddings, we used a linear layer for text and audio models. Since ViT models cannot process sequences, we addressed this issue by applying a single layer of cross-attention to a fixed number of learnable latent tokens.

As shown in Table 4, decoder-only models outperformed both encoder-only and encoder-decoder models in event sequence encoding in almost all setups. Specifically, experiments with text models demonstrated that the decoder-only GPT2 model outperformed the encoder-decoder T5 model, and the BERT model training did not converge. Similarly, audio architectures, which are primarily decoder-based, also showed superior performance. In response to the concerns about the performance of larger models within the same family, as observed in Table 4, our analysis suggests that the enlargement of the encoder size contributes to overfitting. This overfitting is the primary reason for the degradation in performance outcomes.

**Table 4: Table comparing different architectures for predicting default of the client on the AlfaBattle dataset. The best results are highlighted in bold and the second best results are underlined.**
| Architecture | Type | Number of parameters | AUC |
| --- | --- | --- | --- |
| GPT2 Base | Decoder | 124M | 0.7869 |
| GPT2 Medium | Decoder | 355M | 0.7833 |
| GPT2 Large | Decoder | 774M | 0.7747 |
| Whisper-tiny | Decoder | 29M | 0.7892 |
| Whisper-small | Decoder | 153M | 0.7894 |
| Whisper-medium | Decoder | 456M | 0.7715 |
| T5 Small | Encoder-Decoder | 60M | 0.7721 |
| T5 Base | Encoder-Decoder | 223M | 0.7756 |
| T5 Large | Encoder-Decoder | 770M | Diverged |
| BERT Base | Encoder | 110M | Diverged |
| BERT Large | Encoder | 335M | Diverged |
| ViT Base | Encoder | 85M | 0.7822 |
| ViT Large | Encoder | 302M | 0.7639 |

After determining the type of architecture (i.e., the decoder), we conducted further experiments to identify the specific type and size of the decoder architecture. We compared Whisper-tiny, Whisper-small, and GPT2-base, as they produced the best results. Additionally, we evaluated various types and sizes of recurrent architectures: GRU-1, GRU-6, GRU-12, LSTM-1, and LSTM-4, where the number indicates the number of layers used in each model. The embedding size for all recurrent models was set to 1024.

**Table 5: Table comparing different decoder architectures presented in Section A.1 for predicting default and attributes of the next transaction on the AlfaBattle dataset. The best results are highlighted in bold and the second best results are underlined.**
| Architecture | # params. | Amount | MCC Category | 24-hour acc | Default |
| --- | --- | --- | --- | --- | --- |
| Metric |  | MSE | Accuracy | Accuracy | AUC |
| Whisper-tiny | 29 M. | 0.0660 | 0.4861 | Diverged | 0.7892 |
| Whisper-small | 153 M. | 0.0656 | 0.4896 | 0.645 | 0.7894 |
| GPT-2-base | 100 M. | 0.0657 | 0.4888 | Diverged | 0.7869 |
| GRU-1 | 0.3 M. | 0.0668 | 0.4817 | 0.418 | 0.7854 |
| GRU-big | 16 M. | 0.0670 | 0.4805 | Diverged | 0.7578 |
| GRU-large | 35 M. | 0.0662 | 0.4815 | Diverged | 0.7732 |
| LSTM-1 | 0.4 M. | 0.0669 | 0.4830 | 0.634 | 0.7710 |
| LSTM-4 | 2 M. | 0.0664 | 0.4858 | 0.655 | 0.7664 |

Table 5 indicates that transformer architectures outperformed recurrent models. Scaling up recurrent models did not significantly enhance their quality and sometimes even degraded their performance. Given the similar results among transformer architectures, we selected Whisper-small as the optimal model for all ESQA experiments.

### A.2 Connector architecture selection

**Table 6: Table comparing different connector architectures for better modalities alignment. Q-Former architecture based connectors with initialisations from BLIP-2 (Li et al., ) pretrained weights are labelled ‘w. init.’, without initialisation are indicated by ‘w/o. init.’. The best results are highlighted in bold and the second best results are underlined.**
| Architecture | # params. | MCC code | MCC category | Amount |
| --- | --- | --- | --- | --- |
| Metric |  | Accuracy | Accuracy | MSE |
| Linear | 197 k. | 0.501 | 0.574 | 0.0174 |
| 2 x Linear | 920 k. | 0.523 | 0.561 | 0.0196 |
| RNN (LSTM) | 3.94 M. | 0.509 | 0.558 | 0.0220 |
| Transformer | 1.1 M. | 0.478 | 0.529 | 0.1361 |
| 2 x Transformer | 2.09 M. | 0.519 | 0.555 | 0.0168 |
| Q-Former-small | 14.7 M. | 0.519 | 0.579 | 0.0162 |
| Q-Former-base (w/o init.) | 96 M. | 0.526 | 0.570 | 0.0189 |
| Q-Former-base (w. init.) | 96 M. | 0.527 | 0.569 | 0.0177 |

Integrating multiple modalities within a single approach centered around an LLM requires mapping new modalities into a textual model. Employing a separate encoder for each modality simplifies the task to finding an efficient architecture for mapping each modality’s vector space to the LLM embedding text space. When analyzing event sequences, processing extended data sequences presents challenges due to increased context length, which leads to higher computational complexity. In some instances, the sequence length may surpass the maximum context length of the language model.

To address these challenges, we conducted experiments to determine the optimal architecture for the connection layer between the event sequence encoder and the LLM. We evaluated several potential implementations: a single linear layer, a transformer layer, and two model sizes of the Q-Former architecture. Additionally, we investigated the impact of initialization on problem-solving quality and training speed by initializing the Q-Former with weights from the pre-trained visual-text model BLIP-2, based on FLAN-T5. In all experiments, we tackled three tasks in a multi-task mode using the AlfaBattle dataset. The components used in all experiments included Whisper-tiny as the transaction encoder and FLAN-T5-small as the language model. Performance was measured at 20 epochs, with fixed batch size, learning rate, and optimization parameters. We used multi-class accuracy for classification tasks and MSE for numerical response prediction tasks as target metrics.

The results revealed that simply increasing the number of trainable parameters does not necessarily enhance task solution quality. A linear layer with a small number of parameters performed worse than Q-Former-small, which also trained much faster. However, adding more simple identical blocks within a single connector, such as ${}^{\prime}2xLinear^{\prime}$, did not significantly improve performance. On the other hand, more complex blocks, such as ${}^{\prime}2xTransformer^{\prime}$, showed substantial quality gains. Increasing the model size to Q-Former-base yielded mixed results: while MCC code prediction quality improved by 2%, the metrics for MCC category prediction and numerical attribute Amount declined.

Additional initialization with weights from visual-text pre-training marginally improved the MCC code prediction task but slightly degraded the metrics for the other two tasks. The overall impact of initialization was minimal, indicating few common patterns between extracting salient information from images and deriving dependencies from event sequences. This discrepancy is expected due to the lack of temporal dependence within a single image, in contrast to the strong temporal dependence between events in a sequence.

Therefore, we selected the Q-Former-base model without initialization, anticipating an increase in the number of tasks our approach can handle simultaneously. This model offers a sufficient margin for increasing the complexity of future experiments.

## Appendix B Implementation details

#### B.0.1 Training and hyper-parameters

All experiments for ESQA described below utilised consistent hyperparameters and approach components, unless otherwise specified. We employed the AdamW optimizer (Loshchilov and Hutter, ) with parameters $\beta_{1}=0.9$, $\beta_{2}=0.98$, and a weight decay of 0.01. Cosine learning rate decay with restarts was applied, featuring different peak learning rates for each dataset and varying numbers of warm-up steps. In our experiments, LoRA (Hu et al., ) with a rank of $r=16$ was applied only to the matrices $W_{q}$ and $W_{v}$ of the self-attention and encoder-decoder attention layers. The LoRA scaling factor was set to 32, and the dropout rate to 0.05. The number of trainable parameters in the language model was calculated as $\theta^{train}=2\times L\times d_{model}\times r$, where $L$ is the number of layers and $d_{model}$ is the internal dimensionality of the language model. The rank of trainable decomposition matrices is denoted by $r$. Therefore, the number of trainable parameters in each FLAN-T5 model did not exceed 0.9% of the total parameters (Table 8). All models were trained using 6 Nvidia A100 (80G) GPUs. The training hyperparameters are summarised in Table 7.

**Table 7: Hyperparameters used for ESQA training. In all experiments, the Whisper-small model architecture was used as the encoder.**
| Dataset | AlfaBattle2 | Age | Gender | X5 | Taiwan |  |
| --- | --- | --- | --- | --- | --- | --- |
| LLM | flan-T5-xl | FLAN-T5-xl | fla-T5-large | FLAN-T5-xl | FLAN-T5-xl |  |
| Emb. size | 201 | 110 | 74 | 163 | 100 |  |
| Learn. rate | 3e-4 | 3e-4 | 1e-4 | 1e-4 | 1e-4 |  |
| warmup steps | 4 k. | 1 k. | 1 k. | 4 k. | 1 k. |  |
| Max. epochs | 40 | 10 | 10 | 20 | 30 |  |
| Batch size | 300 | 250 | 50 | 250 | 50 |  |
| Min seq. len. | 50 | 0 | 0 | 0 | 6 |  |
| Max seq. len. | 750 | 1500 | 1500 | 750 | 6 |  |

**Table 8: Trainable parameters of LLM with LoRA.**
| Model | % trainable params. | # trainable params. |
| --- | --- | --- |
| FLAN-T5-small | 0.8862 | 0.688 M. |
| FLAN-T5-base | 0.7096 | 1.77 M. |
| FLAN-T5-large | 0.5989 | 4.72 M. |
| FLAN-T5-xl | 0.3301 | 9.44 M. |
| FLAN-T5-xxl | 0.1692 | 18.87 M. |

#### B.0.2 Evaluation strategy

We employed several classical machine learning metrics to thoroughly evaluate the proposed approach. As previously mentioned, ESQA is designed to handle tasks that can be framed as binary or multi-class classification as well as regression settings.

Classification Metrics. We utilised classification metrics for tasks that involved predicting a categorical feature of the next event or a characteristic of the entire sequence (e.g., default of a bank customer). For non-binary target tasks, we used Accuracy and F1-score. For binary target tasks, we employed the Area Under the Receiver Operating Characteristic curve (ROC-AUC). The model with the highest performance on these metrics was deemed the best.

To calculate the classification metrics Accuracy and F1 score using the language model’s response in the question-answer format, we applied the following process. The question body was followed by an instruction specifying the format of the answer to clearly define the structure of the language model’s output. The tokens predicted by the language model were then decoded into text, and the segments containing the desired answer were extracted. These extracted values $y$ were compared to the target $\hat{y}$ in a classification format, where the number of classes matched the cardinality of the predicted value. Subsequently, Accuracy and F1 were calculated as follows:

$$ $\texttt{Accuracy}(y,\hat{y})=\frac{1}{n_{\text{samples}}}\sum_{i=0}^{n_{\text{ samples}}-1}1(\hat{y}_{i}=y_{i})$ $$

$$ $\text{F1}=\frac{2*\text{TP}}{2*\text{TP}+\text{FP}+\text{FN}}$ $$

In this context, $TP$ represents the number of true positives, $FP$ stands for the number of false negatives and $FP$ denotes the number of false positives.

In calculating the ROC-AUC metric, we utilised the difference between the probabilities of the positive and negative response tokens.

Regression Metrics. To evaluate prediction performance for tasks with real-valued target variables, we employed Mean Absolute Error (MAE) and Mean Squared Error (MSE) metrics. For calculating these regression metrics, each question was accompanied by instructions specifying the format and range of the expected answer. The required numerical values (both real and integer) were then extracted from the LLM’s textual predictions according to the given response structure. Instances where the prediction could not be interpreted as a number were excluded from the final metric calculation(^1^11We made this assumption based on the rarity of such instances, given the clarity of the questions and the accompanying guidance provided for answering them.). The selected numerical responses, denoted as $y$, were compared with the target values $\hat{y}$ for accurate assessment:

$$ $\text{MAE}(y,\hat{y})=\frac{1}{n_{\text{samples}}}\sum_{i=0}^{n_{\text{samples }}-1}\left|y_{i}-\hat{y}_{i}\right|$ $$

$$ $\text{MSE}(y,\hat{y})=\frac{1}{n_{\text{samples}}}\sum_{i=0}^{n_{\text{samples }}-1}(y_{i}-\hat{y}_{i})^{2}$ $$

### B.1 Detailed datasets description

A complete list of the datasets and a description of each dataset is given below. Main statistics and descriptions for each dataset are provided in Table 9.

**Table 9: Statistics of the datasets used for models evaluation.**
| Dataset | AlfaBattle | Age | Gender | X5 | Taiwan |
| --- | --- | --- | --- | --- | --- |
| # events | 443 M. | 44 M. | 6,85 M. | 45,8 M. | 0.18 M |
| # sequences | 1,47 M. | 30 K. | 9,2 K. | 400 K. | 30 K. |
| Avg, seq. len. | 881.7 | 862.4 | 446.6 | 114.3 | 6 |
| # numeric | 3 | 1 | 3 | 3 | 3 |
| # categorical | 15 | 2 | 2 | 3 | 5 |
| # classes | 2 | 4 | 2 | 4 | 2 |
| train/val split % | 70/30 | 90/10 | 90/10 | 90/10 | 90/10 |

AlfaBattle2.0 dataset. The AlfaBattle2.0 dataset Evgeny and Max () consists of transaction activity records of bank customers over a two-year period, capturing spending, payments, and transfers. The primary goal is to estimate the probability of a customer defaulting on a loan within a given timeframe. The default rate in this dataset is 2.76%. Each customer is associated with a sequence of transactions, each described by 18 features: 3 numeric and 15 categorical. The numeric features include the normalized transaction amount, the number of hours since the customer’s last transaction, and the number of days until the loan is disbursed. The categorical features encompass various identifiers: the merchant’s code and category, the currency and type of payment card, and the city, country, etc. All categories are encoded with numeric values to ensure the dataset remains anonymized. The temporal component is defined by the attributes of hour, day of the week and week of the year, which in combination form the transaction date and time.

Age Group Prediction Competition. This dataset (Sirius, ) comprises anonymized transaction records of bank customers, with the aim of predicting the age group of each client based on their transactions. Each transaction is characterised by three features: a discrete MCC (Merchant Category Code) identifying the type of merchant, the transaction date, and the transaction amount. Transactions can be grouped according to the unique customer identifier specified in the transaction description. The merchant identifier is also provided in text form, with categories such as ’bookshop’, ’ATM’, ’pharmacy’, etc. This allows for a more detailed and nuanced analysis of spending patterns related to different age groups.

Gender Prediction Competition. The primary goal of this competition is to predict the gender of bank customers based on their transaction activity (Max, ). The dataset includes historical transaction and transfer data spanning one year and three months. Each transaction record is associated with a unique client ID and contains the time and date of the transaction, its type, the transaction amount, and a discrete identifier for the merchant point. The transaction amount is not normalised and can indicate both inflows and outflows of funds. A negative value signifies a debit, while a positive value denotes a credit to the account.

Taiwan Default of Credit Card Clients. This dataset (Yeh and hui Lien, ) includes customer transaction data from April to September 2005, and it is used to predict whether a customer will repay their borrowed credit. Each record in the dataset contains 8 real-valued attributes. Some attributes describe the customer’s characteristics, such as age, education level, and marital status, while the remaining attributes provide details about the history of loan repayments.

X5 Retail Hero: Uplift Modeling for Promotional Campaign. Initially designed for an uplift modeling competition, this dataset focuses on predicting a customer’s age based on their purchasing activity (Babaev et al., ). Each purchase in the dataset is characterized by the time of the transaction, product type, segment, purchase amount, and the type of loyalty program associated with the customer.

### B.2 Baselines implementation details

Below, we provide details about the architectures and hyperparameters of the baseline approaches used in our study.

Handcrafted features with LightGBM: This baseline aggregates numerical feature values across buckets and includes statistics such as count, mean, variance, minimum, and maximum. The LightGBM classifier (Ke et al., ) is then used for prediction.

Randomly initialised RNN encoder: This approach utilizes a randomly initialized and untrained RNN sequence encoder based on a unidirectional Gated Recurrent Unit (GRU) with a single hidden layer of size 1024. The resulting 1024-dimensional event sequence representations are used with LightGBM to solve the downstream task.

CoLES (Contrastive Learning for Event Sequences): This method employs a self-supervised contrastive pretraining approach called CoLES (Babaev et al., ) to generate vector representations of event sequences. The encoder is a recurrent neural network (RNN) GRU with one hidden layer of size 1024, producing final embeddings of the same size. A supervised classifier based on LightGBM is then trained using the pretrained embeddings.

CPC (Contrastive Predictive Coding): This approach uses a similar sequence encoder architecture to CoLES but applies the Contrastive Predictive Coding (CPC) method (van den Oord et al., ) for pretraining. CPC is a self-supervised technique for learning vector representations using an autoregressive model for non-discrete data sequences.

Barlow Twins: This method follows the same scheme and sequence encoder architecture as CoLES and CPC but implements a Barlow Twins Loss (Zbontar et al., ) for encoder pre-training. LightGBM is then used on the obtained embeddings for solving the downstream problem.

NSP (Next Sequence Prediction): This baseline employs an RNN sequence encoder with a unidirectional GRU and a single hidden layer of size 1024, pretrained on the Next Sequence Prediction task (Devlin et al., ). The resulting 1024-dimensional embeddings are used with LightGBM for the downstream task.

RTD (Replaced Token Detection): Similar in architecture to the NSP baseline, this approach uses the Replaced Token Detection loss function from the ELECTRA paper (Clark et al., ).

SOP (Sequences Order Prediction): Identical in architecture to NSP and RTD, this baseline uses the Sequences Order Prediction loss function from the ALBERT work (Lan et al., ).

MLM NSP (Masked Language Modelling with Next Sentence Prediction): This approach uses a LongFormer (Beltagy et al., ) with 4 attention heads, 8 hidden layers of dimension 2048, and a maximum of 2000 positions as an event encoder. The output embedding size is 2048. The encoder is pretrained using a combination of Masked Language Model and Next Sentence Prediction tasks as in BERT (Devlin et al., ). LightGBM is then used on the obtained embeddings for the downstream task.

TabFormer: This approach implements the TabFormer method (Padhi et al., ), utilizing a LongFormer (Beltagy et al., ) with 4 attention heads, 8 hidden layers of dimension 2048, and a maximum of 2000 positions as the sequence encoder. The output embedding size is 2048. The encoder is pretrained using the Masked Language Modelling (MLM) task (Devlin et al., ). LightGBM is then used on the obtained embeddings for solving the downstream problem.

GPT: This approach uses a GPT-2 architecture (Radford et al., ) as the event sequence encoder, with 12 layers, 12 heads per layer, and position encoding up to 2056 positions. The embedding dimension is 768. The encoder is pretrained on an autoregressive task of predicting the fields of the next transaction, each using a separate head. LightGBM is used on the obtained embeddings for the downstream task.

RNN with CoLES: This baseline differs from the standard CoLES approach by adding several MLP heads to the event sequence encoder after contrastive pre-training. This architecture is then end-to-end trained on the target task.

GPT with descr.: This approach modifies the conventional GPT-2 baseline by applying discretization to the numerical features of events.

CatBoost: A simple implementation of the CatBoost algorithm (Ostroumova et al., ) trained on event features.

Text LLM: This text-based LLM approach serializes event features into a string using a template, selecting only the attributes necessary for the task while ignoring others due to the long token sequence. The length of event sequences is also reduced to fit the language model’s context. For this baseline, we used the FLAN-T5-xl (Wei et al., ) model.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Why did you choose only a few datasets for transactions and not for other domains? You have mentioned in the introduction section that event prediction is widely used in various fields such as healthcare, sociology etc. I have read the sources of the datasets you have used and found that they are rarely used by other works. So, I think the datasets are not representative.
2. Why are all the large language models you use for experiments the FLAN-T5 series? Why don't you use the newer LLaMA series or the Mistral series?
3. How did you get the two empirical values of 0.6 and 1.56 in rows 168-169? Have you tried other values?
soundness: 2
strengths: 1. The authors take advantage of categorical and numeric features from tabular neural networks at the same time in Section 3.2.
2. In Section 3.3, the choice of Whisper as an Encoder after extensive comparative experiments is convincing.
summary: The authors present a new approach to apply large language models to the task of event sequence prediction. This approach is a little fine-tuned and less expensive to use. It takes into account both time and numerical sequence processing. The authors chose FLAN-T5 as the base model and conducted experiments on several datasets related to customer transactions. In addition to experiments on specific downstream tasks, the generalization ability of the large language model for event sequence prediction was tested.
weaknesses: 1. I think the biggest problem with this paper is the lack of sensible baselines. The authors ignore some recent applications of large models for event prediction in the last year. For example [1], [2]. The authors use earlier baselines and therefore the results obtained are not convincing. In particular, according to Table 3, ESQA does not achieve a consistent lead compared to NPPR either.
[1] Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding
[2] MIRAI: Evaluating LLM Agents for Event Forecasting

2. I think the novelty of the ESQA method itself is limited. The authors have mentioned time-series prediction in the related work section. In fact, there is a lot of recent work dealing with complex data structures such as time-series, e.g., [3], [4]. They have embedded complex data structures individually and used the combination of freezing and tuning. So, I don't think similar ideas used for event sequence prediction task is very novel.
[3] TIME-LLM: TIME SERIES FORECASTING BY REPROGRAMMING LARGE LANGUAGE MODELS
[4] LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters

3. The authors' descriptions of their methods and the results are often confusing. For example, the Encoder model is used in Figure 1, and the Event Sequence Encoder is used in Figure 3.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Refer to the weakness.
soundness: 3
strengths: S1: Experimental Design: The paper supports its proposed approach with a series of experimental setup, utilizing multiple datasets that span various event sequence types and domains, enhancing the robustness and applicability of the findings. 

S2: Each component of the ESQA framework is backed by experimental results, validating the selection and effectiveness of different modules. 

S3: ESQA achieves state-of-the-art or competitive results on a range of tasks, highlighting its versatility and effectiveness in handling diverse event-sequence tasks.
summary: This paper introduces ESQA (Event Sequences Question Answering), a framework designed to handle question-answering tasks in the domain of event sequences. Leveraging large language models (LLMs), ESQA addresses unique challenges in processing temporally structured data, such as managing long input sequences and encoding time and numeric features effectively. The framework showcases its adaptability across various downstream tasks without extensive fine-tuning, achieving comparable performance across multiple event-sequence datasets in domains like finance and e-commerce.
weaknesses: Overall, the paper lacks novelty beyond the designed QA framework with backbone encoder and LLM. Moreover, the organization and experiment settings in paper can be further optimized. 

W1: Inadequate Background Explanation: The paper contains some strongly worded statements in the background, such as “no effort was made to adapt LLMs to event sequences.” However, event sequence tasks share similarities with event prediction tasks, making it essential for the authors to clarify the differences between these two. Since LLMs have already been applied to various event prediction tasks, it would be valuable to consider these models as baselines for comparison. Additionally, there is considerable prior work on LLMs in question answering, and including these models as baselines could further strengthen the experimental design. 

W2: Lack of Rigorous Annotation: The methodology section lacks precise annotation and contains numerous typographical issues. For example, “Q-Former” in the related work section is not properly cited, which could lead to misunderstandings. Some symbols, such as special tokens, are only briefly introduced in figures without corresponding explanations in the text. This could create confusion among readers, highlighting the need for careful annotation and early definition of terms. 

W3: Insufficient Justification of Component Selection: The paper could enhance the methodology by explaining the motivations behind selecting each component of the framework, beyond the experimental advantages. A discussion of why each component was chosen in alignment with the framework’s goals would strengthen the readers' understanding of the architecture. 

W4: Mismatch Between Methodology Text and Figures: Certain descriptions in the methodology section do not fully align with the figures. For example, how the Input prompt, query, and Question are integrated is not clearly explained, and the division in the embedding matrices shown in Figure 2 could lead to misinterpretation. Moreover, although the authors state that the LLM is not fine-tuned, LoRA, a traditional fine-tuning technique—is used. These inconsistencies should be clarified to avoid potential misunderstandings. 

W5: Limited Experimental Baselines: The experimental design could benefit from considering more comprehensive baseline models from existing LLM research, especially LLM applications in question answering and traditional event prediction, to provide a more robust evaluation of the proposed framework.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Converting the structured features of an event is not difficult. Once converted, it becomes a sequence of embedding, which is identical to other LLM-based sequence modeling methods, e.g., those for time series. What is the uniqueness of this paper compared to other LLM-based sequence prediction problems?

2. What is the reason for only considering numeric and categorical features?

3. Why FLAN-T5 is used as the backbone? Have you tested other LLMs such as GPT4, Llamma3, etc.?

4. Only evaluating the method on fintech data cannot justify its generalisability. Why not compare these methods on social media data and healthcare data?

5. What is the exact reason that the proposed method underperforms the competitors on AlfaBattle and Gender datasets? What are the unique properties of these datasets that lead to the performance degrade of the method? You mentioned the two datasets exhibit class imbalance. Does this mean your method is vulnerable to imbalanced data?

6. What are the tasks used for training ESQA m/t, and how are they different from the tasks in the testing phase?
soundness: 2
strengths: 1. The paper proposes a method to encode features of different formats and integrate the embedded features with questions as input to LLMs. The motivation for using LLMs is clearly presented.
2. By leveraging LLMs, the proposed method can generalise to unseen tasks.
3. The proposed method outperforms the baseline methods on some of the datasets.
4. In general, the paper is well-written and easy to follow.
summary: This paper introduces an LLM-based method for modeling event sequences. By leveraging LLMs, this research aims to improve the generalization and modeling quality of the existing methods for event sequence prediction tasks. Benefiting from LLM's knowledge from other domains, the proposed method is able to perform unseen tasks. The experimental results demonstrate the capability of the proposed method in achieving state-of-the-art results on certain tasks compared to the existing methods.
weaknesses: **Novelty**:
1. The paper claimed it is the first to apply LLMs to event sequences. This claim does not seem to be correct. There have studies [1,2] using LLMs for event sequences.
2. The technical novelty of this paper is weak. The event encoder adopts several well-known machine learning techniques, such as binning, to convert the raw features to learnable embeddings. The connector is a direct application of an existing technique Q-former. The training protocol applies LORA, a well-known fine-tuning technique for LLMs. It is hard to see the originality of the proposed method.

**Method Design**:

1. Although discretisation is one way to convert numeric features into categorical features, it also increases the dimensions and parameters of the model. If an event has a large number of features, the event embedding (obtained by concatenating the embedding of each feature) can be very large, which may lead to a large number of parameters in the projection layer. Discretisation may not be parameter-efficient in this case.
2. The method is limited to categorical and numeric data. It lacks support for other data types. For instance, in social media, plain text and images are very common attributes.
3. The motivation for choosing only FLAN-T5 as the backbone is unclear.

**Experiments**:

1. Although the paper claimed event sequences are prevalent in real-world applications, including social networks and healthcare, the paper only evaluates the methods on fintech data.
2. The latest methods compared in the paper were published in 2022. More recent methods could be included as baselines, such as [1,2]. Besides, the proposed method does not have fundamental differences compared to LLMs for time series or temporal knowledge graphs, except for the feature encoder; it is a question why the authors do not compare with those LLM-based methods in other domains.
3. The proposed method underperforms the baselines on certain datasets. However, there is no in-depth analysis of the reason. 
4. LLMs are known to be unstable in text generating. The output of independent runs might be different. However, there is no standard deviation is reported in the experimental results. Compared to existing methods, LLM-based method might have a larger variance. The authors need to justify that the proposed method is stable and that the improvements are statistically significant.
5. The experimental setting for the generalization abilities is not clearly presented. The method ESQA m/t is trained in a multi-task setting. Do these tasks include the task in the testing phase? The low performance of ESQA z/s is not surprising because it is trained only one a single task. 

**Minor comments**: 

1. The citation format seems to be incorrect. For instance, "This type of data is widely spread in domains like geoscience Bergen et al. (2019)" --> "This type of data is widely spread in domains like geoscience (Bergen et al. 2019)".
2. The definition $c_j^{cat} \in |c_{j}|$ is confusing. $|c_{j}|$ is often used to denote the size of a set.
3. On page 3, "a sequence of f-dimensional vectors in$E_{\theta}$". There should be spacing between "in" and "$E_{\theta}$".

**Reference**:

[1] Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Zhang, Jun Zhou, Chenhao Tan, Hongyuan Mei. Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning. NeurIPS 2023.

[2] Matthew McDermott, Bret Nestor, Peniel Argaw, Isaac S Kohane. Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events. NeurIPS 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. What is the compress rate between the query embeddings and the textual representation of the structured data?
2. Have the authors considered alternative approaches for dealing with imbalanced classes, such as integrating class weights into the cross-entropy loss function?
soundness: 3
strengths: - Originality: The paper introduces a creative adaptation of LLMs to the ES domain that has not been extensively explored with such models. While prior work has adapted LLMs to other non-language data (e.g., time series), the specific application to ESs with the question-answering format is novel.
- Quality: The paper offers extensive experiments on relevant datasets from real-world domains. The use of LLMs for ES modeling and the integration of Q-Former as a connector between the LLM and event sequence encoder are sound technical contributions.
- Clarity: The paper is generally clear in its description of the methods and experiments.
- Significance: The proposed approach is highly significant in the context of applied machine learning, particularly in industries reliant on ES data. The ability to leverage LLMs for ES tasks with minimal fine-tuning could lead to substantial efficiency improvements in data processing and decision-making systems.
summary: This paper presents a novel approach to handling event sequences (ESs) by adapting LLMs to the ES domain for multiple downstream tasks like extraction and prediction, without the need for extensive fine-tuning. This work addresses challenges in processing ESs, including long sequence lengths and temporal features. Key contributions include: (1) proposing a novel framework for processing long ESs using LLMs, and (2) leveraging the Q-Former to efficiently compress and integrate ESs into LLMs. State-of-the-art results on multiple ES tasks across several datasets are achieved.
weaknesses: - Quality: The proposed method has certain limitations, particularly in handling imbalanced classes and discretization errors in numeric features, as noted by the authors.
- Clarity: There are several typos and styling issues, for example:
  - Numerous citations should be in parenthesis using \citep command.
  - The authors have mentioned that "the second best results are underlined" in Table 4's caption, but the table contains no underlines.
  - In the captions of Table 3-5, "baseline approaches presented in Section 4.2.1" should actually be Section 4.1 or Appendix B.3.
- Significance: The model has been tested primarily on financial data, but not on more diverse domains, such as healthcare or social media interactions, thus the domain generalizability of the model is not demonstrated.