CoLES: Contrastive learning for event sequences with self-supervision
Authors: Dmitrii Babaev
Nikita Ovsov
Ivan A Kireev
Gleb Gusev
Maria Ivanova
Alexander Tuzhilin
Venue:  (2021)
Keywords: representation learning
contrastive learning
neural networks
event sequiences

Abstract: We address the problem of self-supervised learning on discrete event sequences generated by real-world users. Self-supervised learning incorporates complex information from the raw data in low-dimensional fixed-length vector representations that could be easily applied in various downstream machine learning tasks. In this paper, we propose a new method CoLES, which adopts contrastive learning, previously used for audio and computer vision domains, to the discrete event sequences domain in a self-supervised setting. Unlike most previous studies, we theoretically justify under mild conditions that the augmentation method underlying CoLES provides representative samples of discrete event sequences. We evaluated CoLES on several public datasets and showed that CoLES representations consistently outperform other methods on different downstream tasks.


Under review as a conference paper at ICLR 2021
COLES: CONTRASTIVE LEARNING FOR EVENT SE- QUENCES WITH SELF-SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
We address the problem of self-supervised learning on discrete event sequences generated by real-world users. Self-supervised learning incorporates complex in- formation from the raw data in low-dimensional fixed-length vector representations that could be easily applied in various downstream machine learning tasks. In this paper, we propose a new method CoLES, which adopts contrastive learning, previ- ously used for audio and computer vision domains, to the discrete event sequences domain in self-supervised setting. Unlike most previous studies, we theoretically justify under mild conditions that the augmentation method underlying CoLES provides representative samples of discrete event sequences. We evaluated CoI.ES on several public datasets and showed that CoLES representations consistently outperform other methods on different downstream tasks.
INTRODUCTION
A promising and rapidly growing approach known self-supervised learning is the main choice for pre- pre-training in situations where the amount of labeled data for the target task of interest is limited.
Most of the research in the area of self-supervised learning has been focused on the core machine
learning domains, including NL. (e.g., ELMO (Peters et al.] 2018). BERT (Devlin et al., (019h). speech (c.g.. CPC (van den Oord et al. 2018) and computer vision (Doersch et al.) 2015 van den Oord et al.) 2018). However, there has been very little research on self-supervised learning in the
domain of discrete event sequences, including user behavior sequences (Ni et al. 2018) such as credit card transactions at banks, phone calls and messages at telecom, purchase history at retail and click-stream data of online services. Produced in many business applications, such data is major key to the growth of modern companies. User behavior sequence is attributed to a person and captures regular and routine actions of a certain type. The analysis of these sequences constitutes an important sub-field of machine learning (Laxman et al. 2008 Wiese and Omlin 2009 Zhang et al. 2017
Bigon et al. 2019).
NLP, audio and computer vision domains are similar in the sense that the data of this type is "continuous": short term in NLP can be accurately reconstructed from its context (like pixel from its neighboring pixels). This fact underlies popular NLP approaches for celf-supervision such as BERT's s Cloze task (Devlin al. 2019) and approaches for sufersuperiiion in audio and computer vision, like CPC (van den Oord et al. 2018). In contrast, for many types of event sequence data, single token cannot be determined using its nearby tokens, because the mutual information between a token and its context is small. For this reason, most ofate-of-the-ar the-art self- self-supervised methods not applicable to event sequence data.
In this paper, propose the COntrastive Learning for Event Sequences (CoLES) method that learns low-dimensional representations of discrete event sequences. It is based on a novel theoretically grounded data augmentation strategy, which adapts the ideas of contrastive learning (Xing et al.) 2002 Hadsell et al.] (006)) to the discrete event sequences domain in a self-supervised setting. The aim of contrastive learning is represent semantically similar objects (positive pairs of images, video, audio, etc.) closer to each other, while dissimilar ones (negative pairs) further away. Positive pairs are obtained for training either explicitly, e.g., in manual labeling process or implicitly using different data augmentation strategies (Falcon and Cho (220)). We treat explicit cases as
See, e.g., keynote by Yann LeCun ICLR-20: http://wwwwiccrccc//intulll hatp:///ccccccuuuuaaa/00/////rrr 7.hhml

---

Under review as a conference paper at ICLR 2021
supervised approach and implicit cases as self-supervised one. In most applications, where each person is represented by one sequence of events, there are no explicit positive pairs, and thus only self-supervised approaches applicable. Our CoLES method is supervuue and based on the observation that event sequences usually possess periodicity and repeatability of their events. We propose and theoretically justify new augmentation algorithm, which generates sub-sequences of an observed event sequence and uses them as different high-dimensional views of the same (sequence) object for contrastive learning.
Representations produced by the COLES model can be used directly as a fixed vector of features in some supervised downstream task (e. g. classification task) similarly to (Mikolov et al. 2013] Song
2017 ZZhai et al.) (019). Alternatively, the trained CoLES model can be fine-tuned (Devlin et al.
it al.
2019 for the specific downstream task. We applied CoLEES to several user behavior sequence datasets with different downstream classification tasks. When used directly as feature vectors, CoLES representations achieve strong performance comparable to the hand-crafted features produced by data scientists. We demonstrate that fine-tuned CoLES representations consistently outperform methods based on other representations by a significant margin. We provide the full source code for all the experiments described in the pape
This paper makes the following contributions: (1) We present the CoLES method that adapts contrastive learning in the self-supervised setting to the discrete event sequence domain. (2) We propose a novel theoretically grounded augmentation method for discrete event sequences. (3) We demonstrate that CoLES consistently outperforms previously introduced supervised, self-supervised and semi-supervised learning baselines adapted to the event sequence domain. We also conducted a pilot study on event sequence data of a large European bank. We tested CoLES against the baselines and achieved superior performance on downstream tasks which produced significant financial gains, measured in hundreds of millions of dollars yearly.
The rest of the paper is organized as follows. In the next section, we discuss related studies on self-supervised and contrastive learning. In Section 3 we introduce our new method CoLES for discrete event sequences. In Section 4 we demonstrate that CoLES outperforms several strong baselines including previously proposed contrastive learning methods adapted to event sequence datasets. Section is dedicated to the discussion of our results and conclusions.
2 RELATED WORK
Contrastive learning has been successfully applied to constructing low-dimensional representa- tions (embeddings) of various objects, such images (Chopra et al.; [2055] |Schroff et al.) (001)). texts (Reimers and Gurevych] 2019), and audio recordings (Wan et al. 20.8). The aim of these studies is identify the object based its sample (Schroff al.] 2015: Hu et al.] 2014 Wan et al.. 2018). Therefore, their training datasets explicitly contain several independent samples per each particular object, which form positive pairs as a critical component for learning. These supervised approaches are not applicable to our setting.
For situations when positive pairs are not available their amount is limited, augmentation techniques were proposed in the computer vision domain. One of the first frameworks with augmentation was proposed by Dosovitskiy et al. (2014). In this work, surrogate classes for model training were introduced using augmentations of the same image. Several recent works (Bachman et al. 2019 Hel jtt al. 2019, Chen et al. 2020) extended this idea by applying contrastive learning methods, they nicely summarised by Falcon and Cho (2020). Although augmentation techniques proposed in these studies provide good performance empirically, we note that no theoretical background behind different augmentation approaches has been proposed so far.
Contrastive Predictive Coding (CPC) is self-supervised learning approach proposed for non-discrete sequential data (van den Oord et al. 2018). CPC extracts meaningful representations by predicting latent representations of future observations of the imput sequence and using autoregressive methods. CPC representations demonstrated strong performance on four distinct domains: audio, computer vision, natural language and reinforcement learning. We adapted the CPC based approach to the domain of discrete event sequences and compared with our CoLES approach (see Section 4.2)
hhtps///githu..co///s////// (the link was anonymized for the double-blind peer review purposes)
2

---

Under review as a conference paper at ICLR 2021
Independently of our study, several papers appeared in the past few months self supf-uuuen for user behavior sequences in the recommender systems domain. Zhhou et al. (2020a) proposed to use CPC-like approach for self-supervised learning user clicks history. Ma et al. (2020) used an auxiliary self-supervised loss on click sequences. (Zhou et al. (2020b) proposed "Cloze" task from BERT (Devlin al.) (2099) for self-supervision on purchase sequences. Finally, Yao et al. (2020)
adapts a SimCLR-like approach for text-based tasks and tabular data. Although there has been significant progress on contrastive learning, augmentation for contrastive learning does not have any theoretical grounding and is understudied in the domain of discrete event sequences.
PROBLEM FORMULATION AND OVERVIEW OF THE COLES METHOD
3.1 PROBLEM FORMULATION
While the method proposed in this paper could be studied in different domains, in this paper we focus on discrete sequences of events. Assume there are some entities e, and the life activity of each entity e Entities could be people or organizations or
is observed as a sequence of events Ie == (Ie (t=et)))= some other abstractions. Events xee (t) may have any nature and structure (e.g., transactions of client, click logs of a user), and their components may contain numerical, categorical, and textual fields (see
datasets description in Section
According to theoretical framework of contrastive learning proposed in Saunshi et al. (2019), each entity is latent class, which is associated with distribution P. over its possible samples (event sequences). However, unlike the problem setting of Saunshi et al. (2019), we have no positive pairs, i.c. pairs of event sequences representing the same entity e. Instead, we have only one sequence ze per entity e. Formally, each entity is associated with a latent stochastic process (X.(t) we observe only one realisation {x={ generated by the process (X), Our goal to learn an encoder M that maps event sequences into a feature space Rd in such way that the obtained embedding ce M ({xe]) E Id of sequence {t) encodes essential properties of e and disregards any randomness and noise contained in the sequence. That is, embeddings M( }) and (((r2)) should be close to each other, if æ1 and i2 are sequences generated by the same process {Xe (t)}. and they should be further away, when generated by distinct processes. The quality of representations can be examined by downstream tasks in the two ways: (1) c, can be used as a feature vector for task-specific model, and (2) encoder M can also be (jointly) fine-tuned Yosinski et al. 2014).
and
3.2 SAMPLING OF SURROGATE SEQUENCES AS AN AUGMENTATION PROCEDURE
While we have no access to the latent processes (4), we need to use augmentation. Most augmentation techniques proposed earlier for continuous domains (such as image jätter, color jitter or random gray scale in computer vision, see Falcon and Cho (202)) are not applicable to discrete events. A possible approach for augmentation is generating sub-sequences of the same event sequence {t)) The idea proposed below resembles the bootstrap method (Efron and Tibshirani 1994). which enables to generate several bootstrap samples using only one sample of independent datapoints of latent distribution. However, our setting is different, since we have no independent observations, so we should rely on different data assumptions. The key property of event sequences that represent life activity is periodicity and repeatability of its events (see Figure 4 in the Appendix [D for the empyrical observations of these properties for the considered datasets). This is motivation for the Random slices sampling method applied in CoLES, as presented in Algorithm [ Each sub-sequence is generated from the initial sequence as its connected segment ("slice") using the following three steps. First, the length of the slice is chosen uniformly from possible values. Second, its starting position is uniformly chosen from all possible values. Third, too short (and optionally too long) sub-sequences are discarded. It could seem that the mean length of obtained sub-sequences are less than the mean length of sequences in the dataset. However, show in the next section that the distribution of subbeeeueeces is close to the initial distribution in some realistic assumptions. The overview of the CoLE method is presented in Figure
3

---

Under review as a conference paper at ICLR 2021
User 1 event sequence
Embedding vectors
Jub-sequence
CoLES
Minimize distance
Encoder
sub-sequence 2
User event sequence
distance
Embedding vectors
sub-sequenie 3
CoLES Encoder
Minimize distance
sub-sequence
Figure 1: General framework
Algorithm 1: Random slices sub-sequence generation strategy
hyperparameters: m. M: minimal and maximal possible length of sub-sequence k: number of trials.
input: A sequence S of length T.
output: S: sub-sequences of S.
for i + 1 to k do
Generate random integer uniformly from [1, []; i; E [m, .M then
Generate random integer from [0, T; 1]; Add S; == SIs s + T; 1] to
end
3.3 THEORETICAL ANALYSIS
segment of a latent process (1)E) which generates sequence of all events in the potentially infinite life of entity That is, assume that Xx
Assume that process (X)(1)
(t se) for some random starting point s, E {0, I.... and horizon Te. Thus we observe, in our data, segment [s. + 1, Tel of the life of e. We also make the following Assumptions:
1. Process {t) is cyclostationary (in the strict sense) (Gardner et al. 2006) with some period? 7.
2. Starting is independent, and the distribution of (se mod T) is uniform over [0, 1].
3. Horizon T, independent and follows power-law distribution on [m, oo).
These assumptions correspond to a scenario where some persons become clients of a service at random moment and for some random time span and their behaviour obey some periodicity.
Theorem 1.
If sequences (xx(t)} in the dataset are generated from latent processes (t)} as described above with a lower bound m for the length of sequence (x)(t)) then sub-se seq-eeqeessss obtained by Algorithm| from { follow the same distribution as {x)(t) up to slight alteration of the distribution of the length T.. Namely, T. follows power law with an exponent a < -1, then the density function for the length TV of a sub-sequence satisfy
-a
k
-a p(T =k) for any is E [m, oo]. (1)
PTT k)
S p T 13)
m
1/2
This theorem means that a sub- sub-sequence obtained by Algorithm I is a representative sample of entity and follows its latent distribution P.. Combining this result with generalization guarantees proved for setting with explicitly observed positive pairs (Saunshi et al. (2019),, we obtain theoretical background for our implicit self-supervised setting. See Appendix for the proof of Theorem
4

---

Under review as a conference paper at ICLR 2021
3.4 MODEL TRAINING
Batch generation. The following procedure creates a batch during CoLES training. N initial sequences are randomly taken and K sub-sequences produced for each of them. Pairs of sub- sequences of the same sequence are used as positive samples and pairs from different sequences are used negative ones.
We consider several baseline empirical strategies for the sub-sequence generation to compare with Algorithm [ The simplest strategy random sampling without replacement. One more strategy is to produce uub-sequences by the random splitting of the initial sequence to several connected segments without intersection between them (see Appendix [A].
Contrastive loss We consider classical variant of the contrastive loss, proposed by (Hadsell al. 2006): L = (1 Y) (Diy)* + Y 2 [mar)(0, m )3, where Diy a distance function between embeddings in i-th labeled sample pair, Y is binary variable identifying that the pair is positive. As proposed (Hadsell et al. 2006). we use euclidean distance function: B.)².
Di = D(A, B) -
(A, -
Pair distance calculation. In order to select negative samples, we need to compute the pairwise distance between all possible pairs of embedding vectors of batch. For the purpose of making this procedure more computationally effective perform normalization of the embedding vectors, i.c. project them onto a hyper-sphere of the unit radius (see Appendix[B]
Negative sampling is way to address the following challenge of the contrastive learning approach: using all pairs of samples can be inefficient: for example, some of the negative pairs are already distant enough, thus these pairs eet al. 2015). Hence, only a part of possible negative pairs in the batch are used during loss calculation. We compared the most popular choices for negative sampling applied for CoLES, see Section 4.2 for details.
not valuable for the training (Simo-Serra al. 2015 Schroff
3.5 ENCODER ARCHITECTURE
To embed a sequence of events to the fixed-size vector, we use an encoder network, which consists of two conceptual parts: the event encoder and the sequence encoder subnetworks.
The event encoder e takes the set of attributes of each single event 2t and outputs its representation in the latent space Rd: 2, e(x;)) The event encoder consists of several embedding layers and batch normalization layers. Each categorical attribute is encoded by its corresponding embedding layer. Batch normalization is applied numerical attributes of events. Outputs of all embedding and batch normalization layers are concatenated to produce latent representation 21.
The sequence encoder s takes latent representations of the sequence of events: 21:7 -1, 22. 2T
and outputs the representation of the whole sequence C₁ in the time-step t: = 5((1=) Several
approaches can be used to encode a sequence (Cho et al.) 2014 Vaswani et al.) 2017)
our
experiments we use the recurrent network (RNN) similarly to (Sutskever et al. 2014). The output
produced for the last event is used to represent the whole sequence of events. the case of RNN the last output he is representation of the sequence.
To summarise, the CoLES method consists of three major ingredients: event sequence encoder, positive and negative pair generation strategy and the loss function for contrastive learning.
EXPERIMENTS
We compare method with existing baselines several publicly available datasets from various data science competitions. We chose datasets with sufficient amounts of discrete events per user.
Age group prediction competition The dataset of 44M anonymized credit card transactions representing 50k persons was used to predict the age group of a person. The label is known for 30k persons, other 20k are unlabelled. The group ratio balanced in the dataset. Each transaction includes the date, type, and amount being charged.
5

---

Under review as a conference paper at ICLR 2021
Churn prediction competition" The dataset of 1M anonymized card transactions representing 10K clients was used to predict a churn probability. Each transaction is characterized by date, type, amount and Merchant Category Code. 5k clients have labels, 5.2k clients haven't labels. Target is binary, almost balanced with proportions 0.55 and 0.45.
Assessment prediction c The task is predict the in-game assessment results based on the history of children's 's gameplay data. Target is one of 4 grades, with proportions 0.50, 0.24, 0.14, 0.12. The dataset consists of 12M gameplay events combined in 330k gameplays representing 18k children. 17.7k gameplays are labeled, the remaining 312k gameplays are not labeled. Each gameplay event is characterized by timestamp, event code, the incremental counter of events within game session, time since the start of the game session, etc.
Retail purchase history age group prediction The task is to predict the age group of client based on its retail purchase history. The group ratio is balanced in the dataset. Only labeled data is used. The dataset consists of 45, .5,8M retail purchases representing 400k clients. Each purchase is characterized by time, product level, segment, amount, value, loyalty program points received.
As we can see Figure 3 (Appendix D) these datasets satisfy the power law assumption for the sequence length distribution of Theorem Also, as shown in Figure 4 (Appendix D the datasets satisfy the periodicity and repeatability assumption.
Dataset split. For each dataset, we set apart 10% persons from the labeled part of the data as the test set that we used for evaluation of different models. The rest 90% of labeled data and unlabeled data constitute our training set used for learning, For all methods, a random search on 5-fold cross- validation over the training set is used for Hyper-paramette selection. The hyper- parameters with the best out-of-fold performance are then chosen. For the learning of semi- seeiiissursssseellf--ssese superrvsss techniques (including CoL.ES), we used all transactions of training sets including unlabeled data. The unlabelled parts of the datasets were ignored while training supervised models.
Performance. Neural network training was performed on a single Tesla P-100 GPU card. For the training part of CoLES, the single training batch is processed in 142 milliseconds. For example, in the age group prediction dataset the single training batch contains 64 unique persons with sub sequences per person, i.e. 320 training seeueeeeeeess in total, the mean number of transactions in sub-sequence is 90, hence each batch contains about 28800 transactions.
Hyperparameters Unless we explicitly specify, we use contrastive loss and random slices pair generation strategy for CoLES in our experiments (see Section 4.2 for motivation). The final set of hyper-parametery used for CoLES is shown the Appendix Table
4.1 BASELINES
LightGBM. We consider the Gradient Boosting Machine (GBM) method (Friedman 2001) on hand-crafted features. GBM can be considered as strong baseline cases of tabular data with heterogeneous features. (Wu et al. 2009 Vorobev et al. 2019 Zhang and Haghani 2015 Niu et al. 2019). GBM based model requires a large number of hand-crafted aggregate features produced from the raw transactional data. An example of an aggregate feature is an average spending amount in some categories of merchants, such as hotels of the entire transaction history. We used LightGBM (Ke) ett al. (017)) implementation of the GBM algorithm with nearly 1,000 hand-crafted features for the application. The details of producing hand-crafted features can be found in the Appendix E.
Self-supervised baselines.
NSP. We consider a simple baseline inspired by the next sentence prediction task used in BERT (De vlin et al. 2019). Specifically, we generate two sub-sequences A and B, in way that 50% of the time B is the sub-sequence from the same sequence as A and follows it (positive pair), and 50% of the time it is a random sub-sequence taken from another sequence (negative pair).
htttps///bootttrsspp//hhmmmimmmp https://www.kaggee htt.g///..cc//caa-ssccccceeooo/o "http:///ds. http://dd...oooooooo////////alad

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of batch generation strategies
Dataset
Random samples
Random disjoint samples
Random slices
Age group (Accuracy)
0.613 0.006
0.619 + 0.011
0.639 +0.006
Churn (AUROC)
0.820 + 0.014
0.819 + 0.011
0.823 =0.017
Assessment (Aceuracy)
0.563 + 0.004
0.563 + 0.004
0.618 +0.009
Retail (Accuracy)
0.523 0.001
0.505 + 0.002
0.542 #0.002
5-fold cross cross-validation metric +95% is shown
SOP. Another simple baseline is the same as sequence order prediction task from ALBERT (Lan et al. 2020). It uses two consecutive sub-sequences as a positive pair, and two consecutive sub-sequences with swapped order as a negative pair.
RTD. We also adapt the replaced token detection approach from ELECTRA CClark et al. 2020) for event sequences as a baseline for our research. We replaced 15% of events from the sequence with random events, taken from other sequences and train model predict whether event is replaced or not.
CPC. As the last self-supervised baseline, selected the recently proposed Contrastive Predictive Coding (CPC) (van den Oord et al. 2018), a self-supervised learning method that produced an excellent performance on sequential data of such traditional domains as audio, computer vision, reinforcement learning and recommender systems (Zhou et al. 2020a).
Supervised learning. In addition to the aforementioned baselines, we compare our method with supervised learning approach where the encoder network (see Section and the classification sub-network h are jointly trained on the downstream task target, e. the classification sub-network h(e(r)). One-layer neural net with softmax
takes encoder output and produces a prediction: activation is used as h. Note that no pre-training used in this case.
Note that all neural network baselines use the same architecture of the encoder model as CoLES.
4.2 RESULTS
Features of CoLES. To evaluate the proposed method of sub-sequence generation we compared
with two alternative strategies described Section 3.4 The results are presented in Table [
The proposed random slices sub-sequence generation strategy significantly outperforms alternative
strategies, what confirm theoretical results (see Section 3.3). Also, note that the random samples
strategy is similar to the augmentation strategy proposed by Yao et al. (2020), and the random disjoint
samples strategy is similar to sub-sequence generation proposed by] Ma et al. (2020).
We evaluated several possible loss functions and found that contrastive loss that can be considered as the basic variant of contrastive learning loss, performs on par or better than other losses on the downstream tasks (see Appendix F.T Table 7). This means that improvements obtained by more recent losses in object recognition tasks does not necessarily lead to gains in other downstream tasks.
We also compared popular negative sampling strategies (distance-weighted sampling (Manmathal let al. (201)), and hard-negative mining (Schroff et al. 2015)) with random negative sampling strategy. The results are shown in the Appendix [.1 Table g We found that hard negative mining leads to
significant increase in quality on downstream tasks in comparison to random negative sampling.
Comparison with baselines. We compared CoLES with baselines described Section 4.1 two scenarios. First, we compared embeddings produced by the CoLES encoder with other types of embeddings and with manually created aggregates by using them as input features of downstream task model. The downstream task model is trained by LightGBM (Ke et al., 2017) independently from the sequence encoder. As Table demonstrates, our method generates sequence embeddings of sequential data that achieve strong performance results in comparison to the case of manually crafted features when used on the downstream tasks. In particular, Table 2 shows that even unsupervised CoLES embeddings perform par and sometimes even better than hand-crafted features. Also note, that CoLES embeddings outperform embeddings produced by the other self-super baselines on each dataset.
7

---

Under review as a conference paper at ICLR 2021
(a) Age group
(b) Assessment
0.650
0.60
0.625
0.600
Acouracy
Accuracy
0.575
0.56
0.550
0.525
lightGBM hand-crafted
Setup
0.51
lightGBM hand-crafted features
Setup
CPC Fine-tuning
CPC Fine tuning
0.500
COLES Fine-tuning Supervised learning
CoLES Fine tuning
Supervised learning
1.56%
3.12% 6.25% 12.5% 25.0% 50.0% 100.0% Shares of Labeled Datapoints
3.12%
6.26%
12.5%
25.0%
50.0%
100.0%
Shares of Labeled Datapeints
Model quality for different dataset sizes The rightmost point corresponds all labels and supervised setup.
Figure 2:
Table 2: Accuracy on the downstream tasks: Metric increase against baseline
Age group Accuracy
Churn AUROC
Assessment Accuracy
Retail
Method
Accuracy
LightGBM:
Designed features
0.631 + 0.004
0.825 + 0.005
0.602 + 0.006
0.547 0.001
SOP embeddings
-21.9% = 0.6%
-5.3% 0.8%
-4.1%
-22.8% + 0.2%
NSP embeddings
-1.5% 0.9%
+0.6% 0.77
-.5%
-22.3% 0.4%
RTD embeddings
+0.1% 0.6%
-2.9% = 0.8%
-3.6% 1.1%
-5.0% = 0.3%
CPC embeddings
-5.9% 0.6%
-2.9% 0.6%
-2.3% 1.8%+0.0 0.9%
-4.0% = 0.3%
CoLES embeddings
+1.1% + 1.2%
+2.2% +0.6%
-0.1% 0.9%
-1.4% = 0.2%
Supervised learning
0.628 + 0.005
0.817 = 0.012
0.602
0.542 = 0.001
RTD fine-tuning
+1.2% + 1.2%
+0.3% = 1.3%
227%
-0.5% = 0.4%
CPC fine-tuning
-2.1% + 1.6%
-0.9% 1.4%
+0.7% + 1.1%
-1.2% 0.3%
CoLES fine-tuning
+2.5% 11.0%
+1.1% +1.3%
+2.2% +1.1%
+1.9% +0.2%
test quality metric 1.95% is shown
In the second scenario, we fine-tune pre-trained models for specific downstream tasks. The models are pre-trained using CoLES and other self-supervised learning approaches and then additionally trained on the labeled data for the specific task in the same way as we trained a neural net for the supervised learning (see Section 4. neural net without pre-training also added to the comparison. As Table shows, fine-tuned representations obtained by our method achieve superior performance on the considered datasets, outperforming other methods by statistically significant margins.
Semi-supervised setup. evaluate our method in case the restricted amount of labeled data, we performed the series of experiments where only a fraction of available labels used to train the downstream task model. As in the case of the supervised setup, we compare the proposed method with LigthGBM over hand-crafted features, CPC, and supervised learning without pre-training (see Section
The results of this comparison presented in Figure Note that the difference in performance between CoLES and supervised-ong methods increases as we decrease the number of available labels. Also note that CoLESS consistently outperforms CPC for different volumes of labeled data.
8

---

Under review as a conference paper at ICLR 2021
Business applications. In addition to the described experiments on public datasets, have per- formed extensive testing our method on the private data in a large European bank. We ve observed significant increase in model performance (+ 2-10% AUROC) after the addition of CoLES embed- dings to the existing models many downstream tasks, including credit scoring, marketing campaign targeting, product recommendations cold start, fraud detection and legal entities connections predic- tion.
5 CONCLUSIONS
In this paper, present Contrastive Learning for Event Sequences (CoLES), novel sele-suue method for building embeddings of discrete event sequences. In particular, the CoLES method can be effectively used for pre-training neural networks in semi-supervised settings. It can also be used to produce embeddings of complex event sequences that can be effectively used in various downstream tasks.
We also empirically demonstrate that our approach achieves strong performance results on several downstream tasks and consistently outperforms both classical machine learning baselines on hand- crafted features, as well as other previously introduced self superviee and semi-supervised learning baselines adapted the event sequence domain. In the cumi-supervised setting, where the number of labeled data is limited, our method demonstrates even stronger results: the lesser is the labeled data the larger performance margin between CoLES and supervised-only methods.
The method is especially adapted for event sequence data which is extensively used by the core businesses of many large companies, including financial institutions, internet companies, retail and telecom.
REFERENCES
Bachman, D. Hjelm, and W. Buchwalter
2019. Learning representations by maximizing mutual information across views. ArXiv, 00.01906.00910.
Bigon, L.,, G. Cassani, C. Greco, L. Lacasa, M. Pavoni, A. Polonioli, and J. Tagliabue 2019. Prediction very hard, especially about conversion. predicting user purchases from clickstream data in fashion e-commerce. ArXiv, abs/1907.00400.
Chen, T., S. Kornblith, M. Norouzi, and G. E. Hinton 2020. A simple framework for contrastive learning of visual representations. abs/2002.05709.
ArXiv,
Cho. K., B. Merrienboer, Çaglar Gülççhhre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. ArXiv, abs//406.100
Chopra, S., R. Hadsell, and Y. LeCun
2005, Learning a similarity metric discriminatively, with application face verification. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '05). 1:539-50 13339-546000 vol. 1.
Clark, K., M.-T. Luong, Q. V. Le, and C. D. Manning
2020. Electra: Pre-training text encoders as discriminators rather than generators. ArXiv, abs/2003.1053:
Devlin, J.. M.-W. Chang, K. Lee, and K. Toutanova 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805.
Doersch, C.. A. Gupta, and A. A. Efros 2015. Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference Computer Vision (ICCV), Pp. 1422-1430.

---

Under review as a conference paper at ICLR 2021
Dosovitskiy, J. T. Springenberg, M. A. Riedmiller, and T. Brox
2014. Discriminative unsupervised feature learning with convolutional neural networks. In NIPS.
Efron, B. and R. J. Tibshirani
1994. An introduction to bootstrap. CRC press.
Falcon, W. and K. Cho
2020. A framework for contrastive self-supervised learning and designing new approach. ArXiv, abs/2009.00104.
Friedman, J. H.
2001. Greedy function approximation: A gradient boosting machine. The Annals of Statistics.
Gardner, W. A., A. Napolitano, and L. Paura
2006. Cyclostationarity: Half century of research. Signal processing, 86.4:639-697.
Hadsell, R., S. Chopra, and Y. LeCun
2006. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society
Conference on Computer Vision and Puttern Recognition (CVPR 06), 2:1735-1742.
He, K., H. Fan, Y. Wu, S. Xie, and R. B. Girshick 2019. Momentum contrast for unsupervised visual representation learning. ArXiv, abs/1 911.91722
Hoffer, E. and N. Ailon
2015. Deep metric learning using triplet network. In SIMBAD.
Hu, J. Lu, and Y.-P. Tan
2014. Discriminative deep metrie learning for face verification in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition, Pp. 1875-1882.
Kaya, M. and H. Şakir Bilge 2019. Deep metric learning: A survey. Symmetry, 11:1066.
Ke, G., Q Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu 2017. Lightgbm: A highly efficient gradient boosting decision tree. In NIPS.
Lan. Z., M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut
2020. Albert: lite bert for self-supervised learning of language representations. ArXiv, Abs/1909.11944
Laxman, S., V. Tankasali, and R. W. White
2008. Stream prediction using a generative model based on frequent episodes in event sequences. In KDD.
Ma, J., C. Zhou, H. Yang. Cui, X. Wang, and W. Zhu 2020. Disentangled self-supervision in sequential recommenders. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Manmatha, R., C.-Y. Wu, J. Smola, and P. Krähenbühl
2017. Sampling matters in deep embedding learning. 2017 IEEE International Conference on Computer Vision (ICCV), Pp. 2859-2867
Mikolov, T., Chen, G. Corrado, and J. Dean 2013. Efficient estimation of word representations in vector space. CoRR, abss/1301.777
Ni, Y., D. Ou, S. Liu, X. Li, W. Ou, Zeng, and L. Si 2018. Perceive your users in depth: Learning universal user representations from multiple e- commerce tasks. Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining.
Niu, X., L. Wang, and X. Yang
2019. A comparison study of credit card fraud detection: Supervised versus unsupervised. ArXiv, abs/1904, 10604.
10

---

Under review as a conference paper at ICLR 2021
Peters, M. E., M. Neumann, M. lyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer
2018. Deep contextualized word representations. ArXiv, abs/ Abb//1802.0565.
Reimers, N. and I. Gurevych
2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP/JJCNLP.
Saunshi, N., O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar
2019. A theoretical analysis of contrastive unsupervised representation learning. In In International Conference on Machine Learning, Pp. 5628-5637.
Schroff, F., D. Kalenichenko, and J. Philbin
2015. Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Pp. 815-823.
Simo-Serra, E., Trulls, I.. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer 2015. Diseriminative learning of deep convolutional feature point descriptors. 2015 IEEE International Conference on Computer Vision (ICCV), Pp. 118-126.
Song, Y., L., B. Wu, C.-Y. Chen, X. Zhang, and H. Adam
2017. Learning unified embedding for apparel recognition. 2017 IEEE International Conference Computer Vision Workshops (ICCVW), Pp. 224-2246..
Sutskever, I., O. Vinyals, and Q. V. Le
2014. Sequence sequence learning with neural networks. ArXiv, (bs//409.32255
Ustinova, E. and V. S. Lempitsky 2016. Learning deep embeddings with histogram loss. In NIPS.
van den Oord, A., Y. Li, and O. Vinyals
2018. Representation learning with contrastive predictive coding. ArXiv, abs/1807.03748.
van der Maaten, L. and G. E. Hinton
2008. Visualizing data using t-sne. Journal of machine learning research.
Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin 2017. Attention is you need, ArXiv, abs/1706.03762.
Vorobev, A., A. Ustimenko, G. Gusev, and P. Serdyukov 2019. Learning to select for a predefined ranking. In JCML.
Wan, L., Q. shan Wang, A. Papir, and Lopez-Moreno 2018. Generalized end-to-end loss for speaker verification. 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), Pp. 4879-4883.
Wiese, B. and C. W. Omlin
2009. Credit card transactions, fraud detection, and machine learning: Modelling time with Istm recurrent neural networks. In Innovations in Neural Information Paradigms and Applications.
Wu, Q., C. J. C. Burges, K. M. Svore, and J. Gao 2009. Adapting boosting for information retrieval measures. Information Retrieval, 13:254-270.
Xing, E. P., Ng. M. Jordan, and S. J. Russell
2002. Distance metric learning with application to clustering with side-information. In NIPS.
Yao, T., Yi, D. Cheng, F. Yu, A. Menon, L. Hong, E. H. hsin Chi, Tjoa, J. Kang, and E. Ettinger 2020. Self-supervised learning for deep models in recommendations, ArXiv, abs/2007.128653
Yi. D., Z. Lei, and S. Li
2014. Deep metric learning for practical person re-identification. ArXiv, abs/1400.4497
Yosinski, J., J. Clune, Y. Bengio, and H. Lipson
2014. How transferable are features in deep neural networks? ArXiv, abs/1411.1792
11

---

Under review as a conference paper at ICLR 2021
Zhai, A., H.-Y. Wu, E. Tzeng, D. H. Park, and C. Rosenberg
2019. Learning a unified embedding for visual search at pinterest. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Zhang. Y. and A. Haghani
2015. A gradient boosting method to improve travel time prediction. Transportation Research
Part C-emerging Technologies, 58:308-324.
Zhang. Y., D. Wang, Y. Chen, H. Shang, and Q. Tian
2017. Credit risk assessment based on long short-term memory model. In ICIC.
Zhou, C., J. Ma, J. Zhang. J. Zhou, and H. Yang
2020a. Contrastive learning for debiased candidate generation in large-scale recommender systems. ArXiv.
Zhou, K.. H. Wang, W. X. Zhao, Y. Zhu, S. Wang, F. Zhang. Z. yuan Wang, and J. Wen
2020b. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. ArXiv, abs/2008.07873.
A
BATCH GENERATION
We consider several empirical strategies for the sub-sequence generation to compare with the random slices algorithm described in section "Sampling of surrogate sequences" of the main paper. The simplest strategy is random sampling without replacement. One more strategy is to produce sub- sequences by the random splitting of the initial sequence to several connected segments without intersection between them. To generate k sub-sequences, the following procedure should be repeated k times: take random number of elements from the sequence without replacement.
Algorithm 2: Disjointed sub-sequences generation strategy
hyperparameters: k: number of sub-sequences be produced.
input: A sequence S of length l.
output: S,,.., Sy: sub-sequences of S.
Generate vector inds of length with random integers from [1.k]. for i + 1 to k do S, S[inds
end
B PERFORMANCE OPTIMIZATIONS
Here we describe several performance optimizations that can be used for model training and inference.
Pair distance calculation. In order select negative samples, need compute pair-wise distance between all possible pairs of embedding vectors of a batch. For the purpose of making this procedure more computationally effective we perform normalization of the embedding vec-
tors, i.e. project them on a hyper-sphere of unit radius. Since D(A, B) (A, B,)²
47 + E, B 2 2, A,B, and ||||| = | 1, to compute the euclidean distance we only need to compute: 2 2(A. B).
To compute the dot product between pairs in a batch we just need to multiply the matrix of all embedding vectors of batch by itself transposed, which is highly optimized computational procedure most modern deep learning frameworks. Hence, the computational complexity of the negative pair selection is Onn²) where h is the size of the output embeddings and is the size of the batch.
Embedding update calculation. Encoder. based on RNN-type architecture like GRU (Cho et al. 2014), allows to calculate embedding C++* by updating embedding c, instead of calculating em- bedding Ct from the whole sequence of past events 21: Ck rnn(e,, 244)). We use this
12

---

Under review as a conference paper at ICLR 2021
optimization reduce inference time to update already existing person embeddings with events, occurred after the calculation of embeddings. This is possible due to the recurrent nature of RNN- networks.
C PROOF OF THBOREM
In this section, we provide proof of Theorem (Section 3.3) that justifies the Random Slices sub- sequence generation strategy proposed in the paper.
Proof. First, we state the following straightforward lemma:
Lemma 1. Let stochastic process {Y (+) be shift of another stochastic process {Y (t) by independent random time s, i.e. (t)) Y( + s) with integer 2 0. If process {Y) is cyclostationary with period T and (se mod T) is uniform over [0, 1], then process {t) is stationary.
Lemma implies that process {x.(t) is stationary. and all its segments (Xx(t)) 1 of
T.
given length Th define the same distribution over sequences as its starting segment {x. (4) (0) does. Furthermore, integrating over s', conclude that the conditional distribution of a sub-sequence
obtained via Random Slices generation strategy given its length TV follows the process {x, (t) 1t To finish proof, remains to prove Equation] [
Assume ((T k) ka for l zo]. By the law of total probability, we have PPT?" kn) Σe PT, = (TT ko To = k), that is,
P(T. Ko) Σ
ka-1
k=
where C' is the normalization constant. To estimate the sum of the series, notice that
No- dx
k
dx.
where the former inequality follows from the fact that | f(z) = -01 is convex. After integration, we rewrite Equation as follows:
pk+1/2 a > kn long as function
ko
ki
2
Ao
Using these inequalities, we obtain the upper bound for P(T. = k) in the following way:
P(T" = kn) = Σ
kp
2
k=
I=m k=
i=
kn Ko
kn
kg
P(T. = -k))
1/2
Ko
1/2
L=
At last, the lower bound for P(T2
k)) can be obtained using Equation 3 as follows:
P(T = ku) = Σ k
k
KO
1
>
2
Amm ko
Jm km/
Imm
m 1/2
m
1/2
k
1a
P(T. = =ko)-
1=
1-
The latter inequality in these calculations follows from the fact that
<
for > m.
13

---

Under review as a conference paper at ICLR 2021
Table 3: Data structure for single credit card
Date
Time
Amount
Currency
Country
Merchant Type
Jun 21
16:40
230
EUR
France
Restaurant
Jun 21
20:15
5
USD
US
Transportation
Jun 22
09:30
40
USD
US
Household Appliance
Table 4: Click-stream structure for a single user
Time
Date
Domain
Referrer Domain
17:40
Jun amazon.com
google.com
17:41
Jun 21
amazon.com
amazon.com
17:45
Jun
en. en.wikipedia.org
google.com
D DATASETS
We designed the method specially for the user behavior sequences (Ni et al. 2018). These sequences consist of discrete events per person in continuous time, example, behavior on websites, credit card transactions, etc.
Considering credit card transactions, each transaction has set of attributes, either categorical or numerical including the timestamp of the transaction. An example of the sequence of three transactions with their attributes presented in Table 3 The merchant type field represents the category of merchant, such as "airline", "hotel", "restaurant", etc.
Another example of user behavior data click-stream: the log of internet page visits. The example of click-stream log of a single user is presented in Table
In our research we chose several publicly available datasets from data science competitions.
Age group prediction competition the task is to predict the age group of person. The group ratio is balanced in the dataset. The dataset consists of 44M anonymized transactions representing 50k persons with a target labeled for only 30k of them (27M out of 44M transactions), for the other 20k persons (17M out of 44M transactions) label is unknown. Each transaction includes date, type (for example, grocery store, clothes, gas station, children's goods, etc.) and amount. We use all available 44M transactions for contrastive learning, excluding 10% for the test part of the dataset, and 5% for the contrastive learning validation.
2. Churn prediction competition The dataset of IM anonymized card transactions repre- senting 10K clients was used to predict churn probability. 5k clients have labels (0.49M out of 1M transactions), 5.2k clients haven' 't labels (0.52M out of 1M transactions). Target is binary, almost balanced with proportions 0.55 and 0.45. Transactions of the same type and month are grouped and represented as single pseudo-transaction, which amount is the sum of grouped transactions. Each transaction is characterized by date, type, amount and Merchant Category Code.
Assessment prediction competition the task to predict the results of the in-game assessment based on the history of children gameplay data. Target is one of 4 grades, with proportions 0.50, 0.24. 0.14, 0.12. The dataset consists of 12M gameplay events combined in 330k gameplays representing 18k children. 17.7k gameplays (0.9M out of 12M gameplay events) are labeled, the remaining 312k gameplays (11.6M out of 12M gameplay events) are not labeled. Each gameplay event is characterized by timestamp, event code, the incremental counter of events within game session, time since the start of the game session, etc.
14

---

Under review as a conference paper at ICLR 2021
Figure 3: Event sequence length distribution
1.0
Age group
Churn
0.8
Assessment
Retail
Frequency
0.6
0.4
0.2
0.0
16
64
256 Events per client
1024
4096
4. Retail purchase history age group prediction TD
the task is to predict the age group of client based on its retail purchase history. The group ratio is balanced in the dataset. The dataset consists 45,8M retail purchases representing 400k clients. Only labeled data used. Each purchase is characterized by time, product level, segment, amount, value, points received.
To check that sequence lengths the considered datasets follow the power-low distribution we measured their distribution of lengths. As Figure 3 shows, the event sequence length distribution is close to the power-law distribution for every considered dataset.
To check that considered datasets follow Section (octioo[3.2jam a.d and used for theoretical analysis in Section 333 we performed the following experiments. We measure the KI-ddvergeence two kinds of samples:
repeatability and periodicity assumption made in
(1) between random sub-samples of the same sequence, generated using modified version of Algorithm [ where overlapping events are dropped and (2) between random sub-samples taken from different sequences. The results are shown in Figure 4 As Figure 4 shows, the KL-divergence between sub-sequences of the same sequence of events relatively small compared the typical KL-divergence between sub-samples of different sequences of events. This observation supports our repeatability and periodicity assumption. Also note that additional plot (e) provided as an example for data without any repeatable structure.
E EXPERIMENT SETUP
For all methods, a random search on 5-fold cross-validation over the train set is used for hyper- parameter selection. The hyper-parameters with the best out-of-fold performance on the train set are then chosen. The final set of hyper-parameters used for CoLES is shown in Table The number of sub-sequences generated for each sequence was always for each dataset.
E. HAND-CRAFTED FEATURES
Here we describe the details of producing hand-crafted features. All attributes of each transaction are either numerical (e. g. amount) or categorical (e.g. merchant type (MCC code), transaction type, etc.). For the numerical type of attribute we apply aggregation functions, such as 'sum', 'mean', 'std', 'min', "max', over all transactions per user. For example, if we apply 'sum" for the numerical field 'amount"
15

---

Under review as a conference paper at ICLR 2021
(a) Age group
(b) Churn
7000
type
type
Same client sample
500
Same client sample
Random client sample
Random client sample
6000
5000
Count
4000
Count
300
3000
200
2000
1000
10
15
20
25
15
20
KL
(c) Assessment
Retail
20000
type
type
Same client sample
17500
Same client sample
Random client sample
Random client sample
12500
Count
Count
10000
10000
7500
7500
5000
5000
2500
KL
KL
(c) Texts
type
Same post sample
Random post sample
25000
20000
Count
25000
10000
5000
0
15
KL
Figure 4: Periodicity and repeatbility of the data. KL-divergence between event types of two random sub-sequences from the same sequence is compared with KL-divergence between sub-sequences of different sequences.
16

---

Under review as a conference paper at ICLR 2021
Table Flyper-parametter for CoLES training
Output
Learning rate
N samples in batch
N
Mín seq length
Max seq
Dataset
Encoder
size
epochs
length
Age group
800
0.001
64
150
25
200
GRU
Churn
1024
0.004
128
60
15
150
LSTM
Assessment
100
0.002
256
100
100
500
GRU
Retail
800
0.002
256
30
30
180
GRU
Table 6: Comparison of encoder types
Dataset
LSTM
GRU
Transformer
Age group (Accuracy)
0.621 +0.008
0.638 #0.007
0.622 0.006
Churn (AUROC)
0.823 #0.017
0.812 + 0.010
0.780 + 0.012
Assessment (Aceuracy)
0.620 #0.007
0.618
#0.009
0.542 0.007
Retail (Accuracy)
0.535 + 0.003
0.542 +-0.002
0.499 + 0.002
5-fold cross-validation metric +95% shown
we obtain a feature 'sum of all transaction amounts per user'. For the categorical type of attribute apply aggregation functions in slightly different way. For each unique value of categorical attribute we apply aggregation functions, such as 'count', 'mean', 'std" over all transactions per user' numerical attribute. For example, if we apply 'mean' for the numerical attribute 'amount' grouped by categorical attribute 'MCC code' we obtain a feature 'mean amount of all transactions for each MCC code per user'. For example, for age prediction task we have one categorical attribute (small group) with 200 unique values, combining it with amount we can produce 200 3 features ( grooppp x amount x count', 'groupl x amount x count', mean', ...). In total we use approx 605 features for this task. Note, that hand-crafted features contain information about user spending profile but omit information about transactions temporal order.
"group 190up199. amount x count', "groupO x amount x
F RESULTS
F.1 DESIGN CHOICES OBSERVATIONS
We consider several contrastive learning losses that showed promising performance on differ- ent datasets (Kaya and Şakir Bilge 2019) and some classical variants: contrastive loss (Hadsell al., 2006), binomial deviance loss (Yi al. 2014). triplet loss (Hoffer and Ailon 2015), his- togram loss (Ustinova and Lempitsky 2016), and margin loss (Manmatha et al. 2017). The results of comparison are shown in the Table
It is interesting to observe that even contrastive loss that can be considered as the basic variant of contrastive learning loss allows to get strong results on the downstream tasks (see Table 7 Our hypothesis is that an increase in the model performance on contrastive learning task does not always lead to an increase performance on downstream tasks.
As shown in Table 6 different choices of encoder architectures show comparable performance on the downstream tasks.
F.2 EMBEDDING SIZE
Figure 5|iressssoows that the performance quality on the downstream task increases with the dimensionality of an embedding. After the best quality is achieved, a further increase in the dimensionality of an embedding dramatically reduces quality. These results can be interpreted as the bias-variance trade-off. When the embedding dimensionality is too small, too much information can be discarded (high bias). On the other hand, when embedding dimensionality is too large, too much noise is added (high variance). Note, that increasing the embedding size will also linearly increase the training time and the volume of consumed memory the GPU.
17

---

Under review as a conference paper at ICLR 2021
Figure 5: Embedding dimensionality vs. quality
(a) Age group
(b) Chum
0.64
0.63
0.82
0.81
Accuracy
AUROC
0.61
0.79
0.59
0.78
32
64
96
160224
480
8001200
2400
32
64
128
256
512
1024
3072
Embedding
Embedding size
(c) Assessment
(d) Retail
0.540
0.620
Accuracy
0.610
Accuracy
0.605
D.530
0.600
0.525
0.595
0.520
Embedding size
Embedding size
18

---

Under review as a conference paper at ICLR 2021
Table 7: Comparison of contrastive learning losses
Contrastive (margin=0.5)
Binomial deviance
Dataset
Histogram
Margin
Triplet
Age group
0.639
0.621
0.632
0.638
0.636
(Accuracy)
+0.006
+0.005
+0.008
+0.007
+0.004
Churn
0.823
0.769
0.815
0.823
0.781
(AUROC)
+0.017
+0.018
#0.018
10.012
10.021
Assessment
0.618
0.589
0.615
0.612
0.600
(Accuracy)
#0.009
#0.004
#0.007
+0.005
10.004
Retail
0.542
0.535
0.533
0.541
0.541
(Accuracy)
++0002
+-0.004
+0.002
+0.001
+0.001
S-fold cross-validation metric 195% shown
Table 8: Comparison of negative sampling strategies
Hard negative mining
Random negative sampling
Distance weighted sampling
Dataset
Age group (Aceuracy)
0.639 +0.006
0.626 0.008
0.629 0.004
Churn (AUROC)
0.823 +0.017
0.815 + 0.013
0.821 + 0.014
Assessment (Accuracy)
0.618 +-0.009
0.593 + 0.002
0.603 0.010
Retail (Accuracy)
0.542 #0.002
0.530 0.002
0.536 0.002
5-fold cross-validation metric +95% is shown
F.3 SEMI-SUPERVISED SETUP
To evaluate our method in case of the restricted amount of labeled data, only part of the available target labels for the semi-supervised experiment, see Section 442 for details. As in the case of the supervised setup, we compare the proposed method with LigthGBM over hand-crafted features, CPC, and supervised learning without pre-trainne training. In figure 6 we provide learning curves for all considered datasets.
F.4 EMBEDDING VISUALIZATION
In order to visualize CoLES embeddings in 2-dimensional space, applied tSNE transforma- tion (van der Maaten and Hinton 2008) on them. iSNE transforms high-dimensional space to low- dimensional based local relationships between points, neighbor vectors in high-dimensional embedding space are pushed to be close in 2-dimensional space. We colorized 2-dimensional vectors using the target values of the datasets.
Note, that embeddings was learned in a fully self-supervised way from raw user transactions without any target information. The sequence of transactions represent user' behavior, thus the CoLES model captures behavioral patterns and outputs embeddings of users with similar patterns nearby. As shown below, local clusters embedding space correspond the distribution of user's attributes either age or churn fact.
1SNE vectors from the age prediction dataset are presented in Figure 7a We can observe clusters: clusters for group '1' and "2' are on the opposite side of the cloud, clusters for groups '2' and '3' are the middle.
Taking into account that age is an ordinal attribute, we can make an assumption about the ordering of age groups: age(1) < age(3) (0e < age(2) or vice versa. (age(bin) returns age of user for specific group).
ISNE points from the churn prediction dataset are presented in Figure There are areas where one type of label dominates over the other.
19

---

Under review as a conference paper at ICLR 2021
(a) Age group
(b) Churn
0.650
0.625
0.600
Accurary
AUROC
0.575
0.78
0.550
0.76
Setup
Setup
0.525
lightGBM on crafted features
lightGBM hand-crafeee features
CPC Fine tuning
CPC Fine tuning
0.500
CoLES Fine-tuning Supervised leaming
0.74
CoLES Fine-tuning Supervised learning
3.12% 6.25% 12.5% 25.0% Shares of Labeled Datapoints
6.25%
12.5%
25.0%
50.0%
100.0%
Shares Labeled
(c) Assessment
Retail
0.54
0.60
0.52
0.58
Acturacy
Accuracy
0.48
0.56
0.54
0.44
Setup
Setup
lightGBM hand-crafted features
lightGBM hand-crafted features
CPC Fine-tuning
CPC Fine-tuning
0.52
CoLES Fine-tuning Supervised learning
CoLES Fine-tuning Supervised learning
0.40
3.12%
6.26%
25. 2%.0% Shares Labeled Datapoints
12.5%
50.0%
100.0%
0.17%0.33%
1.56%
6 6.25%
25.0%
100.0%
Shares Labeled Datapoints
Figure 6: Model quality for different dataset sizes The rightmost point corresponds to all labels and supervised setup.
20

---

Under review as a conference paper at ICLR 2021
Figure 7: 2D (SNE mapping of CoLES embeddings colored by target labels
(a) Age group
(b) Churn
20
chum
10
15N8111
-15
20
-10
(SNE(0]
1SNE[0]
(c) Assessment
Retail
Accuracy group
Age group
7.5
1SNE11
0.0
-500
-7.5
(SN[(0]
(SNE(0]
21


Reviews:
[{"review_id": "oCk_lA1U9Lc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Self-supervised learning is a rapidly growing domain where the amount of labeled data is limited. Contrastive learning with data augmentation enables us to design a loss function which makes semantically similar objects closer to each other, while dissimilar ones further away. The authors argue that self-supervised learning approaches have been focused on NLP or Computer vision, rather than sequential user behaviors datasets (mostly tabular). In order to address this problem, this paper proposed a random slices subsequence generation strategy (CoLES) for contrastive learning in event sequences.\n \nThe proposed CoLES algorithm is a data augmentation strategy that randomly selects subsequences from the full event sequence from each user. By theoretical analysis, the authors claimed that subsequences generated by CoLES are representative samples of the original entity and follows its latent distribution. I am not sure about it (See the end of this review).\n \nThe CoLES algorithm is tested with several publicly available datasets: 1) Age group prediction, 2) Churn prediction, 3) Assessment prediction, 4) Retail purchase history age group prediction, these datasets include sufficient amounts of discrete events per user. It was not clearly stated how CoLES was used to solve the classification (1, 3, 4) and regression (2) problems. However, considering the contents of chapter 3.4-3.5, I was able to infer that K subsequences are created with CoLES as well as several negative samples. Then each subsequence is encoded by the event encoder, then, the contrastive loss is minimized together with the final classification or regression using the sequence encoder (RNN) described in chapter 3.5. Describing the overall procedure will help readers to understand the framework better.\n \nThrough various experiments, the authors show that the CoLES sampling strategy is superior. (Table 1) Compared with baseline sampling strategies for sequential data, (Figure 2) Augmentation by COLES is helpful in small-labeled SSL scenarios, (Table 2)  Pretrained model using COLES performs is effective on fine-tuning stage for downstream classification. Supplementary results such as changing encoder types, contrastive learning losses, negative sampling strategies, embedding sizes, full-results on dataset sizes are also available in the appendix.\n \nThe CoLES algorithm is very simple and intuitive. Although there is no technical advance,  CoLES showed a strong performance and it seems that CoLES can be widely used because of its simplicity. But, I still have some concerns about the experiments that the performance was compared only within the proposed subsequence augmentation problem setting, so it was not possible to find out how excellent CoLES is in the global solution space. For example, training a model by using additional augmented data generated by CTGAN [1], or tabular learning framework by masking some entities [2] can be strong baselines for wider experiment scope. \n\n- [1] CTGAN: https://github.com/sdv-dev/CTGAN \n- [2] TabNet: https://github.com/dreamquark-ai/tabnet\n \n \nThe contents are easy to follow, but the overall presentation should be improved. Here are Some typos and issues that I found:\n- Footnote 1: LeCunn -> LeCun\n- Page 7: self-superivsed -> self-supervised\n- Page 8: pre-traning -> pre-training\n- Page 8: Descriptions/Results of real-world business applications can be removed / or can be explained in a little more detail in the Appendix.\n- Page 9-11: Reference format does not look very natural\n- Page 12-13: The notation for probability is broken.\n- Page 14: Numbering of bullet points, ???M retail purchases representing ???k clients\n- Page 15: Event sequence lenght  -> Event sequence length (Also, it is difficult to say that event sequence length follows power-law distribution)\n- Figure 4 on Page 17: Would be better to have equally-spaced x-ticks.\n- Page 19: Terse writing\n \nAbout the theorem: To prove that the subsequences are representative samples, the upper- and lower-bounds of the conditional probability of the length of the subsequences were calculated, with underlying three assumptions. But, the reviewer could not understand why this theorem proved the above claim. So, I want to leave the verification of this proof to other reviewers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lLFUimOkQ5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n1. The paper targets the problem of event sequence prediction in a contrastive self-supervised learning framework .They train this contrastive learning method by generating positive and negative samples via data augmentation method proposed as random slicing, which creates overlapping sub-sequences from each task sequence. This method follows the same distribution and properties as the original sequence which is supported by proofs. \n2.This approach shows improvements in the semi-supervised setting and supervised setting compared to other self-supervised training baselines and also against hand-crafted feature generation techniques. \n3.The problem is formulated properly that it is easy to follow. \n\nCons: \n\n1.The paper applies the ideas of data augmentation, random sampling and self-supervised learning in the event sequence prediction domain. Although these strategies may have been used first time in this setting, they have minor contributions in these individual methods. \n2.The framework shown in Fig 1., is a very classic contrastive learning based approach which has gained popularity in self-supervised literature as well. When the authors use CPC as a baseline, do they consider the same data augmentation and random sampling techniques as shown in Fig1.?  It is not very convincing that CPC will perform worse than the regular contrastive loss function. \n3.Are the final embeddings fine-tuned when they are trained for the downstream task or are they fixed ?\n4.As accuracy is being used as a metric in the paper, it is not very clear what are the number of classes for all datasets used. It will be good to add this.\n5.Why is the accuracy negative in Table2, is it the relative improvement with respect to the supervised setting? The authors should put absolute accuracy instead of relative improvement for each. \n6.Is the supervised model also trained with data augmentation? Self-supervised doing better than supervised is not so intuitive and could you discuss why the supervised performance in general is low for these tasks ?\n7.The paper has a lot of information as a laundry list that makes it harder to find the motivation and intuition behind different design choices. I will highly recommend to restructure the paper with stronger motivation on the choice of model in fig 1. \n\n####### Post rebuttal #########\n\nI thank the authors for their well formed responses. They address all my concerns effectively. I keep my original rating and recommendations. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "gh3fVpKl2_f", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n- The paper proposes CoLES that uses contrastive learning to learn representations of event sequence related to user behavior (e.g., credit card transactions, retail purchase history). The method trains in a self-supervised manner by randomly slicing event sequences to generate sub-sequences. Sub-sequences from the same user event sequence are positive samples (otherwise are negative samples) for contrastive learning. The authors provide theoretical analysis for the random slicing approach, under certain assumptions of the data. They compared CoLES with other supervised, self-supervised, and semi-supervised baseslines on multiple datasets, and show the proposed method outperforms baselines. \n\nStrong points: \n- The paper is well-written and the related work provides a good context of recent advances.\n- The evaluation is very thorough and extensive.\n- The random slicing approach is theoretically analyzed.\n\nWeak points: \n- While the evaluation is solid, the improvements over baselines seem limited (1~2%).\n- Modeling-wise the technical novelty seems limited. Using contrastive learning for self-supervision was proposed by past work (Chen et al 2020 and references therein). Apart from the analysis for using random slicing as the data augmentation scheme, limited novelty was introduced to adapt contrastive learning for discrete event sequences.\n- The theory proves that random splicing generates representative samples. However, this doesn’t seem very interesting when the assumption is that the data is cyclostationary.\n\nRecommendation: \n- I’m on the borderline but slightly inclined to accept the paper. The main reason is that the paper is well-written and the evaluation is solid. The main reason for rejection would be not enough technical novelty and limited improvements.\n\nComments & questions:\n- The paper assumes “periodicity and repeatability” in the data and bases the theoretical analysis on this assumption. However, no supporting data was shown that these event sequences do meet these assumptions. Does this assumption really make sense? Answering that will make the claim much stronger and might further explain the results across different datasets.\n- How different is event sequence modeling vs language modeling? The sequence of words resembles the events. The encoder in this paper seems overly simple compared to SOTA NLP models. Would the method perform better with a better encoder design?\n- What is the ratio of labeled and unlabeled data in the evaluation? This is important for comparing results with supervised approaches.\n- The paper uses a lot of phrases like “significantly outperforms”, “significant increase in performance”. However, the results show 1~2% improvements which do *not* match the statements. I think the authors should revise the claims for the paper to get accepted, so that the statements reflect the results more truthfully.\n\nMinor comments:\n- In Theorem 1, the definitions of variable m & k are unclear just from the text.\n- In Table 2, adding plus signs before positive results may improve readability.\n\n==== Updates after the response ====\n\nI thank the authors for answering my questions and the updated manuscript. I’m keeping my score and recommendation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work is well written and accurately covers the context and recent related work. It's a good example of how to apply self-supervised training to the event sequence domain. However, the combination of a lack of technical originality (composing a set of previously explored ideas) and significant improvements in results (results with CoLES overlap in error bars with RTD results) limits the impact of this paper.

Pros:
- Well written.
- Extensive evaluation.
- Well formulated problem.

Cons:
- Lack of technical novelty. The method appears to be general to all sequences rather than specialized for event sequences so the motivation for this design is not crystal clear.
- Minor improvement in results from using the method despite written claims that the method 'significantly outperforms'.
- Limited analysis that shows the periodicity and repeatability in the data.
Decision: Reject