Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
- The paper proposes CoLES that uses contrastive learning to learn representations of event sequence related to user behavior (e.g., credit card transactions, retail purchase history). The method trains in a self-supervised manner by randomly slicing event sequences to generate sub-sequences. Sub-sequences from the same user event sequence are positive samples (otherwise are negative samples) for contrastive learning. The authors provide theoretical analysis for the random slicing approach, under certain assumptions of the data. They compared CoLES with other supervised, self-supervised, and semi-supervised baseslines on multiple datasets, and show the proposed method outperforms baselines. 

Strong points: 
- The paper is well-written and the related work provides a good context of recent advances.
- The evaluation is very thorough and extensive.
- The random slicing approach is theoretically analyzed.

Weak points: 
- While the evaluation is solid, the improvements over baselines seem limited (1~2%).
- Modeling-wise the technical novelty seems limited. Using contrastive learning for self-supervision was proposed by past work (Chen et al 2020 and references therein). Apart from the analysis for using random slicing as the data augmentation scheme, limited novelty was introduced to adapt contrastive learning for discrete event sequences.
- The theory proves that random splicing generates representative samples. However, this doesn’t seem very interesting when the assumption is that the data is cyclostationary.

Recommendation: 
- I’m on the borderline but slightly inclined to accept the paper. The main reason is that the paper is well-written and the evaluation is solid. The main reason for rejection would be not enough technical novelty and limited improvements.

Comments & questions:
- The paper assumes “periodicity and repeatability” in the data and bases the theoretical analysis on this assumption. However, no supporting data was shown that these event sequences do meet these assumptions. Does this assumption really make sense? Answering that will make the claim much stronger and might further explain the results across different datasets.
- How different is event sequence modeling vs language modeling? The sequence of words resembles the events. The encoder in this paper seems overly simple compared to SOTA NLP models. Would the method perform better with a better encoder design?
- What is the ratio of labeled and unlabeled data in the evaluation? This is important for comparing results with supervised approaches.
- The paper uses a lot of phrases like “significantly outperforms”, “significant increase in performance”. However, the results show 1~2% improvements which do *not* match the statements. I think the authors should revise the claims for the paper to get accepted, so that the statements reflect the results more truthfully.

Minor comments:
- In Theorem 1, the definitions of variable m & k are unclear just from the text.
- In Table 2, adding plus signs before positive results may improve readability.

==== Updates after the response ====

I thank the authors for answering my questions and the updated manuscript. I’m keeping my score and recommendation.
review_title: Well-written with extensive evaluation, but novelty seems limited

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:

1. The paper targets the problem of event sequence prediction in a contrastive self-supervised learning framework .They train this contrastive learning method by generating positive and negative samples via data augmentation method proposed as random slicing, which creates overlapping sub-sequences from each task sequence. This method follows the same distribution and properties as the original sequence which is supported by proofs. 
2.This approach shows improvements in the semi-supervised setting and supervised setting compared to other self-supervised training baselines and also against hand-crafted feature generation techniques. 
3.The problem is formulated properly that it is easy to follow. 

Cons: 

1.The paper applies the ideas of data augmentation, random sampling and self-supervised learning in the event sequence prediction domain. Although these strategies may have been used first time in this setting, they have minor contributions in these individual methods. 
2.The framework shown in Fig 1., is a very classic contrastive learning based approach which has gained popularity in self-supervised literature as well. When the authors use CPC as a baseline, do they consider the same data augmentation and random sampling techniques as shown in Fig1.?  It is not very convincing that CPC will perform worse than the regular contrastive loss function. 
3.Are the final embeddings fine-tuned when they are trained for the downstream task or are they fixed ?
4.As accuracy is being used as a metric in the paper, it is not very clear what are the number of classes for all datasets used. It will be good to add this.
5.Why is the accuracy negative in Table2, is it the relative improvement with respect to the supervised setting? The authors should put absolute accuracy instead of relative improvement for each. 
6.Is the supervised model also trained with data augmentation? Self-supervised doing better than supervised is not so intuitive and could you discuss why the supervised performance in general is low for these tasks ?
7.The paper has a lot of information as a laundry list that makes it harder to find the motivation and intuition behind different design choices. I will highly recommend to restructure the paper with stronger motivation on the choice of model in fig 1. 

####### Post rebuttal #########

I thank the authors for their well formed responses. They address all my concerns effectively. I keep my original rating and recommendations. 
review_title: This paper uses a self-supervised learning based approach to learn efficient representations for downstream discrete event sequence prediction domains. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Self-supervised learning is a rapidly growing domain where the amount of labeled data is limited. Contrastive learning with data augmentation enables us to design a loss function which makes semantically similar objects closer to each other, while dissimilar ones further away. The authors argue that self-supervised learning approaches have been focused on NLP or Computer vision, rather than sequential user behaviors datasets (mostly tabular). In order to address this problem, this paper proposed a random slices subsequence generation strategy (CoLES) for contrastive learning in event sequences.
 
The proposed CoLES algorithm is a data augmentation strategy that randomly selects subsequences from the full event sequence from each user. By theoretical analysis, the authors claimed that subsequences generated by CoLES are representative samples of the original entity and follows its latent distribution. I am not sure about it (See the end of this review).
 
The CoLES algorithm is tested with several publicly available datasets: 1) Age group prediction, 2) Churn prediction, 3) Assessment prediction, 4) Retail purchase history age group prediction, these datasets include sufficient amounts of discrete events per user. It was not clearly stated how CoLES was used to solve the classification (1, 3, 4) and regression (2) problems. However, considering the contents of chapter 3.4-3.5, I was able to infer that K subsequences are created with CoLES as well as several negative samples. Then each subsequence is encoded by the event encoder, then, the contrastive loss is minimized together with the final classification or regression using the sequence encoder (RNN) described in chapter 3.5. Describing the overall procedure will help readers to understand the framework better.
 
Through various experiments, the authors show that the CoLES sampling strategy is superior. (Table 1) Compared with baseline sampling strategies for sequential data, (Figure 2) Augmentation by COLES is helpful in small-labeled SSL scenarios, (Table 2)  Pretrained model using COLES performs is effective on fine-tuning stage for downstream classification. Supplementary results such as changing encoder types, contrastive learning losses, negative sampling strategies, embedding sizes, full-results on dataset sizes are also available in the appendix.
 
The CoLES algorithm is very simple and intuitive. Although there is no technical advance,  CoLES showed a strong performance and it seems that CoLES can be widely used because of its simplicity. But, I still have some concerns about the experiments that the performance was compared only within the proposed subsequence augmentation problem setting, so it was not possible to find out how excellent CoLES is in the global solution space. For example, training a model by using additional augmented data generated by CTGAN [1], or tabular learning framework by masking some entities [2] can be strong baselines for wider experiment scope. 

- [1] CTGAN: https://github.com/sdv-dev/CTGAN 
- [2] TabNet: https://github.com/dreamquark-ai/tabnet
 
 
The contents are easy to follow, but the overall presentation should be improved. Here are Some typos and issues that I found:
- Footnote 1: LeCunn -> LeCun
- Page 7: self-superivsed -> self-supervised
- Page 8: pre-traning -> pre-training
- Page 8: Descriptions/Results of real-world business applications can be removed / or can be explained in a little more detail in the Appendix.
- Page 9-11: Reference format does not look very natural
- Page 12-13: The notation for probability is broken.
- Page 14: Numbering of bullet points, ???M retail purchases representing ???k clients
- Page 15: Event sequence lenght  -> Event sequence length (Also, it is difficult to say that event sequence length follows power-law distribution)
- Figure 4 on Page 17: Would be better to have equally-spaced x-ticks.
- Page 19: Terse writing
 
About the theorem: To prove that the subsequences are representative samples, the upper- and lower-bounds of the conditional probability of the length of the subsequences were calculated, with underlying three assumptions. But, the reviewer could not understand why this theorem proved the above claim. So, I want to leave the verification of this proof to other reviewers.
review_title: Simple and powerful augmentation strategy for sequential data

Meta-review: This work is well written and accurately covers the context and recent related work. It's a good example of how to apply self-supervised training to the event sequence domain. However, the combination of a lack of technical originality (composing a set of previously explored ideas) and significant improvements in results (results with CoLES overlap in error bars with RTD results) limits the impact of this paper.

Pros:
- Well written.
- Extensive evaluation.
- Well formulated problem.

Cons:
- Lack of technical novelty. The method appears to be general to all sequences rather than specialized for event sequences so the motivation for this design is not crystal clear.
- Minor improvement in results from using the method despite written claims that the method 'significantly outperforms'.
- Limited analysis that shows the periodicity and repeatability in the data.
Decision: Reject