Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment
Authors: Chenxiao Yang
Qitian Wu
Qingsong Wen
Zhiqiang Zhou
Liang Sun
Junchi Yan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sequential Event Prediction
Sequential Recommendation
Causal Inference
Out-of-Distribution Generalization

Abstract: The goal of sequential event prediction is to estimate the next event based on a sequence of historical events, with applications to sequential recommendation, user behavior analysis and clinical treatment. In practice, the next-event prediction models are trained with sequential data collected at one time and need to generalize to newly arrived sequences in remote future, which requires models to handle temporal distribution shift from training to testing. In this paper, we first take a data-generating perspective to reveal a negative result that existing approaches with maximum likelihood estimation would fail for distribution shift due to the latent context confounder, i.e., the common cause for the historical events and the next event. Then we devise a new learning objective based on backdoor adjustment and further harness variational inference to make it tractable for sequence learning problems. On top of that, we propose a framework with hierarchical branching structures for learning context-specific representations. Comprehensive experiments on diverse tasks (e.g., sequential recommendation) demonstrate the effectiveness, applicability and scalability of our method with various off-the-shelf models as backbones. 

Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment
Chenxiao Yang', Qitian Wu', Qingsong Wen', Zhiqiang Zhou', Liang Sun?, Junchi Yan
DDepartment of Computer Science and Engineering, Shanghai Jiao Tong University
DAMO Academy, Alibaba Group
{chr26195, echo740, yanjunchi (anchii@sjt...... edu. en, fqingsong. wen, zhouzhiqiang zzq, liang......... @alibaba -nw.
Abstract
The goal of sequential event prediction is to estimate the next event based on a sequence of historical events, with applications sequential recommendation, user behavior analysis and clinical treatment. In practice, the next-event prediction models are trained with sequential data collected at one time and need to generalize to newly arrived sequences in remote future, which requires models to handle temporal distribution shift from training to testing. this paper, we first take a data-generating perspective to reveal negative result that existing approaches with maximum likelihood estimation would fail for distribution shift due to the latent context confounder, i.e., the common cause for the historical events and the next event. Then we devise a new learning objective based on backdoor adjust- ment and further harness variational inference to make it tractable for sequence learning problems. On top of that, we propose a framework with hicrarchical branching structures for learning context-specific representations. Comprehensive experiments on diverse tasks (e.g., sequential recommendation) demonstrate the effectiveness, applicability and scalability of our method with various off-the-shelf models as backbones.
Introduction
world problem scenarios are flooded with sequential event data consisting of chronologically arrived events which reflect certain behaviors, activities responses. A typical example is user that aims to harness user's recent activities to estimate future ones, which could help downstream target advertisement in online platforms (c.g., e-commerce or social networks). Predicting future events also plays a central role in some real situations such as clinical treatment for promoting social welfare.
activity prediction [56] 12 32
20
44 [
common nature of (sequential) event prediction lies the different time intervals within which training and testing data are generated. Namely, models trained with data collected at one time are supposed to predict next event in the future [59] 36. where the underlying data-generating distributions may have gone through variation due to environmental changes. However, most existing overlook this issue in both problem formulation and empirical evaluation. which may leave the real problems under-resolved with model mis mnisspecciiicatio, and result in over-cstimation for model performance on real data.
approaches (2 14 t7l 24 Erl
19 [
As concrete example in sequential recommenders [ the data-generaiee generating distribution for user clicking behaviors over items is highly dependent on user preferences that are normally correlated
'The SJTU authors are also with MoE Key Lab of Artificial Intelligence, SJTU. Junchi Yan is the correspon- dence author who is also with Shanghai Al Laboratory.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Sparious Correlation
Sparions Corcelation
Prior Art
Past Events
Future Event
Past Events
Prediction
Figure 1:
toy example in rec- ommendation. Prior art would spu- riously correlate non-causal items ('ice cream" and 'beach T-shirt") and produce undesired results un- der new environment in the future. Our approach endeavors to allevi- ate the issue via counteracting the effects of contexts (seasons).
C
Truin
Test
(Offline)
(Online)
Ours V.S.
Time Lime
Summee
Winter
Past Events
Funure Event
Time uom
Past Events
Prediction
Causality
Cassality
with some external factors, like the dynamic fashion trends in different years 1K
seasonal
influences on item popularity (43]. These highly time-sensitive external factors would induce distinct
user preferences leading to different behavioral data time goes by. This issue could also partially explain the common phenomenon of model performance drop after adaptation from offline to online
environments
41 36 [2
Handling distribution shift in sequential event data poses several non-trivial challenges. First, the temporal shift requires models' capability for out-of-distribution (OoD) generalization [00] i.e., extrapolating from training environments to new unseen environments in remote future. Prior art that focus model learning and evaluation over in-distribution data may yield sub-optimal results on OoD testing instances. Second, as mentioned before, there exist external factors that impact the generation of events. These external factors, which we term as contexts, are unobserved in practice. To eliminate their effects, one may also need the latent distributions that characterize how the contexts affect events generation. Unfortunately, such information is often inaccessible due to constrained data collection, which requires the models to learn from pure observed sequences.
1.1 Our Contributions
To resolve these difficulties. in this paper, adopt generative perspective investigate the temporal distribution shift problem and propose new variational context adjustment approach with instantiations solve the issue for sequential event prediction.
generative perspective on temporal distribution shift. We use proof-of-concept Structural Causal Models (SCMs) characterize the dependency among contexts, historical sequences and labels (i.e.. next event type) in terms of both data generation and model prediction. We show that contexts essentially act confounder, which leads the model to leverage spurious correlations and fail to generalize to data from new distributions. See Fig. for toy example to illustrate the issue.
Variational context adjustment. We propose variational context adjustment approach to resolve the issue. The resulting new learning objective has two-fold effects: 1) helping to uncover the underlying relations between latent contexts and sequences in data-driven way, and 2) canceling the confounding effect of contexts to facilitate the model to explore the true causality of interest (as illustrated in Fig. DT. We also propose mixture of variational posteriors approximate context prior via randomly simulating pseudo input sequences.
Instantiation by flexible framework. We propose a framework named CaseQ to instantiate ingredients the objective, which could combine with most off-the-shelf sequence backbone models. To accommodate temporal patterns under different environments, we devise novel hierarchical branching structure for learning context-specifet representations of sequences. It could dynamically evolves its architecture to adapt for variable contexts.
Empirical results. We carry out comprehensive experiments on three sequential event prediction tasks with valuation protocols designed for testing model performance under temporal distribution shift. Specifically, when we enlarge time gap between training and testing data, CaseQ can alleviate performance drop by 47. 77% w.r.t. Normalized Discounted Cumulative Gain (NDCG) and 35.73% w.r.t. Hit Ratio (HR) for sequential recommendation, which shows its robustness against temporal distribution shift.

---

1.2 Related Works
Sequential Event Prediction aims to predict the next event given historical event sequence with applications including item recommendation in online commercial platforms, user behavior prediction in social networks, and symptom prediction for clinical treatment. Early works [39 [8] rely on Markov chains and Bayesian networks to learn event associations. Later, deep learning based methods are proposed to capture non-linear temporal dynamies using CNN (45), RNN 120 27 54],
etc. These models are usually trained offline using event sequences collected during a certain period time. Due to temporal distribution shifts, they may not generalize well to online environments [+] [3 [22 Our work develops principled approach for enhancing
attention models [24] 50 5.
sequential event prediction with better robustness against temporal distribution shifts via pursing causality behind data.
cat-of-Distributionn Generalization butions. It is a common yet challenging problem, and has received increasing attention due to its For event prediction, distribution shift inherently exists because of different time intervals from which training and testing data are generated. Most existing sequence learning models assume that data are independent and identically distributed 220 14 377. Despite various approaches of OoD generalization designed for machine learning problems in other research areas (e.g., vision and texts), it still remains under-explored for sequential event prediction and sequential recommender systems.
30
deals with distinct training and testing distri-
significance 26 22 52 55
a fundamental way to identity causal relations among variables, and pursuit stable and robust learning and inference. It has received wide attention and been applied in various domains e.g., computer vision [57 28 natural language processing 40 F2] and recommender
Causal Inference B3
35
systems (5% [1 Some existing causal frameworks for user behavior modeling BB
I6.
50)
61: 17
aim to extract causal relations based on observed or predefined patterns, yet they often require domain knowledge or side information for guidance and also do not consider temporal distribution shift.
m2 6 48 adopt counterfactual learning to overcome the effect of an ad-doc bias in recommendation task (e.g., exposure bias, popularity bias) mitigate clickbait issue, whereas they do not focus on modeling sequential events and are still based on MLE as learning objectives.
Problem and Model Formulation
We denote X' = 21, ,M. as the space of event types, and assume each event is assigned with an event type x; E X. Events occurred in chronological order consist of a sequence S .as (21,"2g external factors, i.e., context c. We use S, Y, Y and C' to denote the random variables of historical event sequence S, ground-truth next event y. predicted next event ý and context c, respectively. The data distribution can be characterized as C)) = P(S)S))(Y1S,C))
). As mentioned before, the data distribution normally affected by time-dependent
Problem Formulation. Given training data ((S,, y:) generated from data distributions with (S,, ys) ~ P(S, Y|C =cc==tt =tt) where (2) denotes the specific context when the i-th training sample generated, we are to learn a prediction model ÿ, = f(S; 0) that can generalize to testing data {(S,, ) from new distributions with (S). vy) ~ P(S, = (i)), ). where ( denotes the specific context when the j-th testing sample is generated. The distribution shift stems from different contexts that change over time, which we call temporal distribution shift in this paper.
2.1 Understanding the Limitations of Maximum Likelihood Estimation
Most of existing approaches target maximizing the likelihood (o(y)S) as the objective for optimiza- tion. Here we use P-(-) to denote the distribution induced by prediction model fe. Based on the definitions, we can build two Structural Causal Models (SCMs) that interpret the causal relations among 1) C, S, Y (given by ganarggeng process) and 2) C. S and Y (given by model learning), as shown in Fig. a) and (b).
For Fig. a), the three causal relations are given by the definitions for data generation, i.e., YIS, P(S)C) We next illustrate the rationales behind the other two causal relations S + Y and C + Y.
3

---

C
Varietinal Infannnes Decomftond
C
P(S(C)
P(S(C) Ti:P1
C(VIS
f(S:))
8
Real-world data generation P(Y
Traditional model
(c) Our interventional model Pe (Y))
Ps (Y
Figure Structural causal model for sequence learning.
S This relation induced by the prediction model f(S: () that takes historical event sequence S as input and outputs the prediction for next event ý. The relation from S to is deterministic given fixed model parameters 0.
C Y:
This relation is implicitly embodied in the learning process that optimizes the model parameters with given dataset collected at one time. By our definition, the training dataset is generated from latent distribution affected by context Ctr and the MLE algorithm yields
arg min E - (f(S;
(1)
/
where I(.,-) denotes a certain loss function, e.g., cross-entropy. This indicates that the learned model parameters 6* is dependent the distribution of ctr. Also, due to the fact á f(; 0), conclude the proof for relation from C' Y. In short, intuitive explanations for such causal relation lie in two facts: C affects the generation of data used for model training, and Y is the output of the trained model given input sequence S.
Confounding Effect of C. The key observation is that C acts as the confounder in both Fig. 2 and (b), which could play crucial role leading to undesirable testing performance of MI.F-based approaches once the distribution is shifted. As implied by the causal relations in Fig. 2 a). there exists partial information in S that predictive for Y yet highly sensitive to C. usually fast-changing variable in real-world scenarios. As result, the correlation between S and Y in previous contexts may become spurious in future ones. This also explains the failure MLE-based models for generalizing to data from new distributions, according the similar causal pattern in Fig. (
We can reuse the toy example in Fig. as an intuitive interpretation for the failure. The 'summer' season (a context) acts confounder that correlates buying 'ice cream' (a historical event) and buying 'T-shirt' label), between which exists no obvious causal relation. However, the model would "memorize" their correlation and tend to improperly recommend "T-shirt" in the 'winter' season (new context). Whereas in fact the user purchases ice cream winter most possibly because he/she is a dessert lover, in which case recommending other desserts would be better decision.
Intervention. To address the confounding effect of C' and endow the model with robustness to temporal distribution shift, we propose to target model learning with P& (Y{))) instead of the conventional Po SY) As shown in Fig. (]c). the do-operator cuts off the arrow (i.c., causal relation) coming from C' to S. which essentially simulates an ideal tata-generating process where sequences
are generated independently from contexts. This operation blocks the backdoor path S <- C - Y that spuriously correlates S and Y, and enables the model to learn the desired causal relation S Y which is invariant to environmental change.
2.2 Variational Context Adjustment
An ideal way to compute Po (o)) is to carry out randomized controlled trial (RCT) 34 by recollecting data from prohibitively large quantity of random samples under any possible context, which is infeasible since could neither control the environment nor collect data in the future. Fortunately, there exists a statistical estimation of Po (Y (do())) by leveraging backdoor
adjustment 34], wherein the confounder C is stratified into discrete pieces C {c By using

---

basic rules induced by the do-operator (see derivation in Appendix A we have:
|C|
Po(Y)do(S)) Σ C
(2)
Intuitively, the backdoor adjustment approximates an ideal situation where cj is enumerated according to the context prior P(C) independent from input sequences, which serves to counteract the effect of C' on the generation of S. Nevertheless, optimizing with Eq. (2) is intractable since oj is usually unobserved or even undefined, and its prior distribution P(C) is also unknown. Furthermore, computing Eq. 2 requires awareness of the relation between contexts and generated sequences, which is implicit in the data-generating process behind the data.
To address the difficulty for targeting Eq. 2 we introduce a variational distribution Q(CIS) as the estimation for latent contexts given input sequences. By treating C latent variable and using variational inference technique, we can obtain the following tractable evidence lower bound (ELBO) as learning objective (see derivation in Appendix B
log Po(SS = =S)
2Eo- [log
DKL (Q(CIS = S)IP)(
where the last step is given by Jensen's Inequality and the equality holds if and only if Q(CIS) exactly fits the true posterior P(C Y). which suggests it successfully uncovers the latent context from observed data. The first component Eq. 3 the negative reconstruction error (i.e. prediction error). The second component is the Kullback-Leite
(KL) divergence of the variational distribution and context prior distribution. Similar with 19)] we could re-write the second term summation of the entropy of Q(C)S). i... H[[[(IIS distribution, i.e., (QC)), P(C)). The former enforces high variance of contexts for each event sequence, and the latter aligns the aggregated posterior with the prior.
S)] and the cross-entropy between Q(C')S) and prior
Learnable Prior via Mixture of Posteriors. The distribution P(C) characterizes the true context prior in real world. It serves to regularize ((CIS) through the KL term in Eq. 5 Choosing an ap-
propriate P(C) is important yet challenging. One straightforward solution adopted by some previous
works 8 57 is to use a pre-defined prior such as uniform distribution. However, such a simplistic distribution may not reflect the true context prior, and potentially lead to over-regularization [5] An alternative way is to estimate the prior by computing the average of the variational posterior 21 i.c.,
P(C) R 1/N Σ Q(CIS St). However, such method is computationally expensive 44 and
would often result in biased estimation given a limited quantity of training data collected within
certain time interval. Inspired by [ using mixture of Gaussian flexible and learnable prior, we propose to use a mixture of pseudo variational posteriors as an estimation:
1 P(C) = R E ((C|s=
where S/ is a randomly generated pseudo event sequence and we set R < N to reduce the computa- tional cost. Note that the variational distribution Q(C1) (C)).)) is given by the prediction model (see details in Section 3. Therefore, the prior estimation by Eq. 4 general reflection of how the model favors each context given uninformative inputs. Also, the estimated prior is learned with the model in a fully data-driven manner.
3 Model Instantiations
We next parameterize the components in Eq. 3 i.... Po C)) and Q(CIS), and propose a flexible framework named as CaseQ by taking 'cas" from 'causal' and 'seq" from 'sequence'.
Event Embedding. Given event sequence S = {#1, Ø2, 3 as input, we use an embedding to represent each type of event. We consider global embedding matrix H, IR Mx xd to map each type of event into d-dimensional embedding space:
h = OneHot(( H
(5)
where M denotes the number of event types. OneHot(.) Z (0, 13 transforms event into one-hot column vector, and hy e gd denotes the initial representation for event Im.-
5

---

Inference Unit: Context-Specific Encoder. To encode input sequences and accommodate the effect of C on prediction, we consider an encoder (--) conditioned on a specific context. takes sequence of event embeddings and estimated context representation C(t) at time step / (given by the branching unit detailed next) as input, and outputs sequence of hidden states:
.,:), [h h2 h ((b), h2. 1,,...,,,,,, C(e):
(6)
where h is the m-th hidden state for the event sequence. Assume that there are K types of contexts, i.e., C = [cx] and C(t) E C. Each type of context C₁ is represented as a K -ddimensional one-hot column vector C2 {0, 13 whose k-th dimension is 1. Based on the SCM in Fig. 2 the context would change the underlying mechanism how the next event is generated real-world. Therefore, we consider context-specifie inference units {&x (: 0s)) as sub-networks of the encoder (-)) to
learn context-aware sequence representations, where + {Ox k Then, we have
K
o (b). by. ... ,h],, Ceji θ)
colk (x,), .M h/):
where C(t) [k]] returns the k-th entry of the vector C(t). The inference unit can be specified arbitrary off-the-shelf sequence models such as recurrent neural network (RNN) or self-attention (SA) 47
Branching Unit: Dynamic Model Selection. We next introduce branching unit (((() to parame- terize the variational posterior ((CIS) that aims to obtain (()) given the sequence. At each time step, it takes the sequence representation as input and outputs a probability vector E [0, 1] whose k-th entry represents the probability of the corresponding context ck2 i.e.,
K
q: = (h); where Σ
q [k]
1.
(8)
We use a d'-dimensional context embedding Wk. E Rdd to accommodate the information of each context C₁ by an embedding matrix H. E Kx where Wx = Ck T He and d' = d(d + 1). Then, we split each context embedding into several fix-sized parameters We E Raddd ag. E Rdd
W = wol: d rreshape(d, d), a wa d(d + 1)].
(9)
The attribution score Stk. that measures how likely sequence up to time step / belongs to context Ck can be calculated via
Sik= = Tanh (Wyh:))
(10)
Namely, we first project the sequence representation into a new d-dimensional space and then use dot product to measure the similarity. The variational distribution Q(CIS S) can be specified as a probability vector qe that is computed by using Softmax function over Su:- i.e., q. Softmar([[ K T), where controls the confidence level. To implement the random sampling procedure e ~~ QCCIS) in Eq. G), one can also apply the categorical reparameterizztion trick that uses differentiable samples from Gumbel-SSottma
2 distribution, i.e..
exp ((su + 9)/T)
qr [k]
g ~ Gumbel(O, 1).
(11)
k== exp ((SLk + g)//))
Then, we can revise the right term of Eq.
as EK 1 qi [k) & (h). (b))))))))))))
Hierarchical Branching Structure. The aforementioned single-layered architecture requires independent inference unit for each type of context, i.e., |C| = K, which has several drawbacks: 1) It is impractical to accommodate a large amount context types due to limited computational resources; 2) Real-world contexts not entirely isolated, and thus independently parameterizing them may lead to over-fitting and undesired generalization performance given limited training data. To address these limitations, further devise hierarchical branching structure for CaseQ framework. Instead of using a one-hot vector, we denote a context as a 0-- matrix, i.e.,
CaseQ:
C₁ stack([ck, c ,c 10, DxK
(12)

---

where D denotes the number of layers and K denotes the number of parallel inference units in each layer. Each row C₁ one-hot vector indicating a certain inference unit layer, and each type of context corresponds to certain combination of inference units. The entire network could then be represented as the stack of multiple parallel inference units and branching units:
K
[h
1+ 1/2
h Σ
h1-1
[ o (L), b/, ... hz]:
(13)
where qf = V/l (h.),, and I E (1, D). The final prediction result at each time step can be obtained by ranking the relevance scores of current hidden states and event embeddings:
mll.a (Hg [mp+)
(14)
The above computation acts as a realization of Po(Y|S S. C = c) and c~ Q(C S), required by the first term in Eq. B3. Besides, the KL term in Eq. 3 also requires density estimation from
Q(CIS = S). To achieve this, we use the continued Kronecker product of the probability vector q in each layer to represent the final variational posterior:
q: Flatten
X
d
E ERK
(15)
where qf E R and @ denotes Kronecker product. The produced variational posterior is a KDD dimensional vector, where each entry is the probability of a certain path (i.e., combination of inference units across layers) the network. This design has some merits. First, we could accom- modate IC = K types of contexts that grow exponentially w.r.t. model depth D. This helps to endow the model with larger capacity for learning diversified contexts. Second, different contexts may associate with each other by sharing the same inference unit in some layers, which plays as an effective inductive bias that guides the model to generalize to new environments (see more details and justifications in Appendix C
According to Eq. 3 the final loss function can be written
|S|-1
R E
ΣΣ
Uyt, Dr) + aDKL
q:
(
(16)
where a is weight parameter for balance, (S))) denotes the produced variational posterior distribu- tion with a pseudo sequence S as input, and I(-,-) can be arbitrary loss functions (c.g., cross-entropy) depending on specific tasks.
4 Experiments
The goal of our experiments is to test whether CaseQ can effectively handle temporal event prediction under distribution shift and we consider several temporal prediction tasks with particular evaluation protocols for the purpose. The codes are available at battps://giii
com/ hhtp:/ch2211///// /
Datasets. We evaluate the effectiveness of CaseQ with different applications for event prediction, including sequential recommendation, ATM maintenance and user awarded badges prediction. We use four datasets with variable length and number of event types: Movielens, Yelp, Stack Overflow and ATM. The detailed description of datasets and their statisties are deferred to Appendix
Evaluation Protocol. The commonly adopted leave-one-out evaluation for sequential event predic- tion [2]] 44 451
uses the latest event for testing, the event right before the last for validation, and the rest for training. To test the effectiveness of Case for handling temporal distribution shift, we generalize such evaluation protocol by enlarging the time gap between training and testing data. We use the last G 1 events for testing, the first [S] G 2 events for training. and the (IS) -- event for validation. We call G maximal gap size and g E [0, G gap size which means the number of events between the validation event and specific testing event. When G = 0 and = 0, it degrades to the conventional leave-one-out evaluation. By increasing y in our case, we could enlarge the time gap between training and testing data, so as to simulate temporal shift in real-world scenarios where data collection and online serving are during different periods of time. More details about the evaluation protocol are in Appendix G
7

---

Gap Size
GRU4Ree
AV)
Case (GRU)
SASRec
24
Case (SA)
SSE-PT
H9
Case (PI)
NDCG
HR
NDCG
NDCG
HR
NDCG
HR
NDCG
HR
NDCG
HR
0.436
0.730
0.448
0.453
0.742
0.465
0.753
0.476 (12.3%)
0.763
Moviclens
(+1.15g)
(12.0%)
30
0.371
0.603
0.406
0.660
0.384
0.627
0.429
0.675 (17.3%)
0.397
0.642
0.443
0.680
(16.7%)
(18.758)
(130.0983
(1 (130.8%)
(.0098)
Drop (%)
15.0
4 17.4
9.3
10:3
15.6
7.1
4 14.7
7.0
4 10.9
s50y2
(-40 7%)
(-53.196)
60 696;
(-51 895)
20 6%)
0.428
0.714
0.440
0.731
0.436
0.734
0.452
0.744
0.451
0.745
0.462
0.751
(12.8%)
(12.358)
(13.755)
(11.4%)
(12.3%)
0.995
Yelp
20
0.395
0.663
0.426
0.702
0.402
0.682
0.433
0.719
0,405
0.685
0,440
0.717
(+7:2%)
(+5. 596)
(+ (+7.076)
(+5.2%)
(+5.0%)
(-4 (-4.55
Drop (9)
7.7.
7.2
1 3.2
4.1
7.6
7.1
1 4.3
4 3.4 550
10.3
8.0
4.7
4.5
(-57.9%5
(-13115)
(-13,355)
(-M.355)
(-13.75)
Thee parcentage messures the improvement of CaseQ over the counterpart baseline with respect to one metric (e.g.. NDCG).
2 The percentage measures the degree of CaseQ helping to alleviste the performance drop compared to the counterpart.
Table Experiment results of Case with different base models. The Drop measures the percentage of decrease of a metric when time gap size increases.
0.650
-0.10%
0.31
3.57%
0.625
-44554
Accuracy
0.30
100%
Accuracy
0.600
-2.1074
5.0%
13 11.24%
25%
0.575
0.29
0.550
17.97
0.20
GRU
Ensentle
0.525
LSTM
DMst
DMCE
0.26
Taraformar
CasetyOOuu)
Tansfomer
CasiOlbura)
225555
0
10
20
30
40
50
0
10 Gap Size
20
Gap Size
(a) Stack Overflow
(b) ATM
Figure 3: Performance comparison of CaseQ and baselines w.r.t. different gap sizes.
Evaluation Metrics. For recommendation datasets, i.e., Movielens and Yelp, we adopt two widely used evaluation metrics: Hit Ratio at K (HR @ K), and Normalized Discounted Cumulative Gain K (NDCG@ K), where K = 10 in practice. For other two low-dimensional datasets with fewer number of event types, i... Stack Overflow and ATM, we use Accuracy (i.e., Precision) as the metric.
4.1 Sequential Recommendation
We adopt three sequence models as backbones for sequential recommendation: 1) GRU4Rec uses RNNs to model users' sequential behaviors and predict the next clicked item; 2) SASRec
20
24
adopts Self-Attention (SA) in Transformer's encoder for sequential recommendation; 3) SSE-PT further considers user's ' personalized embedding which incorporates user information into the input sequence. We use MLE for training these models. For fairness, our model uses the same number of layers the baselines. We set gap size g 0, 30 for Movielens and 0, 20 for Yclp since the average sequence length of Movielens is greater than Yelp's. s. As shown in Table [ our approach consistently outperforms the counterparts with different backbones throughout all the cases. While the performance degrades when we increase the gap size, CaseQ is significantly more robust to temporal distribution shift than conventional models. The percentage of the performance drop for CaseQ is is significantly lower than the counterpart model. Also, CaseQ is agnostic to the sequential prediction backbones and can be further combined with other advanced architectures.
49
4.2 Event Prediction
We further study the performance of CaseQ in Stack Overflow and ATM datasets, which have distinct statistical features compared to sequential recommendation datasets. We consider three commonly adopted models for predicting discrete sequential events, i.e., GRU, LSTM and Transformer [47]. We also consider two ensemble frameworks that has the same model size with ours and are built on parallel structures: Ensemble 7 adopts mean pooling over the hidden states of different inference
units; 2) DMoE [00 combines the hidden states by learning gating network in each layer. Also,
8

---

basclines show Time Stop 0
Time Soep
Time Stop
Time Stop: 30
Layer
Layer
Layer
Lager
Caper
(a) LSTM as base model
Time Stop 0
Time 50op
Time Stop 20
Time Step 59
Layol
Layer
Laye
(b) Transformer encoder as base model
Figure 4: Visualization of probabilities for different contexts recognized by well-trained CaseQ
(D 2, = 4,7 1). The results reveal the temporal distribution shift behind data.
we use multiple heads for Transformer to enlarge its model size. For Stack Overflow, the gap size varies from 0 to 50. For ATM, it varies from 0 to 20. As shown in Fig. 3 CaseQ achieves the best results throughout all the cases in both datasets. As the gap size increases, the overall accuracy of baseline models exhibit a sharp downward trend in general, which shows the severe negative impact of temporal distribution shift. Despite with larger model size, both ensemble- little improvement over the single models and also suffer serious performance drop with larger gap sizes. In comparison, when the gap size goes large, the performance improvement of Case over other competitors becomes significant. This suggests that our approach is indeed effective for improving the robustness of sequence models w.r.t. temporal shift.
4.3 Case Study
Visualization of Temporal Shift. To further investigate temporal distribution shift, in Fig. 4 we visualize the probabilities for each type of context at different time steps in a sequence. We set
K = 4, D 2. The two sub-figures show the results with different base models (LSTM and Transformer). The probability value reflects the intensity of certain context one time. As shown in the figure, while some contexts dominate at the beginning. they exhibit clear variation over time. The results imply that there indeed exists temporal distribution shift behind the data collected at different time intervals. Therefore, prior art use the past sequential data for training may not successfully transfer to new environments in the future. By contrast, our approach could first identifies latent contexts behind data and then alleviate its impact on learning stable prediction model.
Learning of Context Embeddings. Context embeddings are latent representations for each type of context and dynamically affect the branching unit for selecting the proper inference unit. Conceptually, each type context is expected to represent an environment in physical world, so each context embedding is supposed to be informative and distinctive. To verify whether our model could learn meaningful context representations without collapse or over-regularization, we visualize the context embeddings (K 3, D 2) through the training process in Fig. by using dimensional reduction. Each color corresponds to the embedding the same initialized positions, they gradually separate and move towards different directions the epochs increase. This indicates that the model manages to learn informative context representations.
certain context. While the context embeddings have
4.4 Scalability Test
We further test the scalability of CaseQ w.r.t. the number of context types. Fig. 6 shows the training time and inference time per mini-batch of data. We use a single layer when the number of context types is within [1, 6] and two layers when it further increases. As we can see in the range of [1, 6). the training and inference time exhibit linear increase with a single-layer architecture. The increasing trend goes down when we increase the model depth. The results suggest CaseQ's 's decent scalability.

---

Traiting Time
Inference Time
0.12
0220
Inforrnce Time
Trann Time
Raira
Far
Tugg Co
3.00
0.023
0.00
0.20
3.00
173456
41143)
19jtet)
-Tpess Context
Figure 5: Visualization of context embeddings (reduced into a 2D space, K 3) different training epochs.
Figure Train and inference time (s) of Case w.r.t. number of context types.
0x2 0225
02
Dm4
4.31
300
0+2
677
0-2
CAN
0-2
0-2
0-3 6381
0-3
0-2
044
0387
0m
0.00
0+4
(a) Gap Size = =30
(b) Gap Size
(c) Gap Size 30
(d) Gap Size
Figure 7: Performance of CaseQ recommendation task w.r.t. different K and D.
4.5 Hyper- Parameear Analysis
An appropriate number of context types are important for guaranteeing the performance and general- izability of our model. Recall that there are in total K types of contexts, given model whose depth (i.e. number of layers) is D and width (i.e. number of parallel inference units in layer) K. Fig. shows the performance of CaseQ w.r.t. different K and D in sequential recommendation task. The results show that increasing K and D in a certain range is effective for improving the performance. Notably, the improvements are more sigmificant with larger gap size, which again verifies that CaseQ helpful for addressing temporal shift. Still, out-of-range -o--ge K and D would harm the performance. which is presumably due to over-fitting
Limitations and Conclusion
Limitations and Potential Impacts. One of the main limitations of our variational context ad- justment framework is that it stratifies discrete contexts (which may be infinitely many) into fixed context set, in order to make the objective tractable. However, this remains common issue for all works that are dealing with infinitely many discrete confounders, and our framework has already made a progress in this direction increasing the size of context set to KDD. We expect see more works to be done in this direction. Also, due to the nature of the problem setting where the context is defined as a latent and abstract factor (which is a major challenge we aim to tackle and serves as our contribution), it would be hard or even impossible to identify its meaning in the real-world. Finally, we do not foresee any direct negative societal impacts.
Conclusion. In this paper, we endeavor to deal with temporal distribution shift in sequential event prediction. Namely, the training and testing data are generated from distinct distributions with different contexts. We first show that existing approaches with maximum likelihood estimation would leverage spurious correlation and produce undesirable prediction. Then we propose new learning objective with backdoor adjustment and variation inference techniques. Based on this, we devise a novel model framework for learning context-specific representations. Extensive experiments demonstrate the practical efficacy of proposed method. We hope our work could enlighten new research focus for subsequent works and inspire more frontier research in sequential event prediction against distribution shift.
10

---

Acknowledgement
This work was partly supported by National Key Research and Development Program of China (202AAAA0107000), National Natural Science Foundation of China (61972250, 127003)) and Shanghai Municipal Science and Technology (Major) Project (2021 SHZDZX0102, 2251 1051000.
References
[1] Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. A general framework for counterfactual learning-to-rank, In SIGIR, pages 5 5-14, 2019.
[2] Aman Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Li, Mare Najork, and Thorsten Joachims. Estimating position bias without intrusive interventions. In WSDM, pages 482,482, 2019.
[3] Juan-Jose Beunza, Enrique Puertas, Ester Gareía-Ovejero, Gema Villalba, Emilia Condes, Gergana Koleva, Cristian Hurtado, and Manuel F Landecho. Comparison of machine learning algorithms for clinical event prediction (risk of coronary heart disease). JBI, 97:103257, 2019,
[4] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifica- tion tasks to new unlabeled sample. In Advances in Neural Information Processing Systems, pages 2178 2186, 2011.
[5] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. 2015.
[6] Konstantina Christakopoulou, Madeleine Traverse, Trevor Potter, Emma Marriott, Daniel Li, Chris Haulk, Ed Chi, and Minmin Chen. Deconfounding user satisfaction estimation from response rate bias. In RecSys, pages 450-455, 2020.
[7] Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1-15, 2000.
[8] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. International Conference on Learning Representations, 2015.
[9] Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In SIGKDD, pages 1555-1564, 2016.
[10] David Eigen, Marc' Aurelio Ranzato, and llya Sutskever. Learning factored representations in deep mixture of experts. arXiv:/312.44: 2013.
[11] Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. TOIS, 39(1):1-42, 2020.
[12] Shanshan Feng, Xutao Li, Yifeng Zeng. Gao Cong, Yeow Meng Chee, and Quan Yuan. Person- alized ranking metric embedding for next new poi recommendation. In IJCAI, 2015.
[13] Wei Gong and Laila Khalid. Aesthetics, personalization and recommendation: A survey deep learning in fashion. arXiv:2101.08301, 2021.
Branroth-Wilddng [14] Mark Granroth- Wilding and Stephen Clark. What happens next? event prediction using compositional neural network model. In AAAI, volume 30, 2016.
[15] F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. pages 1-19, 2015.
[16] Chikara Hashimoto, Kentaro Torisawa, Julien Kloetzer, and Jong-Hoon Oh. Generating event causality hypotheses through semantic relations. In AAAI, 2015.
[17] Chikara Hashimoto, Kentaro Torisawa, Julien Kloetzer, Motoki Sano, István Varga, Jong-Hoon Oh, and Yutaka Kidawara. Toward future scenario generation: Extracting event causality exploiting semantic relation, context, and association features. In ACL, pages 987-997, 2014.
11

---

[18] Ruining He, Wang-Cheng Cheng Kang, and Julian McAuley. Translation-basee recommendation. In RecSys, pages 161-169, 2017.
[19] Ruining He and Julian McAuley. Fusing similarity models with markov chains for sparse sequential recommendation. In ICDM, pages 191-200. IEEE, 2016.
[20] Balázs Hidasi and Alexandros Karatzoglou. Recurrent neural networks with top-k gains for session-based recommendations, In CIKM, pages 843-852, 2018.
[21] Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NeurIPS, 2016.
[22] Guangda Huzhang, Zhenjia Pang, Yongqing Gao, Yawen Liu, Weijie Shen, Wen-Ji Zhou, Qing Da, Anxiang Zeng, Han Yu, Yang Yu, et al. Aliexpress learning-to-rank: maximizing online model performance without going online. TKDE, 2021.
[23] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. International Conference on Learning Reprexentations, 2017.
[24] Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In ICDM, pages 197-206, 2018.
[25] Benjamin Letham, Cynthia Rudin, and David Madigan. Sequential event prediction. Machine learning, 93(2):357-380, 2013.
[26] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-leaming for domain generalization. In AAAI, 2018.
[27] Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural attentive session-based recommendation. In CIKM, pages 1419-1428, 2017.
[28] Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference predict invariant conditional distributions. Advances in Neural Information Processing Systems, 2018.
[29] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adver- sarial autoencoders. arXiv:1511.05644 2015.
[30] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In COLT, 2009.
[31] Krikamol Muandet, David Balduzzi. and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML, pages 10-18, 2013.
[32] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Kian-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look at language bias. In CVPR, pages 12700-12710, 2021.
[33] Judea Pearl et al. Models, reasoning and inference. Cambridge University Press, 19, 2000.
[34] Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. Causal inference in statisties: primer. John Wiley & Sons, 2016.
[35] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic Books, 2018
[36] Ladislav Peska and Peter Vojtas. Off-line vs. on-line evaluation of recommender systems in small e-commerce. In HT, pages 291-300, 2020.
[37] Zhi Qiao, Shiwan Zhao, Cao Xiao, Xiang Li, Yong Qin, and Fei Wang. Pairwise-ranking based collaborative recurrent neural networks for clinical event prediction. In IJCAI, 2018.
[38] Kira Radinsky, Sagie Davidovich, and Shaul Markovitch. Learning causality for news events prediction. In WWW, pages 909-918, 2012.
12

---

[39] Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thiee Factorizing personalized markov chains for next-basket recommendation. In WWW, 2010.
[40] Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen. Adjusting for confounding with text matching. American Journal of Political Science, 64(4):887-99) 803) 2020.
[41] Marco Rossetti, Fabio Stella, and Markus Zanker. Contrasting offline and online results when evaluating recommendation algorithms. In RecSys, pages 31-34, 2016.
[42] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri. Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross graddiett training. 18X4 2018.
[43] Henrik Stormer. Improving e-commerce recommender systems by the identification of seasonal products. In AAAI, pages 92-99, 2007.
[44] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In CIKM, pages 1441-1450) 2019.
[45] Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In WSDM, pages 565-573, 2018.
[46] Jakub Tomezak and Max Welling. Vae with vampprior. In AISTATS, 2018.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages 5998 6008, 2017.
[48] Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. " "click" is not equal to" like": Counterfactual recommendation for mitigating clickbait issue. SIGIR, 2021.
[49] Liwei Wu, Shuqing Li, Cho-Jui Hsieh, and James Sharpnack. Sse-pt: Sequential recommenda- tion via personalized transformer. In RecSys, pages 328-337. 2020.
[50] Qitian Wu, Yirui Gao, Xiaofeng Gao, Paul Weng, and Guihai Chen. Dual sequential prediction models linking sequential recommendation and information dissemination. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pages 447-457, 2019.
[51] Qitian Wu, Chenxiao Yang. Shuodian Yu, Xiaofeng Gao, and Guihai Chen. Seq2bubbles: Region-basee embedding learning for user behaviors in sequential recommenders. In Proceed- ings of the 30th ACM International Conference Information & Knowledge Management, pages 2160-2169, 2021.
[52] Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. In International Conference on Learning Representations, 2022.
[53] Shuai Xiao, Junchi Yan, Xiaokang Yang. Hongyuan Zha, and Stephen Chu. Modeling the intensity function of point process via recurrent neural networks. In AAAI, volume 31, 2017.
[54] Chenxiao Yang, Jiale Zhang, Xiaofeng Gao, and Guihai Chen. Mattrip: Aulti-functional attention-based neural network for semantic travel route recommendation. In 2021 IEEE
International Conference on Web Services (ICWS), pages 145-154. IEEE, 2021.
[55] Nianzu Yang, Kaipeng Zeng. Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for -uitof-distriuutinn molecular representations. In Advances in Neural Information Processing Systems, 2022.
[56] Jihang Ye, Zhe Zhu, and Hong Cheng. What's your next move: User activity prediction in location-based social networks. In SDM, pages 171-179, 2013.
[57] Dong Zhang, Hanwang Zhang, Jinhui Tang, Xiansheng Hua, and Qianru Sun. Causal interven- tion for weakly-supervised semantic segmentation. Advances in Neural Information Processing Systems, 2020.
13

---

[58] Shengyu Zhang, Dong Yao, Zhou Zhao, Tat-Seng Chua, and Fei Wu. Causerec: Counterfactual user sequence synthesis for sequential recommendation. In SIGIR, pages 367-377, 2021.
[59]
Liang Zhao. Event prediction in the big data era: A systematic survey. arXiv:2007.09815, 2020.
[60] Sendong Zhao, Ting Liu, Sicheng Zhao, Yiheng Chen, and Jian- Yun Nic. Event causality extraction based connectives analysis. Neurocomputing, 173:1943-1950 1950, 2016.
[61] Sendong Zhao, Quan Wang, Sean Massung, Bing Qin, Ting Liu, Bin Wang, and ChengXiang Zhai. Constructing and embedding abstract event causality networks from text snippets. In WSDM, pages 335-344, 2017.
[62] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click-through rate prediction. In AAAI, volume 33, pages 5941-5948, 2019.
[63] Kaiyang Zhou, Ziwei Liu, Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, arXiv:2103.02503, 2021.
[64] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and JinRong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In CIKM, 2020.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "vVPIloiWVgM", "reviewer": "Reviewer_XNzH", "summary": "This paper aims to address the task of event sequence modeling, where the aim is to predict the next event in sequential data, focusing on the scenario where there is a substantial gap between the time the data is collected and when the prediction is made.  In this setting, the data distribution may have shifted over time, and the events' context may also have changed.  The authors propose a strategy that combines several ideas: a causal \"backdoor adjustment\" to correct for the confounding nature of the latent context, variational inference to infer the distribution over the latent context, and a neural network with a \"hierarchical branching structure\" for the inference.", "questions": "Can the authors identify a scenario in which this approach is more appropriate than a continuous-time model that leverages timestamps to model the gap between training and deployment and also models the context's temporal dynamics?", "limitations": "Since there is little focus on the potential application of the work, there is not much to say about its societal impacts.\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed methodology combines a diverse range of ideas from different subfields of machine learning (temporal modeling, causality, variational inference, neural networks) into a sophisticated, cohesive approach.  The proposed methodology seems reasonable (though with this many moving parts, I could easily have missed something if there was an issue).\n\nWhile the problem formulation is interesting, I am not convinced that it is well motivated for real applications.  Arguably, if the goal is to predict events in the remote future, a model which predicts the next event in a sequence is not appropriate, since the event to be predicted will most likely not actually be the \"next\" event.  In this scenario, a continuous time model that uses timestamp information would be more appropriate to properly model how the gap between training and prediction would impact the prediction.  E.g., in the example where the context is the season, one can easily incorporate seasonality into a model that takes the timestamp into account.  There may be specific applications where the proposed method is needed, but the paper has not provided one.\n\nI am also not convinced that the method needs to be this complicated, and the experimental results have not confirmed this.  To justify this complicated formulation, it would have been valuable to include comparisons to simplifications of the proposed framework.\n\nThe analysis regarding the limitations of maximum likelihood estimation (Section 2.1) was very nice, though not really surprising.  If you train a model that does not account for latent confounders in a scenario where those confounders have changed, it is clearly not going to work well.  The issue is not really with maximum likelihood estimation, but with the misspecification of the model in not modeling the confounder, as well as using a non-causal method in a scenario where causality is needed.\n\nThe paper is reasonably well written.  If it is hard to follow in parts, this is mostly because of the complicated combination of disparate ideas.\n\nStrengths:\n\n  - Innovative approach combines ideas from different subfields in a unique way.\n  \n  - The use of causal methods for this problem is worth pursuing.\n  \n  - Experimental results include both quantitative and qualitative analyses, and investigate scalability and hyper-parameter settings.\n\nWeaknesses: \n\n  - The problem formulation is not well motivated (see above).  It is a lot of effort to try to fix a model which is not appropriate for its task (event sequence modeling for predicting events in the remote future) instead of just using an appropriate model (continuous time models, particularly those which explicitly model the context and distribution shift).  Those types of more traditional models should also be compared to as baselines.\n  \n  - The approach, although interesting, is complicated, and there are no experimental results that help to justify this complexity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ocJpNSIVloP", "reviewer": "Reviewer_k7dp", "summary": "The paper tackles the problem of event sequence modelling and forecasting by allievating problem of adapting to OOD data. The authors propose a variational learning algorithm that encodes the context embeddings which model changing data distribution of past event sequences and motivate it as a Structural Causal model where the causal relation between context and future even sequence is deconfounded so that the causal chain only goes through past event sequence.", "questions": "1. How does model deal with novel contexts?\n2. Will the generative model adapt to uncertain situations or provide good uncertainty bounds?\n3. EValuation on a benchmark where changes in context is known to test if the learned contexts map to domain knowledge would be useful.", "limitations": "1. .It is not clear how sensitive the model is to the number of contexts when dealing with datasets of differing scales and changes in distributions.\n2. It is not clear how the model deals with novel or derivative contexts.\n3. The underlying assumption of using backdoor adjustment to effectively deconfound context to events causal relationship may not in many real-world cases.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Paper is strongly motivated and tackles an important problem in deploying real-world predictive systems with rapidly changing data distributions.\n2. The method is a novel application of SCM for event sequence prediction and modeling choices are well justified both rationally and empirically.\n3. The empirical results show improvement in some standard benchmarks. Ablation and secondary results show that the model indeed automatically learns to detect changes in context and switches across them resulting in increase in performance.\nWeaknesses\n1. % improvement is used as the primary metric of comparison. A better approach would be to provide absolute results and note statistical significance.\n2. Model doesn't adapt to novel context in test set and there is no evidence of good uncertainty estimation in unknown scenarios when facing novel contexts/data distributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jVBvA33OOv", "reviewer": "Reviewer_HB3W", "summary": "The authors propose the CaseQ framework for temporal event prediction under distribution shift. They identify a clear problem in typical temporal event prediction approaches: learning spurious correlations between items in the training set that do not generalize to the test set, which consists of events at later times. They hypothesize that these spurious correlations are due to latent contexts that may change from the training to the test set. They consider structural causal models for the data generating process that incorporate context and propose to use a backdoor adjustment to deconfound the context. They then propose a variational approximation that is tractable and treat the evidence lower bound (ELBO) as the learning objective. They demonstrate that their CaseQ framework can be used to mitigate accuracy drops when the gap between the training and test sets grows over time.\n", "questions": "1. Which components of the model instantiations section are novel?\n2. The proposed framework considers a very specific reason for distribution shift--dependence on a hidden context. What happens if there are distribution shifts for other reasons, e.g. changes in user preferences over time that may not be related to a context?\n", "limitations": "Limitations are not really discussed aside from some effects of hyperparameters in Section 4.5. There could be many different reasons for temporal distribution shift, and the proposed work considers one specific reason: dependence on a latent context. The discussion of limitations could be improved to discuss other potential reasons for context shift and how they could be incorporated into the model, or how their current model may try to capture these other reasons.\n", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Addresses the distribution shift problem for temporal event prediction in an innovative manner (to the best of my knowledge) using a causal inference approach combined with variational inference.\n- Strong experimental results showing that the proposed CaseQ framework can mitigate drops in recommendation accuracy by about 50% as the gap between training and test set grows to about 30 events.\n- Paper is mostly well written and explains the reasoning behind the proposed generative perspective on temporal distribution shift in a friendly manner.\n\nWeaknesses:\n- Novelty of the specific model instantiations are unclear. It appears to me that the authors are taking standard sequence models (e.g. GRU) as the inference unit, but then then hierarchical branching structure is novel. It would be helpful to describe in the first paragraph of Section 3 which components are novel.\n- Several of the figures are quite small (e.g. Figure 4) because the authors are trying to present a lot of results. I would suggest moving some of the results to the supplementary material, e.g. the hyperparameter analysis in Section 4.5. I'm not sure what I'm supposed to understand from Figure 4 aside from seeing some changes over time.\n- Code is not provided with the submission.\n\nOther minor issues:\n- \"Prior art\" and \"prior work\" are already plural, not \"prior arts\" and \"prior works\"\n- Time unit is missing in Figure 6. Train time of 0.12 in what unit of time?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F7dl77V9uEf", "reviewer": "Reviewer_t8MQ", "summary": "The paper investigates the the temporal distribution shift problem or sequential event prediction and propose a new variational context adjustment approach. The main idea of the approve leveraged variational inference to estimate and adjust for the casual contexts, treating them as latent variables to be learned by the model. Lastly, a hierarchical branching structure was proposed to accommodate large number of events  . ", "questions": "Could the authors comment on the model's performance where the context shifted to a point where previously common events now become are and previously rare events now become common. The latter case would suggest that the model will have trouble learning good embeddings for the rare/possibly unseen events during training.", "limitations": "The reviewer agrees with the authors that there are no imminent societal impact with the current work.\n\nHowever, the reviewer would recommend the authors to address potential limits of this work, per the check list. It was marked as yes but no reference section. Mainly, given the potentially large number of event types for real-world problems, it seems like the branching factor of K^D is quite limiting.\n\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1) Clarity of writing. The paper is very well-written and easy to follow, which should help with reproducing the results of this work.\n2) Originality. Although the idea of leveraging variational inference for casual effect inference is not new [1], the application of it to event prediction is worthy of investigation, in addition to the novelty from the proposed hierarchical structure that can scale up to large number of events and the mixture of posteriors as prior.\n3) Evaluation results looks promising, especially the visualization of context probabilities over time.\n\nCons:\n1) Lack of synthetic evaluations. For works with casual inference, it's necessary that the authors should experiment with a synthetic evaluation where ground-truth contexts/confounders are known and are manually manipulated. This allow one to verify exactly how good are the estimated contexts compared to ground-truths and how the model adjust for the context to arrive at the next-step prediction. I don't recall seeing it in the paper.\n2) Lack of insights for the real-world datasets in the evaluation section. Is there a way to relate how the changing context causes certain events  to happen more/less? Although the proposed approach did well at detecting shifts and adjusting for these shifts to improve event predictions, can we draw any practical insights from these shifts w.r.t the real-world datasets like movies reviews / preferences.\n\n[1] Louizos, C., Shalit, U., Mooij, J. M., Sontag, D., Zemel, R., & Welling, M. (2017). Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept