Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims to address the task of event sequence modeling, where the aim is to predict the next event in sequential data, focusing on the scenario where there is a substantial gap between the time the data is collected and when the prediction is made.  In this setting, the data distribution may have shifted over time, and the events' context may also have changed.  The authors propose a strategy that combines several ideas: a causal "backdoor adjustment" to correct for the confounding nature of the latent context, variational inference to infer the distribution over the latent context, and a neural network with a "hierarchical branching structure" for the inference.
strengths_and_weaknesses: The proposed methodology combines a diverse range of ideas from different subfields of machine learning (temporal modeling, causality, variational inference, neural networks) into a sophisticated, cohesive approach.  The proposed methodology seems reasonable (though with this many moving parts, I could easily have missed something if there was an issue).

While the problem formulation is interesting, I am not convinced that it is well motivated for real applications.  Arguably, if the goal is to predict events in the remote future, a model which predicts the next event in a sequence is not appropriate, since the event to be predicted will most likely not actually be the "next" event.  In this scenario, a continuous time model that uses timestamp information would be more appropriate to properly model how the gap between training and prediction would impact the prediction.  E.g., in the example where the context is the season, one can easily incorporate seasonality into a model that takes the timestamp into account.  There may be specific applications where the proposed method is needed, but the paper has not provided one.

I am also not convinced that the method needs to be this complicated, and the experimental results have not confirmed this.  To justify this complicated formulation, it would have been valuable to include comparisons to simplifications of the proposed framework.

The analysis regarding the limitations of maximum likelihood estimation (Section 2.1) was very nice, though not really surprising.  If you train a model that does not account for latent confounders in a scenario where those confounders have changed, it is clearly not going to work well.  The issue is not really with maximum likelihood estimation, but with the misspecification of the model in not modeling the confounder, as well as using a non-causal method in a scenario where causality is needed.

The paper is reasonably well written.  If it is hard to follow in parts, this is mostly because of the complicated combination of disparate ideas.

Strengths:

  - Innovative approach combines ideas from different subfields in a unique way.
  
  - The use of causal methods for this problem is worth pursuing.
  
  - Experimental results include both quantitative and qualitative analyses, and investigate scalability and hyper-parameter settings.

Weaknesses: 

  - The problem formulation is not well motivated (see above).  It is a lot of effort to try to fix a model which is not appropriate for its task (event sequence modeling for predicting events in the remote future) instead of just using an appropriate model (continuous time models, particularly those which explicitly model the context and distribution shift).  Those types of more traditional models should also be compared to as baselines.
  
  - The approach, although interesting, is complicated, and there are no experimental results that help to justify this complexity.
questions: Can the authors identify a scenario in which this approach is more appropriate than a continuous-time model that leverages timestamps to model the gap between training and deployment and also models the context's temporal dynamics?
limitations: Since there is little focus on the potential application of the work, there is not much to say about its societal impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates the the temporal distribution shift problem or sequential event prediction and propose a new variational context adjustment approach. The main idea of the approve leveraged variational inference to estimate and adjust for the casual contexts, treating them as latent variables to be learned by the model. Lastly, a hierarchical branching structure was proposed to accommodate large number of events  . 
strengths_and_weaknesses: Pros:
1) Clarity of writing. The paper is very well-written and easy to follow, which should help with reproducing the results of this work.
2) Originality. Although the idea of leveraging variational inference for casual effect inference is not new [1], the application of it to event prediction is worthy of investigation, in addition to the novelty from the proposed hierarchical structure that can scale up to large number of events and the mixture of posteriors as prior.
3) Evaluation results looks promising, especially the visualization of context probabilities over time.

Cons:
1) Lack of synthetic evaluations. For works with casual inference, it's necessary that the authors should experiment with a synthetic evaluation where ground-truth contexts/confounders are known and are manually manipulated. This allow one to verify exactly how good are the estimated contexts compared to ground-truths and how the model adjust for the context to arrive at the next-step prediction. I don't recall seeing it in the paper.
2) Lack of insights for the real-world datasets in the evaluation section. Is there a way to relate how the changing context causes certain events  to happen more/less? Although the proposed approach did well at detecting shifts and adjusting for these shifts to improve event predictions, can we draw any practical insights from these shifts w.r.t the real-world datasets like movies reviews / preferences.

[1] Louizos, C., Shalit, U., Mooij, J. M., Sontag, D., Zemel, R., & Welling, M. (2017). Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30
questions: Could the authors comment on the model's performance where the context shifted to a point where previously common events now become are and previously rare events now become common. The latter case would suggest that the model will have trouble learning good embeddings for the rare/possibly unseen events during training.
limitations: The reviewer agrees with the authors that there are no imminent societal impact with the current work.

However, the reviewer would recommend the authors to address potential limits of this work, per the check list. It was marked as yes but no reference section. Mainly, given the potentially large number of event types for real-world problems, it seems like the branching factor of K^D is quite limiting.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose the CaseQ framework for temporal event prediction under distribution shift. They identify a clear problem in typical temporal event prediction approaches: learning spurious correlations between items in the training set that do not generalize to the test set, which consists of events at later times. They hypothesize that these spurious correlations are due to latent contexts that may change from the training to the test set. They consider structural causal models for the data generating process that incorporate context and propose to use a backdoor adjustment to deconfound the context. They then propose a variational approximation that is tractable and treat the evidence lower bound (ELBO) as the learning objective. They demonstrate that their CaseQ framework can be used to mitigate accuracy drops when the gap between the training and test sets grows over time.

strengths_and_weaknesses: Strengths:
- Addresses the distribution shift problem for temporal event prediction in an innovative manner (to the best of my knowledge) using a causal inference approach combined with variational inference.
- Strong experimental results showing that the proposed CaseQ framework can mitigate drops in recommendation accuracy by about 50% as the gap between training and test set grows to about 30 events.
- Paper is mostly well written and explains the reasoning behind the proposed generative perspective on temporal distribution shift in a friendly manner.

Weaknesses:
- Novelty of the specific model instantiations are unclear. It appears to me that the authors are taking standard sequence models (e.g. GRU) as the inference unit, but then then hierarchical branching structure is novel. It would be helpful to describe in the first paragraph of Section 3 which components are novel.
- Several of the figures are quite small (e.g. Figure 4) because the authors are trying to present a lot of results. I would suggest moving some of the results to the supplementary material, e.g. the hyperparameter analysis in Section 4.5. I'm not sure what I'm supposed to understand from Figure 4 aside from seeing some changes over time.
- Code is not provided with the submission.

Other minor issues:
- "Prior art" and "prior work" are already plural, not "prior arts" and "prior works"
- Time unit is missing in Figure 6. Train time of 0.12 in what unit of time?

questions: 1. Which components of the model instantiations section are novel?
2. The proposed framework considers a very specific reason for distribution shift--dependence on a hidden context. What happens if there are distribution shifts for other reasons, e.g. changes in user preferences over time that may not be related to a context?

limitations: Limitations are not really discussed aside from some effects of hyperparameters in Section 4.5. There could be many different reasons for temporal distribution shift, and the proposed work considers one specific reason: dependence on a latent context. The discussion of limitations could be improved to discuss other potential reasons for context shift and how they could be incorporated into the model, or how their current model may try to capture these other reasons.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the problem of event sequence modelling and forecasting by allievating problem of adapting to OOD data. The authors propose a variational learning algorithm that encodes the context embeddings which model changing data distribution of past event sequences and motivate it as a Structural Causal model where the causal relation between context and future even sequence is deconfounded so that the causal chain only goes through past event sequence.
strengths_and_weaknesses: Strengths
1. Paper is strongly motivated and tackles an important problem in deploying real-world predictive systems with rapidly changing data distributions.
2. The method is a novel application of SCM for event sequence prediction and modeling choices are well justified both rationally and empirically.
3. The empirical results show improvement in some standard benchmarks. Ablation and secondary results show that the model indeed automatically learns to detect changes in context and switches across them resulting in increase in performance.
Weaknesses
1. % improvement is used as the primary metric of comparison. A better approach would be to provide absolute results and note statistical significance.
2. Model doesn't adapt to novel context in test set and there is no evidence of good uncertainty estimation in unknown scenarios when facing novel contexts/data distributions.
questions: 1. How does model deal with novel contexts?
2. Will the generative model adapt to uncertain situations or provide good uncertainty bounds?
3. EValuation on a benchmark where changes in context is known to test if the learned contexts map to domain knowledge would be useful.
limitations: 1. .It is not clear how sensitive the model is to the number of contexts when dealing with datasets of differing scales and changes in distributions.
2. It is not clear how the model deals with novel or derivative contexts.
3. The underlying assumption of using backdoor adjustment to effectively deconfound context to events causal relationship may not in many real-world cases.
review_title: null

Meta-review: metareview: This paper proposes a method for predicting the next event given sequential data. For the prediction under distribution shift, the proposed method uses a backdoor adjustment, variational inference of latent context, and hierarchical branching structure. The proposed method that combines techniques from different fields is interesting. In particular, the use of causal methods for the distribution shift problem in temporal event prediction is novel. The experimental results demonstrate the effectiveness of the proposed method well quantitatively and qualitatively. The paper should be improved according to the reviewers’ comments, e.g., clarifying the motivation in real-world applications.
confidence: Certain
Decision: Accept