Title: Transformers for Mixed-type Event Sequences
Venue: NeurIPS 2025
Accepted: yes
Keywords: marked temporal point processes

generative modeling

autoregressive models

transformers

time series
Abstract: Event sequences appear widely in domains such as medicine, finance, and remote sensing, yet modeling them is challenging due to their heterogeneity: sequences often contain multiple event types with diverse structures—for example, electronic health records that mix discrete events like medical procedures with continuous lab measurements. Existing approaches either tokenize all entries, violating natural inductive biases, or ignore parts of the data to enforce a consistent structure. In this work, we propose a simple yet powerful Marked Temporal Point Process (MTPP) framework for modeling event sequences with flexible structure, using a single unified model. Our approach employs a single autoregressive transformer with discrete and continuous prediction heads, capable of modeling variable-length, mixed-type event sequences. The continuous head leverages an expressive normalizing flow to model continuous event attributes, avoiding the numerical integration required for inter-event times in most competing methods. Empirically, our model excels on both discrete-only and mixed-type sequences, improving prediction quality and enabling interpretable uncertainty quantification. We make our code public at https://github.com/czi-ai/FlexTPP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for learning mixed-type Marked Temporal Point Processes using normalizing flows and finds that it improves predictive performance on a number of tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Marked temporal processes with mixed outcome types are very important\n\nWeaknesses:\n- The conditional handling seems overly engineered. Why not just add it as an event at the start of the sequence?\n- The lack of public source code makes it very challenging to understand and verify many of the results\n- The novelty here is very questionable. Marked Temporal Point Processes with mixed types have been explored before. https://arxiv.org/abs/2306.11547 is a recent example that has a nice open source codebase for handling Marked Temporal Point Process with mixed types.\n- Likewise, using normalizing flows for temporal point processes has also been explored. See https://arxiv.org/abs/1909.12127.\n- Related to the missing code, many of the experiments require more detailed information. Secton 4.2 is particularly lacking. How many patients are there per condition? How are you addressing the issue of multiple labels per patient inherent in EHR data? In Table 3 when you say \"Lab tests w/o values\" are you excluding the values only as prediction targets or as both prediction targets and features?\n- No notion of uncertainty or statistic soundness of many of the comparisons. This is especially important for small datasets like EHRSHOT.\n- Missing ablation experiments to test whether particular components (like the normalizing flows part) are helpful. This is something to remark upon because the current literature (like https://arxiv.org/abs/1909.12127) found that normalizing flows were not helpful.\n\nQUESTIONS:\nHow is this different from https://arxiv.org/abs/2306.11547 and https://arxiv.org/abs/1909.12127? What does this paper add that is not in either of those two papers?\n\nCan you please address the questions about 4.2 listed in weaknesses?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends Marked Temporal Point Processes (MTPPs) to support variable-length marks with mixed discrete and continuous data types, proposing FLEXTPP—a Transformer-based architecture that flattens heterogeneous event data into sequences and models them autoregressively. The approach uses categorical distributions for discrete marks and normalizing flows for continuous values, for distribution modeling.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe paper addresses a practical limitation in existing MTPP methods by supporting variable-length, mixed-type marks, which is important for real-world applications such as electronic health records. The proposed framework is clean and general, and the empirical evaluation is relatively comprehensive, showing good performance across multiple benchmarks.\n\n**Weaknesses:**\n(1) The model conditions each mark component on all previous times and marks in a fully autoregressive manner, which can be unnecessarily restrictive, or misrepresent the data. For example, blood pressure and heart rate are recorded at the same time for a patient, but there is no natural dependence on each other.  \n\n(2) While the approach is well-executed, it largely applies standard sequence modeling to flattened temporal data. I feel the main novelty is in the data representation (mixed-type, variable-length marks) rather than methodological innovation.\n\n(3) The likelihood values are reported in the experiments, but not sure how significance the improvement is without statistical test.\n\n**Suggestion.**\nThe paper complicates its presentation by deriving the approach through intensity functions in Section 3.1, despite ultimately proposing an intensity-free model that directly parameterizes $p(t_i, m_i | H_{t_i})$. The detour through intensity functions $\\lambda(t, m |H_t)$ and the conversion in Equation (2) is unnecessary. Equation (3) merely states the standard likelihood for time series data, not a derived result. The exposition would be clearer and more focused if it began with the goal of modeling $p(t_i, m_i | H_{t_i})$ directly, with only a brief mention of intensity-based methods as a contrasting approach.\n\nQUESTIONS:\nTo the best of my understanding, the proposed approach proceeds as follows:\n(1) Start with events consisting of timestamps and mixed-type marks.\n(2) Flatten each event into a sequence of the form $[t_1,  type_1,  mark_{1,1}, \\dots, mark_{1,n}, t_2, type_2, \\dots]$.\n(3) Feed this sequence into a Transformer.\n(4) Train or sample the model in an autoregressive manner.\n\nCan you specially list what has been done more than above, and what's the contribution/innovation from a point process view.\n\nI may be misunderstanding some aspects and would welcome clarification to reassess my evaluation of this work.\n\nLIMITATIONS:\nYes, the paper discussed their limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a marked temporal point process framework for predicting event type, event time, and variable-length discrete/continuous longitudinal covariates, given event history and static covariates, by leveraging an autoregressive likelihood. It employs an autoregressive transformer for encoding event history and conditioning on static covariates. Additionally, the paper leverages normalizing flows for event time and continuous longitudinal covariates prediction. Experimental results demonstrate that the proposed  approach achieves competitive event time and event type likelihood predictions compared to baselines on two real-world and one synthetic dataset.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is relatively well-written and easy to follow.\n- The use of autoregressive transformers for conditional generation given variable length discrete/continuous longitudinal covariates is interesting.\n- Experimental results demonstrate that the proposed approach achieves competetive ikelihood predictions compared to baselines.\n\n**Weaknesses**\n- The proposed approach seems more complex and might not be scalable relative to baselines, which is worsened by the use of normalizing flows for both continuous marker metadata and event time.\n- The paper seems to focus on the likelihood metric, overlooking model interpretability, which is important in practice. For example, by not modeling the intensity function, it is unclear how the model captures the influence of event interactions on event time. The paper should also benchmark the interpretability aspect and compare it with classical Hawkes and interpretable neural Hawkes approaches, *e.g.*, [1, 2].\n- The proposed approach seems to yield small gains relative to baselines (including FlexTPP vs. FlexTPP-C). The paper should also report error bars of the reported negative likelihood.\n- While the paper clearly describes the ablation study of the proposed model (Equations 10-12), the experimental results do not seem to highlight the benefits of conditioning on the variable length discrete/continous longitudinal covariates, which is the key contribution of this work compared to baselines. The paper should also consistently provide benchmarks for the proposed FlexTPP (with/without the longitudinal covariates), *i.e.*, Eqn 10 *vs.* Eqn 12. \n- Table 4: The paper should report the metrics on event type and event time predictions separately and provide non-likelihood-based metrics, such as RMSE, for event time predictions.\n- *Minor*: Line 91 - The intensity probabilities should be conditioned on the event history.\n\n**References**\n\n- [1] Isik et al., \"Hawkes Process with Flexible Triggering Kernels\",  MLHC 2023.\n- [2] Shixiang et al. , \"Neural spectral marked point processes\",  ICLR 2022.\n\nQUESTIONS:\n- Could you provide a complexity analysis in terms of the number of parameters, training/inference time, and scalability relative to baselines?\n- Eqn 9: Could you clarify how the normalizing flow function is specified, *e.g.*, what is different or shared across marker types and event times? Also what is $f^\\prime$?\n- Could you clarify the empirical benefits of conditioning on longitudinal covariates, *i.e*., Equation 10 *vs.* Equation 12?\n- Tables 3 and 4: (i) Could you clarify the models used for \"only discrete events\" and \"Lab tests w/o values\"? (ii) Could you provide complete results, including baselines from Table 2?\n- Figure 3: The qualitative results in Figure 3 are difficult to follow; could you clarify?\n- How does the model capture the influence of event interactions on event time?\n\nLIMITATIONS:\n- The main limitations of the proposed approach are its complexity relative to baselines and its lack of interpretability. The paper should expand the discussion on these aspects.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends Marked Temporal Point Processes (MTPPs) by introducing a framework that supports mixed-type (discrete/continuous) and variable-length marks through normalizing flows, particularly effective in healthcare scenarios like EHR analysis. It establishes conditional modeling as a core principle using an autoregressive Transformer architecture, which eliminates numerical integration bottlenecks and enhances dependency capture compared to RNN-based model. The proposed intensity-free approach employs direct joint probability modeling, significantly improving computational efficiency in complex event generation. Empirical validation across medical and financial domains demonstrates superior prediction accuracy, especially in long-term forecasting through advanced historical encoding, revealing synergistic interactions between temporal dynamics and mark information.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\nThis study demonstrates significant advancements in marked temporal point process modeling through two core innovations：1）it pioneers hybrid-type (discrete+continuous) and variable-length mark processing using normalizing flows, overcoming the fixed-type mark limitations of traditional MTPPs like the RMTPP framework；2) the conditional modeling framework via autoregressive Transformer，achieving higher conditional information utilization compared to the attention-based improvements, particularly enhancing structured output tasks in cross-modal scenarios.\nWeaknesses：\nThe research exhibits certain limitations: The combined computational load from Transformer and normalizing flow architectures increases parameter count by 5× compared to RNN-based models, potentially hindering edge device deployment. While effective for medium-dimensional marks like medical codes, the approach lacks validation on high-dimensional sparse marks (e.g., genomic data).\n\nQUESTIONS:\nPlease clarify whether complexity calculations exclude attention weight matrix generation and provide quantitative GPU memory usage data, as the self-attention mechanism inherently requires O(N²).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}