Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces A-NHP and A-NDTT that tackle the problem of continuous time sequences and neural-symbolic computing with a time component using transformers and attention. The paper explains their model and techniques mainly with equations. The work seems primarily motivated by previous work on Hawkes processes and neural datalog through time NDTT. They improve by adding a probability model that allows them to embed all possible events and using transformers instead of LSTMs for parallelization. They achieve strong experimental results, especially in StackOverflow and RoboCup.
main_review: While the paper is not too theoretical, the techniques do seem to appear natural and well-motivated by the problem at hand. The experiments are well done and reproducible with the appendix.

I did find the paper a bit difficult to read, but this may be somewhat unavoidable due to the probabilistic modeling. I am not sure about the claim of A-NDTT being simpler and shallower when training time is longer? There is a lot of emphasis on probability models and interpretability with neural-symbolic computing in the exposition that is unexplored in the experiments, too.
summary_of_the_review: I would recommend the paper for acceptance. While there are some issues, the paper is generally well-written, explores new techniques, achieves good results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A model for sequences of events occurring in the continuous time setting is presented. It uses the self-attention mechanism from the Transformer model, together with some domain-specific masking operations, to create event embeddings that are contextualized by their relation to other (relevant) events that occurred. The model is then extended towards a generative process, which in turn is integrated into a neural datalog through time framework.
Experiments show the model to be perform equally or better than baselines based on RNNs or discrete time embeddings.
main_review: This paper is clearly the result of detailed study of the of the NDTT modeling framework and its requirements for underlying models. I should start with saying that I'm not an expert in NDTT.

Positive aspects about the paper:
* It discusses clearly and intuitively why the authors made some architectural choices in Sect. 3 and 4.
* Experimental results indicate that the model works.

Challenges for the reader:
* The paper refers to the architecture as a Transformer-like thing, but important differences exist (no feed forward sublayer, use of the bounded tanh as activation function, no layer norms, positional encodings concatted and fed in at each layer). In Sect. 4, "multi-head selective attention" is introduced, which has no relation to standard "multi-head attention" besides the fact that several subnetworks do different things in parallel - even the aggregation of results is different. These differences are not called out explicitly, nor empirically validated.
* Section 5 is extremely dense and very hard to follow, lacking motivation:
  - If you are trying to introduce NDTT, you should provide an example that helps the reader to understand what you are trying to achieve, BEFORE introducing the formalism.
  - $[h]^{(\ell)}(t)$ is used in Eq(9), visually hard to distinguish from $[[ h ]]^{(\ell)}(t)$, and only discussed 10 lines later.
* Experiments: no ablations of the architecture are studied, even though many choices were made that are non-obvious.

Detailed questions:
* Sect. 4: why is the embedding of the event type not sufficient to drive different attention patterns? (i.e., why does the event type need to be part of parameters rather than part of data)

Update after rebuttal
---------------------------
While the authors provided extensive answers to my questions, the updated submission largely does not reflect these answers, nor is it obvious how it could (given the space constraints). Overall, I think the manuscript is not ready for publication at this time, largely due to a lack of clarity in the writing. I feel unable to judge the core technical contribution, but bad writing alone is sufficient for me to decide not to change my rating.
summary_of_the_review: Overall, I found the first four pages of this paper quite enjoyable, but found the remainder large impenetrable as someone unfamiliar with the area. Given the fact that obvious ablation experiments are missing, I do not think that this paper is ready for publication, lacking both in clarity of writing and strength of evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The main contribution of this work is a new transformer-based temporal point process (TPP) model.
The proposed model defines the conditional intensity at each time $t$ as a function of all past events using the attention mechanism.
This is different form existing neural TPP models that typically embed the event history into a vector.
Additionally, it is shown how the proposed model can be combined with the Neural Datalog Through Time (NDTT) framework [(Mei et al., 2020)](https://arxiv.org/abs/2006.16723), which allows us to specify certain hard constraints on the event occurrences (such as "events of type x can only occur when certain conditions are met").
main_review: **Strengths**
- The proposed model is well-motivated. 
- The architecture and possible variations are described very thoroughly.
- Combining the model with logical rules using the NDTT framework (Section 5) allows to incorporate known constraints on the modelled environment, which may lead to better generalization and predictive performance (Section 8).

-------

I have the following main concerns regarding this paper:

1. **Missing discussion of closely related works**

- The idea of using transformer architectures to define neural TPP models has been explored in multiple existing works. While the THP (Zuo et al., ICML 2020) and SAHP (Zhang et al., ICML 2020) models have been mentioned in this paper, several other published works have been missed and not compared with:

    - [Enguehard et al., ML4H 2020](https://arxiv.org/abs/2007.13794)
    - [Chen et al., ICLR 2021](https://arxiv.org/abs/2011.04583)
    - [Zhu et al., AISTATS 2021](https://arxiv.org/abs/2002.07281)
    - [Sharma et al., KDD 2021](https://arxiv.org/abs/2008.11308)

- The definition of selective attention introduced in Section 4 is equivalent to the notion of Granger causality. GC has been extensively studied in the field of marked TPPs and has been combined with transformer-based neural TPP models ([Zhang et al., WWW 2021](https://dl.acm.org/doi/fullHtml/10.1145/3442381.3450135)).


2. **A more thorough empirical evaluation is needed to support the main claims of the paper**

- One of the claimed advantages of the proposed A-NHP model is its **flexibility** (Section 1, point 3; Section 7). However, the experiments only consider THP and SAHP as baselines. These model assume a rather simple parametrization of the inter-event time distribution, which limits their flexibility, as pointed out in Section 7. There exist multiple more flexible architectures, such as [(Omi et al., 2019)](https://arxiv.org/abs/1905.09690), [(Shchur et al., 2020)](https://arxiv.org/abs/1909.12127) or [(Sharma et al., 2021)](https://arxiv.org/abs/2008.11308), that haven't been compared with. 

    These models have other advantages compared to the A-NHP model such as closed-form likelihood computation and lower memory footprint, while A-NHP requires multiple forward passes to evaluate the intensity at random points during Monte Carlo approximation of the integral. Therefore, comparing to these flexible and more efficient models would be important to answer if the higher runtime/memory consumption of A-NHP is justified by its improved flexibility.

----

**Minor suggestions**
- Section 5: The contents are rather dense and are difficult to comprehend without first reading Mei et al., 2020a. Adding a figure that demonstrates how the rule embedding $[\\![h]\\!]$ $(t)$ affects the conditional intensity for different event types would be really helpful (similar to Figure 1b).

- Figure 2: The figure is difficult to read. Color-coding or using symbols to denote different models in each plot would make it easier to understand.

- Reducing the amount of new / non-standard notation, if possible, would improve the readability of the paper.

- Another claim regarding the A-NHP model that is not explored in the experiments is its claimed ability to learn long-range interactions better than an RNN-based TPP, like NHP (Section 1, point 2). It would be interesting to evaluate this aspect of the model and highlight these failure modes of RNN-based TPPs. One idea would be to do this using multivariate Hawkes processes with long-range triggering kernels and many marks.


------
## Update after the rebuttal

The rebuttal has addressed some of my concerns, so I have raised my score
- An efficient approximation to the log-likelihood is available, so the increased runtime / memory consumption compared to existing models shouldn't be a problem.
- The clarity of writing in Section 5 has been improved

Still, I have several remaining concerns, which is why I'm leaning towards rejection. Here are my suggestions for improving the paper:
- Comparison with additional baselines, such as ADMN and LogNormMix 
- Make the connection to prior work more explicit in Section 4 (selective attention), add citations to other works on transformer-based TPPs, and better frame the contribution of this paper with respect to prior works
- Explain how the upper bound for the thinning algorithm is derived for A-NTPP, since this is a crucial implementation detail
summary_of_the_review: My main criticism of this paper is the lack of discussion and comparison with existing works, as described in point 1 and 2 above. The proposed modifications to the NDTT framework [(Mei et al., 2020)](https://arxiv.org/abs/2006.16723) don't lead to significant improvements over the original methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a continuous-time transformer architecture for event sequence tasks using the Neural Datalog Through Time (NDTT) framework for incorporating logic specifications into the model.
The paper then experimentally compares the proposed model with point-process models and standard NDTT. 
main_review: Overall, the paper provides some interesting contributions on modeling continuous-time event processes and adding specifications via the NDTT framework.
However, the paper also contains several weaknesses, which makes it not ready for publication in its current format.
The writing needs to be substantially improved before this manuscript is ready for publication. What is the intuition on Eq (1-4)? Why would the embeddings of the time be relevant from the second layer onward?  It is unclear what the main contribution of the paper is. On the one hand, the paper proposes continuous-time event transformers, while on the other hand, the paper proposes a neuro-symbolic framework for embedding rules of which event can draw information from which other events. 
The paper assumes the events are discrete-valued, i.e., $e \in \{1,2,\dots E\}$, which limits its scopes for general-purpose modelling.
The paper fails to experimentally compare the proposed continuous-time transformer to existing continuous-time models on standard benchmarks (PhysioNet, etc.) [1,2,3,4]
The paper fails to compare with existing continuous-time attention-based architectures [5].
From the experiments' side, there seems only little to no improvement over standard NDTT.
No code was provided.

[1] Rubanova et al. Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. NeurIPS 2019
[2] Singh et al. Sequential neural processes. NeurIPS 2019.
[3] Li et al. Scalable gradients for stochastic differential equations. AISTATS 2020.
[4] Norcliffe et al. Neural ODE Processes. ICLR 2021
[5] Shukla and Marlin. Multi-Time Attention Networks for Irregularly Sampled Time Series. ICLR 2021
summary_of_the_review: I recommend the authors to improve the writing and clarify whether their approach aims to contribute to general-purpose continuous-time modeling or on incorporating continuous-time logic specification via the NDTT framework. In the current form, neither of these two objectives is fulfilled adequately (see comments above).
review_title: null

Meta-review: The paper builds upon previous work on neural temporal point processes. It mainly proposes the replacement of the LSTMs with Transformers as transformers are widely considered as a more powerful sequence modeling tool and the three advantages listed in the end of section 1 in this paper.

However, on the modeling side, it is not straight-forward how to apply the transformer (the attention architecture) on to the continuous time-sequence problem using NDTT. I think because I read a revised version of this paper, it is actually more understandable to me as compared to the reviewers who read the first draft of the paper. I think A-NDTT is a natural and principled way of introducing the attention mechanism into the continuous time neural symbolic framework, although I agree it unfortunately does not leading to a significant improvement in every experimental setting.

To summarize the discussions, I think the authors did a good job in resolving the concerns on the related work and made the paper easier to understand. I appreciate these efforts from the authors even though I also understand there are concerns left still from the reviewers.

In summary, I am leaning to accept this paper because I think it is an interesting contribution. However, I do agree with the reviewers that the writing of the paper needs to be improved and the experiment section is relatively weak in this paper.
Decision: Accept (Poster)