Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies event streams and proposes a neural-network-based method that learns weighted clock logic to model temporal point process. Compared to traditional combinatorial optimization approaches, this method is more expressive in terms of representing weighted clock logic formulas as neural network activation function; easier to learn through gradient descent. 
strength_and_weaknesses: The approach of using neural networks to model the time interval length is novel. The writing of this paper, especially the technical part, is sound and with minor issues. There are some insights of how to relax logical formulas that people can draw from this paper.

The major concern of this paper is that there are no theoretical insights or motivations for the proposed method. The authors present the architecture of the clock logic neural networks without motivating the reason behind them, nor do they explain how the smooth relaxation or POC/SOC benefits problem-solving, either in the method section or in the experiments section.

In section 2, though the examples are easy to follow, the authors introduce more than enough terminologies which increases the complexity of understanding. For example, it seems only "paired order predicate" is used in the rest of this paper so other terminologies can be simplified. 

The author claims to learn "interpretable" rules without detailed explanation/verification, either in the main method section or in the experiments section. 

The synthetic dataset is too simple and thus easy to learn. The "effectiveness" of the proposed method needs to be verified on larger datasets or real-world datasets. The empirical results on real-world datasets are not competitive enough compared to existing methods. 


clarity,_quality,_novelty_and_reproducibility: The writing is clear and the paper shows enough detail for reproducibility. 
summary_of_the_review: This paper proposes a novel approach to learn temporal point processes using neural networks which learn weighted clock logic formulas. Though the presentation is clear, the major concerns of this paper are the lack of motivation and competitive empirical results. I am not an expert on this topic, but I think the authors can improve this paper by providing clear motivation/theoretical insights, or detailed explanation/verification of all the claims (such as interpretability and effectiveness).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method for learning models of multivariate event streams, which are time series of events that may be may occur irregularly and synchronously at points in continuous time.  Specifically, they introduce clock logic neural networks (CLNNs), which model temporal point processes using formulas to represent conditional intensity rates.  Learning in this context involves a relaxation of the satisfiability predicate on formulae and then gradient-based maximum likelihood optimization.  
strength_and_weaknesses: Strengths:

- Attacks an interesting and useful problem with a well-motivated approach
- The method is performant (in absolute execution time) and has the potential to be a practical tool
- The general approach of relaxation for learning multivariate event streams could generalize to richer representations

Weaknesses

- The experimental results are mixed.  More importantly, it’s not clear when I should expect this method to perform well or poorly.  I would not wish to penalize this paper for not beating SOTA in every situation, but it’s not apparent when and why I should use this approach over any other.  The method performs best on the LinkedIn dataset.  Is this coincidental, or is there some salient property there that makes a difference?
- The primary claimed advantage of CLNNs of SOTA is expressiveness.  While this is to some degree true by construction — CLNNs can represent the relationships between interval lengths and the conditional intensity — there’s not much evidence to demonstrate that it matters in its ability to model a richer class of datasets.
clarity,_quality,_novelty_and_reproducibility: - The main challenge with introducing a new formalism to an existing problem is to demonstrate why it is important or useful relative to what exists or could exist.  To the best of my knowledge the approach taken in this paper is sound and well thought through, but it fails in this particular regard.  The first main claim is on expressiveness, which is well supported by the evidence but more context is needed to determine if the gains are meaningful.  For instance, why are performance numbers on the real datasets absent, when these tests were clearly run for the accuracy results?  The second claim is around expressiveness, which as I have noted elsewhere, is barely explored.
- There’s extensive work in various forms of relaxations of propositional formulae and correspondingly learning formulae from a relaxed search space.  The weighted clock logic formalism is new and well-suited to the domain.  In terms of novelty, its relaxation is in line with existing work.
- The writing is locally good.  Individual sentences and paragraphs are clear, and the formalism is well explained.  However, the overall structure could be improved.  For example, the introduction paragraph of Section 2 could provide more context; at the moment one goes through several definitions without a clear direction of where it is headed until the end of the section.  This could be as simple as giving an example of a wCL earlier in the section.
- The mathematical exposition is precise.  On the other hand, to my knowledge, no code has been provided.  Also, optimization hyperparameters and the general experimental set up are mostly absent
summary_of_the_review: - Well-motivated and interesting problem
- Sound approach but unclear the relevance of the result.  I'm willing to be convinced otherwise
- I’d suggest focusing on the core claims and providing more evidence
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach for learning interpretable logical rules for event data, where each event is represented by a timestamp and a discrete event type.
The main idea of the proposed approach is to represent the logical rules and their combinations as nodes in a neural network. By optimizing the weights of the neural network with gradient descent, the relevant logical rules are learned from the data. This is a significant improvement compared to earlier works, where the rules are learned using combinatorial optimization tools.
strength_and_weaknesses: Strengths:
- The proposed framework has several clear advantages compared to the existing approaches:
    - The learned logic rules take into account the time between the events (not just their ordering), which can be important in some applications
    - Thanks to using a differentiable relaxation of the discrete search space, the training time for the proposed approach is ~100x faster compared to the existing state-of-the-art method
    - The predictive performance of the proposed model is competitive with other approaches
- The examples introduced throughout the paper are helpful for understanding the contents.
- The experimental setup is described very clearly, which should make the results easy to reproduce.


I haven't found any major weaknesses in the paper. The only part that could be explored in more detail is the stability of the learned logic rules with respect to different random initializations of the network weights and different choices of the  threshold parameter  $\alpha$.

Notes:
- Typo: $c$ should be replaced with $l$ in Equation 9
- Figure 1: Would be helpful to mention that the approach can also work with continuous timestamps, and that discrete timestamps are chosen for illustration purposes.

clarity,_quality,_novelty_and_reproducibility: - The contents of the paper are very technical, but the provided examples are helpful for understanding the high-level intuition behind the different model components.

- To my best knowledge, the proposed approach is novel. References are provided for the building blocks based on earlier works.

- The approach and the experimental setup are described clearly enough to be easily reproduced.
summary_of_the_review: The paper advances the state of the art in the problem of learning interpretable logic rules from multivariate event streams.
The proposed approach offers a clear improvement over existing methods both in terms of expressivity and runtime.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a framework of 'clock logic neural networks', which use a special neural network architecture to train temporal process models that can then be interpreted in terms of weighted clock logic formulae.  The paper provides a detailed overview of weighted clock logic, and proves that the network architecture enforces the desired properties necessary for this interpretation.  Training can be achieved directly through projected gradient descent.  The experimentation on synthetic and real datasets shows that the CLNN learns more informative rules (incorporating interval duration information) than alternative SOTA approaches, and achieves competitive performance in terms of the out-of-sample log-likelihood.
strength_and_weaknesses: Strengths:

- The paper is presented very thoroughly, and the theoretical properties of the model are well established.
- Developing neuro-symbolic approaches is an important problem, both from the point of view of model interpretability, and modeling complex dependencies; the paper offers an elegant solution by formulating a differentiable model which has a logical interpretation.
- The experimentation is somewhat condensed but convincing on both synthetic and real problems.

Weaknesses:

- As above, the experimentation could be expanded.  Mostly, the interpretation is done by simply listing the formulae extracted by the network (Tables 1 and 4) and inviting comparison by inspection.  It would be beneficial to develop a metric to assess the formulae found against the ground truth for various methods.
- The network training incorporates constraints on architecture and parameters (via projected gradient) to ensure interpretability.  It would be interesting to see the log-likelihood performance of similarly sized NN models without these constraints, to gauge if the interpretability is helping or hindering performance.
clarity,_quality,_novelty_and_reproducibility: The paper has good clarity, quality, novelty and reproducibility.
summary_of_the_review: An interesting neuro-symbolic approach for training NN models with interpretability in terms of temporal logic.  The paper will be of interest to those working in logic, deep-learning and model interpretability.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a neuro-symbolic model for continuous event streams. Reviewers appreciate the proposed model but had some reservations about the empirical results, which are sufficiently addressed in the author response.
justification_for_why_not_higher_score: Empirical results are not strong enough for a higher score.
justification_for_why_not_lower_score: It's a thorough interesting paper, certainly something that can be built on.
Decision: Accept: poster