Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a reinforcement temporal logic rule learning
algorithm to jointly learn temporal logic rules (before-like kind of
rules) and their weights from event data. The proposed learning
algorithm alternates between a rule generator stage and a rule
evaluator stage, where a neural search policy is learned by
risk-seeking gradient descent to discover new rules in the rule
generator stage.
strength_and_weaknesses: The work is quite interesting and shows good results in the
application of predicting sepsis from the MIMIMC-III database by
distinguishing time predictions for LowUrine from NormalUrine rules. Various state of the
art systems were compared against the method proposed in this work.

I wonder if classical methods based on (Probabilistic) Inductive Logic
Programming wouldn't work as well as your proposed method. Why isn't a
method like that compared in your work? Is it very different? TTE
problems may be also modeled using ILP and variants that are
neurosymbolic.

Why did you use only one application? How well would your method work
if you used other examples?

Why do you concentrate on grounded terms and not on first order logic terms with a logical variable?
clarity,_quality,_novelty_and_reproducibility: Clarity: good
Quality: good
Reproducibility: authors give enough materials in the paper itself and in complementary material.  It may be possible to reproduce, but I haven't checked
summary_of_the_review: This work presents a reinforcement temporal logic rule learning
algorithm to jointly learn temporal logic rules (before-like kind of
rules) and their weights from event data. The proposed learning
algorithm alternates between a rule generator stage and a rule
evaluator stage, where a neural search policy is learned by
risk-seeking gradient descent to discover new rules in the rule
generator stage.

The work is quite interesting and shows good results in the
application of predicting sepsis from the MIMIMC-III database by
distinguishing LowUrine from NormalUrine rules. Various state of the
art systems were compared against the method proposed in this work.

I wonder if classical methods based on (Probabilistic) Inductive Logic
Programming wouldn't work as well as your proposed method. Why isn't a
method like that compared in your work? Is it very different? TTE
problems may be also modeled using ILP and variants that are
neurosymbolic.

Why did you use only one application? How well would your method work
if you used other examples?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on the problem of learning the intensity functions for temporal point processes (TPP) to model the occurrence of events in irregular time intervals. Specifically, the authors consider the representation of the intensity functions as parameterized by an exponentiated weighted sum of kernels switched by temporal logic rules. Given this parameterization, the overall objective is to find a set of rules and their corresponding weights to maximize the likelihood of data.

To optimize this objective, the authors break the MLE problem into two steps, a master step and a substep. In the master step, the weights are optimized via convex optimization. In the substep, the logic rules are selected. In order to optimize in the combinatorial space of logic rules, the authors formulate the problem as an RL problem, where the policy outputs a token of logic rules at a time and receives a reward when the sequence of tokens is completed. The authors then apply the risk seeking policy gradient method to solve this problem.

The paper includes empirical evaluations of the proposed method on synthetic and real datasets, and the results suggest that the proposed method achieves good likelihood and absolute error.

strength_and_weaknesses: Overall I think the paper presents an interesting idea of applying RL to solve combinatorial optimization problems in temporal logic rule learning. I’m not familiar with the field of temporal logic point process (TLPP) and the related works in this field, so I will focus on the optimization method part in this review.

### Pro

The paper is well presented and the proposed method is easy to follow. The authors include a detailed introduction about the preliminaries and make it easy for readers to understand the problem. Although I’m not familiar with TLPP, I am able to quickly understand its formulation and the optimization problem associated with learning TLPP thanks to the high quality introduction of this paper.


The empirical results of the proposed method seem strong. The proposed method outperforms many baselines in real datasets.


### Con

While the problem of searching for logic rules can certainly be framed as an RL problem, I’m not convinced that this is a good formulation, and I want to argue that formulating it as an RL problem here makes it unnecessarily more difficult. First of all, the problem has no inherent Markov state transition structure of an MDP because all tokens are directly generated by the agent, unlike in an MDP, where the states can only be influenced by the actions of the agent rather than directly chosen by the agent. Moreover, the reward here is the result of an entire sequence rather than the result of a single time step. Combining these two characteristics, I believe the problem is better formulated as a contextual bandit problem, and solved using a bandit algorithm instead of an RL algorithm. Using a bandit algorithm will directly remove the long horizon credit assignment difficulty of RL and likely result in much stabler training and better performance. 


Given that the optimization problem is a standard back box optimization problem, it would be important to compare to many widely used back box optimizer baselines. Some examples of these algorithms include cross-entropy methods, genetic algorithms and simulated annealing methods. There are many well-developed software packages for these blackbox optimization algorithms that can be easily integrated into the specific problem, so it would be important to try them out and include them in baseline comparisons.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and the proposed method is easy to follow. Because I’m not familiar with the field of temporal logic point process, I cannot evaluate the novelty and reproducibility of this work.

summary_of_the_review: While this paper presents an interesting method of applying RL to optimize the log likelihood of TLPP, I’m not really convinced that RL is the right approach and I believe that simpler approaches could work better for this problem. Hence, I cannot recommend acceptance of this paper in its current state. I highly encourage the authors to try applying bandit methods and including some black box optimization baselines during the author response phase.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to learn a set of temporal logic rules to build a probabilistic model that maximizes the likelihood of the observed data that come in the form of temporal event sequences. Compared to brute-force search in the space of logic rules that are enormous, the proposed approach leverages a neural policy to generate logic rules guided by heuristic reward signals to maximize the likelihood of the probabilistic model that can be obtained using the new rules. Empirically, the paper demonstrates that the proposed approach can improve upon prior baselines that utilize brute force for logic rule generations on a set of synthetic event data proposed by the authors. The proposed approach also outperforms prior methods on the MIMIC-III health care dataset by achieving lower prediction errors. 
strength_and_weaknesses: *Strength*

- The idea of formulating the logic rule generation problem as a reinforcement learning problem is interesting. 

*Weaknesses*

- The paper has a lot of incorrect statements and confusing sentences which make the technical content hard to understand. 
- The empirical section seems to only contain results generated using one single run of the algorithm, which makes it very difficult to judge the effectiveness of the proposed algorithm. 
- Many details of the baseline algorithms are missing in Table 1 (e.g., Transformer architecture, network size, how the data are fed into the transformer models) which could pose major reproducibility issues. 
clarity,_quality,_novelty_and_reproducibility: In general, I found that many statements in the paper are either inaccurate or poorly justified:
- Section 1 - last paragraph above the contributions -- "1) We make the temporal logic rule search subproblem differentiable" -- I am not sure if this statement is accurate. The sentence seems to imply that there are gradients that can be computed through the search procedure, but I think only policy gradient is computed for optimization as far as what I understand.
- Why does $w_f$ need to be positive in Equation (4, 5)? 
- Section 4.1 - "For example, we can formulate $\Omega(\mathbf{w}) = \lambda\_0 \sum\_{f \in \bar{\mathcal{F}}} c_f w_f$ where $c_f$ is the rule length". I am confused by this regularization function. If $w_f$ is a vector of parameters and $c_f$ is a scalar, how can the multiplication results in a scalar?
- Should the minimization be over $f$ rather than $\phi\_f$ in Equation (7)? It does not make much sense to minimize the logic-informed function as the function itself (from my understand) remains fixed (when conditioned on $f$). Although I could be misunderstanding it as it was not very clear in the setup whether this feature function is fixed. 
- Section 5.1 accuracy and scalability - "1) whether our reinforcement temporal
logic learning algorithm can truly uncover the rules from the noisy variable set" -- what is the noisy variable set referring to? How is the noise being introduced?

There are also many typos and grammar issues through out the paper that makes the paper hard to read and understand. To name a few:
- Page 3 above Equation 1 -- "By some simple proof Rasmussen (2018)"
- Page 4 Section 3.2 Temporal Logic Rule -- "Add a temporal dimension to the predicates.": incomplete sentence. 
- Page 4 Section 3.3 Temporal Logical Point Process -- "Introduce a logic function $g_f(\cdot)$ to check ...": incomplete sentence. Also, is $g_f(\cdot)$ a learnable function? What does it output? 

Many notations are not introduced properly:
- What is $X_u(t_u)$ in Equation (2)? 
- How is $g_f(\cdot)$ defined in Equation (3)? The paper mentions that it is a logic function that checks the body conditions of $f$, but this is very vague and it is unclear what the output of the function really is.

summary_of_the_review: In general, I found the paper to be a bit difficult to understand. I can roughly get the main idea of the propose method, but I might have misunderstood many technical details due to the writing issues. In addition, I found that the empirical evaluation of the paper to be a bit lacking and the details of some baselines are missing. These altogether make it hard to evaluate the significance of the results. Because of these reasons, I would not recommend acceptance of the paper at its current state.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a method for identifying temporal logic rules.
This method uses reinforcement learning to select rules and separately optimizes the weights of these rules.

This method is evaluated on finding rules on benchmark tasks, in a transfer setting, and on a real-world problem.
strength_and_weaknesses: The method is an intuitive improvement over the existing enumeration approach.
My understanding of the method is that the selection of rules is done in a greedy way.
Additionally, further rules are not added when the next greedy rule is no longer useful.
As a result, this method is akin to performing prioritized enumeration along with a stopping condition.

My understanding is that the state only includes information about the tokens for the current rule (i.e., no information is included about existing rules).
As a result, the typical MDP properties do not hold in this setting.
The choice of an RL formulation is therefore a bit odd, but the sequential nature does make it a better fit than alternatives with which I am familiar.

This work seeks to also perform transfer. Transfer in RL is a non-trivial problem.
The assumption of "same or similar ground truth rules" substantially simplifies the problem. This assumption is akin to assuming similar optimal policies, at which point policy reuse (without adaptation) is expected to substantially improve early performance.
Benefiting from past knowledge when able while not hurting performance in other cases is difficult. This aspect is stripped away in the performed transfer experiments as transfer is known to be helpful.
Additionally, this assumption cannot be made in real-world applications of this method as ground truth rules cannot be known prior to applying methods.

Section 5.2 with human experts has experts looking at rules and determining if these rules seem reasonable.
This approach to evaluation is only considering one side: do doctors generally agree with the rules identified.
The other side is missing: do doctors disagree with the rules not selected by the system.
Since the system is supposed to distinguish between useful and non-useful rules, this categorization should be evaluated by doctors.
Ideally, this should be done close to the decision boundary, so comparisons should be made between the last included rules and the first ones not included (rather than comparing random rules to the top rules, for example).
Additionally, the goal of explanations is to provide trust in _performant_ systems.
Evaluation can be repeated with rules generated by a faulty model, and doctors should be able to distinguish between the properly functioning and faulty models based on the produced explanations.
clarity,_quality,_novelty_and_reproducibility: This work notes that enumeration is intractable for long temporal logic rules.
Tractability of their methods was successfully demonstrated.

Section 5.1 would have benefited from results for additional methods within the main body for comparison.
Likewise, the baselines mentioned in 5.2 are not used for earlier comparisons.

Minor comments:
- Several times, the authors use a phrase akin to "We make the temporal logic rule search subproblem differentiable." This is not entirely true. The RL policy is differentiable, but the search problem itself is not differentiable. Only the problem of finding logic rules _with this given policy_ is differentiable.
- Parenthetical and non-parenthetical citations are each sometimes used when the other would be more appropriate. Another editing pass for this would be helpful.
- 4.2.2: "wihch" -> "which"
- 5.2: "studies suggests" -> "studies suggest"

The results in Table 1 (OURMETHOD) are prior to doctor intervention, right?
What is the performance after the doctors have made modifications?
summary_of_the_review: The new method is motivated well and appears to perform better than existing alternatives.
Evaluation could be improved, though. For example:
For the real-world study, by only considering the useful rules found by a successful system, the usefulness of explanations is not evaluated.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper focuses on the problem of learning temporal logic rules for temporal point processes (TPP). The overall objective is to find a set of rules and their corresponding weights to maximize the likelihood of data.

The MLE problem is solved in two steps, a master step, in which the weights are optimized via convex optimization, and a substep, in which the logic rules are selected. The authors formulate the rule induction problem as an RL problem, where the policy outputs one rule token at each step and receives a reward when the sequence of tokens is completed. The authors then apply a risk-seeking policy gradient method to solve this problem.

Strengths: All the reviewers thought this was an interesting approach and the results seem promising. The writing is reasonably clear and there is sufficient introductory material to help readers, such as myself, who are not expert in TPP.  The paper includes empirical evaluations of the proposed method on synthetic and real datasets, such as medical diagnosis, and the results suggest that the proposed method achieves good likelihood and absolute error.I appreciate the author's engagement during the review process and their running of additional experiments.

Weaknesses: There were questions about competing approaches such as ILP (or neural ILP) techniques that might make reasonable baselines but were not discussed in the paper. The authors argue that they would not be performant but it is perhaps not entirely clear still. There were also doubts raised about the suitability of RL approaches for this problem, given that the underlying formulation relies on history and therefore is non-Markovian.  The authors responded to these concerns but the reviewers were not convinced that other approaches such as bandits might not be more appropriate. Finally, there was a question about how doctors would assess the learned rules (and those that were rejected). The authors mention that doctors found the results useful and agreed with the model's assessment but the reporting of this is anecdotal and perhaps a user study would be appropriate to make the case stronger.

Given the concerns and the current scores I can't recommend acceptance at this point. Clearly this is good work in a difficult setting. I hope the authors take the reviewer concerns into account in a rewrite and resubmit.
justification_for_why_not_higher_score: This was a tough one.  Two of the reviewers were at 6 and two at 3 and there wasn't any movement after engagement with the authors. This is not my area and I had a tough time adjudicating the differences so perhaps erred on the negative. If someone who knows more about TPPs would champion it I could revise upwards.
Decision: Reject