Reinforcement Logic Rule Learning for Temporal Point Processes 
Authors: Chao Yang
Lu Wang
Kun Gao
Shuang Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: temporal point processes
explainable models
rule learning

Abstract: We aim to learn a set of temporal logic rules to explain the occurrence of temporal events. Leveraging the temporal point process modeling and learning framework, the rule content and rule weights are jointly learned by maximizing the likelihood of the observed noisy event sequences. The proposed algorithm alternates between a master problem, where the rule weights are updated, and a subproblem, where a new rule is searched and included. The formulated master problem is convex and relatively easy to solve, whereas the subproblem requires searching the huge combinatorial rule predicate and relationship space. To tackle this challenge, we propose a neural search policy to learn to generate the new rule content as a sequence of actions. The policy parameters will be trained end-to-end using the reinforcement learning framework, where the reward signals can be efficiently queried by evaluating the subproblem objective. The trained policy can be used to generate new rules, and moreover, the well-trained policies can be directly transferred to other tasks to speed up the rule searching procedure in the new task. We evaluate our methods on both synthetic and real-world datasets, obtaining promising results.

Under review as a conference paper at ICLR 2023
REINFORCEMENT LOGIC RULE LEARNING FOR TEM- PORAL POINT PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
We aim to learn set of temporal logic rules to explain the occurrence of temporal events. Leveraging the temporal point process modeling and learning framework, the rule content and rule weights are jointly learned by maximizing the likelihood of the observed noisy event sequences. The proposed algorithm alternates between master problem, where the rule weights are updated, and a subproblem, where a new rule is searched and included. The formulated master problem convex and relatively easy solve, whereas the subproblem requires searching the huge combinatorial rule predicate and relationship space. To tackle this challenge, we propose neural search policy to learn generate the new rule content as a sequence of actions. The policy parameters will be trained end-to-end using the reinforcement learning framework, where the reward signals can be efficiently queried by evaluating the subproblem objective. The trained policy can be used to generate new rules, and moreover, the well-trained policies can be directly transferred to other tasks to speed up the rule searching procedure in the new task. We evaluate our methods on both synthetic and real-world datasets, obtaining promising results.
INTRODUCTION
Understanding the generating process of events with irregular timestamps has long been an interesting problem. Temporal point process (TPP) an elegant probabilistic model for modeling these irregular events in continuous time. Instead of discretizing the time horizons and converting the event data into time-series event counts. TPP models directly model the inter-event times as random variables and can be used to predict the time-to-event as well as the future event types. Recent advances in neural-based temporal point process models have exhibited superior ability event prediction (Du al. 2016 Mei & Eisner, 2017). However, the lack of interpretability of these black-box models hinders their applications in high-stakes systems like healthcare.
In healthcare, it is desirable to summarize medical knowledge clinical experiences about the disease phenotypes and therapies to a collection of logic rules. The discovered rules can contribute to the sharing of clinical experiences and aid to the improvement of the treatment strategy. They can also provide explanations to the occurrence of events. For example, the following clinical report
"A 50 years old patient, with chronic lung disease since 5 years ago, took the booster vaccine shot March 1st. The patient got exposed to the COVID-19 virus around May 12th, and afterward within week began have mild cough and nasal congestion. The patient received treatment as soon as the symptoms appeared. After intravenous infusions at healthcare facility for around 3 consecutive days, the patient recovered...
contains many clinical events with timestamps recorded. It sounds appealing to distill compact and human-read luman-reedabbe readable temporal logic rules from these noisy event data. In this paper, we propose an efficient reinforcemení temporal logic rule learning algorithm to automatically learn these rules from event sequences. See Fig. for a better illustration of the types of temporal logie rules we aim to discover, where the logic rules are in disjunctive normal form (i.c., OR-of-ANDs) with temporal ordering constraints.
Our proposed reinforcement rule learning algorithm builds upon the temporal logic point process (TLPP) models (Li et al. 2020), where the intensity functions (i.c., occurrence rate) of events are informed by temporal logic rules. TLPP intrinsically a probabilistic model that treats the temporal

---

Under review as a conference paper at ICLR 2023
logic rules as soft constraints. The learned model can tolerate the uncertainty and noisiness in events and can be directly used for future event prediction and explaination. Given this TLPP modeling framework, our reinforcement rule learning algorithm jointly learns the rule content (i.e., model structures) and rule weights (i.e., model parameters) by maximizing the likelihood of the observed events. The designed learning algorithm alternates between solving convex master problem, where the continuous rule weight parameters are easily optimized, and solving a more challenging subproblem, where a new candidate rule that has the potential to most improve the current likelihood is discovered via reinforcement learning. New rules are progressively discovered and included until by adding new rules the objective will not be improved.
Specifically, we formulate the rule discovery subproblem as reinforcement learning prob- lem, where a neural policy is learned to ef- ficiently navigate through the combinatorial search space to search for good explanatory temporal logic rule to add. The constructed neural policy emits distribution over the pre- specified logic predicate and temporal relation libraries, and generates the logic variables as ac- tions in a sequential way to form the rule content, The generated rules can be of various lengths. Once a temporal logic rule is generated, ter- minal reward signal can be efficiently queried by evaluating the current subproblem objective using the generated rule, which is computationally expedient, without the need to worry about the insufficient reward samples. The neural policy is gradually improved by a risk-sceking policy gradient to learn to generate rules to optimize the subproblem objective, which is rigorously formulated from the dual variables of the master problem the current likelihood.
X1
f
1
X
R.le-2((.)
Before
+ Nore
- Lqual
Figure 1: Example of temporal logic rules: fi Y t- 21 A A 1 Before x2) A(22 None 22)) (xa Before x4). fz Y + Is A 22 A Ws A 2s A (ar None 22) (I2 Equal xta) A (#s Before 21))
None means no temporal order constraints.
search for rule that has the potential to best improve
This proposed reinforcement logic rule learning algorithm has the following advantages: 1) We utilize differentiable policy gradient to solve the temporal logic rule search subproblem. All the policy parameters can be learned end-to-end via policy gradient using the subproblem objective as reward. 2) Domain knowledge or grammar constraints for the temporal logic rules can be easily incorporated by applying specific dynamic masks to the rule generative process at each time step. 3) The memories of how to search through the rule space have been encoded in the policy parameters, The well-train well-trained neural policies for each subproblem can directly transferred to similar rule learning tasks to speed up the computation in new tasks, where we don't 't need to learn rules from scratch.
Contributions Our main contributions have the following aspects:
We propose an efficient and differentiable reinforcement temporal logic rule learning algorithm, which can automatically discover temporal logic rules to predict and explain events. Our method will add flexibility and explainability to the temporal point process models and broaden their applications in scenarios where interpretability is important.
tii All the well- neural policies in solving each subproblem can be readily transferred to new tasks. This fits the continual learning concept well. The quality of the rule search policies can be continually improved across various tasks. For a new task, we can utilize the preceding tasks' memories even though we cannot get access to the old training data. We empirically evaluated the transferability of our neural policies and achieved promising results.
tii) Our discovered temporal logic rules are human- readablee Their scientific accuracy can be easily judged by human experts. The discovered rules may also trigger experts in thinking. In our paper, we considered real healthcare dataset and mined temporal logic rules from these clinical event data. We invited doctors to verify these rules and incorporated their feedback and modification into our experiments.
2 RELATED WORK
Temporal point process (TPP) models. TPP models can be characterized by the intensity function. The modeling framework boils down to the design of various intensity functions to add the model flexibility and interpretability (Mohler et al. 2011). Recent development in deep learning has
2

---

Under review as a conference paper at ICLR 2023
significantly enhanced the flexibility of TPP models. (Du et al. 2016) proposed neural point process model, named RMTPP, where the intensity function is modeled by a Recurrent Neural Network.
(Mei & Eisner (017) improved RMTPP by constructing continuous-time RNN. (Zuo et al. 2020) and (Zhang et al. 2020a) recently leveraged the self-attention mechanism to capture the long- ferm dependencies of events. Although flexible, these neural TPP models are black-box and are hard to
interpret. To add transparency, (Zhang et al. (020b) used Granger causality as a latent graph to explain point processes, and the structures are jointly learned via gradient descent. However, Granger causality is still limited to the mutual triggering patterns of events. Recently, (Li et al. (020) proposed an explainable Temporal Logic Point Process (TLPP), where the intensity function built on the basis of temporal logic rules. TLPP model enables one to perform symbolic reasoning for events however their rules are required to be pre-specified. A follow-up work (Li et al. 2022) designed a column generation type of temporal rule learning algorithm. However, their subproblems are solved by enumeration, which will be intractable for long temporal logic rules and their search memories cannot be reused for future tasks. By contrast, our proposed algorithm makes the subproblem differentiable and the trained neural policies can be reused and transferred.
Logic rule learning methods. Learning logic rules without temporal relation constraints has been studied from various perspectives. Recently, (Wang al., (017) tried to learn an explanatory binary classifier using the Bayesian framework. SÄTNet (Wang et 2019aa transformed rule mining into SDP-relaxed MaxSAT problem. Attention- -baed methods Yang & Song 2019) were also introduced. Neural-LP Yang et al. (017)) provided the first fully differentiable rule mining method based on TensorLog CChhe 2016), and (Wang et al. 2019b) extended Neural-LP to learn rules with numerical values via dynamic programming and cumulative sum operations. In addition, DRUM (Sadeghian cet al.
2019) connected learning rule confidence scores with low-rank tensor approximation. (Dash al. 2018 Wei et al.) 2019) introduced a column generation (i.e., branch and price) type of MIP algorithm to learn the logic rules. However, all these above-mentioned logic learning methods cannot be directly applied to event sequences with timestamps. By contrast, we designed differentiable algorithm to learn temporal logic rules from event sequences.
Learning model structures via reinforcement learning (RL). RL provided a promising approach to automatically finding the best-fittingg model structures, which inspired us to apply it to rule learning. For example, in AutoMI., the famous NAS (Zoph & Le (017) trained recurrent neural network by RI. to design the architectures of deep neural networks and has achieved comparable performance with the human-designed models. Similar ideas have been adopted to aid the design of the explainable machine learning models. For example, the RL algorithm has been successfully introduced to learn the causal graph to explain the data (Zhu al. 2020). Recently, the RL. algorithm has been used in symbolic regression (Petersen) [021; Landajuela| 2021), which aims to learn the set of compact mathematical expressions to explain the dynamics of complex dynamic systems. our paper, we customized the RL algorithm to learn the temporal logic rules to explain the event sequences.
3 BACKGROUND
3.1 TEMPORAL POINT PROCESSES
Given an event sequence Hi- = {ti, ft, < f} up to f, which yields counting process (N()), f 2 0}. the dynamics of the TPP can be characterized by conditional intensity function, denoted as X({(H)). By definition, we have A(t]Ht)dt = [[, t + di)) [H.], where N( dtl) denotes the number of points falling in an interval [t, + di]. By some simple proof (Rasmussen 2018), one can express the joint likelihood of the events Hr as (- A(TT(TL)))))))
f))
II X(t exp L.E:.
The TPP modeling boils down to the design of intensity functions and the model parameters can be learned by maximizing the likelihood. Recent neural-based event models start modeling Ao H.) as deep learning model, such as RNN, which greatly increases the model flexibility but the learned model is hard to interpret. In this paper, we consider TLPP, where the intensity is constructed by temporal logic rules.
3

---

Under review as a conference paper at ICLR 2023
3.2 TEMPORAL LOGIC RULES
Predicate Define set of entities C = {c1, (2, Cx) The predicate is defined as the property or relation of entities, which a logic function that defined over the set of entities, i.e., xx.)) C. x C... .....CC+) x {0, 1}. For example, Smokes(c) is the property predicate and Friend(c, c') is the relation predicate.
Temporal Logic Rule A first-order logic rule is a logical connectives of predicates, such as Ve, ve', Smokes(c') + Friend(c, c') A Smokes(c). A temporal dimension is added to the predicates. The temporal logic rule a logic rule with temporal ordering constraints. For example, (1) A Before(ti, (2).
f Ve, Covid(c, (a) < Symptomss Aypear(e, (2) )) A ExposedToo
For discrete events, we consider three types of temporal relations,
[efore(41, (2) 141 l2 < 0}, Afler(4,, (2) {41 l2 0}, Equal (2) = 114 (2)
We also treat the temporal relation as the temporal predicate, which is a boolean variable. Formally, consider the following general temporal logic rule (for simplicity, omit the entity index and assume all the predicates are defined for the same entity).
Y(y)
(tut
A
Ttur(tuttu" twr)
(2)
MEX,
property predicates
temporal relations
where Y (ty) is the head predicate evaluated at time ty. Ny is the set of predicates defined in rule f. Tuw' denotes the temporal relation of predicate and u' that can take any mutually exclusive relation from the set { fore, A fter, Equal, None). Note that N None indicates there is no temporal relation constraint between predicate and u'. ty. tus and ty the occurrence times associated with the predicates.
3.3 TEMPORAL LOGIC POINT PROCESS
The grounded temporal predicate src, f), such as Smokes(c, t), generates sequence of discrete events {t,, ..,,,,.... with the time that the predicate becomes (i.e., True) is recorded.
Logic-informed intensity function The main idea of TLPP Li et 2020) is to construct the intensity functions using temporal logic rules. TLPP considers complicated logical dependency patterns, which enables symbolic reasoning for temporal event sequences. Gven rule as Eq. (2). TLPP builds the intensity function conditional on history. Only the effective combinations of the historical events that makes the body condition True will be collected to reason about the intensity of the head predicate. We introduce a logic function 9)(+) to check the body conditions of f. gr(-) can also incorporate temporal decaying kernels to capture the decaying effect the evidence like Hawkes process. The logic- infor-informed feature Φ gathers the information from history, which is computed
(r(()= = - Σ
91 ((tu)uex,)
(3)
MEX, (t=)EH"
where Hd indicates the historical event specific to predicate up Suppose there is rule set F that can be used to reason about Y. For each f F, one can compute the features dy(t) as above. Assume that the rules are connected in disjunctive normal form (OR-of-ANDs). TLPP models the intensity of the head predicate {Y Ft) as a log-lineat function of the features,
(4) where w = [wy]rer 2 ( are the learnable weight parameters associated with each rule, and by is the learnable base intensity. All the model parameters can learned by maximizing the likelihood as defined in Eq. [T). Note that given this intensity model, the likelihood is convex with respect to wr
A(t He) = exp bo + Σ₁er Wy (
(Fahrmeir et al. 1994).
4 REINFORCEMENT TEMPORAL LOGIC RULE LEARNING
Our goal is to jointly learn the set of OR-of-ANDs temporal logic rules and their weights by MLE. Each rule has a general form (2). To discover each rule, the algorithm needs to navigate through the combinatorial space considering all the combinations of the property predicates and their temporal relations. Moreover, each rule can have various lengths and the computational complexity grows exponentially with the rule length. To tackle this challenging problem, we propose a RL search
4

---

Under review as a conference paper at ICLR 2023
Add rule to Restricted Master Problem
Rule Evaluator
Rule Generator (RL Policy)
Yes
Train LSTM using policy gradient
Master Problem
Restricted Master Problem
Generate rules via LSTM
Increased gain negative?
End (Rule set is found)
Figure 2: Overall Learning Framework: alternating process between rule generator and rule evaluator
algorithm discover new rules one-by-one. The overall learning framework is shown in Fig. Where the algorithm alternates between a master problem (rule evaluator) and subproblem (rule generator).
4.1 OVERALL LEARNING OBJECTIVE: A REGULARIZED MLE
We formulate the overall model learning problem as a regularized MLE problem, where the objective function is the log-likel with rule set complexity penalty, i.e.,
Original Problem: w bb arg min - ((w, bo) + 2(w)) s.t. Wy 2 0, fEF
(5)
w w.b.
where F the complete rule set, and ((w)) is a convex regularization function that has high value for "complex" rule sets. For example, we can formulate s?(w) = Ào Ejer cryy where cy is the rule length.
4.1.1 RESTRICTED MASTER PROBLEM: CONVEX OPTIMIZATION
The above original problem is hard to solve, due that the set of variables is exponentially large and can not be optimized simultaneously in a tractable way. We therefore start with a restricted master problem (RMP), where the search space is much smaller. For example, we can start with an empty rule set, denoted as Fo C F. Then we gradually expand this subset improve the results, this will produce a nested sequence of subsets Fo F C c... C Fx C ..... For each FR. k = 0, 0,1,..., the restricted master problem is formulated by replacing the complete rule set F with FR:
Restricted Master Problem: w(k) (ū.()) = arg min - (w, bn) + ((w)) s.l. wy 2 0, E (6)
Solving the RMP corresponds to the evaluation of the current candidate rules. All rules in the current set will be reweighed. The optimality of the current solution can be verified under the principle the complementary slackness for convex problems, which in fact leads the objective function of our subproblem. More proof can be found in the Appendix
4.1.2 SUBPROBLEM: COMBINATORIAL PROBLEM
subproblem is formulated to propose a new temporal logic rule, which can potentially improve the optimal value of the RMP most. Given the current solution w(ky (R)))) for the restricted master problem a subproblem formulated to minimize the increased gain:
de (w. bo) ()(w) Owy
Subproblem:
min Gy
Owy
where oy is a rule-informed feature. We aim to search for 17.y)55.(1) rule that the corresponding feature minimizes the above objective function. If the optimal subproblem value is negative, we will include the new rule the set. Otherwise, the optimal subproblem value is non-negative, we reach the optimality and we can stop the overall algorithm. However, this search procedure is also computationally expensive. It requires search rule structures have feature evaluation. In the following, we will discuss how to solve the subproblem more efficiently using reinforcement learning.
We want to emphasize here that, although the idea above as how to formulate the master and subproblems have been widely used in machine learning, including the gradient grafting algorithm for learning high-dimensional linear models (Perkins et al. (2003), column-genee generatiognnnnionn algorithm for solving mixed integer programming (Savelsbergh] 2002 20hel Nemhauser 2012 Lübbecke Desrosiers 2005) and large linear programming (Demiriz et al.] 2002), and for learning ordinal logic rules (Dash et al. 20188 Wei et al. 2019). our overall learning framework is gradient- baadd..t-baaed. For the master problem, solve convex optimization with continuous variables by gradient descent (or SGD). For
5

---

Under review as a conference paper at ICLR 2023
the subproblem, although it requires searching over the combinatorial space, convert the problem into learning a neural policy where the policy parameters can be learned by policy gradient.
4.2 SOLVING SUBPROBLEMS VIA REINFORCEMENT LEARNING
Head
Predicate
T(A.8
T(A.C
T(S.C
T(A
7LA D)
TTC,
1
Categorical
I|II
In
JIJJ
IInI
II|
Distribution
Library 1: Property Predicates
Library 2: Temporal Relations
Tg
T2
Figure 3: Illustration of generating a temporal logic rule using a neural policy (LSTM). The subproblem formulated Eq. 7 proposes criterion propose The subproblem itself is a minimization, which aims to attain the most negative increased gain. However, explicitly solving the subproblem requires enumerating all possible conjunctions of the input property predicates and all possible pairwise temporal relations among the selected predicates, which extremely computationally expensive. In this paper, instead of trying to enumerate all the conjunctions, we propose to learn neural policy by reinforcement learning to generate the best-explanatory rules.
temporal logic rule.
4.2.1 GENERATING RULES WITH RECURRENT NEURAL NETWORK
We leverage the fact that the temporal logic rules can be represented as a sequence of "tokens" subject to some unique structures. Using the pre-ordered traversal trick, we parameterize the policy by RNN or LSTM, combined with dynamic masks, to guarantee that the generated tokens can yield a valid temporal logic rule.
We generate the rules one token at time according the pre-order traversal, as demonstrated in Fig. the state, which is the embedding of the previously generated tokens. We model the policy as T (a)s) with learnable parameter 0, which quantifies the token selection probability given s. Each token/action can be chosen from the two predefined libraries: the property predicate libraries, and ii) the temporal relation libraries.
3 Specifically, denote
Given the head predicate, we generate the body predicates and their temporal relations in a sequential way. Every time property predicate is generated, we need to consider its temporal relation with all the previously generated property predicates. Note that the temporal relation token can be None, which means there no temporal relation constraints. All these generative prior knowledge can be incorporated as constraints by designing dynamic masks.
4.2.2 RISK-SBEKING POLICY GRADIENT
The standard policy gradient objective (() defined as an expectation. This the desired objective for control problems in which one seeks optimize the average performance of policy. However, rule learning problems described in our paper are to search for best-fitting rules. For such problems, (0) may not appropriate, there is mismatch between the objective being optimized and the final performance metric.
We consider risk-secking policy gradient like (Petersen 2021 Landajuela| 202
which proposed an alternative objective that aims to maximize the best-case performance. According to the original work (Landajuela 2021 Petersen 2021), we first define (.(O) ()) the (1 -)-qunnt of the distribution of rewards under the current policy. Then the new objective Jrisk (6; is given by:
(0;e) Er-x(rio) [R(T) R(T) > (O)
(8)
Then the risk risk-seeking policy gradient can be estimated using the roll-out samples, i.c.,
VoJrien(O;e) eN Σ
Σx
R
R( (0) (-)2 Vo log no (0))
(9)

---

Under review as a conference paper at ICLR 2023
Rules with different structures
Tree-Like Rules
Group-1: Case-4
Group-2: Case-4
Group-3: Case-4
Group-4: Case-4
Group-S: Case-4 Jaccard
Group-6:
Case-4
Jaccard
1.00
Jaccard
0.67
Jaccard
0.33
Jaccard
1.00
1.00
Jaccard
0.33
Learned
Learned
Learnd
Learned
Learned
Learned
3
2
2
3
2
Truth
Truth
Truth
Truth
Truth
Truth
Truth Leaned
Trum Learscd
Learned
Lcorred
Lcarred
Laaned
Value
Value
Volue
Figure 4: Rule discovery ability and rule weight learning accuracy of our proposed model based on Case-4 for all 6 groups. Blue one indicates ground truth rule and red/yellow one indicates learned rule in different groups.
where N is the number of episodes, and K is the length of tokens (actions). R is the empirical -uuuntie of the batch of rewards, and 1 returns if condition true and otherwise. We use this estimated policy gradient to update the policy 0.
(10) where a is the learning rate. Further, according to the maximum entropy reinforcement learning framework (Haarnoja et al.),, bonus can be added to the loss function proportional to the entropy to help the policy do the exploration.
0 - 0 av-aaal a (0;c)
5
EXPERIMENTS
5.1 SYNTHETIC DATA
We prepared groups of synthetic event data, each group with different set of ground truth rules. We considered the following baselines: TELLER (Li et al. 2022). policy gradient without risk seeking,
and brute-force method (cnumerating all possible rules).
Accuracy and Scalability For each group, further considered cases, with the to-be-searched property predicate library being sized 8, 12, 16, 20, and 24, respectively. Note that only small amount of the predicates will be in the true rules. Many of the predicates are redundant information and they will act as background predicates. We aim to test: 1) whether our reinforcement temporal logic learning algorithm can truly uncover the rules from the noisy variable set, 2) how accurate can the rule weights be estimated, 3) and how the performance will evolve if we gradually increase the variable set with more and more redundant variables.
The ground truth rules of different groups are with different length and various content structures. For some groups, each rule shares many common predicates content, while for some groups, the designed rules are quite distinct in content. For example, the ground truth rules in group-(1,2) 2. are quite different in their content, and the ground truth rules in group- (4, 5, 6} share many common predicates. On the other hand, in group- (1, 4) and group- 5). the number of property predicates in one ground truth rule is and 7 respectively. We set the number of property predicates of ground truth rules to be group-(3. 6) to craft relatively long rules, especially considering intricate temporal relation at the same time. When uncovering ground truth rules in every group, we fix 2 predicates as prior knowledge and ask the algorithm to complete the temporal logic rules. Complete results for all the datasets can be found in Appendix !
In Fig. 4 we reported the learning results for the case with predicate library size 20 for all groups (with different rule content). We used 1000 samples of event sequences training data. Each plot in the top row uses a Venn diagram to show the true rule set and the learned rule set, from which the Jaccard similarity score (area of the intersection divided by the area of their union) is calculated. Our proposed model discovered almost all the ground truth rules. Each plot in the bottom compares the true rule weights with the learned rule weights, with the Mean Absolute Error (MAE) reported. Almost all the truth rule weights are accurately learned and the MAEs are quite low. In group-3 and group-6, crafted long and complex rules with 8 body property predicates and various temporal
7

---

Under review as a conference paper at ICLR 2023
relations, which yield extremely huge search space, but our model still discovered almost all the ground truth rules.
Fig. [ illustrates the Jaccard similarity score and MAE for all cases in all 6 groups using 1000 samples. For all 6 groups, as the number of predicates in the predicate set increases, the Jaccard similarity scores decrease slightly and the MAE increases slightly, but it is still within acceptable range. This is because as the number of redundant predicates increases, the search space expands exponentially and the complexity of searching is dramatically increased. But if the number of predicates in the predicate set is appropriate and the samples are sufficient, our model is very stable
and reliable.
Group-1
Group-2
Group-3
Group-4
Group-5
Group-6
0.
0.4
0.8
08
0.
0.8-
6A
accar)
Broand
0.4
0.
0.4
64
0.2
0.2
6.2
62
0.3
12
16
20
12
16
20
24
12
16
20
24
0.
12
20
24
Figure 5: Jaccard similarity score and MAE for all 6 groups. X-axis indicates the predicate library size and Y-axis indicates the value of Jaccard similarity and MAE
Computational Efficiency As shown in Fig (left), we displayed the curve of the log- oogliik function versus the running time for our method and baselines. Our method uses the risk-seeking policy gradient in solving subproblems (refer to the flat-line period in figure). As a comparison, vanilla (normal) policy gradient still optimize the expectation of the reward to solve the subproblem. For TELLER, it uses enumerative search to generate new rules in subproblems although adopts the depth-first type of heuristic to try to append predicates to important short rules to generate long rules. As for the brute-force method, first considers all the one-body-prede likelihood by learning the rule importance, then adds all the boed-prediccate rules to optimize the rules to the model, and so on. From the results, we see that our method uncovers the ground truth rules faster and more accurately compared with all the baselines in the long-run. It almost intractable for the brute-force method. Compared with the normal policy gradient method, the performance and accuracy of our model are also better, mainly because by using risk-seeking policy gradient, the model can focus learning only on maximizing best-case performance. We did more experiments to compare the normal policy gradient and risk- seeking policy gradient. Please refer to Appendix foror more details. Log-Likellhood Trajectory
Log-Likellhood Trajectory
-1.00-
-1.25
-1:75
-2.5
PG Risk Seeking
PG Normal
22.00
i-hwined
Log
Snute
2,25
Transfer 02
Transfer
Transfer 03
-2.50
Transfer
Transfer
2.75
Same Rules 01
Same Rulls 02
33,00
Same Ruless
20000 40000
60000
a0000
320000
5000
10000 15000
20000
Time(s)
Timels)
Figure 6: Log Likelihood Trajectory. Left: Compare our model (risk secking policy gradient) with several baselines (normal policy gradient, TELLER, Brute force way). Right: Transfer experiments. First completely train collection of rule generators and save the model parameters, then use the well-trained model to uncover new rules on different datasets. Transfer ## indicates the new datasets with ground truth rules that are slightly different compared with the datasets used by the well-trained model. Same Rule # # indicates the new datasets generated by same ground truth rules.
Transferability Our well-trained neural search policies also transferable. As shown in Fig. 6 (right), once we get a collection of train other different datasets which generated by the same similar ground truth rules. The
policies in solving each subproblem, we can use it to
8

---

Under review as a conference paper at ICLR 2023
results showed that this will speed up the process of solving subproblems and improve the efficiency and accuracy of uncovering ground truth rules in these datasets.
5.2
HEALTHCARE DATASET
MIMIC-III is an electronic health record dataset of patients admitted to the intensive care unit (ICU) (Johnson et al. 2016). We considered patients diagnosed with sepsis (Saria 2018 Raghu et al.) 2017 Peng et al. 2018), since sepsis is one of the major causes of mortality in ICU. Previous studies suggest that the optimal treatment strategy still unclear. It unknown what the optimal treatment strategy in terms of using intravenous fluids and vasopressors to support the circulatory system. There also exists clinical controversy about when and how use these two groups of drugs to reduce the side effect for the patients. For this real problem, we implemented our proposed reinforcement temporal logic rule learning algorithm to learn explanatory rules and their weights and gain insight into this problem.
Discovered temporal logic rules In Appendix C we reported all the uncovered temporal logic rules and their weights learned by our algorithm. We use LowUring ine and rma Urrine as the head predicates respectively. We also invited human experts (doctors) to check the correctness of these discovered rules and the doctors think most of these rules have clinical meaning and are consistent with the pathogenesis of sepsis. Doctors' modifications and suggestions for these algorithm- discovered rules are also provided. Experts think that Rule 1-9 capture the major lab measures, like low systolic blood pressure, high blood urea nitrogen and low central venous pressure (appeared in Rule 2), that usually emerge before extremely low urine. Rule 10-18 shed light on drug and treatment selection. For example, reflected in Rule 12, Crystalloid and Dobutamine together yields a weight of 0.4535 for patient's 's normal urine.
Compared with baselines in event prediction We considered the following SOTA baselines: 1) Recurrent Marked Temporal Point Processes (RMTPP) (Du et al.; 2016), the first neural point process (NPP) model, where the intensity function is modeled by a Recurrent Neural Network (RNN); Neural Hawkes Process (NHP) (Mei & Eisner 2017). an improved variant of RMTPP by constructing a continuous-time LSTM; 3) THP. Transformer Hawkes Process (Zuo et al. 2020). which leverages the self-attention mechanism to capture long-term term dependencies and meanwhile enjoys computational efficiency; 3) Tree- Pree-Regularized GRU (TR-GRU) (Wu et al., 2018). deep time-series model with designed tree regularizer to add model interpretability; 5) Hexp (Lewis) & Mohler 201T). Hawkes Process with an exponential kernel; 6) Transformer (Vaswani et al. 2017). an advanced model which follows an encoder-decoder structure, but does not rely on recurrence and convolutions in order to generate output. 7) TELLER (Li et al. 2022). al- ternating between master problem and subproblem (enumerative search) to learn logic rules; 8) PG-normal, neural search policy learned by normal policy gradient, without risk seeking.
Table 1: Event prediction results.
We used mean absolute error (MAE) as the evaluation metrics for the event prediction tasks of the two head predicates: LowUrine and 2) No rmalUrine, which are evaluated by predicting the time when these events will occur. Lower MAE (unit is hour in this example) indicates better performance of model. The performance of our model and all baselines are compared in Tab. I from which one can observe that our model outperforms all the basclines in this experiment.
LowUrine (MAE)
NormalUrine
Method
(MAE)
RMTPP
1.989
2.170
NHP
1.784
1.977
THP
1.658
1.759
TR-GRU
1.627
1.692
Hexp
2.578
2.433
Transformer
1.704
1.872
TELLER
1.887
1,532
PG-nommal
2.050
1.647
OURMETHOD
1.622
1.305
6 CONCLUSION
In this paper, we proposed a reinforcement temporal logic rule learning algorithm to jointly learn temporal logic rules and their weights from noisy event data. The proposed learning algorithm alternates between a rule generator stage and a rule evaluator stage, where a neural search policy is learned by risk-seeking gradient descent to discover new rules in rule generator stage. The use of the neural policy makes the subproblem differentiable, and the well-trained policies can be easily transferred to other tasks. We empirically evaluated our method on both synthetic and healthcare datasets, obtaining promising results.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Frederick Callaway, Antonio Rangel, and Thomas L Griffiths. Fixation patterns in simple choice reflect optimal information sampling. PLoS computational biology, 17(3)e 1008863, 2021.
William W Cohen. Tensorlog: A differentiable deductive database. arXiv preprint
2016.
Sanjeeb Dash, Oktay Gunluk, and Dennis Wei. Boolean decision rules via column generation. Advances in neural information processing systems, 31, 2018.
Ayhan Demiriz, Kristin P Bennett, and John Shawe-Taylo Linear programming boosting via column generation. Machine Learning, 46(1):225-254, 2002.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez and Le Song. Recurrent marked temporal point processes: Embedding event history vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1555-1564, 2016.
Ludwig Fahrmeir, Gerhard Tutz, Wolfgang Hennevogl, and Eliane Salem. Multivariate statistical modelling based on generalized linear models, volume 425. Springer, 1994.
Tuomas Haarnoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with stochastic actor.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Kim Santiago Glatt Nathan Mundhenk Petttit Faissol Landajuela, Petersen. Discovering symbolic policies with deep reinforcement learning. pp. 5979-5989, 2021.
Erik Lewis and George Mohler. A nonparametric em algorithm for multiscale hawkes processes. Journal of Nonparametric Statistics, 1(1):1-20, 2011.
Shuang Li, Lu Wang, Ruizhi Zhang, Xiaofu Chang, Xuqin Liu, Yao Xie, Yuan Qi, and Le Song. Temporal logic point processes. In International Conference on Machine Learning, pp. 5990-6000. PMLR, 2020.
Shuang Li, Mingquan Feng, Lu Wang, Abdelmajid Essofi, Yufeng Cao, Junchi Yan, and Le Song. Ex- plaining point processes by learning interpretable temporal logic rules. International Conference on Learning Representations, 2022.
Marco E Lübbecke and Jacques Desrosiers. Selected topics in column generation. Operations research, 1306)13007-1023, 2005.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self- doff-modulaiing multivariate point process. Advances in neural information processing systems, 30, 2017.
George O Mohler, Martin B Short, Jeffrey Brantingham, Frederic Paik Schoenberg, and George Tita. Self-exciting point process modeling of crime. Journal of the American Statistical Association, 106(4933::00-108, 2011.
George L Nemhauser. Column generation for linear and integer programming. Optimization Stories, 20(65):U73, 2012.
Xuefeng Peng, Yi Ding, David Wihl, Omer Gottesman, Matthieu Komorowski, Li-wei H Lehman, Andrew Ross, Aldo Faisal, and Finale Doshi-Velez. Improving sepsis treatment strategies by com- bining deep and kernel-based reinforcement learning. AMIA Annual Symposium Proceedings, volume 2018, pp. 887. American Medical Informatics Association, 2018.
Simon Perkins, Kevin Lacker, and James Theiler. Grafting: Fast, incremental feature selection by gradient descent in function space. The Journal of Machine Learning Research, 111333-1356 2003.
10

---

Under review as a conference paper at ICLR 2023
Nathan Mundhenk Santiago KiKim T.Kim Petersen, Larma. Deep symbolic regression: recovering mathematical expressions from data via risk seeking policy gradients. 2021.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzych Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:171 1.09602. 2017.
Jakob Gulddahl Rasmussen. Lecture notes: Temporal point processes and the conditional intensity function. arXiv preprint arXiv: 00221, 2018.
Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to-end differentiable rule mining on knowledge graphs. arXiv preprint arXiv:1911.000 2019.
Suchi Saria. Individualized sepsis treatment using reinforcement learning. Nature medicine, 24(11): 1641-1642, 2018.
Martin WP Savelsbergh. franch-and-priee Integer programming with column generation, bp. 2002.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using differentiable satisfiability solver. In International Conference on Machine Learning, pp. 6545-6554. PMLR, 2019a.
Po-Wei Wang, Daria Stepanova, Csaba Domokos, and Zico Kolter. Differentiable learning of numerical rules in knowledge graphs. In International Conference on Learning Representations, 2019b.
Tong Wang, Cynthia Rudin, Finale Doshi Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. A bayesian framework for learning rule sets for interpretable classification. The Journal of Machine Learning Research, 18(1) 1811)2357-2333, 2017.
Dennis Wei, Sanjeeb Dash, Tian Gao, and Oktay Gunluk. Generalized linear rule models. In International Conference on Machine Learning, pp. 6687-6696. PMLR, 2019.
Mike Wu, Michael Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi- Beyond sparsity: Tree regularization of deep models for interpretabilityy In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning, arXiv preprint arXiv: 2017.
Yuan Yang and Le Song. Learn explain efficiently via neural logic inductive learning, In International Conference Learning Representations, 2019,
Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive hawkes process. In International conference on machine learning, pp. 11183-11193. PMLR, 2020a.
Wei Zhang, Thomas Panum, Somesh Jha, Prasad Chalasani, and David Page. Cause: Learning granger causality from event sequences using attribution methods. In International Conference on Machine Learning, pp. 1 11235-11 1335-11245... PMLR, 2020b.
Shengyu Zhu, Ignavier Ng, and Zhitang Chen. Causal discovery with reinforcement learning. 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. International Conference on Machine Learning, 2017.
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In International conference machine learning, pp. 11692-1 1170. PMLR, 2020.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
In the following, we will provide supplementary materials to better illustrate our methods and experiments.
presents different ground truth rule structures and some experiment results on synthetic data.
presents all learned rules MIMIC-III data and expert identification and revision suggestion.
Section provides the definitions of all types of temporal relations considered in our model.
Felaborate on the necessary proofs, which justify our model and learning framework.
Section and
Section G provides pseudocode to illustrate our rule generator policy, subproblem formula- tion, and the complete algorithm.
Section provides computing infrastructure.
Section anddKKssoww the comprehensive experiments on synthetic data. In|K we compared the performance between standard policy gradient and risk seeking policy gradient.
Section D introduces the background the MIMIC-III data and provides list of the chosen predicates used in our real experiment. Section M presents more experiment results in terms of the modified rules by experts.
Dectionn[intrrduceee another application using our proposed method, which is an experiment about understanding shoppers' purchase patterns given their eye fixation event data.
B ALL LEARNED RULES ON SYNTHETIC DATA
We did experiments on 6 groups of event data, each group with different set of ground truth rules. For each group. we further considered cases. To illustrate different rule structures in different structure, we listed all the discovered rules for Case-2 of all groups in Tab. 2 to limited space, we did not show all learned rule content for all cases of all groups. Please reter | for the complete results of experiments synthetic data.
Tab. 3 and Tab. Due
Table 2: Results of Case-2 for Group-1, Group-4. *Ground truth rules which are learned. truth rules which are not learned. * Rules that are wrongly learned.
Ground
Group-1 Case-2
Group-4 Case-2
* Y + AAB C AAAA
Y t AABACAAAAAA AAAA
A A Before A B Before
Before
Before
A C Before D ADD Before
A.C. Before D A Before E
A Before
AE Before
# Y + CA-AABAF ADAEAG
*Y + AABACO CAACAAAES
AC Before A B Before F
A A Before B A Before C
AF Before DAD Before
A. Before DAD Before
AE Before G
AE Before G
#Y C.CAAAAAAAAAAA
* AAAAAAА
AC Before BAB B Before G
AA Before В A B Before C AC Before D DAD Before E
AG Before D AAD Before
AE Before
AE Before H
AD Before A Before
AC Before A A A Equal E
AE Equal
12

---

Under review as a conference paper at ICLR 2023
Table 3: Results of Case-2 for Group-2 and Group-5. *Ground truth rules which learned.
Ground truth rules which are not learned. Rules that are wrongly learned.
Group-2 Case-2
Group-5 Case-2
Y c- AABACCAAI A AEAFAG
Y Y A AABACAA GBAAAAARAFA
A A Before B BAB Before с
A Before BAB Before C
A C Before D DAD Before
AC Before D DAD Before E
AE Before F FA Equal
AE Before FAF Equal G
* C
DAEA GAH
* TY+AABE CADAAAAR
AC Before B A B Before F
A Before B A B Before C
Before DAD Before
AC Before DAD Before
A Before GAC G Equal H
AE Before FAF Equal H
# CAB A AADAAAA AAG
+ A A C A DADAD FAAFF
AC Before A B Before
AA Before B B Before C
A A Before D AAD D Before
AC Before DA D Before E
AE E Before НА H Equal
A Before FAF Before H
Table 4: Results of Case-2 for Group-2 and Group-5. *Ground truth rules which are learned.
Ground truth rules which are learned. Rules that are wrongly learned.
Group-3 Case-2
Group- Case-2
+
A.-AAAACC A #Y=AABACADEEAEAGM
+ T
AA Before B BAB B Before C A Before B A Before C
AC Before D A D Before E
AC Before D DAD Before E
AE Before F F Equal G
E Before FAF Equal G
AG Equal H
A G Equal H
*
*Y CACAAAA BABBFFADAER AAAAAAGAНA
* * A AADAAAAAGAE
AC Before в BAB Before F
A Before B A B Before C
AF Before DAD Before E
AC Before
Before
AE Before GAG G Equal H
AE Before F FAF Equal G
АH Equal A
AG Before
# CA A AABAAAAAAHHAAAA
* + AA CAB.AAAAAAAAAA.C.......AAAAA
AC Before в A B Before A
A Before В A B Before C
A Refore AD Before E
AC Before DAD Refore
AE Before Н AA Equal G
A E Before FAF Equal H
AG Equal
Before
# #Y
AETAACAAL
AA Before C A Before H
AH Before G Before B
AB Before D DAD Before E
Before
C ALL LEARNED RULES ON MIMIC-III DATA
The complete sets of learned rule on MIMIC-III data are shown in Tab. and Tab. 6 We invite human experts to check our learned rules and provide several revision suggestion. Rules shown in these tables have been identified by experts as clinically meaningful and consistent with sepsis pathology. Some of the rules are directly learned by our method without any modification (marked with blue asterisk). And some are basically consistent with clinical facts, but some contents of the rules are not pathological. These rules have been slightly modified by experts (marked with a red asterisk).
13

---

Under review as a conference paper at ICLR 2023
Table 5: Part of learned rules with LowUri the head predicate. *Rules that are clinically meaningful confirmed by experts. *Rules that are modified by experts. Weight
Rule
"Rle 1: Lowe + LowSysBP A HighBUN A LowArterialph
0.0015
(LowSysBP Equal HighBUN)
A (Hi [HigN Equal ArterialpH)
Rule 2: Low + LowSysBP A HighBUN A LowCVP
0.1846
(LowSysBP Before HighBUN)
(HighBUN Equal LowCVP)
Rule Low LowSodium LowChloride Highheetiiiiii HighCreatinine
0.1175
A (LowSodium Equal LowChloride)
(LowChloride Reual HighCreatinine)
Rule 4: Low LowSodium Lowchlorrdeo LOwSVR
0.8226
(LowSodium Equal LowChloride)
(LowChloride Equal LOWSVR)
Rule 5: tow
+ LowSodium A LowChloride HighBUN
0.3694
(LowSodium un Before LowChloride)
(LowChloride de Before HighBUN)
Rule 6: Low +
HighSVR A LowBUN A
LowArterialpH
0.0727
A (Highhrteerllllcctttee Equal HighSVR)
(HighSVR Equal LOWBUN)
A (LowBUN Equal LowArterialpt)
*ule Low Aitia Higghs HighsVR A
Hi SgghPttsssiumAAooeeaaaaa
0.5554
tnighhrreeriallactttee Equal HighSVR)
(HighSVR Equal HighPotassium)
(HighPotassium Equal LOwCVP)
*Rle 8: hoooommmne-iiigttttriaaaatttttt HighSVR
HighHCo3 A LowScdium
0.0546
(#ignArteeiallLateetttttt
Equal HighSVR)
A (HighSVR Equal HighHC03) A (HighHC03 Equal LowSodium)
Rule 9:
+ Miw.hmes--LooyysEPEaAaaaaaaalaat
HighPotassium/ Highhhoooieeie HighhUN
A (LowSysBP Equal HighArterial
0.4935
(Higharteriallattaaa Equal HighPotassium)
A (HighPotassium Equal HighChloride)
(HighChloride Equal HighBUN)
D DETAILED EXPLANATION TEMPORAL RELATION
In this paper, the temporal relation defined among events. For any pairwise events, denoted A and B, there exist only three types of temporal relations, which can be grounded by their occurrence times, denoted as t and tg. See below Table 7 for illustrations.
The temporal relation of any two events will be treated as temporal ordering constraints and can be included in a temporal logic rule Eq. (4). Note that when included in the rule, the temporal relation can be none, which indicates that there is no temporal relation constraint between the two events in order to satisfy the rule.
E PROOF OF THE LIKELIHOOD FUNCTION OF TLPP
The likelihood function of the TLPP is straightforward result from TPP. Readers can refer to the proof's in (Rasmussen) 2018). To be self-contained, we will provide a sketch of proof here.
For specific entity given all the events associated with the head predicate (ti, .....) [0. t). the likelihood function is the joint density function of these events. Using the chain rule, the joint likelihood can be factorized into the conditional densities of each points given all points before it.
14

---

Under review as a conference paper at ICLR 2023
Table 6: All learned rules with Norma Unmiaurrima the head predicate. *Rules that clinically meaningful confirmed by experts. *Rules that are modified by experts.
Weight
Rule
*Rule 10:NormalUrine < LowUrine A Crystalloic A Phenylephhiee (LowUrine Equal Crystalloid)
0.2231
A (Crystal loid Equal Phenylephrine)
"Rule 11:NormalUrine + LowUrine A Phenylephrine A Dopamine
0.0773
(LowUrine Before Phenylephrine)
(Phenylephrine Before Dopamine)
*Rule 12:NomalUrrie + LowUrine Lowwrrin Crystalloid i Dobutamine
0.4535
A (LowUrine Equal Crystalloid)
(Crystalloi) Equal Dobutamine)
*Rule 13tNormlUrinee f- LowUrine P Phenylephrine A Norepinephrine
0.2113
(LowUrine Equal Phenylephrine)
A (Phenylephrine Equal Norepinephrine)
*Rule 14:NomalUrine <- LowUrine A Norepinephrine A Dopamine A
he
NormalArteriale:
0.5459
(LowUrine Equal Norepinephrine)
(Norepirephring Equal Dopamines
A (Dopamine Equal NozmalArteriaaBB)
"Rule 15:NormalUrine + LowUriie A ine A Norma 1RBCcount
NormalBUN (LowUrine Equal Norepinephrine)
0.3926
A (Norepinephrine Equal NormalRBCcount)
(NormalRECcount Equal NormalBUN)
*Rule I6:NormaUUrine <- L A Norma AAtterialp
Dobutami ne (LowUrine Equal Colloid)
0.6430
(Colleid Equal NormalArteriiaptt]
(NormalAAteeriaappt Equal Dobutamine)
* "Ruee 17:NormalUrine < LowUrine Norepinephrine L Dobutamine
A NormalSysB? A (LowUrine Equal Norepinephrine)
0.3464
(Norepinephrine Equal Dobutamine)
A (Dobutamine Equal NormalSysBP)
*Rule 18:Normalurine < LowUrine A Phenylephrine NormalSysB?
Dopamine A NormalcvP
(LowUrine Equal Phenylephrine)
0.5669
(Phenylephrine Equal NormalSysBP)
A (Normal8ys3P Equal Dopamine)
(Dopamine Equal NormalCVP)
Table 7: Event-based temporal relations. Mathematical Expression
Temporal Relation
Illustration
Before
t < tB
t
After
fA > tB
t
t
A Equals B
ta tB
t
tB
For entity c, this yields:
L of (1 Ho) rf (15 Hey) ...pp (1a Hi: ) F (t Hig))
(11)
where (t Ho,,) represents the conditional density and F H...) refers to its cumulative
distribution function for any fn. (1- - Fe (t H4,)) appears the likelihood since the unobserved point 1 hasn't 't happened up to t. Further, by the hazard rate definition of the intensity function
p² (t H.r)
x²(t) 1 F (E H...)
(12)
15

---

Under review as a conference paper at ICLR 2023
will have
p (t (x)))exp exp (- (-fr- X((S)ds)
(13)
Using the above equation,
get
X²t
c =
( Heg--1)
p=(1 He,))
II (1))) exp (-Je- (((r)dr x=
exp (-Jr x((r)dr)
(14)
II
x(()åT)
exp
Now consider the likelihood function of all entities C according to the entities, the likelihood can be written as
{cl, C2, Cnt. which can be factorized
II II X((t))He,) . exp (((((((H))))))
Likelihood:
(15)
IEEM,
which completes the proof.
F OPTIMALITY CONDITION AND COMPLEMENTARY SLACKNESS
We will provide more descriptions the optimality condition and the complementary slackness, which provides sound guarantee to our learning algorithm.
Given the original restricted convex problem,
Orignial Problem: w", w",bi bú = arg min -((w, 60) + S(w)); s.t. wy 20, fEF (16)
w.Sbg
where ((w) is convex regularization function that has high value for "complex" rule sets. For example, we can formulate (?())= Ao ΣFEF cywy where cs the rule length.
The Lagrangian of the original master problem is L(w, byy) = (w. bo) + +((w)
Σ Dyws,
(17)
JEF
where Vy 2 0 is the Lagrange multiplier associated with the non-negativity constraints of wy. As is a convex problem and strong duality holds under mild conditions. Define w", bg as the primal optimal, and v" as the dual optimal, then: -I(w". b))
inf L(w, v²)
(strong duality)
wroo
inf w.bo
(E(w,bo) +2((w)
Σugus IEF
(18)
< - (w", .b6) + +[(w")
Σ Vijug
JEF
< - (w", .b)) + +(ww)).
Therefore, Es = 0. 0, for f E F. This implies the complementary slackness, i.e..
wi + 1/ 2 0, wi + 1/ =
(19)
Given the
(KKT) conditions, the gradient of Lagrangian L(w", bbg v*) w.r.t.
w², by vanishes, i.e.,
a [[(ww, bb) - R(w w)]
vj =
(20)
Owy
" bb
In summary, combining conditions 19 and 20 we obtain the optimalitiy condition of the original problem,
1. if w/ > O, then 1 = 0;
2. if w = O, then 3 2 O,
where the gradient 1/ can be computed via 20 At each iteration, we solve the subproblem to find the candidate rule that most violates this optimaliyy condition, i.e., yields the most negative Eq. (20)
16

---

Under review as a conference paper at ICLR 2023
G ALGORITHM BOX
Our method alternates between solving a restricted master problem and a subproblem. When executing the subproblem, we need generate several candidate rules. We summarize the algorithm in Algorithm [| Algorithm[2 and Algorithm 3 RG refers to the Rule Generator used to generate new candidate rule when solving the subproblem. Here, we will parameterize the RG LSTM. SP is the abbreviation of Sub-Probue Lub-Problem which is optimized to construct new rule. RMP indicates the Restricted Master Problem used to update model parameters.
Algorithm 1: Rule Generator (RG)
Input: RuleLen, HeadPred
Output: CandidateRule
PredLibrary B....... J:
TempRelationLibrrayy- [Before, After, Equal, N None];
Sequence + EmptySet; BodyPredSet +- EmptySet;
DynamicMaskk All-Ones
10
while BodyPredNum RuleLen do NewBodyPred LSTM(Sequence, PredLibrary DynamicMask)
DynamicMask <- Set the location of NewBodyPred zero (DynamicMask) Sequencce.addNeewBBBl
12
13
14
15
for BodyPred in BodyPredSer do // We need to consider the pairwise temporal relation between the new selected body predicate and all the body predi cates that already have been selected in the body predicate set.
T + LSTM(BodyPred NewBodyPred, TempRelationLibraary) Sequenee.addTeemRRlaaiin
17
18
19
20
CandicateRule Convertor(Sequuncee): // Convert sequence of tokens to rule template.
23 return CandicateRule
17

---

Under review as a conference paper at ICLR 2023
Algorithm 2: Sub-Problem (SP)
Input: RuleLen, HeadPred
Output: NewRule
while tter < 'Total_Ite lter do CandidateRuleBatch +- EmptySet
while Batch sirch_idx=5 & BatchSize do
RandidateRule +- RG(RuleLen, HeadPred);; // Generate candidate rule.
Policy <- Policy.update// Policy gradient.
10
if PolicyGradientNorm & threshold then
12
FinalCandidateRull: + RG(leee,,n, HcadPred);
return FinalCandidateRulle
FinaCCandidateRule RG(RuleLen, HeadPred); return FinulCandilateRule
15
Algorithm 3: Complete model
Input: RuleLen, HeadPred, TotalRuleNum
Output: ruleSet
ruleSet <- EmptySet;
0: b, w + RMP(b, w. ruleSet); // Initialize weights and base terms.
while CurrentRuleNums TotalRuleNum do NewRule +- SP(RuleLen, HeadPred); // Cenerate candidate rule.
if IncreasedGain<0 then muleSet.add(Newwulee))
11
12
RMP(b, w, ruleSet); // After adding new rule, update weights and base terms.
13 return b, w, ruleSer
H COMPUTING INFRASTRUCTURE.
All experiments performed Ubuntu 20.04 LTS system with Intel(R) Xeon(R) CPU ES-2690 v3 @ 2.60GHz CPU, 112 Gigabyte memory and single NVIDIA Tesla P100 accelerator.
1 COMPLETE RESULTS OF EXPERIMENTS ON SYNTHETIC DATA
Fig. and Fig. g demonstrate the learning results for the cases with predicate size 8, 12, 16, 20, and 24 (predicates the library that we need to search) for all groups using 1000 samples of networked events. We set the learning rate in solving the subproblem to be x 10 Hidden state size of LSTM was 32. The learning rate in solving the restricted master problem was -0- 3, Our proposed model discovered almost all the ground truth rules for all cases in all groups. And almost all the truth rule weights are accurately learned and the MAEs are quite low. For cases in group-3 and group-6, we considered long and complex rules with 8 body property predicates and the associated temporal relations, which yield a very big search space, but our model still discovered most of the ground truth rules.
18

---

Under review as a conference paper at ICLR 2023
We also evaluated our model using 500 and 2000 samples with the same experiment settings. Our model also achieved satisfactory performance when sample size is small, and the performance the model was further improved as the sample size increases.
Case-1: 8
Case-2: 12
Case-3: 16
Case-4: 20
Case-5: 24
Learned
Leamed
Learned
Learned
Group-1
3
3
3
Truth
Truth
Truth
Truth
Truth
Rules with
Learned
Learned
Learned
Learned
Learned
different
Group-2
2
Truth Truth
Truth
Truth
Truth
structures
Learned
Learned
Learned
Learned
Learned
Group-3
3
2
2
Truth
Truth
Truth
Learned
Learned
Learned
Learned
Group-4
3
Truth Truth Truth
Truth
Truth
Tree-Like
Learned
Learned
Learned
Learned
Learned
Group-5
3
2
Rules
Truth
Truth
Truth
Truth
Truth
Learned
Learned
Learned
Learned
Learned
Group-6
3
2
2
Truth
Truth
Truth
Truth
Truth
Figure 7: Jaccard similarity score for all 6 groups and all cases using 1000 samples. Blue one indicates the ground truth rule and red/yellow indicates the learned rule different groups.
REUSE THE LSTM MEMORY AND EARLY STOP MECHANISM
When to reuse the LSTM memory for different subproblem iterations? The ground truth rules group-(1 1, 2, 3) are quite distinct in their content with almost common predicates in each rule. Given this prior knowledge, when training the datasets in these groups, whenever the algorithm enters the subproblem to search for new candidate rule, we reset our LSTM model and clean the memory. If not, we will get the convergence results as illustrated in Fig. 9 where we observe that keeping the L.STM memory will hinder the convergence speed of the subproblems. As illustrated in Fig. 1T (left) where refresh the LSTM model parameters whenever the algorithm enters the subproblem stage. As a comparison, by doing this, the convergence the subproblem is much faster. The ground truth rules in group- (4, 5, 6) similar in content and share many predicates. Given this prior knowledge, reuse the LSTM across subproblems may help the convergence.
Early stop mechanism. To speed up our algorithm, propose an early stop mechanism. We consider that when the iteration times reach a pre-set reasonable number, and when LSTM model consecutively generates identical logic rules (like identical 10 rules), conclude that the LSTM model has been trained enough and is able to generate good-perte poood-efforming logic rule. We don't need to train the LSTM until the norm of policy gradient is within very small tolerance. Hence, may early stop our training. Fig. 10 (right) illustrates the training trajectories of solving subproblem for group- 1 case-1, where reset the LSTM in the beginning of the subproblems and early stop mechanism. It's obvious that this mechanism can reduce redundant iterations and help us complete the training process in short time.
K COMPARE WITH STANDARD POLICY GRADIENT
For standard policy gradient, the policy mo (a)s) is trained end-to-end by minimizing the subproblem objective using policy gradient, i.c.,
max J(0) max E ( [R(r)]
19

---

Under review as a conference paper at ICLR 2023
Case-1
Case-2
Case-3
Case-4
Case-4 24 predicates
8 predicates
12 predicates
16 predicates
20 predicates
0.8
Tuub Leamned
0.5
Fhthh
0.8
Tuth
08
Truns
0.8
Trubh
Group
0.4
0.3
Learned
0.5
Leamez
0.
Leamned
0.5
Leamed
0.4
0.4-
0.4-
0.4
0.4-
0.2
0.
Tulbs
Thuth
0.8
Trlth
0.8
Truits
0.8
Trub Learned
0.00
Learred
Learthed
0.5
Learned
0.6
Learned
0.
0.4
0,4
m
Trub's
Leamted
Thuth
Learned
Tuth Leamed
Truth
Leamed
0.8-
Truehs
Leamned
0.2
Truca
0.8
Group a
Loameed
0.6
0.5
0.5-
0.6
0.8
0.4
0.4-
0.0
w0
0.0
W3
0.0
W0
W2
0.0
0.
0.8
in
0.7
0.7
0:7-
0.7
0.3
Group
0.5
0.5
0.5
0.5
0.5
0.3
0.3
0.5
0.2
0:2-
w9
M1
W2
0.3
WB
M1
W2
0.9
MD
w1
W2
0.0
W0
W2
0.0
W2
Truth
0.7
0.7
Truits
0.8
Group
0.5
0.5
0.5-
0.1:
0.3
0.1
w2
wi
wa
w2
Rule weight
Rule weight
Rule weight
Rule weight
Rule weight
Figure 8: MAE values for all groups and all cases using 1000 samples. Blue one indicates the ground truth rule and red/yellow one indicates the learned rules different groups.
where is the generated tokens (i.e., candidate rule) and R(7)
(F(an,bu) Bwy
0t((w)
The
duey
w %0.00
policy gradient can be estimated using the roll-out samples, i.e.,
VoJ(O) R E
i-1 Ex k= Tologme(uf) (()
(21)
where N is the number of episodes, and K is the length of tokens (actions). We use this estimated policy gradient to update the policy 0, 0 +- + VoJ(()) where a is the learning rate.
However, in our problem, the final performance of our model measured by the single or few best-performing rules found during training. So normal policy gradient may not be satisfactory while risk-seeking policy gradient may be more suitable.
We compare the performance of the risk-seeking policy gradient model with the normal policy gradient model for cases in group-3 and group-6, mainly because the ground truth rules in these two groups are long and complex with many body property predicates and various temporal relations. Due to the difficulty of recovering ground truth rules in these groups, it may be more obvious to distinguish the performance of the models. We set to be 0.3. The learning rate in solving the subproblem was 100 and the hidden state size of LSTM was 32. The learning rate in solving the restricted master problem x 10-3 The results are shown in Fig. 11 and Fig. 12
20

---

Under review as a conference paper at ICLR 2023
Not Reset LSTM Model
Loss
3-
2
50
100
150
200
250
Iteration number
Figure 9: Training loss trajectories of solving different subproblem stages if L.STM model parameters are inherited from the previous subproblem. Reset LSTM Model - Not Early Stop
Reset LSTM Model Early Stop
18
Loss
50
100
150
200
250
0
10
20
30
40
50
60
70
80
Iteration number
Iteration number
Figure 10: Training loss trajectories of solving different subproblem stages for group-1 case-1. Left: The LSTM parameters are refreshed in the beginning of each subproblem. Right: Early stop mechanism used.
Group-3
Predicates in Library
Case-1:
Case-2: 12
Case-3: 16
Case-4: 20
Case-5: 24
Learned
Leamed
Learned
Learned
Policy Gradient: Normal
2
2
3
Learned
Truth
Truth
Truth
Truth
Truth
Learned
Learned
Learned
Learned
Learned
Gradient:
Policy
3
2
Risk Seeking
Truth
Truth
Truth
Truth
Truth
Figure 11: Jaccard similarity score for all 5 cases in group-3 using 1000 samples. Blue one indicates the ground truth rule, green one indicates the learned rule using normal policy gradient, and one indicates learned rule using risk-seeking policy gradient.
By using risk- seeking policy gradient, there actually some significant improvements. The results show that more, and more importantly, more accurate ground truth rules were learned by the risk- seeking policy gradient.
21

---

Under review as a conference paper at ICLR 2023
Group-6
Predicates Library
Case-1: 8
Case-2: 2: 12
Case-3: 16
Case-4: 20
Case-5: 24
Learned
Learned
Learned
Learned
Leamed
Policy Gradient:
Truth
2
Truth
Truth
Truth
3
2
Truth
3
Normal
Learned
Learned
Learned
Learned
Policy
Gradient:
3
2
2
Risk Seeking Truth
Truth
Truth
Truth
Truth
Jaccard similarity for all 5 in group-6 using 1000 samples. Blue indicates the ground truth rule, green one indicates the learned rule using normal policy gradient, and yellow one indicates the learned rule using risk-sceking policy gradient.
Figure 12:
L PREDICATE DEFINITION IN MIMIC-III
MIMIC-III is an electronic health record ICU dataset, which is released under the PhysioNet Creden- tialed Health Data License 1.5.d" was approved by the Institutional Review Boards of Beth Israel Deaconess Medical Center (Boston, MA) and the Massachusetts Institute of Technology (Cambridge, MA). In this dataset, all the patient health information was deidentified. We manually checked that this data do not contain personally identifiable information offensive content.
We defined 63 predicates, including two groups of drugs (i.e., intravenous fluids and vasopressors) and lab measurements, ested in reasoning about two predicates and define them as head predicates: 1) LowUrine and 2) No rma 1rrine. We treated real time urine head predicates since low urine the direct indicator of bad circulatory systems and the signal for septic shock; normal urine reflects the effect of the drugs and treatments and the improvement of the patients physical condition. our experiments, lab measurement variables were converted to binary values (according to the normal range used in medicine) with the transition time recorded. For drug predicates, they were recorded as when they were applied to patient. We extracted 2023 patient sequences, and randomly selected 80% of them for training and the remaining 20% for testing. The average time horizon is 392.69 hours and the average events per sequence is 79.03.
Tab. 8 for more details. Among all these predicates were inter-
M EXPERTS' MODIFICATION ALSO IMPORTANT
We also invite human experts (i.e. doctors in ICU) to justify correctness and modify our learned rules. We compare the log-like oog-likelihood trajectories of directly updating the rule weights and directly updating the weights of the modified rules by human experts, and the results are shown in Fig. I3 The results show that the modification suggestion from human experts can indeed help improve the performance of our model, since the log- ligeliielhoood trajectory of modified rules rise higher and faster.
N REAL-WORLLD EXPERIMENT: EYE FIXATION
Simple choices are made by integrating noisy evidence that is sampled over time and influenced by visual attention. result, fluctuations in visual attention can affect choices. We aim understand shoppers' purchase patterns given their eye fixation event data (Callaway et al., 2021).
Our conjecture is: the location of the items, shopper-assessed values the items, and the shopper's visual habits (usually looking from left to right) will affect their final item choice. We learned temporal logic rules and their weights to quantitatively understand this.
Dataset Description: Three items randomly placed on the "left", "middle" and "right" the supermarket shelf, each has unique "price" (value).
Each shopper evaluated three items by eye fixation until they identified an item purchase.
22

---

Under review as a conference paper at ICLR 2023
Table 8: Defined Predicates in Our MIMIC-III Experiment.
Low/NoormallHHihh-SS//
Low//oozmaal/ighhccV
Low//Noziaal/Hiih--SVV
Low./Normaal/Highh-Pootaaiiiii
hov//Norraal/Htgg---5ddddu
hoo//Noormaa//iiigh-hhh0000:11
Low//Normaal/High-BBBB
Lab Measurements
Loww//oorma///HghhhCrreeaiinno
Low/NNoznaa/HighhCCRP
Lov//Normaal/iigh-BBCCCCuuut
Lowp/Noommal//Atgg--WBBBCCoonn
Lox//Nozmaal/High--H5C00
Low.NNormaa//Hig00--/0022222222
Output
LowUrine,,Norma111111
Input
Collooddcrrystalllldd..aa
Drugs
grrepinneprrine..Biiepphee
ceanttaainn,DDpmmiiiii Phenyleprhen
Temporal Relation Type
Before, After, Equal
Log-Likelihood Trajectory
-3.13
-3.14
Log Likellhood
-3.15
-3.16
-3.17
-3.18
Learndd Rules
Modified by Experts
1000
2000
3000
4000
5000
6000
Timets)
Figure 13:
Log-likelihood trajectory for just rumning master problem update rule weights. Blue curve indicates that we put all learned rules directly into master problem and update their weights. Red curve indicates that we put the modified rules directly into master problem update their weights.
The data record each shopper's eye fixated items, when and where, and their final purchased item. There are 30 participants, each with at most 100 independent trials. At each trial, participants were asked to look at these three items and choose the item that they think is most valuable. There 2966 trials in total. On average, for one trial, participant has 4.301 eye fixations.
Predicate Definition: We are interested in explaining three final choices of a shopper:
1) finally choose the item with the actual (not shopper-assessed) largest value
2) finally choose the item with the last eye fixation, and
3) finally choose the item with the longest eye fixation.
23

---

Under review as a conference paper at ICLR 2023
These final choices define the head predicate set. Another 18 predicates about the location of the items, value of the items, and the time of eye fixation of shopper on one specific item. Please refer the Tab. beloo for complete predicate set.
Table 9: Defined predicates for eye fixation trials. For eye fixation predicates, there are three parts: location of item, value item, duration of eye fixation
Left Lote_ Left Seff___axxvalue_hhhrttiiettooo
Lefe_MM.cVallle__oongglxxtt.....
Leef__KtnvValle_loongiixxxi..11
Leff_MMnnaalee_SShrttttttttt11
Macdlee_JaaaaaaeeeLooooll1111111
Eye Fixation
Ri.ght_/aavaallee_oooggixxxx111
Riggh__MaavaalleeBhhott1xxx1xx11
Rigntt/WlnyaauueeSShotttttttto.t
Finalchoiic_JaargeettWllle
Final Choice
FinalChoiiceJaasf7laatiooo
FinallChoice_longesstitetttoo
Temporal Relation Type
Before
Rules Discussion: We displayed the discovered important rules in Tab, I0 which summarize the eye fixation patterns before shoppers making choices. From the results, have the following discoveries:
1) the final fixation shorter
2) the later (but not the final) fixations are longer
3) people are more likely to begin to look from the left or from the middle.
Specifically, if shopper finally chooses the item with the largest value, he may first glance over all three items at least once, after looking all three items, go back to check the item he wants to choose, and then make a choice (Rule 1, 2, 5, and 8).
And people more used to looking from left right (Rule 2, 3, 5, and 7).
a shopper finally chooses the item with the last eye fixation, he may only take a quick look at these items and may miss item with the longest eye fixation, he may spend a lot of time on most of these items, reevaluating the value of these items back and forth in his mind (Rule and 10).
two of these items (Rule 3, and 9). If shopper finally chooses the
In summary, our discovered temporal logic rules provide insight into shopper's perchance behaviors terms of eye fixation patterns.
24

---

Under review as a conference paper at ICLR 2023
Table 10: Temporal Logic Rules Discovered for Eye Fixation and Final Choice. Since there is only one type of temporal relation "Before", in the following representation of temporal logic rules, we just ignore the temporal relation Weight
Rule
Rule 1:
0.0550
A Leet_MidValll
Rule achhahheoeennnnnnnnnaoaeeeeeeeea
A Middle ||g.aaaaaooaooaaaaaaaaaaaaaaaaan
0.0976
Rule haJiaaaehhhannnaaaraaaaaaaaaaaan
0.0278
Rule hht.hhasschhheegggiioooor.rreeer
0.0479
Rule otteaaacoooooaaaaaaioooeaeeeeaar
0.0648
A Maaelpaiictuuiiiiorrrrtatotttiir hett_hawwalleeShhhorrrriaa..n...
Rule Rateriachhheeaaaaaiiiiiiiiiiiiih
0.0015
A hLeef_Jawvlee.Shoottttttioo.....
Rule hht.:hhhichceeegggeiiinnnnitiiir
0.0457
A Attollaeeaarralllcccoootaaaiinnn
Rule hht.ahaceooooonaaacoooooeeeaeiir
0.0241
A hegeg//wwillegggggoooooo/gggaiia
Rule hat.ghaieeeu...aaaaianieeeeeieer
0.0243
A Nateahairiiiagiooaaatttoteaaaaer
Rule htt.hiohhhhceenneiiioiiieieeieer
0.0145
A Wat.aaaeee.enniiaaaataooaaaaaaaa
25


Reviews:
[{"review_id": "ngNuUNivZj", "reviewer": "Reviewer_AuUW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a reinforcement temporal logic rule learning\nalgorithm to jointly learn temporal logic rules (before-like kind of\nrules) and their weights from event data. The proposed learning\nalgorithm alternates between a rule generator stage and a rule\nevaluator stage, where a neural search policy is learned by\nrisk-seeking gradient descent to discover new rules in the rule\ngenerator stage.", "summary_of_the_review": "This work presents a reinforcement temporal logic rule learning\nalgorithm to jointly learn temporal logic rules (before-like kind of\nrules) and their weights from event data. The proposed learning\nalgorithm alternates between a rule generator stage and a rule\nevaluator stage, where a neural search policy is learned by\nrisk-seeking gradient descent to discover new rules in the rule\ngenerator stage.\n\nThe work is quite interesting and shows good results in the\napplication of predicting sepsis from the MIMIMC-III database by\ndistinguishing LowUrine from NormalUrine rules. Various state of the\nart systems were compared against the method proposed in this work.\n\nI wonder if classical methods based on (Probabilistic) Inductive Logic\nProgramming wouldn't work as well as your proposed method. Why isn't a\nmethod like that compared in your work? Is it very different? TTE\nproblems may be also modeled using ILP and variants that are\nneurosymbolic.\n\nWhy did you use only one application? How well would your method work\nif you used other examples?", "strength_and_weaknesses": "The work is quite interesting and shows good results in the\napplication of predicting sepsis from the MIMIMC-III database by\ndistinguishing time predictions for LowUrine from NormalUrine rules. Various state of the\nart systems were compared against the method proposed in this work.\n\nI wonder if classical methods based on (Probabilistic) Inductive Logic\nProgramming wouldn't work as well as your proposed method. Why isn't a\nmethod like that compared in your work? Is it very different? TTE\nproblems may be also modeled using ILP and variants that are\nneurosymbolic.\n\nWhy did you use only one application? How well would your method work\nif you used other examples?\n\nWhy do you concentrate on grounded terms and not on first order logic terms with a logical variable?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mjIeOXWaau", "reviewer": "Reviewer_XGxF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of learning the intensity functions for temporal point processes (TPP) to model the occurrence of events in irregular time intervals. Specifically, the authors consider the representation of the intensity functions as parameterized by an exponentiated weighted sum of kernels switched by temporal logic rules. Given this parameterization, the overall objective is to find a set of rules and their corresponding weights to maximize the likelihood of data.\n\nTo optimize this objective, the authors break the MLE problem into two steps, a master step and a substep. In the master step, the weights are optimized via convex optimization. In the substep, the logic rules are selected. In order to optimize in the combinatorial space of logic rules, the authors formulate the problem as an RL problem, where the policy outputs a token of logic rules at a time and receives a reward when the sequence of tokens is completed. The authors then apply the risk seeking policy gradient method to solve this problem.\n\nThe paper includes empirical evaluations of the proposed method on synthetic and real datasets, and the results suggest that the proposed method achieves good likelihood and absolute error.\n", "summary_of_the_review": "While this paper presents an interesting method of applying RL to optimize the log likelihood of TLPP, I’m not really convinced that RL is the right approach and I believe that simpler approaches could work better for this problem. Hence, I cannot recommend acceptance of this paper in its current state. I highly encourage the authors to try applying bandit methods and including some black box optimization baselines during the author response phase.", "strength_and_weaknesses": "Overall I think the paper presents an interesting idea of applying RL to solve combinatorial optimization problems in temporal logic rule learning. I’m not familiar with the field of temporal logic point process (TLPP) and the related works in this field, so I will focus on the optimization method part in this review.\n\n### Pro\n\nThe paper is well presented and the proposed method is easy to follow. The authors include a detailed introduction about the preliminaries and make it easy for readers to understand the problem. Although I’m not familiar with TLPP, I am able to quickly understand its formulation and the optimization problem associated with learning TLPP thanks to the high quality introduction of this paper.\n\n\nThe empirical results of the proposed method seem strong. The proposed method outperforms many baselines in real datasets.\n\n\n### Con\n\nWhile the problem of searching for logic rules can certainly be framed as an RL problem, I’m not convinced that this is a good formulation, and I want to argue that formulating it as an RL problem here makes it unnecessarily more difficult. First of all, the problem has no inherent Markov state transition structure of an MDP because all tokens are directly generated by the agent, unlike in an MDP, where the states can only be influenced by the actions of the agent rather than directly chosen by the agent. Moreover, the reward here is the result of an entire sequence rather than the result of a single time step. Combining these two characteristics, I believe the problem is better formulated as a contextual bandit problem, and solved using a bandit algorithm instead of an RL algorithm. Using a bandit algorithm will directly remove the long horizon credit assignment difficulty of RL and likely result in much stabler training and better performance. \n\n\nGiven that the optimization problem is a standard back box optimization problem, it would be important to compare to many widely used back box optimizer baselines. Some examples of these algorithms include cross-entropy methods, genetic algorithms and simulated annealing methods. There are many well-developed software packages for these blackbox optimization algorithms that can be easily integrated into the specific problem, so it would be important to try them out and include them in baseline comparisons.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kh6ulGm9NO", "reviewer": "Reviewer_Snuq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a method for identifying temporal logic rules.\nThis method uses reinforcement learning to select rules and separately optimizes the weights of these rules.\n\nThis method is evaluated on finding rules on benchmark tasks, in a transfer setting, and on a real-world problem.", "summary_of_the_review": "The new method is motivated well and appears to perform better than existing alternatives.\nEvaluation could be improved, though. For example:\nFor the real-world study, by only considering the useful rules found by a successful system, the usefulness of explanations is not evaluated.", "strength_and_weaknesses": "The method is an intuitive improvement over the existing enumeration approach.\nMy understanding of the method is that the selection of rules is done in a greedy way.\nAdditionally, further rules are not added when the next greedy rule is no longer useful.\nAs a result, this method is akin to performing prioritized enumeration along with a stopping condition.\n\nMy understanding is that the state only includes information about the tokens for the current rule (i.e., no information is included about existing rules).\nAs a result, the typical MDP properties do not hold in this setting.\nThe choice of an RL formulation is therefore a bit odd, but the sequential nature does make it a better fit than alternatives with which I am familiar.\n\nThis work seeks to also perform transfer. Transfer in RL is a non-trivial problem.\nThe assumption of \"same or similar ground truth rules\" substantially simplifies the problem. This assumption is akin to assuming similar optimal policies, at which point policy reuse (without adaptation) is expected to substantially improve early performance.\nBenefiting from past knowledge when able while not hurting performance in other cases is difficult. This aspect is stripped away in the performed transfer experiments as transfer is known to be helpful.\nAdditionally, this assumption cannot be made in real-world applications of this method as ground truth rules cannot be known prior to applying methods.\n\nSection 5.2 with human experts has experts looking at rules and determining if these rules seem reasonable.\nThis approach to evaluation is only considering one side: do doctors generally agree with the rules identified.\nThe other side is missing: do doctors disagree with the rules not selected by the system.\nSince the system is supposed to distinguish between useful and non-useful rules, this categorization should be evaluated by doctors.\nIdeally, this should be done close to the decision boundary, so comparisons should be made between the last included rules and the first ones not included (rather than comparing random rules to the top rules, for example).\nAdditionally, the goal of explanations is to provide trust in _performant_ systems.\nEvaluation can be repeated with rules generated by a faulty model, and doctors should be able to distinguish between the properly functioning and faulty models based on the produced explanations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YDUjO-J7l-J", "reviewer": "Reviewer_Z32A", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to learn a set of temporal logic rules to build a probabilistic model that maximizes the likelihood of the observed data that come in the form of temporal event sequences. Compared to brute-force search in the space of logic rules that are enormous, the proposed approach leverages a neural policy to generate logic rules guided by heuristic reward signals to maximize the likelihood of the probabilistic model that can be obtained using the new rules. Empirically, the paper demonstrates that the proposed approach can improve upon prior baselines that utilize brute force for logic rule generations on a set of synthetic event data proposed by the authors. The proposed approach also outperforms prior methods on the MIMIC-III health care dataset by achieving lower prediction errors. ", "summary_of_the_review": "In general, I found the paper to be a bit difficult to understand. I can roughly get the main idea of the propose method, but I might have misunderstood many technical details due to the writing issues. In addition, I found that the empirical evaluation of the paper to be a bit lacking and the details of some baselines are missing. These altogether make it hard to evaluate the significance of the results. Because of these reasons, I would not recommend acceptance of the paper at its current state. ", "strength_and_weaknesses": "*Strength*\n\n- The idea of formulating the logic rule generation problem as a reinforcement learning problem is interesting. \n\n*Weaknesses*\n\n- The paper has a lot of incorrect statements and confusing sentences which make the technical content hard to understand. \n- The empirical section seems to only contain results generated using one single run of the algorithm, which makes it very difficult to judge the effectiveness of the proposed algorithm. \n- Many details of the baseline algorithms are missing in Table 1 (e.g., Transformer architecture, network size, how the data are fed into the transformer models) which could pose major reproducibility issues. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject