Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper uses a method inspired by column generation to continuously add temporal logic rules to their model to maximize the likelihood of the data. The goal is to generate a model that is highly interpretable. They compare against state of the art baselines and get good improvements.
main_review: Intepretable models are increasingly relevant as ML models are applied to more and more domains where safety and fairness are important -- characteristics that are hard to judge for traditional black box models. The method presented is novel and gets good improvements on existing state of the art methods on synthetic and real data.
summary_of_the_review: Novel method which gets improvements on SOTA
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper builds upon the work of [1] and proposes an algorithm to simultaneously estimate the logic rules and the model parameters.  The estimation problem is formulated as the contained optimization problem (6), with both continuous and discrete parameters to optimize over. The proposed TELLER algorithm divided the original problem into a sequence of Restricted master problems and gradually expand the search space for a global optimum.  Subjective constraints and expert opinions can be incorporated into the search algorithm as well.


[1] Shuang Li, Lu Wang, Ruizhi Zhang, Xiaofu Chang, Xuqin Liu, Yao Xie, Yuan Qi, and Le Song. Temporal logic
point processes. In International Conference on Machine Learning, pp. 5990–6000. PMLR, 2020.
main_review: I find the paper quite interesting. Admittedly, the proposed optimization problem is a very difficult one due to the high dimensionality of the parameter space and it is almost hopeless to find the global optimum. The proposed TELLER algorithm is not perfect, but the authors explain the idea very well and the algorithm makes heuristic sense. The problem this paper aims at solving is an important one and I can see that it will make a difference in many applications. I especially like the fact that domain experts were brought into the real data analysis and the results look interesting. The only concern I have is about the stability of the algorithm, which may be of critical importance in practice. Specifically,

1. In equation (6), how should the complexity $c_{f} $ be determined? I think they determine the sparsity of the solution $w$, hence is important. Simply saying that "$c_f$ can be the rule length" is a bit arbitrary. Wouldn't it make more sense if a tuning parameter, say $\lambda$, is introduced to balance the negative likelihood and complexity penalty? In such case, the Master Probe would become minimizing $-\ell(w,b_0,b_1)+\lambda\sum_{f\in\bar{\mathcal{F}}}c_fw_f$ and $\lambda$ can be chosen by some sort of cross-validation.

2. How much impact does the initial subset $\mathcal{F}_0$ have on the final solution? Can some simulations be conducted to show the impact of $\mathcal{F}_0$？

3. In synthetic data experiments, the number of true logic rules is at most 4. What would happen if this number continues to grow? In this case, the search space for the optimum will be much larger and how well would the algorithm work?

summary_of_the_review:  The problem this paper aims at solving is an important one and I can see that it will make a difference in many applications.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a model for learning temporal logical rules for the temporal point process. Specifically, the key idea is to formulate the rule learning as a restricted master problem and gradually optimize the objective function, which can significantly lower down the search space.
The authors conduct extensive experiments on both synthetic data and real data. 

main_review: Strength: 
* The problem of learning temporal logical rules is very important and interesting. 
* The key idea is novel and interesting. 
* The experiments are extensive, which are conducted on both synthetic data and real data. 
* There are many case studies and learned rules, which is interesting. 

Weakness:
* It would be better if the authors can provide references for the temporal logic formula in the background section. 
* There is a number of works for learning logical rules (e.g., Neural-lp, DRUM). It would be better to claim that the key difference between temporal logical rules and ordinary logical rules, and how temporal logical rules generalize them.
* The following paper is also relevant, please discuss this paper in the related work section. 

Wang, Po-Wei, et al. "Differentiable learning of numerical rules in knowledge graphs." International Conference on Learning Representations. 2019.


summary_of_the_review: In summary, it seems that this is a novel and interesting paper for learning temporal logical rules with solid experiment results. I am familiar with logical rule learning but not the temporal point process, so I am not sure about my evaluation of the temporal process part. But in general, learning temporal logical rules is very interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces TELLER, a framework with novel algorithms based on the temporal point process model to discover interpretable temporal logic rules. To this end, the authors designed a “rule generation - evaluation” two-stage way to solve the problem efficiently. The proposed method is evaluated using one synthetic and two real data sets. Experts have also verified that the learned temporal logic rules are interpretable and reflect many critical principles in real practices.
main_review: 
1.	Structure and presentation

The paper is not self-contained on the theoretical basis. It especially lacks necessary explanations on the relationship between interpretable temporal logic rules and temporal point processes. From the problem to the output (such as the derived rules in Table 2), the paper does not seem to show any strong dependencies on the temporal point process (TPP) model. Unfortunately, it neither fully demonstrates and constructs the complete problem background nor provides sufficient technical details on the critical process (rule generation, searching/expanding algorithm, expert knowledge templates). Instead, it took a lot of space to define a two-stage solution framework. For me, this is not the most valuable contribution.

Figure 1 tries to convey too many technical details that are difficult for readers to understand, especially when reading Section 1, until it was cited in the discussion of “Search algorithms” in Section 3.1 to show the necessary details of designing the REFS algorithm. Therefore, figure 1 should focus on TELLER’s core idea rather than technical details in the paper’s introduction part.

The situation in Figure 2 is similar, where the author uses the concept of TLPP before its theoretical preliminaries are introduced in the main text. In the discussions of “Logic-informed intensity function”, the author may consider guiding readers to understand the concept and calculation method of intensity function using case studies designed in Figure 2, rather than directly referring to it without any explanation.

The position of Figure 3 is also problematic. It appears in Section 3.1 and is not quoted by any text until the end of Section 3.4.

2.	Novelty and contribution

For the concepts introduced in Section 2, the authors should clarify which are cited and new tools designed by TELLER. More importantly, it is necessary to reflect the differences between the related work built on TPP model and this study. 

To my understanding, it is not an outstanding innovation to define rule discovery as a two-stage optimization problem. The most valuable contribution is in designing a sound rule generation or expanding algorithm in the sub-optimization problem (near-optimal).

Although the paper claimed: “Our problem is even more challenging compared to traditional inductive logic programming cases due to the sequential properties of the data and the presence of temporal information.” However, we do not see any theoretical analysis to show how TELLER has reduced the calculation?

Since the event sequences and values are discrete, is this problem challenging to solve in the case of a limited number of basic events (the number of variables considered in the case analysis is small in reality)? That is, how big is the size of the searching space of the $P_Master problem (Equation 4) in real practices?

In response to this issue, the paper skipped several key details when designing the search algorithms in Section 3.3: (1) Why the complexity is O(d)? In such a not-complex solution space, why should we resort to a complicated searching scheme instead of enumerating? (2) When extending the original rule, TELLER considers “the signs and possible temporal relations with the head predicate”. However, the paper did not show specific principles and rules to do this. Many detailed questions need to be explained: Is the extending process following any temporal sequential way? Can it support the negation pattern? Can it discover the loop structure? And logical branch structure? These factors will significantly affect the ability of the extending algorithm to find effective rules

In Section 3.4, the author mentioned that TELLER's scalability is an advantage over brute-force solving algorithms. Although there can be many event sequences in a complex example, what kind of problem will have so many predicates? For example, we see the cases in the experiment part are: 
1. Synthetic data: predicate set {A,B,C,D}, head {E}.
2. MIMIC-III:  62 predicates as stated, head {LowUrine, Survival}.

Finally, we can also see from Crime experiments that users still need to define promising predicates to guide the algorithm to discover rules. It seems the distance between reasonable predicates and truth rule is not far away, or we can say that in the case of limited predicates and temporal relations, the best results can be obtained through brute-force enumeration and evaluation. Therefore, the authors should clarify whether there are problem scenarios that need to deal with massive predicates to prove the necessity of designing TELLER.

3.	Technical details

The authors can consider discussing why we need to introduce rule set weights in logic-informed intensity function.

The paper mentioned that “the initial rule set can be an empty set or any pre-defined small set”. Will an accurate pre-defined set contribute to the accuracy of the algorithm? Better to provide some experimental results and analysis.

In the analysis of TELLER’s scalability, the author mentioned: "In practice, setting the time limit is proven to work well and the most important rules are likely to be discovered." This is a key observation in evaluating TELLER. Therefore, it is recommended to design different time window parameters to verify the effect of the algorithm.

4.	The subjectivity of the experimental results

In Table 1, the author gives five properties but does not explain how to decide whether a method is of or not of these properties. Table 1 shows several arbitrary comparison results without seeing any justified process. It is recommended that the authors provide a detailed definition for several fundamental properties such as Parsimonious and Higher-order interaction and justify the results through a case study or statistical results.

5.	Reproducibility

The author did not attach their codes, which affected the evaluation of the paper’s reproducibility. 

6.	Miscellaneous and language problems

Through careful check, it is found that the font size of the paper is not consistent. Starting from the "Logic-informed intensity function" section on page 4, the font suddenly becomes smaller. I am not sure whether this meets the ICLR format requirements because the content from Page 4-8 may exceed the limit.

The first paragraph in Section 3.4 is a redundant summary that deserves to be simplified.

Section 1, Paragraph (Para) 2, “These domain knowledge can be summarized as” -> This domain knowledge

Section 1, Para 3, “without the need to discretize the time horizon into bins.” Not clear, what “bins” refers to ?

Page 2, Para 5, “where the number of variables are too large to be considered explicitly.” -> is too large to

Page 3, Para 6, “the body part of the formula indicate the evidence” -> indicates

Page 4, Para 4, “-1 indicates negative effect,” -> a negative effect

Page 4, the last line, “requires enumerating exponentially large set of combinations” -> an exponentially large set

Page 5, Para 3, “the set of variables (the $w_f$s) we are optimizing on”. It is inappropriate to use plural “s” here.

Page 6, Para 6, “This provides performance guarantee for speedup algorithms.” -> a performance guarantee

Page 6, Para 6, “we do not need to consider rules with length exceeding $H$.” -> with a length exceeding $H$

Page 6, Para 6, “we mean if the conjunctions of the input predicates is important” -> are important

Page 8, the last paragraph, “Among all these predicates we are interested in reasoning about two predicates.” Ambiguous, should be separated by commas

Page 9, Para 1, “is a important signal for septic shock” -> an important 

Page 9, the last paragraph, “We propose a promising TELLER algorithm to learn interpretale temporal logic rules” -> interpretable

summary_of_the_review: In summary, although the paper presents a solid theoretical discussion and a comprehensive verification, it still has defects in the overall structure, the clarity of innovation, the challenges of the problem scene, and some method details. As a result, I do not recommend accepting the paper even though it shows many merits.

Revision Notes (2021-11-26):

Most of the problems have been appropriately resolved or explained in the revision. The quality of the paper has been significantly improved and exceeded the acceptance threshold. Therefore, I have raised my score to 6.
review_title: null

Meta-review: This paper has been independently evaluated by four expert reviewers. After discussion with authors, three of them set their recommendations at marginal acceptance, one at straight accept. Perhaps the key criticism involved limited rigor of theoretical justification for the proposed method, but it appears to be applicable in practice as the empirical results suggest. All things considered, I am leaning towards recommending that this paper is accepted for ICLR 2022.
Decision: Accept (Poster)