Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a neural nets based approach for Temporal Logic Programming. A key claimed contribution is on formulating combinatorial rule space search as predicate selection indicator vectors assignment so as to make it differential to enable end2end training from atomic event probabilities as inputs.  
main_review: Strength

	1.  formulating combinatorial rule space search as predicate selection indicator

Weakness

	1. The proposed model's parameterization depends on the number of events and predicates making it difficult to generalize to unseen events or required retraining. 
	2. The writing needs to be improved to clearly discuss the proposed approach.
        3. The experiments baselines are of the authors' own design; it lacks a comparison to the literature baselines  using the same dataset. If there is no such baseline, please discuss the criteria in choosing such baselines.


Details: 

	1. Page 1, "causal mechanisms", causality is different from temporal relationship. Please use the terms carefully. 
	2. Page 3, it seems to me that M_T is defined over the probabilities of atomic events. The notation as it is used not making it difficult to make sense of this concept. Please consider providing examples to explain M_T. 
		
	3. Page 4, equation (2), it is not usual to feed probabilities to convolution. 
		a. Please discuss in section 3 how your framework can handle raw inputs, such as video or audio? Do you need an atomic event predictor or human label to use your proposed system? If so, is it possible to extend your framework to directly have video as input instead of event probability distributions?  Can you do end2end training from raw inputs, such as video or audio? (although you mentioned Faster R-CNN in the experiment section, it is better to discuss the whole pipeline in the methodology).
		b. Have you tried discrete event embeddings to represent the atomic and composite events so as the framework can learn distributional embedding representation of events so as to learn the temporal rules?
	4. Page 4, please explain what you want to achieve with M_A = M_C \otimes M_D. It is unusual to multiple length by conv1D output. Also please define \otimes here. I am guessing it is elementwise multiplication from the context.
	5. Page 4, "M_{D:,:,l}=l. This can be thought as a positional encoding. It is not clear to me why this can be taken as positional encoding?  
	6. Page 6, please detail how do you sample top c predicates. Please define what is s in a = softmax(s). It seems to me the dimension of s with \sum_i (c i) can be quite large making it softmax(s) very costly. 
	
	

summary_of_the_review: This is a good initial attempt to attack the neural temporal logic programming. However, the steps in the proposal --- temporal compression, predicate modeling, composite event prediction, combinatorial inference, and rule reduction --- requires more clarity and refinement to reach a publication state. Overall, the proposal has little element of neural networks. To me it is more on matrix formulation of ILP problem. It is Okay to me but please consider renaming the paper title.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an end-to-end differentiable strategy (called neural TLP) to learn unknown temporal relations between atomic events (like after(miss, swing), “miss occurs after swing” in the baseball example), subsequently used to predict composite events (like strike). The strategy consists of a cascade of a smoothing stage to filter  out noise from the input time series, an interval time extractor, a stage predicting temporal relations (such as before, after and during) and a linear output layer. Furthermore, the paper proposes a post-hoc procedure to extract propositional logical rules relating atomic events to composite ones from the last layer. The performance of the proposed strategy is evaluated on video recognition (CATER) and healthcare (MIMIC-III) datasets against two baselines, namely a LSTM neural net and a simplified version of the proposed strategy.
main_review: **Strengths**

- The paper is well written.
- The proposed solutions, namely the differentiable layers to predict the temporal relations between atomic events and the post-hoc strategy to induce propositional rules from the last linear layer, are simple but non trivial.
- Results are promising.

**Weaknesses**

- It is not clear how the work positions with respect to recent literature in spatio-temporal reasoning and video question answering. For example, the authors should have a look at [1]-[2]
- While experimental results are interesting and promising, they provide only a partial picture, as they neglect recent neural advances. Indeed, comparisons are made against only simple baselines and should be conducted against more “structured” neural approaches, see for instance graph-based approaches [1]-[2], where code is available.

**Questions**

1. How scalable is the proposed strategy in terms of number of objects, number of atomic events and compositionality of the target events?
2. As far as I understand, the proposed strategy considers only the temporal relations among atomic events and objects are considered independent of each other. Is that correct? If so, why is that the case?
3. While I understand the intuition about the strategy to extract the time intervals of atomic events (as shown in Figure 3), I do not see how well this strategy can perform in the continuous and more noisy setting. How reliable are the obtained estimates?
4. In CATER experiments, is the perception component (i.e. Faster R-CNN) trained jointly to the Neural TLP  or are two components trained separately?
5. The names parameter and structure learning are a bit confusing in this context. In their classical definition, parameter learning considers learning the parametric relations among variables in a given graph. In Neural TLP, the temporal relations are not given, rather learnt. Why is this considered as parameter learning?
6. Can you mention something about the logic expressiveness of the induced rules?
summary_of_the_review: The paper is clearly written and easy to follow. I like the idea around the proposed differentiable operators used to predict the temporal relations among atomic events. Also, the experimental results are interesting and promising. However, they are preliminary. I would like to see more in-depth analysis on the correcteness of the time intervals for atomic events, the scalability of the proposed strategy and a comparison against recent structured neural approaches for spatio-temporal reasoning. Furthermore, the authors should discuss and relate their work against existing literature in the field of video question answering.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a new approach called neural logic programming to recover composite events from complex time series data using atomic events and their temporal relations. 
main_review: Strengths: the two-step approach (i.e., parameter learning followed by structure learning) does seem novel for the problem. 

Weakness: the empirical results are rather unconvincing. Not only that the baselines (two LSTMs) are rather weak, but also that the proposed approach seems to be even worse than the baselines in terms of mean average precision (which is the mainstream metric used on CATER) in Table 1. Besides, Table 3 suggests that the proposed approach is marginally better than logistic regression.

Minor:  In figure 2, “(1) parameter learning” seems to be in the wrong place. That should refer to predicate prediction according to page 3, paragraph 1.
summary_of_the_review: Although there is some novelty in the proposed approach, the empirical results are too weak to validate the effectiveness of the approach. I do not think the paper in its current form is good enough to be published in ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach for learning temporal rules from data. The idea is to extract atomic events, and their temporal relationships between them. Subsequently, composite events are learned using composite event labels for supervision. The whole approach is formulated as an optimisation problem, after which standard techniques are applied to solve it.

main_review: The paper addresses an important problem of learning temporal rules from data, for which many applications exist (as the paper convincingly motivates). 

A very strong point of the paper is to learn interpretable rules using modern neural network machinery. This can certainly inspire future work. 

The paper is not always easy to follow. It would have been useful to more explicitly state the aims of the proposed method. For example, on page 4 the paper refers to Temporal Relation Networks, but they are dismissed because "timelines can be long in our setting", yet, it is not clear what this refers to. For me, it was also quite unnatural to say that composite events are a-temporal (e.g. in the baseball
example on page 2). Why does it make sense that you cannot say anything about the timeline of the 'strike' event? Also in the remainder of Section 3.1, various choices are made, which were not all obvious to me. 

In the structure learning part, I was missing the connection to probabilistic rule learning approaches (e.g. ProbFOIL). The essence
seems to be to learn rules where the temporal atoms have a probability attached to it, unless I am missing something.

In the evaluation, the paper convincingly shows that variations of the proposed method performs less well. What was not convincing to me
were two aspects:
(1) Why previous approaches are unsuitable for the proposed settings, e.g. probabilistic rule-learning, sequence mining, etc. 
(2) I have doubts about the validity of the medical example. For example, isn't it the case that 'oral water' is simply responsible for 
urine flow rather than after(oral_water, spo2_sao2=high) ? I can understand that a doctor would say that you "captured important explanatory variables" in the proposed rules, but that does not mean that it is in any way true. In fact: doesn't this show a downside of the approach in the sense that only temporal events can be used in rules?

summary_of_the_review: Overall, I thought this is an interesting contribution to the ICLR conference. I do have some doubts about the completeness and validity of the experiments.
review_title: null

Meta-review: The paper gives a framework for learning temporal logic rules from noisy unlabeled data. The key novelty is a formulation of combinatorial rule search as an end-to-end differentiable problem. The method is evaluated on a video dataset and a healthcare dataset. 

The reviewers liked the high-level ideas behind the paper. However, the conclusion was that the experimental results, while interesting, are still somewhat preliminary (in particular, the baselines are weak).  I agree with this point and am recommending rejection this time around. However, I urge the authors to develop the paper further and submit to the next deadline.
Decision: Reject