Neural Temporal Logic Programming
Authors: Karan Samel
Zelin Zhao
Binghong Chen
Shuang Li
Dharmashankar Subramanian
Irfan Essa
Le Song
Venue: ICLR 2022 Submitted (2022)
Keywords: logic programming
time series
neuro-symbolic
video
healthcare

Abstract: Events across a timeline are a common data representation, seen in different temporal modalities. Individual atomic events can occur in a certain temporal ordering to compose higher level composite events. Examples of a composite event are a patient's medical symptom or a baseball player hitting a home run, caused distinct temporal orderings of patient vitals and player movements respectively. Such salient composite events are provided as labels in temporal datasets and most works optimize models to predict these composite event labels directly. We focus uncovering the underlying atomic events and their relations that lead to the composite events within a noisy temporal data setting. We propose Neural Temporal Logic Programming (Neural TLP) which first learns implicit temporal relations between atomic events and then lifts logic rules for composite events, given only the composite events labels for supervision. This is done through efficiently searching through the combinatorial space of all temporal logic rules in an end-to-end differentiable manner. We evaluate our method on video and on healthcare data where it outperforms the baseline methods for rule discovery. 

Under review as a conference paper at ICLR 2022
LEARNING TEMPORAL RULES FROM NOISY TIME- SERIES DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Events across timeline are common data representation, seen in different temporal modalities. Individual atomic events can occur in certain temporal ordering to compose higher level composite events. Examples of a composite event are patient's medical symptom or a baseball player hitting a home run, caused distinct temporal orderings of patient vitals and player movements respectively. Such salient composite events are provided as labels in temporal datasets and most works optimize models to predict these composite event labels directly. We focus on uncovering the underlying atomic events and their relations that lead to the composite events within noisy temporal data setting. We propose Neural Temporal Logic Programming (Neural TLP) which first learns implicit temporal relations between atomic events and then lifts logic rules for composite events, given only the composite events labels for supervision. This is done through efficiently searching through the combinatorial space of all temporal logic rules in an end-to-end differentiable manner. We evaluate our method on video and healthcare datasets where it outperforms the baseline methods for rule discovery.
INTRODUCTION
Complex time series data is present across many data modalities such as sensors, records, audio, and video data. Typically there are composite events of interest in these time series which are composed of other atomic events An example is a health symptom that can be observed in a doctor's s report. Atomic events, such as patient vitals and medications, and their temporal relations dictate an underlying causal rule leading to the composite event symptom. These rules may be unknown but useful to recover (Kovačević et al.
certain order (Liu et al.) 1999 Chakravarthy et al., [1994] Hinze (003)).
2013 Guillame-Bas Bert et al.) 2017).
Recent methods leverage the advances in highly parameterized deep architectures to learn latent representations of atomic event data (Pham et al., 2017 Chen et al., 2018, Choi al.) 2019), with the inereasing availability of large temporal datasets. Methods, such as LSTM (Hochreiter & Schmidhuber 1997) or Transformer (Vaswani et al.] 2017) based architectures, provide state- of-the-art performance in terms of composite event inference. However, it is uncertain whether the latent representations learn the underlying causal sequence of events or overfit spurious signals in the training data. Having representations faithful to causal mechanisms is advantageous for interpretability, -fittt-diitiibuttion generalization, and adapting to smaller data sets. Therefore it is important to leverage parametric models that can handle data noise while providing a mechanism to extract explicit temporal rules (Carletti et al. 2019).
Extracting explicit logic rules has been studied through Inductive Logic Programming (ILP) methods (Muggleton) 1991, Muggleton & De Raedi 1994) and have been leveraged in para- metric fashions as well (Yang et al. [2177] Evans Grefenstette] 2018) [Rocktäschel & Riedel 2017). ILP starts with set of background knowledge, consisting of grounded atoms (i.e. facts which do not contain variables) such as Location cate location determines the relationship between the items Braves and Atlanta There are set of labels from which rules should be learned. The task is to construct a set of rules, when executed over the background knowledge, entail the provided labels. Given the label InLeague(Braves
Atlanta), where the predi-
NL East) and the background knowledge (Figure [ ILP Input) as input, candidate rule is InTeagueeTeem, League) in City)

---

Under review as a conference paper at ICLR 2022
ILP
Input
Predicata Labels
Learned Rules
NYC. Queena
dhrtecn
InLobgue LOcatZoRt TESK, cstyl
NL East
Ini InLengue] BBracea AL Eaxt)
InLengue ML East)
diiioon
Brawes
Alannaa
Temporal ILP
Inpues
Composite
Labels
Leamed Temporal Structure
Learned Rules
Pichh
belore
Swingg
before
HB
beloro
Steal tn Before Pitch]
Run
strake is Before(Pitch, Sxing)
Before(Sning, Pisa;
Swing
Dalora
stmal,
Atomic Event
Figure 1: In ILP, the grounded background knowledge and known relational predicates between atoms provided to induce consistent rules. In the temporal case, operating over raw temporal data samples, such as videos, with potentially multiple labels. Therefore the latent temporal structure between the atomic events is recovered, and then the rules for composite events are learned.
Division(City, League). Here InLeague(Team, League) is the head of the rule con- sisting of an atom with variables Team, League items. The body consists of atoms and when these atoms exist on the background knowledge the rule is evaluated as true. We apply ILP over real world temporal data, however learning such rules poses three key challenges.
Temporal Background Knowledge First, ILP methods operate over existing grounded back- ground knowledge. The temporal case does not have this knowledge when operating over raw time series. For example in baseball video, grounded atomic events pitch swing ing grounded predicates such as be beroree feoreppicccs would be labeled with higher level composite event description, such as "Player A's home run" instead of individual atomic events and their corresponding temporal predicates. Such atoms can be extracted using a model in a probabilistic fashion at each time point, and temporal ILP method should handle this uncertainty. The temporal predicates between these probabilistic atomic events can be applied rule-based manner (ex. l < la - before), but due to the noisy nature of extracted atomic events, the predicate predictions should be robust to consistent noise in the atomic event data.
swing) are not explicitly provided. By nature, the video
Atomic Event Relevance Second, ILP works learn consistent rules that satisfy a path in the NL East).
background knowledge given the terms in the labels, such as InLeague(Brave The labels are nullary predicates in the temporal case, so the relevant source and target atomic events and predicates we have a label stri ke, but are not told when it occurred or what other events, such as pitch, swing, and miss are needed compose rule for strike.
for rule induction are unknown. In example, know from the video
Without a prior on which atomic events to search from, we must consider all pairwise temporal relations between atomic events in the input. This leads combinatorial search of all pairwise events for each predicate in the temporal rule body.
Multi-Event Labels Third, ILP domains work on disjoint labels, while in time series, multiple composite events could occur in each input. In our baseball video, such highlight reel, composite event labels strike, steal and their corresponding atomic events can co-occur in a single video. This further extends the search space of atomic events consider for each composite event rule.
2

---

Under review as a conference paper at ICLR 2022
We illustrate these differences in Figure and further discuss these challenges regarding search complexity in Appendix A To address these challenges, Neural TLP operates on two key steps.
Parameter Learning First Neural TLP inputs probabilistic atomic events and learns parameters to infer temporal predicates between atomic events. We represent the atomic event data in an interval- based representation efficiently predict all pairwise predicates between atomic events. The inferred predicates are then projected to predict the composite event labels.
Structure Learning When the predicate parameters are learned, Neural TLP learns sparse vector to select the correct rule over the combinatorial space of possible rules. To prune the search space, we use the learned projected weights to select candidate grounded predicates per composite event.
We evaluate our method on synthetic video dataset to empirically test our temporal rule induction performance. Additionally, we apply our framework to provide relevant rules the healthcare domain, which were verified by doctors.
2 PROBLEM FORMULATION
along a timeline T. These atomic events can be existing features time series data or user defined features of interest. A temporal logic rule r(Krr Tr)) can be defined as using subset of N S | atomic events X {=u C X, and their associated time intervals T, [tw]am S The time intervals consists of start and end times t, = [ buud.
We define the complete set of atomic events X' {
,,,...,, T|x)
These intervals indicate durational events and we can also initialize instantaneous events occurring at one time point where tuma tews A rule is evaluated true if the corresponding atomic events zu are present and are in correct ordering with respect to the intervals t, other events zp;
r(Xr, :
Fa) A
A tettETT
Pa t.))
Zu Z.E,
The temporal predicates p; (before during, after] P represent simplified subset of Allen's Temporal Algebra (Allen 1983). We simplify the notation the rules as conjunction of temporal predicates between observed events, where the event time intervals implicit:
n
Pi(Fu, xe)
For example, the grounded predicate before(pitchh..n swing would evaluate to true.
These underlying causal rules induce the composite event labels yr seen in the data. Multiple composite events of interest can co-occur during the same time series sample T which we denote as y } E {0, 1) Any yr indicates the latent rule r occurred over T resulting in label Wr.
While T contains precise atomic event interval information, the observed time series T consists of a sequence of probabilistic atomic events from times [1, T'. Potentially & different objects 7 compose the final time series data T = - T. Examples of objects be multiple concurrent sensor data, or tracking multiple people moving within video. Then the input 7 formulated as MT E [0, 1| [ across object, atomic event, and probability dimensions respectively.
The temporal ILP task is to recover all underlying rules R given m samples of inputs and labels {(MTs, y:)
In Neural TLP this involves learning parameters for the predicates between atomic events and then learning the combination of grounded predicates that induce each r E R.
y )
3 NEURAL TEMPORAL LOGIC PROGRAMMING
Neural TLP operates in two stages. The parameter learning stage learns how to compress the temporal data and learns parameterized temporal predicates. Once these parameters learned, the structure
3

---

Under review as a conference paper at ICLR 2022
Setoratron pitch)
Overview
(3) rule induction
structure leaming
probabilstic atomic eventts
compression and smoothing
interval extraction
predicate prodiction
[6. 7.9]
afterthat, pischo
sparse selection
21t
Eiaa
beferstawing putch))
otzake
Zosl
[88, 2.4]
beforejesing hit)
aving
[4.1,
dense inferance
(1) parameter
leaming
Figure 2: The first step (1) of Neural TLP involves learning the convolution and predicate model parameters from the raw time series and labels. Then the structure learning step (2) is learning attention 8 over a sparse combinatorial matrix to infer the labels. This attention and sparse matrix is then used carry the final rule induction (3).
learning stage learns which conjunctive combination of pairwise atomic event predicates is associated with each composite event label. This conjunction composes the rule for label y/- and is jointly computed for all R. An overview of the framework is presented in Figure
3.1 PARAMETER LEARNING STAGE
Temporal Compression Starting from the raw probabilistic atomic event data, we first compress the timeline through convolution. This ID convolution over the temporal dimension compresses and smooths the timeline mitigate noise from spurious events. Here the convolution kernel K of length / is learned per atomic event. We also parameterize a as an extra degree of freedom to scale these convolved scores, which is useful when computing the intermediate predicates downstream.
Mc IR xx|
conv ID 10,11 K)
The time information is incorporated by multiplying the time dimension MD into compressed events: MA E x = Mc @ MD. Here Mp has the same dimensions as Mc. but the temporal dimension enumerated from [1, 4]. where MD This can be thought as a positional encoding.
For example if we look at the sample compressed scores for single object i and atomic event Mcs.,.6:1
[6, 7, 8, 9, 10] then MAi.j,::10 [306,.35,55 5.6, 6.3, .3.. Intuitively we can see that from M MAJ.j.: that (1) atomic event j occurs when the scores are high at 5.6 and 6.3 and that (2) score 6.3 occurs after score 5.6 due to the multiplied time index. This temporal representation provides path to compute precise time intervals of atomic event occurrences and define predicates to compare atomic event intervals.
[001, .5, -7, .03) and MD
Predicate Modeling From the compressed timelines, determine the temporal predicates between atomic events. These relations are computed in pairwise manner for all atomic events Væu,, H, E X occurring object through small network which we call Temporal Predicate Network (TPN), For notation sake here, we represent the atomic event u's timeline for object as t -MAia;: E R' and correspondingly for atomic event We denote TPN go gt tt), which takes pairwise atomic event timelines and predicts a temporal predicate p P to indicate the relationship between the atomic events.
Methods such as Temporal Relation Networks (Zhou et al.) 2018) learn these predicates between video events by sampling frames throughout the video. The timelines can be long in our setting, and events can occur sparsely, making sampling timelines expensive and noisy. To efficiently compute these relations, intervals, we can encode strong inductive biases to predict the predicates. We are working with continuous time series scores in E t the intervals have be extracted the first step in TPN.
would like to recover each event's 's underlying start and end time intervals. From
4

---

Under review as a conference paper at ICLR 2022
1
2
event prob
time
5 max
fausk
max
min
Mend
Mstarr
Figure 3: An overview of how intervals are computed from raw data. First the compressed atomic event scores (1) are multiplied with the time scalar (2) to compute MA (3), where we observe single sample vector t In step 4 we find the max value of t.. representing the end of the interval, and use this value to initialize the mask in step 5 (Equation 3). When steps and are summed in step 6, we get a representation whose min corresponds to the start of the interval as shown Equation
To compute the start of an event interval, we create a mask to identify the atomic event noise. Those values will be below some small value e, corresponding noise in the timeline. We learn the convolution scalar a from Equation[2] scale scores corresponding to active atomic event occurrecces above while keeping scores corresponding to atomic event noise below Then the mask is added to the time series, and min is performed to get the start of the active atomic event interval. Afterwards the min of the mask is subtracted to remove any effect of the mask the start value.
fmakk (max(ti,) + E) (t,, <E)
start min(t; + tmask) - min (tmask)
To get the end of the event interval we simply compute Uead max(t!,) since we multiplied the event scores with the time index earlier. This interval computation from the input time series visualized in Figure This is computed similarly for the other pairwise event v: [Veant Vend]. Given the start and end times for the event pairs u, v, the un noromaaleze predicate scores are computed as:
before(u, v) = Ustatt Mend
(5)
after(u, Uatat Vend
(6)
during(u, v) = min( (Vend Ustart, Uend
Ustart
(7)
Although we use 3 predicates in our model, similar scores can be developed for more fine grained predicates. Then the values are aggregated as p [before(u, v); during(u, v); after(u, v)] to compute normalized predictions as p softmax P= Here and 7 and scale and shift parameters learned from data. Our predicates scores assume that intervals for both and occur, if either event doesn't It occur we suppress all predicate predictions:
supp min( ((nndd
(Mnd Ustart Vend Ustant)
(8)
p, min( ((p,, supp))
Since we leverage a simple interval representation compare atomic event objects, we can scale comparing atomic events within the object and between the other k objects: x, E X, E (X'x k). This second-order interaction information is useful if we want know if, for example, two events occurred simultaneously within different objects. For single object i, these relations are computed for all pairwise predicates through TPN in Mp E || ( xk) = I x Aggregating
over all objects k. we get Mq [Mp ... Mp*] E 2 x x|x|x Pl, We marginalize over the
object dimension get final pairwise relation matrix MR Σ, MQ e R X
Composite Event Prediction The final inference step from the pairwise relational predicates to the composite events labels is carried out by fa. This is a linear projection function (MRR) := (dtropout(eec(Mn) W) used to infer the composite event labels y.
5

---

Under review as a conference paper at ICLR 2022
(1) Paramter Learning
(2) Structure Learning
grounded predicates index
W
lup c
befere(swing, pitch)
during(swing, pitch)
3
3
C max len n = 2
181 1.0- [P]
after(ewang, pitch)
beforo(ewang.mas9
2
2
index for strike
strike after(swing pitch) beffore(swinggming
Figure In the parameter learning stage (I) we learn the most relevant grounded predicates used for predicting each label yr through W. In the structure learning stage (2) we select the most relevant predicates and construct the combinatorial matrix C, where each column indicates a conjunction of predicates. Vector is learned to select the most likely conjunction to induce the rule r.
Here flatten MR ())( R |X1---1111 and regularize it by randomly masking out the grounded predicates (Srivastava et al. 2014). This representation is then projected to the label space using W E -((x1)----1P11)
before passing the un- un-normalized results through sigmoid function swing),
W learns what grounded relational predicates p. (z z,), such before correspond to each composite event label. These weights will also be useful for extracting the rules, in the structure learning stage.
3.2 STRUCTURE LEARNING STAGE
Predicate Selection To induce the logic rules R, one method is to look at our projection weights W. Here we can select the highest weighted entries corresponding grounded predicates for each label yr. One can use a conjunction of these predicates to construct the rule r for yr- However, in most cases, the number of predicates needed compose rule is unknown apriori. Additionally, setting thresholds for information gain splits is heuristic- based and error- pror-,rone, especially when one cannot observe the underlying rules for verification.
Combinatorial Inference Instead of setting thresholds, we directly optimize over the space of combinatorial rules to infer composite event label yr-
Starting from the predicted grounded predicates vec(MR) E Rd where = |x( | initialize combinatorial matrix up to max rule body length n:
[P]
we
C =CC C.l; C, C
(10)
Here for each unique column C. will have indicators for the chosen predicates, corresponding to one possible combination. Since (+) can be quite large, we sample the top < predicate weights in W.y.. Then combinations can be initialized over those predicate indices C, E dd
Selecting the most relevant combination across all combinations is done through attention vector a softmax(s) where E E (?), It is used to weight the combinations c E R = E, C.C... and the label can be inferred by ş/ vec(MR), thus [,,, for each r E R]. Note that maintain separate attention parameters s and unique C for each label yr E y, since each rule relies on different predicates.
Rule Induction
To extract the rule we simply choose the column in C, where = arg max corresponds to the maximum attention value. Each indicator value in C correspond to grounded temporal predicate p e P between two events r x X, which are used to construct a rule:
N pf (of x.)) for pi predicate( (C))
(11)

---

Under review as a conference paper at ICLR 2022
Ineuts
Video
contain cone
piok-olace cone
Labels
12.
32.
56.
pick- cube
rotate snitch
12 dur-agtp.chppacc rotate eaitca)
beforelpick-pl.ee rotate amitch) snisch, cone)
Recover Verify
Generate
Ground Truth Rules
Figure 5: In CATER, generative rules used to synthesize the labels from the videos. Neural TLP can then learn these rules from the raw atomic event data and labels. We then verify our rule induction performance over the ground truth rules.
This is followed for every composite event label yr to provide our final set of rules R. This full process is illustrated in Figure
3.3 OPTIMIZATION
Now that we have defined our inference procedure to obtain ý from both parameter and structure learning stages, we describe the overall training. To train over the data ((MT,Y) ) each stage minimizes the standard cross entropy loss between y and denoted as Lents using the Adam optimizer (Kingma & Ba 2014) (details in Appendix [B)
The parameter learning stage is trained first over the convolution a, K, predicate B, y. and projection W parameters. For W we add L1 regularization (denoted as Li) and project the weights between [0, to mimic logic weights (Chorowski & Zurada 2014 Riegel al. 2020). We also constrain the convolution scalar a between [0, IJ. This gives us our final stage objective to optimize = 0.1. The structure learning stage is trained next optimize Lent all
Lent Ai where attention vectors 8 corresponding to each label while freezing all other parameters. Each stage is trained in an end-to-end differential manner, after which the temporal logic rules R induced.
4 EXPERIMENTS
4.1 CATER
We explore composite event prediction over complex videos in the CATER dataset (Girdhar & Ramanan 2019). CATER consists of videos containing objects moving around scene. The object movements correspond to X' 14 distinct atomic events. Every combination of predicates between atomic events yields rule of length
1, and when this combination occurs in the video, it induces that corresponding label. There are R = 301 rules to recover and the average number of labels per video y Σ Ey.ey, Vr 53 out of 301.
In previous works on the CATER, the main metric is mean average precision (mAP) over the labels. Due to its synthetic nature, we know the ground truth rules used to induce the labels, so we can also empirically evaluate how well our models recover these rules in its top k rule predictions @k). The overall task for CATER is illustrated in Figure 5
Baselines We test Neural TLP against two baselines. For the first baseline, we input our My matrix into aitention-based LSTM (Hochreiter & Schmidhuber] 1997) and predict the composite events. Since we are dealing with only single predicate rules, we synthesize each rule combination within MT and assign it to the highest weighted label.

---

Under review as a conference paper at ICLR 2022
We define the second baseline Temporal MAP, which uses the same Neural TLP model. We freeze the parameters and the weight W a count of co-oceurring grounded relational predicates and labels. This setup is akin to processing atomic event relations deterninistically and computing W through MLE. The rule extraction follows the same methods as Neural TLP, and additional baseline details laid out in Appendix
4.1.1 RESULTS
Model
Hits @its@1
Model
Inputs
mAP
LSTM
.00
.04
I3D/3DD
ResNet Features Optical Flow RGB Difference + Optical Flow
.44
LSTM Attn
.00
.04
TSN
664
Temporal MAP
.27
TSM
ResNet Features
.73
Neural TLP
.91
.95
LSTM Atm
Inferred Atomic Events
.75
Neural TLP
Inferred Atomic Events
.69
Table I: Hits when the inputs are inferred (probabilistic) the atomic events. All scores have a reported variance of S .01.
Table 2: mAP scores versus video baselines.
Model
y
53 (orig)
y = 40
y = 30
y y= 20
y = 10
LSTM Attn
.75
.37
.34
.31
.27
Neural TLP
.69
.49
.47
.45
.42
Table 3: We observe mAP performance when testing out of distribution data with respect the average number of labels per sample.
experiment where the atomic events are obtained from a noisy environment or inferred from process upstream, such our baseball video. To infer the atomic events, we detect the objects through a Faster R-CNN (Ren et al.) 2015) and use its cropped image feature and optical flow to predict the shape and movement. These atomic events predicted and cached prior to inputting them in our models.
From the results in Table we that rules. MAP provides a coarse representation of the labels and enumerated rules with no parameters. can express these rules better than LSTMs but lags behind our method. As the number of free parameters increases, the LSTM models are more likely to pick up spurious signals in the data that useful from cross-entropy optimization perspective but deviate from the underlying generative rule representation (Hits). This can be seen with highly parameterized LSTM models and video models: I3D (Carreira & Zisserman 2017). TSN (Wang al. 2016), and TSM (Lin al., 2019).
structured method is the most optimal for extracting
leading to larger gains in mAP in Table
Even for mAP, we show that our underlying rule representation is useful when generalizing out of distribution. Here we fix our trained LSTM and Neural TLP models and test on out of distribution data where the frequency of labels is changed in Table 3 We show that Neural TLP performance degrades gracefully as it is exposed to out of distribution data.
We further ablate our Neural TLP model architecture and hyperparameters D.2
show the effectiveness of method recovering longer dynamic length rules over Temporal MAP in Appendix [.3] Now that we tested Neural TLP on synthetic tasks to empirically verify the rule accuracy, we explore its capabilities on real-world healthcare data.
4.2 HEALTHCARE DATA
We test the rules recovered from Neural TLP on patient data in MIMIC-III (Johnson et al. 2016). We specifically look 2023 patients admitted for sepsis (severe infection) and recover the rules corresponding to stable vitals. This done through predicting urine output as the composite event, an auxiliary variable indicative of the state of the patient's fluids and circulatory system (Komorowski et al., 2018). There are | = 82 different atomic events, composed of drugs administered and patient vitals. The vitals made into boolean events through logic rules provided by doctors, which
8

---

Under review as a conference paper at ICLR 2022
Model
#@50
MRR
Model
Urine Output mAP
Neural TLP
.04
Logistic Regression LSTM Attn (L) Neural TLP
.75
Temporal MAP
0
.74
.77
Table 4: We compute the number of relevant rules in top 50 50) as well as the mean reciprocal ranking (MRR) of the correct rules.
Table 5: Inference results for the urine output task.
indicate the vital severity: low, normal, or high. The model's task is to learn rules corresponding to normal urine outflow. We present the top rules from Neural TLP and MAP to doctors for verification.
From the results in Table 4 see this is a difficult problem due a large number of atomic events and small sample size. However, it indicates that the learning done by Neural TLP is useful to learn and rank important predicates before rule training. Temporal MAP fails at this task with the increased number of atomic events and longer timelines, which led to many grounded predicates (2-3k) per sample. Therefore the W weights are very coarse and filled with common relations. In addition to relevant rules, Neural TLP maintains good inference performance (T which is also an important metric for this domain.
From the doctors' feedback, we also present the rules learned in Appendix but emphasize that these are observed This means that the rules are partially explained by a subset of predicates and not considered treatment rules, the rules didn't 't contain any drugs administered. The doctors did confirm that we captured important explanatory variables in our proposed rules, so we are optimistic about using our framework for feature selection in complex temporal environments.
5 RELATED WORK
Structured Temporal Prediction To optimize over the composite events given observed atomic events, Hidden Markov Models (HMMs) are commonly used model the latent rules that emit consistent atomic events to induce the composite events. Variants of HMMs have been developed to
handle symbolic atomic events (Mutschler & Philippsen| 2012 Kersting et al. 2006 Liu et al. 2017) to more perception-basee
events, such as videos (Tang al. 2012). Deep models have shown to incorporate temporal logic constraints within their outputs by leveraging Transformers architectures (Finkbeiner et al., 2020), knowledge distillation (Ma et al. 2020), and by representing temporal logic as a differentiable loss (Innes & Ramamoorthy, 2020). However, interpreting latent representations
of deep models in order extract explicit rules is still being researched (Arras al. 2019 Chefer
ct al. 2021
Lal et al. 2021)
Rule Representations and Inference To explicitly induce rules over data, space temporal logic rules can be searched to determine appropriate rules. Temporal rules are induced using satisfiability (SAT) based methods (Neider & Gavran 2018: Camacho Mcllraith, 2019) given strong priors to the rule structure and limited data noise. In the presence of noise, softening logic rules using Markov Logic Networks (Richardson Domingos 2006 [Song et al., 2013) or probabilistic logic (ProbLog) (De Raedt et 2007 are intractable for rule training. Recently Yan Julius (202T) proposed network to learn sparse weights for temporal rules given fixed rule format.
et al., 2011)) perform approximate inference well, but
Inductive Logic Programming Extracting expressive rules is explored through Inductive Logic Programming (ILP) methods (Muggleton) 1991] Muggleton & De Raedi 1994) and within Statistical Relational Learning literature (Koller et al. 2007] De Raedt & Kersting 2010). Given fixed back- ground knowledge, parameterized models softly select the relevant facts used to derive the labels and therefore lift logic rules (Yang et al. 2017: Evans & Grefenstette 2018). Atoms can also be represented as latent vectors to provide probabilistic facts for increased generalizability (Rocktäschel & Riedel, (017). Dong et al. (2019) present a more neural architecture to represent first-order logic and scale larger rule search spaces.

---

Under review as a conference paper at ICLR 2022
CONCLUSION
Composite event extraction is a common task across many temporal domains. It is important to understand the underlying atomic events and their predicates that induce the composite event. We propose Neural TLP that learns these composite event rules even when provided noisy temporal data. first learns parameters for atomic event timeline compression and pairwise predicate prediction. Then once grounded predicates are reliably inferred, the structure learning stage learns over the space of combinatorial predicates to induce the final rules. We verified our method on synthetic video tasks and explored rule recovery in real-world healthcare dataset.
10

---

Under review as a conference paper at ICLR 2022
Ethics Statement Neural TLP, like any data-driven model, has potential societal impacts that could include extracting unfairly biased rules or relations. In such cases, we envision using such a framework for knowledge discovery end decision making. We operate hospital patient data, through MIMIC-III which has been de-identified to avoid leaking privileged patient information.
Reproducibility Statement We provide the entire model code for our framework the supplemen- tary materials. The dataset for CATER is publicly available, and we have provided the processing code to generate our custom CATER data. We are the process of sharing the experimental code for the MIMIC-III experiments as well. We further describe our optimization procedures (Appendix B hyperparameters (Appendix D.2), and bascline model architectures (Appendix (C for clarity.
REFERENCES
James F Allen. Maintaining knowledge about temporal intervals. Communications of the ACM, 26 (11):832 843, 1983.
Leila Arras, José Arjona- Arjona-Medina, Michael Widrich, Grégoire Montavon, Michael Gillhofer, Klaus- Robert Müller, Sepp Hochreiter, and Wojciech Samek. Explaining and interpreting Istms. In Explainable ai: Interpreting, explaining and visualizing deep learning, pp. 211 1-288. Springer, 2019.
Alberto Camacho and Sheila A Mcllraith. Learning interpretable models expressed in linear temporal logic. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pp. 621-630. 2019.
Mattia Carletti, Chiara Masicro, Alessandro Beghi, and Gian Antonio Susto. Explainable machine learning in industry 4.0: Evaluating feature importance in anomaly detection to enable root cause analysis. In 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pp- 21-26. IEEE, 2019.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6300 2017.
Sharma Chakravarthy, Vidhya Krishnaprasad, Eman Anwar, and Seung-Kyum Kim. Composite events for active databases: Semantics, contexts and detection. In VLDB, volume 94, pp. 606-617, 1994.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. arXiv preprint arXiv:2103. /5679, 2021.
Pudi Chen, Shenghua Liu, Chuan Shi, Bryan Hooi, Bai Wang. and Xueqi Cheng. Neucast: Seasonal neural forecast of power grid time series. In ISCAI, pp. 3315-3321, 2018.
Edward Choi, Zhen Xu, Yujia Li, Michael W Dusenberry, Gerardo Flores, Yuan Xue, and Andrew M Dai. Graph convolutional transformer: Learning the graphical structure of electronic health records. arXiv preprint arXiv: / 99999997666 04716, 2019.
Jan Chorowski and Jacek M Zurada. Learning understandable neural networks with nonnegative weight constraints. IEEE transactions on neural networks and learning systems, 26(1):62-69, 2014.
Luc De Raedt and Kristian Kersting, Statistical relational learning. 2010.
Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: probabilistic prolog and its application in link discovery. In IJCAI, volume 7, pp. 2462-2467. Hyderabad, 2007.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. arXiv preprint arXiv: /694, 2019.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.

---

Under review as a conference paper at ICLR 2022
Bernd Finkbeiner, Christopher Hahn, Markus N Rabe, and Frederik Schmitt. Teaching temporal logics to neural networks. arXiv preprint arXiv:2003.04218, 2020.
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and temporal reasoning. arXiv preprint arXiv:1910004444, 2019.
Mathieu Guillame-Bas Bert, Artur Dubrawski, Donghan Wang, Marilyn Hravnak, Gilles Clermont, and Michael R Pinsky. Learning temporal rules to forecast instability in continuously monitored patients. Journal of the American Medical Informatics Association, 24(1):47-53, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Annika Hinze. Efficient filtering of composite events. In British National Conference on Databases, pp. 207-225. Springer, 2003.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8):
1735-1780, 1997.
Craig Innes and Subramanian Ramamoorthy, Elaborating learned demonstrations with temporal logic specifications. arXiv preprint arXiv:2002.00784 2020.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Kristian Kersting, Luc De Raedt, and Tapani Raiko. Logical hidden markov models. Journal of Artificial Intelligence Research. 25:425-456. 2006.
Angelika Kimmig, Bart Demoen, Luc De Raedt, Vitor Santos Costa, and Ricardo Rocha. On the implementation of the probabilistic logic programming language problog. Theory and Practice of Logic Programming. 11(--3):235-262, 2011.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 49800 2014.
Daphne Koller, Nir Friedman, Sašo Džeroski, Charles Sutton, Andrew McCallum, Avi Pfeffer, Picter Abbeel, Ming-Fai Wong, Chris Meek, Jennifer Neville, et al. Introduction to statistical relational learning. MIT press, 2007.
Matthicu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(11):1716
2018.
Aleksandar Kovačević, Azad Dehghan, Michele Filannino, John Keane, and Goran Nenadic. Combining rules and machine learning for extraction of temporal expressions and events from clinical narratives. Journal of the American Medical Informatics Association, 20(5):859-866, 2013.
Vasudev Lal, Arden Ma. Estelle Aflalo, Phillip Howard, Ana Simoes, Daniel Korat, Oren Pereg, Gadi Singer, and Moshe Wasserblat. Interpret: An interactive visualization tool for interpreting transformers. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 135-142, 2021.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding.
In Proceedings of the IEEE International Conference Computer Vision. pp. 7083-7093, 2019.
Guangtian Liu, Aloysius K Mok, and Eric Yang. Composite events for network event correlation. In Integrated Network Management VI. Distributed Management for the Networked Millennium. Pro- ceedings of the Sixth IFIP/IEE International Symposium on Integrated Network Management((Cat No. 99EX302), pp. 247-260. IEEE, 1999.
Sympossiu
12

---

Under review as a conference paper at ICLR 2022
Yu- Ying Liu, Alexander Moreno, Shuang Li, Fuxin Li, Le Song, and James M Rehg. Learning continuous- hidden markov models for event data. In Mobile Health, pp. 361-387. Springer, 2017.
Meiyi Ma, Ji Gao, Lu Feng, and John A Stankovic. Stinet: Signal temporal logic enforced multivariate recurrent neural networks. In NeurlPS, 2020.
Stephen Muggleton. Inductive logic programming. New generation computing, 8(4):295-318, 1991.
Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming. 19:629 679, 1994.
Christopher Mutschler and Michael Philippsen. Learning event detection rules with noise hidden markov models. In 2012 NASA/ESA Conference on Adaptive Hardware and Systems (AHIS), pp. 159-166. IEEE, 2012.
Daniel Neider and Ivan Gavran. Learning linear temporal properties. In 2018 Formal Methods in Computer Aided Design (FMCAD), pp. 1-10. IEEE, 2018
Trang Pham, Truyen Tran, Dinh Phung, and Svetha Venkatesh. Predicting healthcare trajectories from medical records: A deep learning approach. Journal of biomedical informatics, 09:218-2299 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 29199,2015..
Matthew Richardson and Pedro Domingos. Markov logic networks. Machine learning, 62(1-2): 107-136, 2006.
Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, et al. Logical neural networks. arXiv preprint arXiv:2006.14 13/55, 2020.
Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. arXiv preprint arXiv:170S.11040. 2017.
Young Chol Song, Henry Kautz, James Allen, Mary Swift, Yuncheng Li, Jiebo Luo, and Ce Zhang. A markov logic framework for recognizing complex events from multimodal data. In Proceedings of the 15th ACM on International conference on multimodal interaction. pp. 141-148, 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-1955 1958, 2014.
Kevin Tang, Li Fei-Fei, and Daphne Koller. Learning latent temporal structure for complex event detection. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 1250-1257. IEEE, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: / 000./706..3.0 03762, 2017.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Ruixuan Yan and Agung Julius. Neural network for weighted signal temporal logic. arXiv preprint arXiv:2104.05435. 2021.
Fan Yang. Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. arXiv preprint arXiv: 170..0370. 2017.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. Proceedings of the European Conference on Computer Vision (ECCV), pp. 803-818, 2018.
13

---

Under review as a conference paper at ICLR 2022
A TEMPORAL ILP COMPARISON
Neural LP In typical ILP problems, such as Neural LP (Yang et al. 2017), the rules are induced over a static knowledge base. This involves learning the walks along the static graph Gs = (Es, Rs). Starting at entity node 6x &s Neural LP learns the associated edge relations Rs to traverse in order to reach the corresponding ending entity node ey. Since between two entities there can be many arbitrary paths, given the data, the most likely path is found. This is done by learning the attention a over relational predicates matrices MR, to traverse from Vx to Vy. Here Vx: Vy are one hot embeddings of the start and entities respectively.
T
R
Py
# MR
Then the objective is to maximize the score v Vy of selecting the correct end entity Ey through paths selected by af The edges along this path compose the predicates of the rule between Ez₁ ey. Refer to the original paper for full details and implementation (Yang et al., 2017).
Knowledge Representation In temporal rule learning one can also compose a graph between the entities (events) on the timeline. However the structure of such a dynamic graph 9p (ED. Rp) over time is different. The dynamic graph typically contains fewer events [Ep] < [Es]. and the graph is dense every event has some temporal relation with respect all other events. The edge relations may not be provided in unified knowledge base, where the relations can vary per sample m: Rp = [R], R...... RB ). Furthermore these temporal relations between samples are rarely annotated with the relations of interest, while in static case, the relations Ry are usually predetermined.
Rule Induction When learning rules in the temporal case, many rules don't conform the chain like rule structure. This can be seen in the form f == after (a, b) A before d) where there no path between the first predicate and the second (see the last rule in Table Das an example). Without path between two events to guide the rule construction, the rule can potentially involve any events and relations. Formulating the problem in terms of walk is challenging to evaluate if we consider all relations between the disjoint events and selecting the most likely rule:
Σ after (a, b)) Abefore (c, d) VX,Y E disjoint( (a, b), (c, d)) P.EP
arg
Here, events between two pairs of predicates are disjoint if there does not exist a common event between the predicates. If not, we would have to sample all combinations of X, Y from each predicate respectively: (a, c), (a, d), (b, c), (b, d). Instead of expensive marginalization, we aim to search for combinatorial combinations of grounded predicates in a differentiable fashion. This search space for Temporal ILP grows faster with larger numbers of atomic events and relations.
Lemma. In Temporal ILP, given the space of events E = | predicates P = IP,, , and a rule with predicates, the search space for rule is generally O((PE²)") IfP consists of symmetric relations, such as be beforel v) a fftervv, u), and an equivalent relation during v) = duri turing(u, u), we keep 1 of the symmetric relations and the unique values in the equivalent relation. Then the F(E+1)
bound can be tightened to @([[_) + yn)) unique combinations of rules.
In ILP given the knowledge base, construct rule for predicates with pair of entity variables E1, Eg and average node degree D, the search space is @(D*).
The corresponding temporal models and rule extraction methods has to reflect these differences. We focus structured time series tasks where we have to learn temporal predicate parameters in addition to the temporal logic formulas from the samples. Our model Neural TLP learns these temporal relations between pairwise atomic events, with considerations of event noise well as computational complexity. Then given the inferred relations, the combinations of relations can be learned such that the correct composite event is inferred.
14

---

Under review as a conference paper at ICLR 2022
B OPTIMIZATION
We used the Adam (Kingma & Bal 2014) optimizer with learning rate of 0.001 for all our experi- ments and the default parameters described in their paper. The batch size during relational training set to 256. Each experiment was run for 100 epochs to train the relation parameters and weights W. For variable rule length search we tested longer epoch lengths, but didn't 't see much improvement in the validation past the first epoch. Therefore epoch of tuning done to optimize the combinatorial attention weights s while freezing all other relation parameters. All experiments were conducted on a server with Nvidia 2080TT GPU with 11BB of VRAM.
We made sure all our model configurations could fit on a single GPU of this size. The memory intensive component came from having different attention weights and combinatorial matrices C for each rule r E R learned. This meant that we had to limit the number of predicates combinations search over. Here e the number of most relevant predicates searched per rule and is the max number of predicates, so the number of combinations for the max rule length is (9) So for each variable rule length of 1,2.3,4 we chose 100, 100, 30, 25 respectively. Due to the larger memory requirements, we reduce the batch size during rule search to 64.
C BASELINES
C.I LSTM
Since we working with multi-hot labels, and co-occurring atomic events, it is difficult to parametrize existing HMM variants. Therefore we test a more neural recurrent baseline, LSTM (Hochreiter & Schmidhuber 1997), over the atomic event stream. To account for object invariance, marginalize over the timelines to produce the inputs MT Ex max MT....)) RR The compressed timelines and temporal indexing are done over MT to produce MA. From MA E g| we input x E | each step t: x;, h, c; = LSTTM(X; h; 1;-1), where e the cell state. Then we perform classification over the labels through linear layer using the last hidden state y Wh; + b. We test both a large (L) and a small (S) version with hidden dimensions [h| 512 and h 160 respectively.
We also test an attention based variant where the attention value for each hidden state computed using the hidden state well as the input atomic events at that step. This helps the model focus time steps that are not empty over long frame sequences, and led to better convergence.
r; [h; x;
04 tanh(b, (r. W))) softmax(a)
h
Σ h;
ý W2h + b
Here W1 E (R(b|++ +xx)xx and b, E Rdd project the concatenated data into attention dimension d, which was set 32 for all our experiments.
To extract the rules, we first enumerate all possible composite events. Then we synthesize each composite events as raw event timeline data. Unlike the original training data where there are multiple atomic events occurring simultaneously, for each composite event we have two atomic events occurring unambiguously before, during, or after one another. Then we pass the synthesized events into the model and take the argmax prediction as the corresponding rule label.
C.2 TEMPORAL MAP
While LSTMs optimize for label prediction, baseline to test rule induction is to maximize the posterior distribution of the enumerated composite event rules given the training labels. This is done by using Neural TLP but with two changes. First we freeze all model parameters done
15

---

Under review as a conference paper at ICLR 2022
in deterministic setting. Second, instead of learning the attention weights W count the occurrences of relations computed through MR and the training labels for each sample.
To compute W. start with all the observed time series samples T, Y (7) (y) Instead of operating over we are interested the grounded relational data R [MRj-r which can be extracted through the prior stages of the Neural TLP pipeline through the TLN network go with fixed relations, as described in 3.1]
Given the inferred MR data, have pairs of M y for each sample of le [1, Then compute the co-occurrences of grounded predicates and labels as:
O =
Σ
1 P.EM
vk [1,
M
i
W W. Σ, O
This W is used for rule learning in the same manner as Neural TLP for both fixed and variable length strategies. To compute mAP we always predict the top y most frequently occurring labels in the training set.
D CATER
We explore composite action prediction over complex videos in the CATER data set (Girdhar & Ramanan 2019). The CATER data set provides synthetic vidcos of multiple objects performing different actions simultaneously over the duration of the videos. The atomic events are con- junction of these movements E (rotate, slide pick-place, contain) and objects (cone, cube, sphere, snitch). Such atomic event is slide cone X' where | = 14 and temporal predicates r E {before, during, after).
The composite event rules are composed of single grounded temporal predicate (n 1) be- tween two atomic events. For example the underlying composite events in the video such as fore (pi lpick-pla alac cube, rotate snitch) is assigned to label 2, providing yy 301 unique composite events. Only the label along with the videos are provided during training, while the rules unknown.
Furthermore all these atomic events occur randomly during the video, thus multiple labels correspond- ing the the underlying temporal rule are active. This provides difficult, yet practical challenge: we know what coarse composite events occur during timeline, but we want to recover the underlying atomic events and predicates (rules) leading to each of these composite events (labels) as shown in Figure 5) Since the videos are generated with underlying rule templates, we can objectively test our models to recover these rules. For our experiments we used the train, validation, and test splits provided in original dataset.
D.1 PREDICTED ATOMIC EVENTS
Modality
Overall Acc
Rotate
Slide
Pick-Place
RGB
85.7% 96.6%
17.6%
3.9%
91.3%
RGB + Flow
76.9%
88.2%
96.9%
Table 6: Here are the accuracies for the predicted atomic events. The results show accuracy predicting the event only using the RGB image features and with the optical flow information. Further rules were used to predict contain.
Using the original CATER video data we performed experiments where the atomic events were provided and where inferred the atomic events. The latter case is more difficult, yet more realistic for rule recovery over collected data where noise exists. To infer the atomic events we first tune Faster R-CNN (Ren et al. 2015) over the object bounding boxes. In real world use cases it may
16

---

Under review as a conference paper at ICLR 2022
Loss
Relation Parameters
Projection
mAP
Rules@
R
Dent
T = 301; no conv
X
.693
.830
.966
Lent
t 150; kernel= 14 x 3, stride=
X
.682
.883
.953
Lens
t = 50; kernel= 14 x 7, stride= 6
X
.679
.873
.960
Lent
150; freeze conv
X
.669
.869
.953
Lent
= 150; freeze a
X
.656
.671
.950
Lent
t 150; freeze B, Y
X
.658
.681
.913
Cent + L1 Lent + L1
150
X .681
.915
.958
t = 150
.675
.913
.956
Table 7: We ablate the different components of Neural TLP over the CATER predicted atomic events data. We start by determining the efficacy of quantization, where is the dimension after quantization. Then we test the contribution of each parameter in relation learning. Finally test additional optimization strategies.
be possible pre-trained detectors lift these bounding boxes, but they typically don't cover synthetic objects. We use these bounding boxes to generate visual feature over the cropped image using a pre-trained ResNet-50 (He al., 2016). We use the image feature and optical flow from the previous and next frame to predict the shape and movement.
There is additional difficulty performing this inference and we present the event accuracy in Table 6 Furthermore, the action contain is hard to distinguish from pick-place so additional rules are used to disambiguate pick-place from contain, leading to additional uncertainty. The rule classified contain if the movement is pick picck-p ck-pkkcceeeeecl and the bounding box of the moving object close to the bounding box of a static object. Since each object has its own atomic actions we assigned a timeline for each object, tracking max of k = 30 objects.
D.2 MODEL ABLATION
We ablate model over the predicted atomic event data in Table In the first section we test different convolution kernel sizes which contains ld covolution weight that learned per atomic event | 14 and stride. We see that convolution smoothing is beneficial at the reduced dimension of = 150 and use this for further ablations.
In the next section test how much each relation parameter contributes to the result. The convolution weights are fixed to 1, and we see that this case they contribute minimally to rule induction. real world cases with more complex events, these learned convolutions could potentially be more useful if systematic noise exists. For the convolution scalar a crucial to shift the compressed timeline to correctly identify event time intervals, as seen when it is fixed to Similarly when shift B 0 and scale = 1, it misses on variations in the relation data.
In the final section we test L1 regularization produce sparser results for W. This shows enable better rule induction. We test projecting W between [0, 1] and while we got similar results, we notice that the optimization converges faster and in more stable fashion. We primarily focused on rule precision by looking at Hits@1, 1, while increasing recall produced similar results for all methods the setting where the rules only have predicate.
D.3 DATA ABLATION
To test different rule lengths n, training samples m, and event complexity y, we generated more CATER-like like data to explore these dataset statistics. Here we use the same atomic events and relations as in the original dataset. Due the number of possible combinations of the dataset statisties, expensive to generate the video data and run our vision pipeline to generate the predicted atomic events. Instead we opted to simulate the atomic event predictions directly. For each sample we:
1. Randomly sample j rules containing up predicates.
17

---

Under review as a conference paper at ICLR 2022
Max Len n
y
Model
Variable
Len 1
Len 2
Len 3
Len
TL.P MAP
2.7.04
.77.04
.54..07
.53+.07
TLP
444.08
280..5
.09=.01
4.0
MAP
.17.000
.34.000 .00
.00g.01
TLP
.15.002
235..0
.05..02
.01+.01
3
5.3
MAP
.11.000
.33..00
.01
.00+.00
TLP
104.01
362.05
.02+.03
.00
.00
4
10.0
MAP
.09
334.01
.04..03
.000 01
.00+.00
Table 8: For samples compare model Hits 10 performances across different rule lengths. We observe the total variable length Hits for rules of all lengths that have to be learned. We break down the combined performance into the individual performance per fixed rule length Len i, that compose the total variable performance.
Score versue Sampled Eventss
Score venur Sample Complesit
TLP
MAP
TLPLe12
0.8
MAP Lan
0.8
TiP1
MAP L
La
MAP
04
02
0.2
00
1.0
1.5
2.5
3.0
4.0
4.5
2002
4000
6000
8003
10000
Sampled Evertts
Sempies
Figure 6: We compare Neural TLP (Blue) and Temporal MAP (Red) with varying number of active labels and samples. Each curve represents the overall variable length accuracy up rule length n. the left we compare the variable length performance when we sampled more event rules per video, increasing the active labels and timeline noise with fixed 10000 samples. On the right we compare performances as the number of samples increase and fix j = 1.
2. Synthesize a timeline for each rule, where atomic events are placed in the timeline consistent with the rule. Fach atomic event prediction sampled from normal distribution, with means centered around the atomic event detection accuracies in Table
3. Based on the synthesized timeline over sampled rules, add any consistent rules induced by the synthesized timeline. These additional consistent rules contribute to our estimate.
4. Gaussian noise was also added along the timeline where events did not occur to simulate detection noise.
Given the synthesized timeline and the consistent rules, the corresponding MT and labels can be generated for each sample. For validation and testing data we generated 2500 samples each, regardless of the number of training samples.
From this base data set, we first vary the max rule length n of the rule samples and fix sample 100 total rules up to the max length for
to isolate
the effect of the rule length. For every level consistency. Breaking down the variable length accuracy in Table we see Neural TLP provides better performance for shorter rules, while rule learning becomes more difficult for longer rules.
We also test the event j and data points m sample complexity in Figure 6 As we sample more events more noise is added to our timeline and makes it harder for models recover the underlying rules. Inversely we also show the performance increases with the number of samples, where Neural TLP is more sample efficient.
18

---

Under review as a conference paper at ICLR 2022
E MIMIC-III
after (oral water, spo2_sao2 high) A after (oral water, paco2 high) after (heo3 high, spo2_sa02 high) A after (spo2_ sao2 high, calcium high) after (hoo3 high, spo2_sao2 high) A before (hoo3 normal, pao2 low)
Table 9: Induced rules for normal urine, verified as correct or plausible by doctors.
oral water spo2_sao2 pao2
patient drank water
pulse oximetry SpO₂ and blood gas SgO₂ in oxygen partial pressure of blood oxygen PgO₂
paco2 partial pressure carbon dioxide in the blood P, 202 body metabolic bicarbonate HCO, patient calcium indication
hco3
calcium
Table 10: Descriptions of MIMIC atomic events in the lifted rules from Table 9
For the healthcare data we use the MIMIC-III dataset (Johnson et al., 2016). The dataset contains measurements, vitals, and medications for intensive care patients, and is already de-identified. We first filter the patients containing ICD codes corresponding to sepsis and sample 2k patients.
Instead of inferring patient survival directly, the medical doctors suggested to look at circulatory indicators, specifically urine flow. Additionally they helped us identify subset of patient vitals and measurements use for urine flow, leading to our | patients we identify timepoints containing urine information. This serves as a label, and all events before the urine event are timeseries inputs. Since patients had multiple indicators of urine during their stay we have 3.8k samples of time series and urine label (normal or low). During training we created an 80/20 split for training and validation respectively.
boolean atomic events. From the 2k
After training and iterative feedback from the doctors, successfully lifted useful indicators of urine flow as presented in Table 9 description of the atomic events is presented here Table TT
19


Reviews:
[{"review_id": "wwoQAR4jb--", "reviewer": "Reviewer_54Fe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n\t1.  formulating combinatorial rule space search as predicate selection indicator\n\nWeakness\n\n\t1. The proposed model's parameterization depends on the number of events and predicates making it difficult to generalize to unseen events or required retraining. \n\t2. The writing needs to be improved to clearly discuss the proposed approach.\n        3. The experiments baselines are of the authors' own design; it lacks a comparison to the literature baselines  using the same dataset. If there is no such baseline, please discuss the criteria in choosing such baselines.\n\n\nDetails: \n\n\t1. Page 1, \"causal mechanisms\", causality is different from temporal relationship. Please use the terms carefully. \n\t2. Page 3, it seems to me that M_T is defined over the probabilities of atomic events. The notation as it is used not making it difficult to make sense of this concept. Please consider providing examples to explain M_T. \n\t\t\n\t3. Page 4, equation (2), it is not usual to feed probabilities to convolution. \n\t\ta. Please discuss in section 3 how your framework can handle raw inputs, such as video or audio? Do you need an atomic event predictor or human label to use your proposed system? If so, is it possible to extend your framework to directly have video as input instead of event probability distributions?  Can you do end2end training from raw inputs, such as video or audio? (although you mentioned Faster R-CNN in the experiment section, it is better to discuss the whole pipeline in the methodology).\n\t\tb. Have you tried discrete event embeddings to represent the atomic and composite events so as the framework can learn distributional embedding representation of events so as to learn the temporal rules?\n\t4. Page 4, please explain what you want to achieve with M_A = M_C \\otimes M_D. It is unusual to multiple length by conv1D output. Also please define \\otimes here. I am guessing it is elementwise multiplication from the context.\n\t5. Page 4, \"M_{D:,:,l}=l. This can be thought as a positional encoding. It is not clear to me why this can be taken as positional encoding?  \n\t6. Page 6, please detail how do you sample top c predicates. Please define what is s in a = softmax(s). It seems to me the dimension of s with \\sum_i (c i) can be quite large making it softmax(s) very costly. \n\t\n\t\n", "summary_of_the_paper": "This paper proposed a neural nets based approach for Temporal Logic Programming. A key claimed contribution is on formulating combinatorial rule space search as predicate selection indicator vectors assignment so as to make it differential to enable end2end training from atomic event probabilities as inputs.  ", "summary_of_the_review": "This is a good initial attempt to attack the neural temporal logic programming. However, the steps in the proposal --- temporal compression, predicate modeling, composite event prediction, combinatorial inference, and rule reduction --- requires more clarity and refinement to reach a publication state. Overall, the proposal has little element of neural networks. To me it is more on matrix formulation of ILP problem. It is Okay to me but please consider renaming the paper title. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kUXLlhiaQ7", "reviewer": "Reviewer_pPed", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem of learning temporal rules from data, for which many applications exist (as the paper convincingly motivates). \n\nA very strong point of the paper is to learn interpretable rules using modern neural network machinery. This can certainly inspire future work. \n\nThe paper is not always easy to follow. It would have been useful to more explicitly state the aims of the proposed method. For example, on page 4 the paper refers to Temporal Relation Networks, but they are dismissed because \"timelines can be long in our setting\", yet, it is not clear what this refers to. For me, it was also quite unnatural to say that composite events are a-temporal (e.g. in the baseball\nexample on page 2). Why does it make sense that you cannot say anything about the timeline of the 'strike' event? Also in the remainder of Section 3.1, various choices are made, which were not all obvious to me. \n\nIn the structure learning part, I was missing the connection to probabilistic rule learning approaches (e.g. ProbFOIL). The essence\nseems to be to learn rules where the temporal atoms have a probability attached to it, unless I am missing something.\n\nIn the evaluation, the paper convincingly shows that variations of the proposed method performs less well. What was not convincing to me\nwere two aspects:\n(1) Why previous approaches are unsuitable for the proposed settings, e.g. probabilistic rule-learning, sequence mining, etc. \n(2) I have doubts about the validity of the medical example. For example, isn't it the case that 'oral water' is simply responsible for \nurine flow rather than after(oral_water, spo2_sao2=high) ? I can understand that a doctor would say that you \"captured important explanatory variables\" in the proposed rules, but that does not mean that it is in any way true. In fact: doesn't this show a downside of the approach in the sense that only temporal events can be used in rules?\n", "summary_of_the_paper": "This paper presents an approach for learning temporal rules from data. The idea is to extract atomic events, and their temporal relationships between them. Subsequently, composite events are learned using composite event labels for supervision. The whole approach is formulated as an optimisation problem, after which standard techniques are applied to solve it.\n", "summary_of_the_review": "Overall, I thought this is an interesting contribution to the ICLR conference. I do have some doubts about the completeness and validity of the experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fFqTaImtBMi", "reviewer": "Reviewer_X4hj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The paper is well written.\n- The proposed solutions, namely the differentiable layers to predict the temporal relations between atomic events and the post-hoc strategy to induce propositional rules from the last linear layer, are simple but non trivial.\n- Results are promising.\n\n**Weaknesses**\n\n- It is not clear how the work positions with respect to recent literature in spatio-temporal reasoning and video question answering. For example, the authors should have a look at [1]-[2]\n- While experimental results are interesting and promising, they provide only a partial picture, as they neglect recent neural advances. Indeed, comparisons are made against only simple baselines and should be conducted against more “structured” neural approaches, see for instance graph-based approaches [1]-[2], where code is available.\n\n**Questions**\n\n1. How scalable is the proposed strategy in terms of number of objects, number of atomic events and compositionality of the target events?\n2. As far as I understand, the proposed strategy considers only the temporal relations among atomic events and objects are considered independent of each other. Is that correct? If so, why is that the case?\n3. While I understand the intuition about the strategy to extract the time intervals of atomic events (as shown in Figure 3), I do not see how well this strategy can perform in the continuous and more noisy setting. How reliable are the obtained estimates?\n4. In CATER experiments, is the perception component (i.e. Faster R-CNN) trained jointly to the Neural TLP  or are two components trained separately?\n5. The names parameter and structure learning are a bit confusing in this context. In their classical definition, parameter learning considers learning the parametric relations among variables in a given graph. In Neural TLP, the temporal relations are not given, rather learnt. Why is this considered as parameter learning?\n6. Can you mention something about the logic expressiveness of the induced rules?", "summary_of_the_paper": "The paper proposes an end-to-end differentiable strategy (called neural TLP) to learn unknown temporal relations between atomic events (like after(miss, swing), “miss occurs after swing” in the baseball example), subsequently used to predict composite events (like strike). The strategy consists of a cascade of a smoothing stage to filter  out noise from the input time series, an interval time extractor, a stage predicting temporal relations (such as before, after and during) and a linear output layer. Furthermore, the paper proposes a post-hoc procedure to extract propositional logical rules relating atomic events to composite ones from the last layer. The performance of the proposed strategy is evaluated on video recognition (CATER) and healthcare (MIMIC-III) datasets against two baselines, namely a LSTM neural net and a simplified version of the proposed strategy.", "summary_of_the_review": "The paper is clearly written and easy to follow. I like the idea around the proposed differentiable operators used to predict the temporal relations among atomic events. Also, the experimental results are interesting and promising. However, they are preliminary. I would like to see more in-depth analysis on the correcteness of the time intervals for atomic events, the scalability of the proposed strategy and a comparison against recent structured neural approaches for spatio-temporal reasoning. Furthermore, the authors should discuss and relate their work against existing literature in the field of video question answering.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MLnWEwIwBXU", "reviewer": "Reviewer_9nKB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: the two-step approach (i.e., parameter learning followed by structure learning) does seem novel for the problem. \n\nWeakness: the empirical results are rather unconvincing. Not only that the baselines (two LSTMs) are rather weak, but also that the proposed approach seems to be even worse than the baselines in terms of mean average precision (which is the mainstream metric used on CATER) in Table 1. Besides, Table 3 suggests that the proposed approach is marginally better than logistic regression.\n\nMinor:  In figure 2, “(1) parameter learning” seems to be in the wrong place. That should refer to predicate prediction according to page 3, paragraph 1.", "summary_of_the_paper": "The paper proposed a new approach called neural logic programming to recover composite events from complex time series data using atomic events and their temporal relations. ", "summary_of_the_review": "Although there is some novelty in the proposed approach, the empirical results are too weak to validate the effectiveness of the approach. I do not think the paper in its current form is good enough to be published in ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper gives a framework for learning temporal logic rules from noisy unlabeled data. The key novelty is a formulation of combinatorial rule search as an end-to-end differentiable problem. The method is evaluated on a video dataset and a healthcare dataset. 

The reviewers liked the high-level ideas behind the paper. However, the conclusion was that the experimental results, while interesting, are still somewhat preliminary (in particular, the baselines are weak).  I agree with this point and am recommending rejection this time around. However, I urge the authors to develop the paper further and submit to the next deadline.
Decision: Reject