Don't Pour Cereal into Coffee: Differentiable Temporal Logic for Temporal Action Segmentation
Authors: Ziwei Xu
Yogesh S Rawat
Yongkang Wong
Mohan Kankanhalli
Mubarak Shah
Venue: NeurIPS 2022 Accept (2022)
Keywords: Temporal action segmentation
linear temporal logic
neurosymbolic methods

Abstract: We propose Differentiable Temporal Logic (DTL), a model-agnostic framework that introduces temporal constraints to deep networks. DTL treats the outputs of a network as a truth assignment of a temporal logic formula, and computes a temporal logic loss reflecting the consistency between the output and the constraints. We propose a comprehensive set of constraints, which are implicit in data annotations, and incorporate them with deep networks via DTL. We evaluate the effectiveness of DTL on the temporal action segmentation task and observe improved performance and reduced logical errors in the output of different task models. Furthermore, we provide an extensive analysis to visualize the desirable effects of DTL.

Don't Pour Cereal into Coffee: Differentiable Temporal Logic for Temporal Action Segmentation
Ziwei Xut* Yogesh Rawat Yongkang Wong" Mohan S Kankanhalli Mubarak Shah
i School of Computing, National University of Singapore 8 Center for Research in Computer Vision, University Central Florida
{ziwei -u www.mohhhmmmmcmm -:oomp.e.om.mm.o nus mus.edu..a.. edu.sg ag yongkang wong@nus. edu.sg
{yogesh, cchah)@rrev.ucf.ee .ddu.euu
Abstract
We propose Differentiable Temporal Logic (DTL), model-agnostic framework that introduces temporal constraints to deep networks. DTL treats the outputs of network truth assignment of temporal logic formula, and computes a temporal logic loss reflecting the consistency between the output and the con- straints. We propose a comprehensive set of constraints, which are implicit data annotations, and incorporate them with deep networks via DTL. We evaluate the effectiveness of DTL. on the temporal action segmentation task and observe improved performance and reduced logical errors in the output of different task models. Furthermore, we provide an extensive analysis to visualize the desirable effects of DTL.
Frames
GT
pour coffee
pour milk
pour_
pour water
pour
W/ DTL
pour coffee
milk
Figure 1: A video of activity "coffee preparation". The colored bars, from the top to the bottom, show the ground truth (GT), the predictions from baseline [[S]], and the predictions from the base- line trained with DTI, respectively. Note that the baseline model erroneously predicts pour_ceree when DTL solves this problem with an exclusivity constraint between pour_coffee coffee and pour_ pour_cereal.
Introduction
Recent years have witnessed significant advances in video action analysis tasks such as action recog- nition [29. 155],, action detection [49]], and temporal action segmentation [[5, [I]. This can be credited to the availability of large-scale datasets [Р7/] [3] and the development of effective deep visual back- bones [EL 50, LZ] Although data-driven backbones can capture useful spatio-temporal features, learning a large number of highly diverse temporal dependencies and correlations over long time spans can be very challenging. In existing approaches, these temporal dependencies are not explicit to the model: it is possible provide framewise annotation during training, however, temporal con- straints like "event X has to occur after event are still implicit in the annotations and not explicitly enforced. Even though these constraints could be statistically leamned from large amount of data, a pure vision model could still be confused by different actions with similar visual appearances. An
TThi work done when Ziwei Xu was visiting the Center Research Computer Vision.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

example of this is shown in Fig. MI where the model confuses pour_ cour_cereal with pour_ milk because both actions involve holding carton.
In this paper, we propose a solution to this problem by employing temporal logic to apply declarative temporal constraints the output of deep networks. Linear Temporal Logic (LTT.) [[3]], for example, defines operators that describe necessities, possibilities, and dependencies of actions in a series of actions. Checking network's output against those constraints (a.K.a. model checking) tells us if the predicted actions are logically correct. Logical correctness can be used as an additional training objective. According to prior works in the neuro- ceuro-symbolic community [541, 52, [8], applying logic constraints improves the performance of deep networks in tasks for graph data [ and images [[2]] by reducing logical errors in the output. Although one would anticipate similar benefits when using the temporal logic for action analysis, most of the focus in literature is still on non-sequential data.
In this work, focus the temporal aspect and consider temporal constraints in videos. Inspired by the foundational works in logie-based losses [54L [&], we propose a Differentiable Temporal Logic (DTL) framework, which uses an extended definition of linear temporal logic to constrain the output of action analysis models. At high level, DTL treats the model output as a truth assignment of variables in LTL formulae. A differentiable evaluator performs model checking on the outputs and yields a satisfaction score, which measures the consistency between the constraints and the outputs. As we optimize the satisfaction score through standard optimization methods, the constraints are enforced on the deep network. Different from existing work [53]], in DTL the relation between formula evaluation and formula satisfaction deterministic, which means that DTL is logically sound. Moreover, we propose comprehensive set of constraints covering both temporal and non- temporal dependencies between actions, and show how they are represented using DTL. We evaluate DTL on the challenging temporal action segmentation task, where modeling temporal dependencies between actions is crucial. The efficacy of our method shown by the improved performance of different task models when constraints are enforced through DTL.
The contributions of this work are as follows:
We present Differentiable Temporal Logic (DTL), a framework providing a model- manner of introducing temporal logic constraints to deep networks (cf. Section B.2).
We propose DTL constraints to deseribe a wide range of relations between actions, which can be automatically procured from dataset annotation (cf. Section B
Experiments with different task models on the temporal action segmentation task show the efficacy of DTL. In addition, we provide an extensive study to show the effect of DTL constraints on task models.
2 Related Works
2.1 Temporal Action Segmentation
temporal action segmentation model takes a set of video frames as input and predict the action category for each frame. The model needs capture both short and long term dependencies be- tween action categories. Earlier methods [3], [G]] use sliding windows to model changes in visual appearance in short time. Long-term dependencies are captured from those short-term information by sequential models like hidden Markov model [[%]] and recurrent networks [52]] 25 Despite the remarkable performance, when these models are used to process long videos they still face forget- ting issues and heavy computation burden. Temporal Convolutional Network (TCN) [36 Ei [b]], [8] enables efficient modelling of temporal dependency in variable time spans with its flexible receptive field. Many advancements are made based on MSTCN [[SS], which used stacked dilated temporal convolution layers for temporal modelling. For example, ASRF [28] uses a boundary regression task on top of MSTCN to improve the quality of segmentation. Huang et al. [[6]] proposed graph- based dependency model to improve the modeling of relations between actions. Finally, Transformer [53I]] is introduced to this task and shows superior performance in [56]] Our view is that these methods can be complemented by DTL,, since many declarative constraints are difficult to learn from the data. On the other hand, DTL explicitly enforce these constraints the task model during training.

---

Tai Yor
Task Model g
LT
Task Loss
1 ayy
(a)
Input Frames
Output y
Temporal Logic Loss
water mast after take_ add teabag after cup: navez pour pour_juice add teabag:
(-pour_wae water take_cup)
A M-add_teabel teabaa cupe_cup)
A(F add_teabag + -F pour_juice)) A...
f(1, ū)
LTL
(b) Declarative Constraints
Temporal Logic Formula &
Formula Evaluator f
An overview DTL framework. (a) shows a typical action segmentation pipeline, where task model g takes input frames and output segmentation y. for each action class. A task loss CT is minimized during training. (b) shows the logic evaluation part of DTL. A set of declarative constraints actions are converted into temporal logic formula th. A formula evaluator f takes vi and y. as input, and evaluates the consistency between the two, resulting in a logic loss LTL.
Figure 2:
2.2 Temporal Logic and Action Analysis
Temporal logic is large family of logic systems that specify the relations between timed events. Linear temporal logic (LTL) [+5] is one of the earliest temporal logic. It is propositional modal logic that states the necessity and possibility about events in linear discrete succession of time steps. Later variants of LTL include Metric Temporal Logic (MTL) [40], which introduces time units as extra parameters temporal operators, and Signal Temporal Logic []4]], which specifies temporal properties for continuous signals. Allen's 's interval algebra [2] and Interval Temporal Logic (ITL) [22] specify the relations between intervals and their composition. Temporal logic has been used to specifyy constraints for program verification [ل robot control [[33 27, [ and linguistics [[I]]
However, its application action analysis remains scarce. CASE [2]] uses ITL as structural language to describe atomic actions for event classification. T-LEAF [> is the first (and so far the only, the best of our knowledge) attempt apply temporal logic action analysis. It uses an embedding model to incorporate LTL formulae when training deep networks for action classification, which has little guarantee on logical soundness and only handles a small set of constraints.
DTL employs LTL. for temporal constraints because LTL. is: (1) more feasible than its successors with more operators and (2) expressive enough benefit action analysis. principle, DTL can be used with any logic system that is sound and whose formulae can be evaluated in differentiable manner.
2.3 Logic Constraints on Neural Networks
Applying symbolic constraints deep networks has been drawing increasing interest [25 339 erd. 20], yet it remains an open problem. An important challenge is that logic propositions generally take discrete truth values, while end-to-end differentiability is usually required optimize network. A workaround for this challenge is to find a continuous consistency measurement between network outputs and constraints. Existing methods constraints are treated as data or procedures. The first branch is embedding which treats constraints and network outputs as data and project them into continuous space. where the distance between the two is used as a measurement. For example. LENSR [>2] represents logical constraints in Deterministic Decomposable Negation Normal Form (d-DNNF) [ and uses GCN [[I]] as the embedder. The second branch is procedure-baat compiles constraints into computational procedures over network outputs and uses the result as consistency measurement. For example, semantic loss [54]] compiles formulae into logic circuits [Р]] along which the probability of output satisfying the formula computed as the measurement. We design DTL procedure-based framework instead of an embedding- based one, since the latter generally lacks soundness guarantee: high similarities in the embedding space does not guarantee logical correctness (or satisfaction).
be categorized into two branches based whether
[65), GUl 52, 53 A2].
[54g] [8.
I9 12, 24]. which
3

---

3 Method
We consider the action segmentation task in video clip with L frames and N candidate actions A = +[a....,,a' .N., ). As shown in Fig. 2 we assume that there is deep task model parameterized by O, which takes frames 20,-.. ....... -t,1 as input, and outputs "a....., -ay, where yo. E R is the unnormalized score for the presence of action a; throughout the L frames. We assume Ja. > 0 indicates a, presents at time I, and Da., I 0 indicates the opposite. Apart from the ground truth label ya....... Maxx there is set of constraints that describe the relation between actions. The design goal of DTL is to incorporate these constraints into the task model 9. so that y complies with both the ground truth y and the constraints. In order to do so, we introduce temporal logic representation V, and an evaluator f that enforces the constraints in formula va E v on g. through its outputs y.
3.1 Syntax of Formulae
The definition of V is an extension of I Linear Temporal Logic (ITT.)) [8S]. A formula tp E v takes any of the forms separated by "I" below:
1 == True False| a -4h (t1 A 4/2 (11 V 2/2) X Fe (+) W122) S112), (I)
where E A is the atomic proposition, and 1/1, 1/2 E V. The connectives X (NEXT), F (EVENTUAL), W (WEAK_UNTIL), and (SINCE) are modal operators that form the temporal relations between propositions. In our context, atomic propositions represents action Note that the definition in Eqn. (1) is recursive. For example, if aj E V, then X.a E V. Semantically, X is satisfied when proposition to is satisfied in the next time step. F is satisfied when se is satisfied by the end of the sequence. 4/1 Wa/z being satisfied means that t/ must always be satisfied until 4/2 is satisfied (and thy might never be satisfied). 2/1 Suby being satisfied means 2/1 is always satisfied after the is satisfied. Section D.I provides a more formal definition of these operators.
3.2 Formula Evaluator
A formula is said to be satisfied by truth value assignment if the assignment is semantically com- pliant with the constraints. In the context of this paper, each atomic proposition a; is assigned D..- Satisfiability is determined by evaluation, which is a function f of a formula yi and ğ:
fe 9) = f(,,, U DoraN, : x R RL-t x
x IR I,
N times
where refers to the Cartesian product. The two arguments of f formula sb and model output aays Parameter f E [0, L - 1] is the start time of the evaluation. For example, f 1= 2 means that the evaluation is between a and the prediction starting from the third frame, i.e. Vo The result of f is a satisfaction score that measures the consistency between ý and vb.
Eqn. (L) is abstract and must be detailed for all possible forms of formula vi can take in Eqn. (I). We aim to expand the definition so that v is logically sound, i.c. there is a determined relation between fa(v), ÿ) and the satisfaction of v given ÿ. Specifically, imply that t is satisfied by ÿ at time t. Moreover, y must be differentiable to be incorporated with task models. Towards these goals, we first define the evaluation for constants and atomic propositions:
would like fe ÿ) 0 to
fe 9) = +00, (False, 9) = -0, [a 9) = Da.1
Indeed, the evaluation result for True and False will always be positive and negative, because the former is always satisfied and the latter is never satisfied. Note that if sh = it is satisfied at time t when Go.t > O, i.e. action a happens at time L.
Next, we define the evaluation for operators "-"" (negation), "X" (logical and), and "V" (logical or):
fe( 9) = (1/1, 9).
(4)
fe(11 A 12. Ø) = min" (f((t), 0), fe(22, 9 fe V th.. x) = max" 0). f1 0)}
(5)
(6)

---

pour water (a:) must after take_cup cup (az).
V (aa,Waz)
W
Declarative Constraints
Temporal Logic Formula
DAG
42
1 9) 1
fou-1(-A, 8o. 10c-1
(5) fo( We min' (... min' Vo.2( (a,
min'
W
(False)
Pay
(3) fau-1(a 9) 80pa-1
= is the min. value s.t. fa(a = True
Figure The evaluation of an example constraint on two actions over time steps. The con- straint is first written as formula v = (~a1 Waz), then represented as a DAG. (1) (5) show how fa Waz, 4) is computed. (1) shows the output ÿ of task model, where green boxes indicate pos- itive values (True) and grey boxes indicate negative values (False). (2) and (3) show the evaluation of leaf nodes d and dg, which is the start of evaluation. (4) takes the evaluation results for a from (2) and evaluate =a, following Eqn. (9). In (5), the W node uses the result for m0g from (4) and d2 from (3) to perform the evaluation for (-a1 Waz) following Eqn. (.).
where y is a parameter function min min" (xi:z-1 4 E r= which approximates the minimum value of [I1.E-1} [R], and max? (81.L-1}
-mminn (- It can be shown [G] that +om min" (=i_((1 min(#1.L-1} }. In Eqn. (.I) the operator flips the sign of fe(11, 9). reflecting the negation semantics of -. In Eqn. (5). min" (fe(1), 9), (( 9)} will be negative (False) if th 1/2 both are False, and will be positive (True) iff. both 1 and 1/2 are True. This is consistent with the semantics of A. The same rationale applies to maxx for V in Eqn. (G).
Finally, we define the modal operators X, F, W, and S that unique in V. Intuitively, evaluating Xah, is equivalent to evaluating 1/1 at the next time step. Ful means there is at least one time step which sh is satisfied. 1/1 Wrlp and v/1 Si/2 require to be always satisfied in the time period specified by 2/2. We formally define them as follows:
fe(X 4) = (1+,1)))))
(7)
Je(Fu D) = max ( f)}
(8)
falt Weiz. ④) = min" (fe 4) where & 2 / the min. integer s.t. (122 4) > 0,
(9)
filbiSuuzz 4) min (f ()), where 2 is the min. integer s.t. fe(22, ÿ) > 0. (10)
In Eqn. (9), min? {ff (v1, })} is positive (True) iff. all elements of {fux (1/1, 8)} are positive, which means that 4h stays True from time to time k. This is consistent with the semantics of W. We use min? for S in Eqn. (II)) for the same reason.
The definitions in Eqn. (0I) provide a soundness guarantee for V, allowing us to use f1 9) 0 optimization objective to enforce the constraints in & on task model. Formally, when 7 + oc, the approximated evaluation (because of min' ") becomes exact, and the following theorem true by construction (a proof is provided in Section D.3);
Theorem 1. (Soundness) With v E V 7 oo: if fe(,, 9) > 0, then 1 is satisfied by ý at time t.
In essence, the evaluation process propagates network predictions from atomic propositions to logic operators and finally to the formula. Equivalently, we can represent formula directed acyelic graph (DAG) where each leaf node is labeled with True, False, or action a; and each internal node labeled with logic operators. The edges of the DAG point from child nodes to their parents, along which the truth value propagates following Eqn. (GI(II). Fig. illustrates how a constraint " person cannot pour water before taking cup" is converted DAG and evaluated.
3.3 Constraints
This section discusses different types of constraints we find useful for action analysis and the way they are represented using V. constraint can be categorized into either a temporal or a non- temporal constraint based on whether it states the possibility and necessity of an action in a specific
5

---

time period. We propose two temporal constraints, namely Backward Dependency (BD) and For- ward Cancellation (FC), and two non-temporal constraints, namely Implication (Ip), and Exclusivity (Ex). We also introduce how constraints of these types are curated from the training annotations.
Backward Dependency (BD) It is important to specify the proper order of actions because it usu- ally determines the semantics of an activity. An order is generally related to temporal dependency: An action cannot be performed if its prerequisite actions did not occur. A way to describe this dependency "one action must occur before another". We write this constraint as
VBD = (a., ME a2) - (a..,)))B(Fa A Fay) + a. -aWaa))
where (X Y) means "X implies Y" and is equivalent (-X V Y). This logic expression means the following: a; and a, occurs in the same video, a, must not occur until aj occurs. Set B contains action pairs that satisfy this constraint. In practice, (a1, .2) B if (2 always occurs before a1 for any co-occurrence of (1 and 02 in the annotation.
Forward Cancellation (FC) Apart from backward dependence, actions can make some other actions impossible in all future time steps. For example, in video of salad preparation, the ac- tion serve_salad_ to plate marks the end of the preparation. Once this action occurs, actions like cut lettuce cut tomato should not occur thereafter. We write this constraint
V/C Ma d:) (eL.a))E=FFa Fay) + (-a,Say);
which reads "if a; and a, occur in the same video, a, cannot occur after the occurrence of az". Set F contains action pairs that satisfy this constraint.
Implication (Ip) There could be some actions that are semantically dependent but temporally independent: these actions are necessary to complete an activity, but the order not crucial. For example, in a video on juice preparation, the person must take_cur _up and peel_ pranger but these two actions are not temporally correlated. We write this constraint as
1P = Aezlaa a,) = Ma,a))z + Faj).
Set I contains action pairs that satisfy this constraint. In practice, (a1, .2) I if a2 always occurs á1 occurs in the annotation.
Exclusivity (Ex) As opposed to what implication constraints describe, some actions are mutu- ally exclusive: they will never co-occur in the same video. For example, if we know that video contains a single activity that is "coffee" or "frying eggs", then pour coffee cannot happen if put_egg__igg into plate occurs at any time in the video. This constraint is written
tbx xxx Ex(a,, a²) Massa))xx + -Faz).
Set X' contains pairs of actions that never occur the same video. Finally, we connect all constraints using A as vé = 1/RD A vFC A thip A tTxx
3.4 Training with Constraints
During training, we hope that the output ý is constrained by both ground truth and logic constraints described by v. The constraints from ground truth are enforced by task-specific task loss L (ū y). for example, framewise cross-entropy loss for the temporal action segmentation task. For logic constraints, we treat ÿ as an assignment of sb and from Theorem II we know fe((', ÿ) 0 if satisfies eu from time t. Therefore, for any g we can minimize the following objective:
== XLT LT(U. v) Xo (Jo( 0)),
(11)
where LT is the loss term for the target task, the weight of the loss of logic LTL. and (w) (e)++)=) penalizes negative evaluation results. We set / in fe(s,, ÿ) in our experiments, since we require the prediction satisfy the constraints from the first frame. Note that possible to set t to different values so that the constraints be applied flexibly at different temporal locations.

---

Table 1: Results of action segmentation on 50Salads dataset.
Task Model
Edit
FI@ 10
F1
Fl @0@
Acc
Base
55.2 + 2.3
63.7 = 2.3
60.5 = 2.7
53.4
79.0 2.4
GRU
Base + DTL
62.1 1.6
69.3 = 1.4
66.5 1.8
58.9 1.9
80.3 2.2
Gain
7.1 + 2.8
5.6 = 2.7
6.0 = 3.2
5.5 3.6
1.3 3.3
Base [1S
67.9
76.3
74.0
64.5
80.7
Base (Rerun)
69.5 1.7
75.7 = 1.7
73.0 = 1.9
80.0 2.2 1.4
MS-TCN
Base + DTL
70.5 + 1.0
78.3 = 1.3
76.5 = 1.1
67.6
81.5 1.5
Gain
2.0
2.1 2.6
3.5
2.2
2.9 3.0
2.0
Base [56]
79.6
85.1
83.4
76.0
85.6
Base (Rerun)
76.9 - 0.9
83.6 = 0.9
81.5 = 0.8
73.9 # 1.3
84.2 1.2
ASFormer
Base + DTL
80.5 1.5
87.1 1.3
85.7 = 1.2
78.5 + 1.6
86.9 1.5
Gain
3.6 + 1.7
3.5 1.6
4.2 + 1.4
4.6 + 2.1
2.7 1.9
Table 2:
Results of action segmentation on Breakfast dataset.
Task Model
Edit
F1
F1 @25
F1 @55
Ace
Base
56.8 2.0
53.3 = 2.3
48.4 + 2.5
38.4 2.1
70.0 1.7
GRU
Base + DTL
58.4 + 1.7
56.5 = 1.1
51.4 = 1.3
40.7
70.3 1.1
Gain
1.6
3.2 2.6
3.0 = 2.8
2.3 2.9
2.9
2.0
Base [LS]]]
61.7
52.6
48.1
37.9
66.3
Base (Rerun)
71.2 + 1.4
71.7 = 1.3
65.7 + 1.5
52.3 + 1.8
71.3 1.2
MS-TCN
Base + DTL
71.6
73.0 0.4
67.7 + 1.2
54.4 0.8
72.3
Gain
0.5 1.8
1.2 = 1.3
2.0 = 2.0
2.1 2.0
1.1 1.3
Base [56]]
75.0
76.0
70.6
57.4
73.5
Base (Rerun)
76.2 1.4
77.8 1.2
72.9 + 1.6
60.5 + 1.7
75.0
ASFormer
Base + DTL.
77.7
78.8 + 1.1
74.5 + 1.5
62.9 + 1.6
75.8 0.9
Gain
1.1
1.1 1.6
1.6 2.2
2.4 2.4
0.8 1.4
4 Experiments
In this section, we first assess the proposed DTL using the temporal action segmentation task. Through this task, we show that DTL can provide dependency information to improve the perfor- mance of action analysis models, in both quantitative and qualitative manners. Then, we perform an ablation study show the effects different types of constraints. Finally, with a gradient-based analysis, explain how DTL affects the task model. All experiments are run with PyTorch 1.10 on an NVIDIA A6000 GPU. More details are covered in the appendix.
4.1 Temporal Action Segmentation
Datasets We use 50Salads [+8] and Breakfast [32] for this task. these datasets, each video spans at least 200 seconds and contains at least five different actions. 50Salads contains 50 videos of salad preparation with frame-level action annotations of 19 actions. We generated total of 313 constraints from its annotation. Breakfast consists of 1,712 videos with 18 video-level activities, and each frame has one of the 47 actions. use only its action-level annotations this task. There are total of 2,145 constraints for this dataset. The details of the constraints are provided in Section E.
Task Models We use three task models in evaluation: a single-layer bidirectional Gated Recur- rent Unit (GRU) [5], a temporal convolution model MSTCN [[S], and transformer model AS- Former [56]. We use all three task models to examine the performance gain brought by DTL. For ablation study and further discussion, we use GRU and MSTCN for their ease of training. The architecture of GRU is detailed in the appendix. For experiments on MSTCN and ASFormer, we retrain the corresponding models using their released source codes. The task models are trained and assessed using the protocol in [ where the inputs to the task models are the 4048ddimension
7

---

Table 3: Performance gain of GRU and MSTCN when trained with individual/all types of con-
straints. There are no exclusivity constraints in 50Salads.
(a) 50Salads
(b) Breakfast
Task Model
Edit
@l@
e5
@0
Acc
Task Model
Edit
F1@10
F1@25
F1
Acc
+BD
0.7
-1.3
-0.3
-0.3
0.5
+BD
0.4
0.8
0.3
0.2
-0.2
+FC
7.0
5.5
6.6
6.3
2.1
+PC
1.0
1.1
-0.1
GRU
GRU
+Ex
0.3
+ip
.0.8
-099
0.8
-1.2
0.6
+ip
1.1
2.2
2.1
1,9
0.8
6.0
+BD
0.3
0.6
0.0
0.1
+BD
+FC
0.2
1.0
1.0
0.6
0.1
+FC
-0.1
0.8
1.3
0.8
0.2
MSTCN
+Ex
MSTCN
+Ex
0.1
-0.7
+lp
0.8
2.6
2.8
1.9
1.0
+1p
0.0
0.4
1.0
1.0
-0.2
+All
1.0
2.6
3.5
3.0
1.5
+AIl
0.5
1.2
2.1
features extracted using 13D [+]] pre-trained on ImageNet [100]. Frame-wise cross-entropy is used the task loss for all the task models. Levenshtein distance (Edit), Fl score with thresholds 0.1, 0.25, and 0.5 (FI @ 100,25,50). and frame-wise accuracy (Acc) are used to measure the quality of the outputs. The results are from a k-fold vols-vaali,,,on, where k 5 for 50Salads and k 4 for Breakfast.
Performance Table MI and D show the performance of task models trained without (Base) and with (Base DTL) temporal logic objective LTL, and the performance difference (Gain), on 50Salads and Breakfast respectively. For completeness, we also include the base performance published in the original papers where applicable. A prominent observation is that the performance gain is con- sistently positive. This indicates the applicability and efficacy of DTL on task models with different architectures and on datasets of different scales. Another observation is that the gains on different metrics are not even: there are noticeable improvements on Levenshtein distance and Fl scores frame-wise accuracy not significant. We conjecture the reason be
while the improvement that frame-wise accuracy treats frames independently and does not distinguish well between models with and without the werrseegmentation problem. Since DTL constrains the relations between seg- ments of actions instead of frames, metrics based on segment differences like Levenshtein distance and F1 -corrs are therefore comparably better reflections of performance improvement.
4.2 Ablation Study
To understand the effects of each constraint type, we perform an ablation study using GRU and MSTCN on both datasets. In each experiment, we apply only one type of constraint and retrain the task model. The results are shown in Table B. The first observation is that while applying constraints helps the task model, the extent of improvement differs for different models on different datasets. On 50Salads. nearly all the action classes are present in each video sample, making Ip constraints less helpful for GRU. However, Ip is the most helpful constraint type for MSTCN, which indicates that MSTCN comparably weaker in modelling action co-occurrence and benefits more from this constraint. On the other hand, the more complicated nature of Breakfast makes both models benefit from all four types of constraints. Another important observation is that more constraints not guarantee better performance. For example, GRU enjoys the highest improvement with only FC constraints - higher than when BD and Ip are added. This reveals a trade-off between precision and completeness of constraints: Because the min? and max? functions used in Eqn. (5I-(I0) are approximations. the evaluation becomes less accurate when the number of constraints increases. For MSTCN on 50Salads and both models Breakfast, the benefit of more complete constraints out- weighs the increased evaluation errors, resulting in more improvements when using all constraints. Understanding the fine- ine-grrined interactions between the constraints and different backbone models remains an open question.
4.3 Qualitative Results
We qualitatively compare predictions from task models trained with and without DTL in Fig. In Fig. BEL show the output of MSTCN on 50Salads. Note that the baseline MSTCN

---

(a) Example output of MSTCN on 50Salads.
(b) Example output of GRU on Breakfast.
Figure 4: Qualitative comparison between model trained with and without DTL. which shows that logical errors are fixed by DTL. In each group, from the top to the bottom show the ground truth (GT), the prediction from baseline task model, and the prediction from task model trained with DTL.
- Implication - Backwward Dependency
mix dressing
cucumber
but cucumber
mix ingredienils
serve
salad
onto plate
add dressing
(Preceding actions omitted.)
RDradd addressing. mix dressing)
(Sacceeding actions omitted.)
Figure 5: A snippet prediction from 50Salads and the most influential constraints for each segmen-
tation. The black arrows represent temporal order. The colored arrows represent constraint types.
action after add add_ drddsrrssingg which is in conflict with a FC constraint predicts a mix_ (iissssgs)ggg) because dressing cannot be mixed after it has been added to the FC(add dressing,
salad. A similar example of GRU on Breakfast is shown in Fig. HR, where the action pour _ou__oefee is erroneously predicted in video for "tea preparation". The baseline output violates an exclusivity constraint F add teabag + -F pour_coffee because, except in special cases, coffee should not be added to tea. Both errors fixed in the models trained with DTL. Moreover, we notice that correct- ing an individual wrong segment improves the quality of the whole output sequence. This is because the task models we consider are: (a) recurrent models that condition current output on past inputs, or (b) convolutiona//raasffrrnn In either case, correcting an error fixes the cascade effect it has on the model states, which improves the quality of all nearby outputs.
models that predict based on receptive field/attention information.
4.4 Analyzing the Effects of DTL
One of the greatest benefits of DTL is that it provides additional supervisory signals for tempo- ral constraints, which are implicit in training data. We would like to understand how those sig- nals affect the task model. Formally, given model 9 parameterized by @ and an input æ, we would like to know if a constraint vi: promotes or suppresses the model's output about action a at time f. We assume that the output of the model is continuous with respect to its parameters, or lim 90+5e (æ) = (x)). With this assumption, the effect of ths on O can be approximated by the changes in the output as we update @ based on fo(V,, 9). Specifically, we first compute 5oh θ = (fo /))/ where (x) = log(1 + e-*) as in Eqn. (LI). Then, we update θ as O' = O where = 10 4 is a small update step. Finally, the difference in output caused by w); is obtained as A v.. = ger (x) ge(x)) (ær,, where A sh.. E N A positive (negative) A the, in- dicates a promotive (suppressive) effect of 1/: on ÿa,t as it increases (decreases) the score for action a at time L.
a,t
We use A 1e. to pinpoint the constraint that makes model predict action time t. This can be done by calculating 10. for all vú in ub and find the most positively influential constraint as Vr.. For example, Fig. D shows a snippet of prediction from MSTCN on 50Salads and arg max o, the constraints we found contributed the most to each predicted segment. One important observation from this is that actions in earlier time steps are generally promoted by later actions, as the former could serve as the prerequisites of the latter.
We also summarize the total effect of v/; all time steps and samples as A wi provides two pieces of information. The first the effect of 4/ on all actions. As illustrated as heatmap Fig. E A suggest that the Ip constraints play mainly promotional roles, where the FC, BD, and Ex constraints are mostly suppressive. This is consistent with our design intuitions: Ip encourages co-occurrence of actions, FC and Ex constraints intrinsically prohibitive, and BD constraints suppress action if its prerequisites are missing.
[ि EmEX Et- the.. A

---

Ex
VFC
BD
UIp
Figure 6: A heatmap showing A vi of Breakfast, the vertical axis represents the actions and the horizontal axis is the 2,145 constraints th. Labels at the bottom indicate constraint types. A brighter color indicates that constraint has more positive effect on an action.
Table 4: Actions in Breakfast dataset, and the most influential constraints. "~"" refers to the action in the leftmost column of the row it is in.
Action stir fruit
Top Positive Constraint
Top Negative Constraint
Ip(cut fruit, ~)
BD(,, put fruit2bowl) ex(ppt_mill ~) FC(put_burr Together,
stir_ tea
Ip(add_teabell ~)
put Top
ip(put_bunTogeein Togetherr
The second information is how action is affected by other actions. This be determined by selecting the most promotive and suppressive constraints for action a, and checking the actions involved in these constraints. Table E shows the relations between example actions in the Breakfast dataset. An interesting observation is that for put ToppingOnTop the most promotive and suppres- sive action is the same put PotBunTogether meaning that relation between two actions could largely change depending on the context. This also indicates that DTL can help the model learn different dependencies between the same pair of actions.
Conclusion
We propose DTL, framework that uses temporal logic to constrain the training of action analysis models. Experimental results on the action segmentation task show that DTL effectively improves the performance of task models with different architectures. An ablation study reveals the divergent effect of different types of rules on different task models. Our work suggests that temporal con- straints can be explicitly provided deep network and reduce logical errors in its output. The source code of our work is accessible at https://diff - tt. wil.giituu.... 1
Limitations There remain some limitations in this work. First, in this work, we only explored subset of temporal knowledge that is expressible as frequency matrices. When such temporal cor- relation is sparse, DTL is less effective. This calls for more flexible knowledge curation methods (e.g. with human involvement). Besides, non-temporal knowledge about actions, such as object af- fordance, can be exploited to handle more complicated actions like the verb-object compositions in Epic-Kitchens [Z]. Second, in terms of knowledge type, DTL can benefit from more expressive logic languages, like Allen's Interval Algebra [D], which supports first-order temporal constraints beyond necessities and possibilities. Moreover, DTL formulae are evaluated in their raw forms, which means that the evaluation efficiency could further improved by better formula compila- tion [R]] and optimized message passing directed acyclic graphs. In addition, the constraints we collected from dataset annotations only form a partial and clean view of real-world scenarios. The real-world constraints more comprehensive and complex. which poses a higher requirement generalization and robustness against uncertainties.
Acknowledgments and Disclosure of Funding
This research is partially supported by the National Research Foundation, Singapore under its Strate- gic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or rec- ommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
10

---

References
[1] Hyemin Ahn and Dongheui Lee. Refining action segmentation with hierarchical video repre- sentations. In CVPR, pages 16302-16310, 2021.
[2] James F. Allen. Maintaining knowledge about temporal intervals. Commun. ACM, 26(1)) 1)::32- 843, 1983.
[3] Subhabrata Bhattacharya, Mahdi M. Kalayeh, Rahul Sukthankar, and Mubarak Shah. Recogni- tion of complex events: Exploiting temporal dynamics between underlying concepts. In CVPR, pages 2240-2250, 2014.
[4] João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pages 4724 4733, 2017.
[5] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence modeling. In NIPS Workshop on Deep Learning, 2014.
[6] Marco Cuturi and Mathieu Blondel. Soft-DTW: a differentiable loss function for time-series. In ICML, pages 894-903, 2017.
[7] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kaza- kos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-I00 Int. J. Comput. Vis., 130(1):33-55, 2022.
[8] Adnan Darwiche. On the tractable counting of theory models and its application to truth maintenance and belief revision. J. Appl. Non Class. Logics, 11(1-2):1 1-34, 2001.
[9] Adnan Darwiche and Pierre Marquis. A knowledge compilation map. J. Artif. Intell. Res., 17:229-266 264, 2002.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009,
(11) Ivan Donadello, Luciano Serafini, and Artur S. d' Avila Garcez. Logic tensor networks for semantic image interpretation. In IJCAI, pages 1596-1602, 2017.
[12] Paolo Dragone, Stefano Teso and Andrea Passerini. Neuro-symbolic constraint programming for structured prediction. In NeSy, pages 6-14, 2021.
[13] Georgios Fainekos, Hadas Kress- Gazit, and George J. Pappas. Temporal logic motion planning for mobile robots. In ICRA, pages 2020-2022 2025, 2005.
[14] Georgios Fainekos and George Pappas. Robustness of temporal logic specifications for continuous-time signals. Theor. Comput. Sci., 410444)442226--42241 2009.
[15] Yazan Abu Farha and Jürgen Gall. MS-TCN: multi-stage temporal convolutional network for action segmentation. In CVPR, pages 3575-3584, 2019.
[16] Alireza Fathi, Xiaofeng Ren, and James M. Rehg. Learning recognize objects in egocentric activities. In CVPR, pages 3281-3288, 2011.
[17] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pages 6201-6210, 2019.
[18] Mare Fischer, Mislav Balunovic, Dana Drachsler-Coke Coheen Timon Gehr, Ce Zhang, and Martin T. Vechev. DL2: training and querying neural networks with logic. In ICML, pages 1931-1941, 2019.
[19] Eleonora Giunchiglia and Thomas Lukasiewicz. Multi-label classification neural networks with hard logical constraints. J. Artif. Intell. Res., 72:759-818, 2021.
[20] Fleonora Giunchiglia, Mihaela Catalina Stoian, and Thomas Lukasiewicz. Deep learning with logical constraints. In IJCAI-ECAI, 2022.

---

[21] Asaad Hakeem, Yaser Sheikh, and Mubarak Shah. CASEE: A hierarchical event representation for the analysis of videos. In AAAL, pages 263-268, 2004.
[22] Joseph Y. Halpern and Yoav Shoham. A propositional modal logic of time intervals. J. ACM, 38(4):935-966) 1991.
[23] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activ- itynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961-970. IEEE Computer Society, 2015.
[24] Nicholas Hoemlle, Rafael-Michal Karampatsis, Vaishak Belle, and Kobi Gal. MultiplexNet: Towards fully satisfied logical constraints in neural networks. In AAAI, 2022.
[25] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard H. Hovy, and Eric P. Xing. Harnessing deep neural networks with logic rules. In ACL, 2016.
[26] Yifei Huang, Yusuke Sugano, and Yoichi Sato. Improving action segmentation via graph-based temporal reasoning. CVPR, pages 14021-14031, 2020.
[27] Craig Innes and Subramanian Ramamoorthy. Elaborating on learned demonstrations with tem- poral logic specifications. In RSS, 2020.
[28] Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, and Hirokatsu Kataoka. Alleviating over- segmentation errors by detecting action boundaries. In WACV, pages 2321-2330, 2021.
[29] Alec Kerrigan, Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. Reformulating zero-shot action recognition for multi-label actions. In NeurlPS, pages 25566-25577, 2021.
[30] Yoon Kim, Yacine Jernite, David A. Sontag, and Alexander M. Rush. Character-aware neural language models. In AAAI, pages 2741-2749, 2016.
[31] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[32] Hilde Kuehne, Ali Bilgin Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, pages 780-787, 2014.
[33] Hilde Kuehne, Juergen Gall, and Thomas Serre. An end-to-end generative framework for video segmentation and recognition. In WACV, pages 1-8, 2016.
[34] Hilde Kuehne, Alexander Richard, and Juergen Gall. A hybrid RNN-HMM approach for weakly supervised temporal action segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 42(4):765-779, 2020.
[35] Phong Le and Willem H. Zuidema. Compositional distributional semantics with long short term memory. In *SEM@ NAACL-HLT, pages 10-19, 2015.
[36] Colin Lea, Michael D. Flynn, René Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolutional networks for action segmentation and detection. In CVPR, pages 1003-1012. 2017.
[37] Peng Lei and Sinisa Todorovie. Temporal deformable residual networks for action segmenta- tion in videos. In CVPR, pages 6742-6751, 2018.
[38] Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming- Ming Cheng, and Juergen Gall. Ms-ten++: Multi- stage temporal convolutional network for action segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
[39] Yitao Liang and Guy Van den Broeck. Learning logistic circuits. In AAAI, pages 4277-4286, 2019.
[40] Angelo Montanari. Metric and Layered Temporal Logic for Time Granularity. PhD thesis, University of Amsterdam, 1996.
12

---

[41] Toshiyuki Ogihara. Tense and aspect in truth-conditional semantics. Lingua, 117:392-418, 2007.
[42] Dhruvesh Patel, Pavitra Dangati, Jay- Yoon Lee, Michael Boratko, and Andrew McCallum. Modeling label space interactions in multi-label classification using box embeddings. In ICLR, 2022.
[43] Amir Pnueli. The temporal logic of programs. In FOCS, pages 46-57, 1977.
[44]
Aniruddh Gopinath Puranic, Jyotirmoy V. Deshmukh, and Stefanos Nikolaidis. Learning from demonstrations using signal temporal logic in stochastic and continuous domains. IEEE Robotics Autom. Lett., ((4):6250-6257 2021.
[45] Alexander Richard, Hilde Kuehne, and Juergen Gall. Weakly supervised action learning with RNN based fine-to-coarse modeling. In CVPR, pages 1273-1282, 2017.
[46] Sungyong Seo, Sercan Ö. Arik. Jinsung Yoon, Xiang Zhang, Kihyuk Sohn, and Tomas Pfister. Controlling neural networks with rule representations. NeurlPS, pages 11196-1 1207, 2021.
[47] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1222.0402, 2012.
[48] Sebastian Stein and Stephen J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In UbiComp, pages 729-738, 2013.
[49] Praveen Tirupattur, Kevin Duarte, Yogesh Singh Rawat, and Mubarak Shah. Modeling multi- label action dependencies for temporal action localization. In CVPR, pages 1460-1470, 2021.
[50] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. In CVPR, pages 6450-6459, 2018.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pages 6000-6010, 2017.
[52] Yaqi Xie, Ziwei Xu, Kuldeep S. Meel, Mohan S. Kankanhalli, and Harold Soh. Embedding symbolic knowledge into deep networks. In NeurlPS, pages
2019.
[53] Yaqi Xie, Fan Zhou, and Harold Soh. Embedding symbolic temporal knowledge into deep sequential models. In ICRA, pages 4267-4273, 2021.
[54] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In ICML, 2018.
[55] Ziwei Xu, Xudong Shen, Yongkang Wong, and Mohan Kankanhalli. Unsupervised motion representation learning with capsule autoencoders. In NewrIPS, pages 3205-3217, 2021.
(56) Fangqiu Yi, Hongyu Wen, and Tingting Jiang. ASFormer: Transformer for action segmenta- tion. In BMVC, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section R
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
13

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] While Theorem M is true by construction, we provide a sketchy proof in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Training details are mostly provided Section while some extra information is in the appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] With an exception that error bars are omitted for clarity in Table F
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We used publically available 50Salads and Breakfast dataset, the original papers have been cited.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [Yes] The code and data are publicly available and the con- sent has been described in the cited papers.
(e) Did you discuss whether the data you using/curating contains personally identifi- able information or offensive content? [N/A] No such information was found in the dataset we use.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zT2i59fW39", "reviewer": "Reviewer_7jYQ", "summary": "This paper proposed differential temporal logic to model temporal dependencies, such as co-occurrence and ordering. Based on logic operators, four constraints were defined, including backward dependency, forward cancellation, implication and exclusivity. The proposed method is evaluated on two datasets of the temporal action segmentation task.", "questions": "Please see \"weaknesses\".", "limitations": "The limitations were briefly discussed in the paper, while the potential negative societal impact was not. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-motivated and well-written. Modeling temporal dependencies are an important contribution for temporal action segmentation. \n\n2. The proposed differential temporal logic is very interesting. This method is principled and extendable to include more constraints. \n\n3. Effectiveness is shown on two datasets.\n\nWeaknesses: \n\n1. [Critical] How the constraints are curated from the datasets is not clear enough. \n- The curation process need to be described more. \n- Are they curated from the training split or the whole dataset? If they are curated from the training split, there should have been k sets of constraints instead of a single set as in the supplementary.\n- Will the curation codes be released?\n\n2. Why the performance of ASFormer reported in this paper is different from the original ASFormer paper?\n\n3. I would recommend add ablation studies on the hyper-parameter \\lamda and \\gamma to see their sensitivities.\n\n4. How the examples are selected for Fig.5 and Fig.6. Are they representative? What is the model g for Fig.6? From Fig.6, we could see that the Ip constraints are much stronger than others, while in Table 3 the gain from Ip constraints is not as stronger than others. Please discuss this.\n\nRecommandations:\n\n1. It would be interesting if the proposed method could be extended to more constraints, such as:\n- the co-occurrence at the same time step\n- combination of actions in the constraints (e.g., action a happens only if action b and action c have both happened before)\n\n2. How many times are the experiments repeated?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ctdFheZXBVE", "reviewer": "Reviewer_Zn5F", "summary": "This submission proposes to place temporal logic for temporal action segmentation model training. Compared to previous methods, the authors leverage a Linear Temporal Logic (LTL) formula to evaluate model's prediction and place a new loss item during the training. Experiments show that the proposed methods achieves improvement in performance on two datasets (50Salads and Breakfast datasets).", "questions": "There two problems for the temporal logic loss:\n\n1. The collected temporal constraints already provided external knowledge (similar to labeling or supervision) during the training. It may be natural to expect there will be improvement in model's performance.\n\n2. The novelty of the submission lies in that the LTL loss can assist most existing models in temporal action segmentation task. However, the generation of the temporal constraints may not be free and is not scalable. For example, in 220-223, the authors mentioned there are 313 and 2145 constraints on the 50salads and breakfast datasets. It would be better to reveal more details of the generation process (e.g., is there an automatical way to generate such constraints, or we may need to manually select different concepts and explicitly place several constraints as explained in section 3.3 and section E in appendix). ", "limitations": "Please check the \"questions\" section above. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The motivation and technical details are clear to understand. The authors provide theoretical analysis for the proposed LTL evaluation system during training.\n\n+ The authors provided ablation studies on the two datasets (50Salads and breakfast) in terms of different existing models, which is appreciated. \n\n+ The authors provided the collected temporal constraints on these two datasets, which can further assist future research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ULmopOvNkej", "reviewer": "Reviewer_TJYF", "summary": "The paper introduces a Differentiable Temporal Logic (DTL), a framework providing a model-agnostic manner of introducing temporal logic constraints to deep networks which can describe relations between actions.\n", "questions": "To conclude the paper needs more discussions/experiments to prove its claims. \n\n- First, it needs a more elaborated comparison with other methods following the same reasoning, such as [1, 2]. \n\n- Secondly, it needs experiments on more complex datasets such as Epic-Kitchen or Charades, to better understand the impact of the proposed approach.\n\n\nAfter rebuttal, the authors answered all my questions convincingly and provided essential details about their proposed method.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n    Enable to define a priori-knowledge (i.e. temporal logic constraints) to improve the reasoning on events (i.e. temporal relations).\n\nWeaknesses:\n\n- The contribution is limited:\n1) First of all the authors should compare their approach to other methods [1, 2] that use this kind of reasoning on events (i.e. temporal relations) to improve the predictions. For example, unlike the proposed approach, [1] does not use a priori-knowledge but they learn the relations between the events. Hence, it could be a good practice to compare the two approaches, as learnt constraints are often more convenient than manually defined constraints. \n2) Also it could be interesting to combine both approaches to see if a priori-knowledge and learnt logical constraints can bring improvement to work that learns events relations directly from the data. How complementary are both approaches?\n3) Moreover, it is not clear whether the model can work in case of datasets where different actions can happen at the same time (co-occurrent events) as in Epic-Kitchen?\n4) The logical constraints may be new to computer vision but it has been used for a long time on deep learning and since the contribution they provide is not on computer vision (add a logical loss that was used in other deep learning studies) makes the contribution limited. \n\n- The method is very data-specific: experiment on more datasets is needed.\n 5) The proposed Declarative Constraints Temporal/Logic Formula is easy to use in datasets such as breakfast and 50 salades (datasets with fixed scripts). However in the case of Epic-Kitchen, there are more events and more complex actions, where it is more challenging to define event constraints and so results on this dataset can better prove the authors claims.\n6) The proposed model shows improvement especially in the case of small amounts of training data and annotations as their results show a bigger improvement on 50Salades compared to breakfast. \n7) The results show 2 important observations: Firstly, when the base network can better model the temporal relations, the improvement brought by the proposed approach is less important as it is the case of MSTCN and Asformer compared to GRU. Secondly, when the dataset is bigger (more instances), the improvement is not as important, as we can see from comparing improvement on 50Salades and BreakFast. \nSo, the impact is limited.\n\n[1] Huang, Y., Sugano, Y., & Sato, Y. (2020). Improving action segmentation via graph-based temporal reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14024-14034).\n[2] R. Dai, S. Das and F. Bremond. CTRN: Class Temporal Relational Network For Action Detection. In Proceedings of the 32nd British Machine Vision Conference, BMVC 2021, United Kingdom, Virtual, November 22-25, 2021. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept