Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This submission proposes to place temporal logic for temporal action segmentation model training. Compared to previous methods, the authors leverage a Linear Temporal Logic (LTL) formula to evaluate model's prediction and place a new loss item during the training. Experiments show that the proposed methods achieves improvement in performance on two datasets (50Salads and Breakfast datasets).
strengths_and_weaknesses: + The motivation and technical details are clear to understand. The authors provide theoretical analysis for the proposed LTL evaluation system during training.

+ The authors provided ablation studies on the two datasets (50Salads and breakfast) in terms of different existing models, which is appreciated. 

+ The authors provided the collected temporal constraints on these two datasets, which can further assist future research.
questions: There two problems for the temporal logic loss:

1. The collected temporal constraints already provided external knowledge (similar to labeling or supervision) during the training. It may be natural to expect there will be improvement in model's performance.

2. The novelty of the submission lies in that the LTL loss can assist most existing models in temporal action segmentation task. However, the generation of the temporal constraints may not be free and is not scalable. For example, in 220-223, the authors mentioned there are 313 and 2145 constraints on the 50salads and breakfast datasets. It would be better to reveal more details of the generation process (e.g., is there an automatical way to generate such constraints, or we may need to manually select different concepts and explicitly place several constraints as explained in section 3.3 and section E in appendix). 
limitations: Please check the "questions" section above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed differential temporal logic to model temporal dependencies, such as co-occurrence and ordering. Based on logic operators, four constraints were defined, including backward dependency, forward cancellation, implication and exclusivity. The proposed method is evaluated on two datasets of the temporal action segmentation task.
strengths_and_weaknesses: Strengths:

1. The paper is well-motivated and well-written. Modeling temporal dependencies are an important contribution for temporal action segmentation. 

2. The proposed differential temporal logic is very interesting. This method is principled and extendable to include more constraints. 

3. Effectiveness is shown on two datasets.

Weaknesses: 

1. [Critical] How the constraints are curated from the datasets is not clear enough. 
- The curation process need to be described more. 
- Are they curated from the training split or the whole dataset? If they are curated from the training split, there should have been k sets of constraints instead of a single set as in the supplementary.
- Will the curation codes be released?

2. Why the performance of ASFormer reported in this paper is different from the original ASFormer paper?

3. I would recommend add ablation studies on the hyper-parameter \lamda and \gamma to see their sensitivities.

4. How the examples are selected for Fig.5 and Fig.6. Are they representative? What is the model g for Fig.6? From Fig.6, we could see that the Ip constraints are much stronger than others, while in Table 3 the gain from Ip constraints is not as stronger than others. Please discuss this.

Recommandations:

1. It would be interesting if the proposed method could be extended to more constraints, such as:
- the co-occurrence at the same time step
- combination of actions in the constraints (e.g., action a happens only if action b and action c have both happened before)

2. How many times are the experiments repeated?

questions: Please see "weaknesses".
limitations: The limitations were briefly discussed in the paper, while the potential negative societal impact was not.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a Differentiable Temporal Logic (DTL), a framework providing a model-agnostic manner of introducing temporal logic constraints to deep networks which can describe relations between actions.

strengths_and_weaknesses: Strengths :
    Enable to define a priori-knowledge (i.e. temporal logic constraints) to improve the reasoning on events (i.e. temporal relations).

Weaknesses:

- The contribution is limited:
1) First of all the authors should compare their approach to other methods [1, 2] that use this kind of reasoning on events (i.e. temporal relations) to improve the predictions. For example, unlike the proposed approach, [1] does not use a priori-knowledge but they learn the relations between the events. Hence, it could be a good practice to compare the two approaches, as learnt constraints are often more convenient than manually defined constraints. 
2) Also it could be interesting to combine both approaches to see if a priori-knowledge and learnt logical constraints can bring improvement to work that learns events relations directly from the data. How complementary are both approaches?
3) Moreover, it is not clear whether the model can work in case of datasets where different actions can happen at the same time (co-occurrent events) as in Epic-Kitchen?
4) The logical constraints may be new to computer vision but it has been used for a long time on deep learning and since the contribution they provide is not on computer vision (add a logical loss that was used in other deep learning studies) makes the contribution limited. 

- The method is very data-specific: experiment on more datasets is needed.
 5) The proposed Declarative Constraints Temporal/Logic Formula is easy to use in datasets such as breakfast and 50 salades (datasets with fixed scripts). However in the case of Epic-Kitchen, there are more events and more complex actions, where it is more challenging to define event constraints and so results on this dataset can better prove the authors claims.
6) The proposed model shows improvement especially in the case of small amounts of training data and annotations as their results show a bigger improvement on 50Salades compared to breakfast. 
7) The results show 2 important observations: Firstly, when the base network can better model the temporal relations, the improvement brought by the proposed approach is less important as it is the case of MSTCN and Asformer compared to GRU. Secondly, when the dataset is bigger (more instances), the improvement is not as important, as we can see from comparing improvement on 50Salades and BreakFast. 
So, the impact is limited.

[1] Huang, Y., Sugano, Y., & Sato, Y. (2020). Improving action segmentation via graph-based temporal reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14024-14034).
[2] R. Dai, S. Das and F. Bremond. CTRN: Class Temporal Relational Network For Action Detection. In Proceedings of the 32nd British Machine Vision Conference, BMVC 2021, United Kingdom, Virtual, November 22-25, 2021. 

questions: To conclude the paper needs more discussions/experiments to prove its claims. 

- First, it needs a more elaborated comparison with other methods following the same reasoning, such as [1, 2]. 

- Secondly, it needs experiments on more complex datasets such as Epic-Kitchen or Charades, to better understand the impact of the proposed approach.


After rebuttal, the authors answered all my questions convincingly and provided essential details about their proposed method.
limitations: yes
review_title: null

Meta-review: metareview: This paper introduces an approach for incorporating declarative temporal constraints in the training of temporal action segmentation models, in a model-agnostic fashion. Reviewers generally appreciated the proposed approach, but questioned the scalability and generalizability of the constraint curation process and asked for experimental results on more complex and challenging datasets beyond 50Salads and Breakfast. The author responses addressed many of the reviewer concerns, and reviewers responded positively overall. However, not all concerns could be addressed within the rebuttal time; for example, the authors promised to add results on more complex datasets such as Epic-Kitchen and Charades where reviewers were concerned it may be more challenging to generate constraints. After reading the paper and all author and reviewer responses, I believe the contributions of the paper are sufficient for acceptance. However, authors are expected to add the experimental results on additional datasets as promised for the final paper. Baseline numbers from original papers for some of the models that needed to be re-implemented should also be included for comparability.
confidence: Certain
Decision: Accept