Title: Activity Grammars for Temporal Action Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: neuro-symbolic approach

Temporal action segmentation

grammar
Abstract: Sequence prediction on temporal data requires the ability to understand compositional structures of multi-level semantics beyond individual and contextual properties of parts. The task of temporal action segmentation remains challenging for the reason, aiming at translating an untrimmed activity video into a sequence of action segments. 
This paper addresses the problem by introducing an effective activity grammar to guide neural predictions for temporal action segmentation.  
We propose a novel grammar induction algorithm, dubbed KARI, that extracts a powerful context-free grammar from action sequence data. We also develop an efficient generalized parser, dubbed BEP, that transforms frame-level probability distributions into a reliable sequence of actions according to the induced grammar with recursive rules. 
Our approach can be combined with any neural network for temporal action segmentation to enhance the sequence prediction and discover its compositional structure. 
Experimental results demonstrate that our method significantly improves temporal action segmentation in terms of both performance and interpretability on two standard benchmarks, Breakfast and 50 Salads.

Activity Grammars for Temporal Action Segmentation

Abstract

Sequence prediction on temporal data requires the ability to understand compositional structures of multi-level semantics beyond individual and contextual properties. The task of temporal action segmentation, which aims at translating an untrimmed activity video into a sequence of action segments, remains challenging for this reason. This paper addresses the problem by introducing an effective activity grammar to guide neural predictions for temporal action segmentation. We propose a novel grammar induction algorithm that extracts a powerful context-free grammar from action sequence data. We also develop an efficient generalized parser that transforms frame-level probability distributions into a reliable sequence of actions according to the induced grammar with recursive rules. Our approach can be combined with any neural network for temporal action segmentation to enhance the sequence prediction and discover its compositional structure. Experimental results demonstrate that our method significantly improves temporal action segmentation in terms of both performance and interpretability on two standard benchmarks, Breakfast and 50 Salads.

1 Introduction

Human activities in videos do not proceed by accident; they are structured being subject to generative rules imposed by the goal of activities, the properties of individual actions, the physical environment, and so on. Comprehending such a compositional structure of multi-granular semantics in human activity poses a significant challenge in video understanding research. The task of temporal action segmentation, which aims at translating an untrimmed activity video into a sequence of action segments, remains challenging due to the reason. The recent methods based on deep neural networks [25, 9, 45, 2, 15, 16, 1] have shown remarkable improvement in learning temporal relations of actions in an implicit manner, but often face out-of-context errors that reveal the lack of capacity to capture the intricate structures of human activity, and the scarcity of annotated data exacerbates the issue in training. In this work, we address the problem by introducing an effective activity grammar to guide neural predictions for temporal action segmentation.

Grammar is a natural and powerful way of explicitly representing the hierarchical structure of languages [14] and can also be applied to express the structure of activities. Despite the extensive body of grammar-based research for video understanding [23, 24, 33, 35, 32], none of these approaches have successfully integrated recursive rules. Recursive rules are indispensable for expressing complex and realistic structures found in action phrases and activities. To achieve this, we introduce a novel activity grammar induction algorithm, Key-Action-based Recursive Induction (KARI), that extracts a powerful probabilistic context-free grammar while capturing the characteristics of the activity. Since an activity is composed of multiple actions, each activity exhibits a distinctive temporal structure based on pivotal actions, setting it apart from other activities. The proposed grammar induction

enables recursive rules with flexible temporal orders, which leads to powerful generalization capability. We also propose a novel activity grammar evaluation framework to evaluate the generalization and discrimination power of the proposed grammar induction algorithm. To incorporate the induced activity grammar into temporal action segmentation, we develop an effective parser, dubbed BEP, which searches the optimal rules according to the classification outputs generated by an off-the-shelf action segmentation model. Our approach can be combined with any neural network for temporal action segmentation to enhance the sequence prediction and discover its compositional structure.

The main contribution of this paper can be summarized as follows:

We introduce a novel grammar induction algorithm that extracts a powerful context-free grammar with recursive rules based on key actions and temporal dependencies.

We develop an effective parser that efficiently handles recursive rules of context-free grammar by using Breadth-first search and pruning.

We propose a new grammar evaluation framework to assess the generalization and discrimination capabilities of the induced activity grammars.

We show that the proposed method significantly improves the performance of temporal action segmentation models, as demonstrated through a comprehensive evaluation on two benchmarks, Breakfast and 50 Salads.

2 Related work

Grammar for activity analysis. Grammar is an essential tool to represent the compositional structure of language [14] and has been mainly studied in the context of natural language processing (NLP) [21, 22, 20, 37]. Grammar has been extensively studied in various research areas [28, 29, 32, 8, 43, 13, 42, 11, 27, 12]. Similarly, a grammatical framework can be used to express the structure of activities. Several work [23, 24, 33, 35] have defined context-free grammars based on possible temporal transitions between actions for action detection and recognition. Vo and Bovick [41] propose a stochastic grammar to model a hierarchical representation of activity based on AND-rules and OR-rules. Richard et al. [34] propose a context-free grammar defined on action sequences for weakly-supervised temporal action segmentation. Qi et al. [30, 32, 31] utilize a grammar induction algorithm named ADIOS [37] to induce grammar from action corpus. However, none of the proposed grammar for activity analysis includes recursive rules, which are a fundamental factor in expressing repetitions of actions or action phrases. In this paper, we propose a novel action grammar for temporal action segmentation based on key action and temporal dependency between actions considering recursive temporal structure.

Temporal action segmentation (TAS). Various methods have been proposed to address the task. Early work utilizes temporal sliding windows [36, 19] to detect action segments, and language-based methods [24, 23] has been proposed to utilize a temporal hierarchy of actions during segmentation. Recently, a deep-learning-based model named the temporal convolutional networks (TCN) has been proposed with an encoder-decoder architecture [25, 9]. Moreover, transformer-based models [45, 2] are recently introduced to leverage global temporal relations between actions based on self-attention and cross-attention mechanisms [40]. Other researches have been proposed to improve the accuracy of temporal action segmentation based on existing models [9, 45]. Huang et al. [15] introduce a network module named Graph-based Temporal Reasoning Module (GTRM) that is applied on top of baseline models to learn temporal relations of action segments. Ishikawa et al. [16] suggest an action segment refinement framework (ASRF) dividing a task into frame-wise action segmentation and boundary regression. They refine frame-level classification results with the predicted action boundaries. Gao et al. [10] propose a global-to-local search scheme to find appropriate receptive field combinations instead of heuristic respective fields. Ahn and Lee [1] recently propose a hierarchical action segmentation refiner (HASR), which refines segmentation results by applying multi-granular context information from videos. A fast approximate inference method named FIFA for temporal action segmentation and alignment instead of dynamic programming is proposed by Souri et al. [38]. Other researches [5, 6] reformulate TAS as a cross-domain problem with different domains of spatio-temporal variations, introducing self-supervised temporal domain adaptation. Xu et al. [44] proposes differentiable temporal logic (DTL), which is a model-agnostic framework to give temporal constraints to neural networks. In this paper, we propose a neuro-symbolic approach where the activity grammar induced by the proposed grammar induction algorithm guides a temporal action segmentation model to refine segmental errors through parsing.

A

C

S

TAS

model

!

(#

∗

, %

∗

)

B

I

(a) activity grammar induction

KARI

BEP

'

Seg.

Opt.

(b) parsing

(c) optimization

#

∗

(

action sequences

! : KARI-induced grammar, " : video, # : prediction, (% ∗ , ' ∗ ) : refined action (sequences, length), Seg. Opt. : segmentation optimization

3 Our approach

̸

Given a video of T frames F = [ F 1 , F 2 , ..., F T ] and a predefined set of action classes A , the goal of temporal action segmentation is to translate the video into a sequence of actions a = [ a 1 , a 2 , ..., a N ] and their associated frame lengths l = [ l 1 , l 2 , ..., l N ] where N is unknown, a i ∈ A for 1 ≤ i ≤ N , a i = a i +1 for 1 ≤ i ≤ N -1 , and ∑ N i =1 l i = T . 2 The resultant output of a and l indicates that the video consists of N segments and each pair ( a i , l i ) represents the action and length of i th segment.

In this work, we introduce an activity grammar that guides neural predictions for temporal action segmentation through parsing. We propose a novel activity grammar induction algorithm named KARI and an efficient parser called BEP. The overall pipeline of the proposed method consists of three steps, as illustrated in Fig. 1. First of all, KARI induces an activity grammar from action sequences in the training data. Using the KARI-induced grammar, BEP then takes the frame-level class prediction Y ∈ R T ×|A| from the off-the-shelf temporal action segmentation model [45, 9] and produces a grammar-consistent action sequence a ∗ . Finally, segmentation optimization is performed to obtain optimal action lengths l ∗ based on a ∗ and Y . In the following, we introduce the activity grammar as a probabilistic context-free grammar (Section 3.1), present KARI (Section 3.2) and BEP (Section 3.3), and describe a segmentation optimization method for final outputs (Section 3.4).

3.1 Activity grammar

We define the activity grammar as a probabilistic context-free grammar (PCFG) [17], designed to derive diverse action sequences pertaining to a specific activity class. The activity grammar, denoted as G = ( V , Σ , P , S ) , follows the conventional PCFG which consists of four components: a finite set of variables V , a finite set of terminals Σ , a finite set of production rules P , and the start symbol S ∈ V . In our context, the set of terminals Σ becomes the set of action classes A , and the production rules P are used to generate action sequences from the start variable S . We use two types of production rules, 'AND' and 'OR', defined as follows:

The AND rule replaces a head variable V with a sequence of variables and terminals α , determining the order of the terminals and variables. In contrast, the OR rule converts a head variable V to a sub-variable V i with the probability p i , providing multiple alternatives for replacement; '|' denotes 'OR' operation. These two types of rules allow us to generate action sequences hierarchically.

3.2 Grammar induction: Key-Action-based Recursive Induction (KARI)

Grammar induction refers to the process of learning grammars from data [37]. In our context, it takes action sequences of a specific activity in the training set and produces an activity grammar that is

! ! = [ pour coffee, pour milk]

! " = [ pour coffee, spoon sugar, pour milk]

! # = [take cup, pour coffee , pour milk,  spoon sugar,  stir coffee]

! $ = [take cup, pour coffee , spoon sugar, stir coffee]

(a) Action sequences in the training set !

(f) KARI-induced activity grammar

take cup

pour coffee

pour milk

spoon sugar

stir coffee

!

#

%

!

#

&

!

#

'

(e) The dependency sequence " ! , " " , and " #

# ' = $ , take cup ,	# % = pour coffee , # & ={ pour milk , spoon sugar , pour milk , [ ] pour milk , spoon sugar , stir coffee , spoon sugar , stir coffee }

+ ' = take cup ,	+ % = pour coffee , + & = { } pour milk , spoon sugar , stir coffee

, ' = [{ take cup }] , , % = [{ pour coffee }] , , & = [{ } pour milk , spoon sugar , stir coffee ]

- → / ' / % / &

/ ' → `take cup ( | $

/ % → `pour coffee (

/ & → / ) & / * &

/ ) & → `pour milk ′ / ) & | `spoon sugar ( / ) & | $

/ * & → `stir coffee ( / * & | $

!

"

!

"

"

"

#

"

$

#

"

%

#

"

$

#

…

…

#

#

"

$

&

#

(c) The set of sub-sequences ! ! , ! " , and ! #

(d) The action set $ ! , $ " , and $ #

: AND

: OR

: variable

: terminal

: terminal (key action)

able to parse action sequences of the activity; the induced grammar should be able to parse unseen sequences of the activity as well as the sequences in the training set for generalization. To obtain an effective activity grammar avoiding under-/over-generalization, we introduce two main concepts for grammar induction: key action and temporal dependency .

The key actions for a specific activity are those consistently present in every action sequence from the training dataset. Specifically, the top N key most frequently occurring actions among these are selected as the key actions. The hyperparameters of the number of key actions N key affects the degree of generalization achieved by the induced grammar. The temporal dependency refers to the relevance of temporal orders across actions. Temporally independent actions do not occur in a specific temporal order. This concept of temporal dependency can also be extended to groups of actions, meaning that some groups of actions can be temporally dependent on others.

We induce an activity grammar based on the key actions and the temporal dependency. Action sequences are divided into sub-sequences using the key actions as reference points, and the temporal dependencies between actions within the sub-sequences are established; temporally dependent actions are represented using AND rules (Eq. 1), while temporally independent actions are expressed with OR rules (Eq. 2). We give an example of grammar induction in Fig. 2; four action sequences are given in Fig. 2a, where the action class 'pour coffee' is chosen as the key action with the number of key actions N key set to 1.

Given the action sequences from the training dataset D , we begin grammar induction by identifying a set of key actions K ⊂ A with the pre-defined hyperparameter N key . Using the key actions, each action sequence a ∈ D is divided into three parts: a = [ a L , a M , a R ] . The sub-sequences a L , a M , and a R denote the portions of the original action sequence that occurred before , between , and after the key actions, respectively; the sub-sequence a M starts from the first key action and includes up to the last key action in K . An example in Fig. 2b shows that the action sequence a 3 is divided into three sub-sequences using the key actions. For notational convenience, we will use the superscript Ω ∈ { L , M , R } to denote one of the three parts. All action sub-sequences a Ω in a specific part Ω are grouped to consist in a corresponding set of sub-sequences D Ω ( cf . Fig 2c). The action set A Ω ⊆ A

is then defined to contain all the actions occurring in D Ω ( cf . Fig 2d). To determine the temporal dependencies among the actions of A Ω , pairwise temporal orders are considered as follows. If one action always occurs before the other in D Ω , then the two actions are temporally dependent and otherwise temporally independent. Based on the concepts, we construct the action group sequence d Ω by collecting the temporally independent actions as an action group and arranging such action groups according to their temporal dependencies ( cf . Fig 2e).

In the following, we describe how to construct the production rules P of the activity grammar G .

Start rule . We first create the rule for the start variable S :

where V L , V M , and V R are variables used to derive left, middle, and right parts of the action sequence, respectively.

Rule for the variable V Ω . For V Ω , Ω ∈ { L , R } , we construct an AND rule of action groups based on action group sequence d Ω :

where the variable V Ω i represents the i th action group in the action group sequence d Ω i . Since actions in an action group are considered temporally independent, we construct an OR rule for each action group:

where d Ω i,j denotes the j th action from the action group d Ω i . The variable V Ω i yields d Ω i,j V Ω i with the probability p Ω i,j . This rule can be recursively used to proceed to the variable V Ω i in the next step. This recursive structure allows for repeated selection of actions within the same action group, leading to the generation of diverse action sequences, which is effective for generalization. To avoid an infinite loop of the recursion, the empty string ϵ with the escape probability p Ω i,ϵ is added to Eq. 5. For the details, refer to the transition probability p Ω i,j and the escape probability p Ω i,ϵ in Appendix A.1.

Rule for the middle variable V M . Since the temporal order of key actions might vary, we consider all the possible temporal orders between key actions in K . A set of temporal permutations of actions is denoted as Π , where each possible temporal permutation is represented by the OR rule:

The rule for the permutation variable V M i is defined by the AND rule:

where all the key actions are included. Note that π i,j represents the j th action of the permutation π i ∈ Π , and the variable V M( i,j ) derives action sub-sequences between actions π i,j and π i,j +1 . The production rule for V M( i,j ) adheres to the rules specified in Eq. 4 and 5. The resultant KARI-induced grammar from the example is shown in Fig. 2f, highlighting the compositional structure of actions.

3.3 Parser: Breadth-first Earley Parser (BEP)

The goal of the parser is to identify the optimal action sequence a ∗ by discovering the most likely grammatical structure based on the output of the action segmentation model [9, 45]. In other words, the parser examines the production rules of the activity grammar to determine whether the given neural prediction Y can be parsed by the grammar G . However, when the grammar includes recursive rules, the existing parser struggles to complete the parsing within a reasonable time due to the significant increase in branches from the parse tree. To address this challenge, we introduce an effective parser dubbed BEP, integrating Breadth-first search (BFS) and a pruning technique into a generalized Earley parser (GEP) [32]. Since the BFS prioritizes production rules closer to the start variable, it helps the parser understand the entire context of the activity before branching to recursive iterations. Simultaneously, pruning effectively reduces the vast search space generated by OR nodes and recursion, enabling the parser to focus on more relevant rules for the activity.

For parsing, we employ two heuristic probabilities introduced in [32] to compute the probability of variables and terminals within the parse tree. Specifically, let Y t,x denote the probability of frame t

being labeled as x . In this context, we denote the last action in the action sequence a as x , i.e. x = a N , where a = [ a 1 , a 2 , ..., a N ] , for simplicity. The transition probability g ( x | a 1: N -1 , G ) determines the probability of parsing action x given the a 1: N -1 and the grammar G .

The parsing probability p ( F 1: T → a | G ) computes the probability of a being the action sequence for F 1: T . The probability at t = 1 is initialized by:

where ϵ indicates an empty string.

Since we assume that the last action of a is classified as x , the parsing probability p ( F 1: t → a | G ) can be represented with the probability of the previous frames:

p ( F 1: t → a | G ) = Y t,x ( p ( F 1: t -1 → a | G ) + g ( x | a 1: N -1 , G ) p ( F 1: t -1 → a 1: N -1 | G ) ) . (9) The prefix probability p ( F 1: T → a ... | G ) represents the probability of a being the prefix of a ∗ . This probability is computed by measuring the probability that a is the action sequence for the frame F 1: t with t in the range [1 , T ] :

The parsing operation is structured following the original Earley parser [7], consisting of three key operations: prediction, scanning, and completion. These operations involve the update and generation of states, where every state comprises the rule being processed, the parent state, the parsed action sequence denoted as a , and the prefix probability denoted as p ( a ... ) . The states are enqueued and prioritized by their depth d within the parse tree.

Prediction : for every state Q ( m,n,d ) of the form ( A → α · Bβ,Q ( i, j, k ) , a , p ( a ... )) , add ( B → · Γ , Q ( m,n,d ) , a , p ( a ... )) to Q ( m,n,d + 1) for every production rule in the grammar with B on the left-hand side.

Scanning : for every state in Q ( m,n,d ) of the form ( A → α · wβ,Q ( i, j, k ) , a , p ( a ... )) , append the new terminal w to a and compute the probability p (( a + w ) ... ) . Create a new set Q ( m +1 , n ′ , d ) where n ′ is the current size of Q ( m +1) . Add ( A → αw · β, Q ( i, j, k ) , a + w,p (( a + w ) ... )) to Q ( m +1 , n ′ , d ) .

Completion : for every state in Q ( m,n,d ) of the form ( A → Γ · , Q ( i, j, k ) , a , p ( a ... )) , find states in Q ( i, j, k ) of the form ( B → α · Aβ,Q ( i ′ , j ′ , k ′ ) , a ′ , p ( a ′ ... )) and add ( B → αA · β, Q ( i ′ , j ′ , k ′ ) , a , p ( a ... )) to Q ( m,n,d -1) .

Additionally, we introduce a pruning technique of limiting the queue size to reduce the vast search space in the parse tree, similar to the beam search. Specifically, the parser preserves only the top N queue elements from the queue in order of the parsing probability of each state. The parsing process terminates when the parser identifies that the parsed action sequence a ∗ has a higher parsing probability than the prefix probabilities of any other states in the queue. For the further details, refer to Appendix B.

The symbols α , β , and Γ represent arbitrary strings consisting of terminals and variables, i.e. α, β, Γ ∈ (Σ ∪ V ) ∗ . The symbols A and B refer to the variables, while w denotes a single terminal. The symbol Q represents the set of states, and the dot ( · ) denotes the current position of the parser within the production rule.

3.4 Segmentation optimization

The objective of segmentation optimization is to determine the optimal alignment between the input classification probability matrix Y and the action sequence a ∗ . In other words, the entire frames are allocated within the action sequences a ∗ = [ a ∗ 1 , a ∗ 2 , ..., a ∗ N ] , obtained from the parser, to determine the optimal action lengths l ∗ = [ l ∗ 1 , l ∗ 2 , ..., l ∗ N ] . In this work, we utilize dynamic programming-based Viterbi-like algorithm [35] for activity parsing. Similar to [26, 32], the optimizer explores all possible allocations and selects the one with the maximum product of probabilities:

…

seen

unseen

$

/

0

generation

induction

parser

…

all

…

O

X

X

"

"

#

:

#

$%

synthetic

"

"

"

&

:

#

$%

induced

"

$

/

1

KARI-induced grammar index

Synthetic grammar index

ADIOS-OR-induced  grammar index

Synthetic grammar index

1.0

20

20

20

- 0.8

40

40

40

- 0.6

60

60

60

0.4

80

80

80

Z'0

0

20

40

60

80

20

40

09

80

0

20

40

60

80

0.0

4 Experimental evaluation and analysis

4.1 Datasets and evaluation metrics

Datasets. We conduct experiments on two widely used benchmark datasets for temporal action segmentation: Breakfast [23] and 50 Salads [39]. The Breakfast dataset, consisting of 1,712 videos, involves 52 individuals preparing 10 different breakfast activities comprised of 48 actions in 18 different kitchens. Similarly, the 50 Salads dataset comprises 50 egocentric videos of people preparing salads of a single activity with 17 fine-grained actions from 25 people. We used I3D [4] features provided by [9].

Evaluation metrics. For evaluation metrics, we report edit score, F1@ { 10 , 25 , 50 } scores, and frame-wise accuracy following the previous work [9, 45].

4.2 Implementation details

For KARI, we set the hyperparameters of the number of key actions N key to 4 for Breakfast, and 3 for 50 Salads. We individually induce separate activity grammar for the ten activity classes within Breakfast and subsequently merge them into a unified grammar. For the comparison with the existing grammar used in the previous work [32, 31], we induce activity grammars of ADIOS [37] provided by [31]. Two types of ADIOS-induced grammar are induced: ADIOS-AND-induced grammar, primarily composed of AND rules with limited generalization capabilities, and ADIOS-OR-induced grammar, predominantly incorporating OR rules, offering improved generalization. Please refer to Appendix C.1 for grammar induction details.

For BEP, we configured the queue size N queue to be 20. For efficiency, we adjust the sampling rate of the input video features to 50 for Breakfast and 100 for 50 Salads. We use two widely used models for the temporal action segmentation: ASFormer [45] based on Transformer and MS-TCN [9] based on CNNs. Since we apply the proposed method to the reproduced temporal action segmentation models, we directly compare and evaluate the performance based on the reproduced results.

4.3 Evaluation framework for activity grammar

We propose a novel evaluation framework to assess the generalization and discrimination capabilities of the activity grammar. Figure 3 shows the overall process of the grammar evaluation framework. We first generate a set of synthetic activity grammars G S randomly. Action sequences a ∈ D all i are generated from each synthetic grammar G S i ∈ G S , and these sequences are randomly divided into two sets: seen seen and unseen . For each seen set, a grammar induction algorithm is applied, resulting in the induced grammar G I i consisting in a corresponding set of induced grammars G I . For grammar evaluation, the induced grammar G I i ∈ G I parses action sequences from the entire unseen sets. The induced grammar should accurately parse the action sequences generated by the original synthetic grammar from which it was induced, while also effectively discriminating those generated by other synthetic grammars.

To simulate real-world video action sequences, we generate the synthetic activity grammars assuming temporal dependencies across actions. This indicates that certain actions follow a temporal order

while others do not adhere to such dependencies. To prevent parsing failures arising from uncovered terminals, we maintain a consistent set of terminals throughout the entire grammar while randomly assigning key actions to these terminals. The number of variables is randomly determined for each synthetic grammar. As evaluation metrics, we use precision and recall similar to the previous work [37, 3]. For the induced grammar G I i , action sequences successfully parsed from the synthetic grammar G S i are classified as positive samples from the entire unseen sets, otherwise considered negative samples.

Details. In our experiment, we generate a total of 100 grammars, each consisting of 20 variables and 20 terminals. We have developed two types of synthetic grammars that differ in terms of temporal hierarchical difficulty. In synthetic grammar I, each terminal is allocated to a single variable, while in synthetic grammar II, terminals are randomly assigned multiple times to different variables. Three types of grammars are evaluated: induced by ADIOS-AND, ADIOS-OR, and proposed KARI.

Results. Table 1 and Table 2 show the results of grammar evaluation by using synthetic grammar I and II, respectively. Our activity grammar demonstrates robust generalization performances, achieving a recall of approximately 1.0 on unseen action sequences compared to others, maintaining comparable precision. The ADIOS-OR-induced grammar shows better generalization ability compared to the ADIOS-AND-induced grammar. We visualize a confusion matrix of the three types of grammar: synthetic grammar, KARI-induced grammar, and ADIOS-OR-induced grammar, as shown in Fig. 4. The confusion matrix shows the parsing accuracy of each unseen set over the synthetic grammar II. Higher accuracy is represented by brighter cells in the matrix. KARI-induced grammars demonstrate similar patterns in their confusion matrix compared to the synthetic grammars. This similarity indicates their capacity to generalize to unseen sets from which each grammar is induced, allowing effective discrimination of action sequences from other synthetic grammars.

4.4 Effects of the grammar-based refinement on temporal action segmentation

Table 3 and Table 4 show the performance of applying the proposed method to temporal action segmentation models [9, 45] across two benchmark datasets. The first row in Table 3 and Table 4 indicates the performance from the original paper [9, 45], whereas the second row represents the

↓

↓

↓

↓

reproduced performance obtained using official codes. The comparison between the second and the last row of each compartment in each table reveals significant improvements in both edit scores and F1 scores. This result validates the effectiveness of leveraging activity grammars to refine segment-wise classification. Remarkably, the KARI-induced grammar shows great performance compared to both ADIOS-induced grammars, demonstrating the importance of generalizing the grammar to cover unseen action sequences during inference effectively.

4.5 Analysis

Ablation studies of KARI. Ablation studies of KARI are conducted on the 50 Salads dataset using ASFormer [45], as shown in Table 5 to demonstrate the effectiveness of each component, including key actions and temporal dependency . The results show that both key actions and recursive rules contribute to the significant improvement of grammar-based refinement. In particular, using recursive rules is essential for the activity grammar to be generalized to the unseen action sequences.

BEP vs. GEP. Table 6 presents the performance comparison of GEP and BEP using KARI-induced grammar. We limit the queue size of both parsers, as the parser without the limitation fails to complete parsing within a reasonable time. The results indicate that our BEP outperforms GEP under the same condition. This is attributed to GEP prioritizing states based on the highest probability, which increases the risk of getting trapped in local optima when performing selective pruning within specific branches. In contrast, BEP, which prioritizes low-depth states, allows for easier escape from cycles and OR nodes, contributing to improved overall performance.

The number of key actions. Table 7 shows the results by adjusting the number of key actions N key of KARI on 50 Salads. We set the value of N key ranging from 1 to 6, where the induced grammar with a smaller value generates the larger activity corpus. We find that setting N key to 3 outperforms the others, demonstrating the importance of achieving an appropriate level of generalization for effective refinement. Both excessive and insufficient generalization can negatively impact performance, highlighting the need to strike a balance in the generalization ability of the activity grammar.

Grammar evaluation on real data. We evaluate the parsing recall on the unseen action sequences of the Breakfast dataset. The results present the average recall across all splits for each activity. The number inside brackets indicates the average length of action sequences of each activity in D . Table 8 compares the generalization capability of the five grammar induction algorithms [23, 35, 37], including KARI (details in Appendix C.1). The result demonstrates that KARI-induced grammar shows better generalization ability on real data compared to others. Remarkably, the KARI-induced grammar shows robust performance with the extended average length of the action sequences, whereas other algorithms exhibit poor generalization.

4.6 Qualitative results

Figure 5 presents a visual representation of the refined segmentation results on benchmark datasets. The proposed method successfully parses and identifies the actions 'pour oil' (red bar in Fig. 5a)

↓

GT

ASFormer

ADIOS-OR

SIL

pour oil

crack egg

stir egg

pour egg2pan

stirfry egg

add saltnpepper

take plate

put egg2plate

put pancake2plate

(a) Breakfast

Rgb-23-2

GT

ASFormer

ADIOS-OR

action start/end

cut lettuce

cut tomato

place lettuce

peel cucumber

cut cucumber

place cucumber

cut cheese

place cheese

mix ingredients

add oil

add vinegar

add salt

add pepper

mix dressing

serve salad

(b) 50 Salads

and 'add saltnpepper', (blue bar in Fig. 5a), which are omitted in the results obtained by using the ADIOS-OR induced grammar. The results show that KARI-induced grammar allows a more flexible temporal structure between actions. Furthermore, our method effectively removes actions such as 'put pancake2plate' that do not correspond to the intended activity. Similarly, qualitative results on 50 Salads in Fig. 5b show the effectiveness of the proposed method with complex action sequences. The overall results show that activity grammar-based refinement for the temporal action segmentation model is effective for correcting the neural predictions by using the grammar as a guide.

5 Conclusion

We have shown that the proposed approach enhances the sequence prediction and discovers its compositional structure, significantly improving temporal action segmentation in terms of both performance and interpretability. However, the improvement is limited by the initial output of the action segmentation network, which remains further research in the future. We believe that the grammar induction and parsing methods can be easily applied to other sequence prediction tasks.

References

H. Ahn and D. Lee. Refining action segmentation with hierarchical video representations. In Proc. IEEE International Conference on Computer Vision (ICCV) , pages 16302-16310, 2021. 1, 2

N. Behrmann, S. A. Golestaneh, Z. Kolter, J. Gall, and M. Noroozi. Unified fully and timestamp supervised temporal action segmentation via sequence to sequence translation. In Proc. European Conference on Computer Vision (ECCV) , pages 52-68. Springer, 2022. 1, 2

P. Belcák, D. Hofer, and R. Wattenhofer. A neural model for regular grammar induction. arXiv preprint arXiv:2209.11628 , 2022. 8

J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6299-6308, 2017. 7

M.-H. Chen, B. Li, Y. Bao, and G. AlRegib. Action segmentation with mixed temporal domain adaptation. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 605-614, 2020. 2

M.-H. Chen, B. Li, Y. Bao, G. AlRegib, and Z. Kira. Action segmentation with joint selfsupervised temporal domain adaptation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9454-9463, 2020. 2

J. Earley. An efficient context-free parsing algorithm. Communications of the ACM , 13(2):94102, 1970. 6, 15

H.-S. Fang, Y. Xu, W. Wang, X. Liu, and S.-C. Zhu. Learning pose grammar to encode human body configuration for 3d pose estimation. In Proc. AAAI Conference on Artificial Intelligence (AAAI) , volume 32, 2018. 2

Y. A. Farha and J. Gall. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3575-3584, 2019. 1, 2, 3, 5, 7, 8

S.-H. Gao, Q. Han, Z.-Y. Li, P. Peng, L. Wang, and M.-M. Cheng. Global2local: Efficient structure search for video action segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16805-16814, 2021. 2

M. Guo, V. Thost, B. Li, P. Das, J. Chen, and W. Matusik. Data-efficient graph grammar learning for molecular generation. In Proc. International Conference on Learning Representations (ICLR) , 2021. 2

Y. Hong, Q. Li, R. Gong, D. Ciao, S. Huang, and S.-C. Zhu. Smart: A situation model for algebra story problems via attributed grammar. In Proc. AAAI Conference on Artificial Intelligence (AAAI) , pages 13009-13017, 2021. 2

Y. Hong, Q. Li, S.-C. Zhu, and S. Huang. Vlgrammar: Grounded grammar induction of vision and language. In Proc. IEEE International Conference on Computer Vision (ICCV) , pages 1665-1674, 2021. 2

J. E. Hopcroft, R. Motwani, and J. D. Ullman. Introduction to automata theory, languages, and computation. Acm Sigact News , 32(1):60-65, 2001. 1, 2, 18

Y. Huang, Y. Sugano, and Y. Sato. Improving action segmentation via graph-based temporal reasoning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14024-14034, 2020. 1, 2

Y. Ishikawa, S. Kasai, Y. Aoki, and H. Kataoka. Alleviating over-segmentation errors by detecting action boundaries. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 2322-2331, 2021. 1, 2

F. Jelinek, J. D. Lafferty, and R. L. Mercer. Basic methods of probabilistic context free grammars . Springer, 1992. 3

D. Jurafsky. Speech & language processing . Pearson Education India, 2000. 18

S. Karaman, L. Seidenari, and A. Del Bimbo. Fast saliency based pooling of fisher encoded dense trajectories. In ECCV THUMOS Workshop , volume 1, page 5, 2014. 2

Y. Kim. Sequence-to-sequence learning with latent neural grammars. Advances in Neural Information Processing Systems , 34:26302-26317, 2021. 2

Y. Kim, C. Dyer, and A. M. Rush. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2369-2385, 2019. 2

Y. Kim, A. M. Rush, L. Yu, A. Kuncoro, C. Dyer, and G. Melis. Unsupervised recurrent neural network grammars. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 1105-1117, 2019. 2

H. Kuehne, A. Arslan, and T. Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 780-787, 2014. 1, 2, 7, 9, 10, 18, 19

H. Kuehne, A. Richard, and J. Gall. Weakly supervised learning of actions from transcripts. Computer Vision and Image Understanding , 163:78-89, 2017. 1, 2

C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks for action segmentation and detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 156-165, 2017. 1, 2

J. Li, P. Lei, and S. Todorovic. Weakly supervised energy-based learning for action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019. 6

Q. Li, S. Huang, Y. Hong, Y. Chen, Y. N. Wu, and S.-C. Zhu. Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In Proc. International Conference on Machine Learning (ICML) , pages 5884-5894. PMLR, 2020. 2

A. Piergiovanni, A. Angelova, and M. S. Ryoo. Differentiable grammars for videos. In Proc. AAAI Conference on Artificial Intelligence (AAAI) , pages 11874-11881, 2020. 2

A. Piergiovanni, A. Angelova, A. Toshev, and M. S. Ryoo. Adversarial generative grammars for human activity prediction. In Proc. European Conference on Computer Vision (ECCV) , pages 507-523. Springer, 2020. 2

S. Qi, S. Huang, P. Wei, and S.-C. Zhu. Predicting human activities using stochastic grammar. In Proc. IEEE International Conference on Computer Vision (ICCV) , pages 1164-1172, 2017. 2

S. Qi, B. Jia, S. Huang, P. Wei, and S.-C. Zhu. A generalized earley parser for human activity parsing and prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(8):2538-2554, 2020. 2, 7, 15

S. Qi, B. Jia, and S.-C. Zhu. Generalized earley parser: Bridging symbolic grammars and sequence data for future prediction. In Proc. International Conference on Machine Learning (ICML) , pages 4171-4179. PMLR, 2018. 1, 2, 5, 6, 7, 18

A. Richard, H. Kuehne, and J. Gall. Weakly supervised action learning with rnn based fineto-coarse modeling. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 754-763, 2017. 1, 2

A. Richard, H. Kuehne, and J. Gall. Action sets: Weakly supervised action segmentation without ordering constraints. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5987-5996, 2018. 2

A. Richard, H. Kuehne, A. Iqbal, and J. Gall. Neuralnetwork-viterbi: A framework for weakly supervised video learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7386-7395, 2018. 1, 2, 6, 9, 10, 18, 19

M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele. A database for fine grained activity detection of cooking activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1194-1201. IEEE, 2012. 2

Z. Solan, D. Horn, E. Ruppin, and S. Edelman. Unsupervised learning of natural languages. Proceedings of the National Academy of Sciences , 102(33):11629-11634, 2005. 2, 3, 7, 8, 9, 10, 18

Y. Souri, Y. A. Farha, F. Despinoy, G. Francesca, and J. Gall. Fifa: Fast inference approximation for action segmentation. In Pattern Recognition: 43rd DAGM German Conference, DAGM GCPR 2021, Bonn, Germany, September 28-October 1, 2021, Proceedings , pages 282-296. Springer, 2022. 2

S. Stein and S. J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing , pages 729-738, 2013. 7, 19

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Proc. Neural Information Processing Systems (NeurIPS) , 30, 2017. 2

N. N. Vo and A. F. Bobick. From stochastic grammar to bayes network: Probabilistic parsing of complex activity. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2641-2648, 2014. 2

B. Wan, W. Han, Z. Zheng, and T. Tuytelaars. Unsupervised vision-language grammar induction with shared structure modeling. In Proc. International Conference on Learning Representations (ICLR) , 2021. 2

Y. Xu, W. Wang, T. Liu, X. Liu, J. Xie, and S.-C. Zhu. Monocular 3d pose estimation via pose grammar and data augmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2021. 2

Z. Xu, Y. S. Rawat, Y. Wong, M. Kankanhalli, and M. Shah. Don't pour cereal into coffee: Differentiable temporal logic for temporal action segmentation. In Advances in Neural Information Processing Systems (NeurIPS) . 2

F. Yi, H. Wen, and T. Jiang. Asformer: Transformer for action segmentation. In Proc. British Machine Vision Conference (BMVC) , 2021. 1, 2, 3, 5, 7, 8, 9, 10, 19

Appendices

In this supplement, we provide detailed descriptions of the proposed method and additional results, which are omitted in the main paper due to the lack of space. In Section A, we will describe the formulation of the probability of activity grammar. Algorithmic details of BEP is included in Section B. Section C compares KARI with the existing grammar induction algorithms for activity grammar and Section D presents additional qualitative results. We conclude this Appendix by discussing the broader impact of our research in Section E.

A Formulation of the probabilities in KARI

In this section, we describe the formulation of transition probability p Ω i,j and the escape probability p Ω i,ϵ and p M ϵ in Eq. 5 and 6 in Section A.1. In the following, the derivation of the expectation of the escape probability is described in Section A.2.

A.1 Formulation of the escape and transition probability

We first introduce and escape probabilities and the transition probabilities introduced in Eq. 5. Pre-processing. Let h Ω i represent a list of action sub-sequences, where the sub-sequence from h Ω i removes actions that does not exist in the action group d Ω i from the action sub-sequence in D Ω . The empty string ϵ remains when the action sub-sequence does not include actions within the action group d Ω i . For example in Fig. 2, a list of sub-sequences h R 1 can be structured as h R 1 = [[pour milk] , [spoon sugar , pourmilk] , [pour milk , spoonsugar] , [spoon sugar]] with the corresponding action group d R 1 = { pourmilk , spoonsugar } . Similary, a list of sub-sequences h R 2 is structured as h R 2 = [ ϵ, ϵ, [ stir coffee ] , [ stir coffee ]] with the action group d R 2 = { stir coffee } . This pre-processing step of generating h Ω i enables us to consider the statistical probabilities associated with actions.

Formulation of the escape probability. The escape probability p Ω i,ϵ and the transition probability p Ω i,j are both defined based on the number of recursion n rec of the current timestep; thereby these probabilities are represented as functions of n rec . We first define the escape probability function:

where ¯ N h Ω i is the average length of the sub-sequences in h Ω i . In the first recursion, i.e. , n rec = 1 , the probability calculation solely considers statistics of the actions. Otherwise, the probability is calculated based on the expected number of recursions, which will be introduced in Appendix A.2. In Fig. 2, p R 1 ,ϵ (1) = 0 , since none of the action sub-sequence a from h R 1 is equal to the empty sequence, and p R 1 ,ϵ ( n rec > 1) = 2 3 , since the average length of sub-strings in h R 1 is 1 . 5 .

̸

Formulation of the transition probability. The action sequence a = [ a 1 , a 2 , ..., a N ] represents the distinct action labels for the video segments, where a i = a i +1 as described in Section 3. In order to prevent the repetition of the same action in Eq. 5, we introduce an additional input q when defining the transition probability. Here, q refers to the index of the actions selected by the rule in the previous step, specifically at ( n rec -1) th step where n rec > 1 . We simply put q to 0 in the first recursion, i.e. n rec = 1 , which does not affect the results. The transition probability p Ω i,j is defined by:

̸

The escape probability p M ϵ and the transition probability p M i,j for the middle variable V M in Eq. 6 is defined in the same way as Eq. 13 and Eq. 14, respectively.

A.2 Derivation of the escape probability

We introduce the formulation of the escape probability p Ω i,ϵ in Appendix A.1. The escape probability is required to avoid an infinite loop of the rules and guarantee the length of sequences from the recursive rules in Eq. 5. Since the number of recursions directly determines the sequence lengths, we determine the escape probability regarding the length of action sequences. For notational simplicity, we denote the escape probabilities p Ω i,ϵ by p , omitting superscripts and subscripts. The expectation of the number of recursions is calculated by:

Since we derive the escape probability when n rec > 1 , the expected number of recursions is equal to ¯ N -1 :

, where ¯ N is the average length of action sequences. Finally, we obtain the escape probability by

where this equation is used in Eq. 13 when n rec > 1 . The derivation of the escape probability p M ϵ of the middle variable V M in Eq. 6 is also formulated as the same.

B Breadth-first Earley Parser (BEP)

B.1 Earley parser

The Earley parser [7] is a classic algorithm that efficiently parses strings for context-free grammar. It operates by maintaining a set of states of the parsing process. Each state consists of a production rule, a position within that rule, and a position in the input string. The parser builds a parse tree for the input string, which records the structure of parsing. The Earley parser is commonly used for natural language processing tasks, such as syntactic analysis and semantic parsing.

The Earley parser consists of three main operations: scanning, prediction, and completion.

Scanning: The parser matches a terminal symbol in the input string with the current position in the production rule. This operation moves the parser forward in the input string.

Prediction: The parser expands a variable in the production rule based on the current position. It adds new states to the set of states for possible future matches.

Completion: When the parser reaches the end of the production rule, it searches for other states predicting the head variable of the current rule. Subsequently, the parser update the positions within the rule of the searched states.

By iterating these three operations, the Earley parser builds a parse chart that represents all possible parse trees for the input string.

B.2 Implementation details

The parsing probability p ( F 1: t → a | G ) (Eq. 9) can suffer from numerical underflow due to its exponential decrease as t increases. To overcome the issue, we compute the probabilities in logarithmic space, following [31]. For simplicity, we denote log( p ( F 1: t -1 → a | G )) as P N and log( p ( F 1: t -1 → a 1: N -1 | G )) as P N -1 below:

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

B.4

Parsing example

In this section, we provide an example to help understand how BEP works. For the sake of simplicity class from the temporal action segmentation networks is equal. First of all, we define toy grammar as

and ease of calculation, we make the assumption that the frame-wise probability of the entire action follows.

for

i

= [1

,

2

,

3

,

4]

Table A1 is the history of parsing with the toy grammar. It shows the currently popped state, visiting wise probabilities are assumed to be the same. Note that the table includes some history after the

order, prefix, previous state, and which states are currently in the queue. The numbers in the column pop, from, and queue indicate m , n , and d of the state. The column p represents the prefix probability excluding the frame-wise probability, which can be considered a parsing probability since all frameFor the computational efficiency, we set the sampling stride of input matrix Y as 50 for Breakfast and 100 for 50Salads. Additionally, we set the maximum length of the refined action sequence as 20 for Breakfast and 25 for 50Salads.

parsed sequence satisfied the early stop constraint to illustrate how BEP prioritizes the states.

S

order 14, even though the probability of state B.3 Parsing algorithm

(1

,

1

,

3)

is higher, BEP visits the state

S

(3

,

0

In a lower depth. C Comparison with other grammar induction algorithms Algorithm 1 shows the parsing procedure of BEP. We utilize a priority queue that sorts the elements in ascending order. The currentSet stores multiple states with the same m , n , and d . See B.4 for the examples. BEP stops parsing when the probability of a ∗ has the highest probability compared to states in the queue while ensuring the current state can reach depth 1 with only completions.

In this section, we introduce the existing grammar induction algorithms for activity grammar induction algorithms will be given in Section C.2 and C.3.

,

2)

with

[2, 5, 6]

in Section C.1. Additional experimental results and analysis to compare KARI with the other grammar B.4 Parsing example

C.1 Grammar induction algorithms The hierarchical context-free grammar proposed by Kuehne et al. introduces a root rule that allows for the selection of activities. In this grammar, the root rule with the starting variable S is defined as In this section, we provide an example to help understand how BEP works. For simplicity, we assume that frame-wise class probabilities from the segmentation model are identical across all action classes. First of all, we define toy grammar as shown in Figure 6. In the context of grammar, S indicates the starting variable. A , B , C , and A i for i = [1 , 2 , 3 , 4] represent the variables, while x j for j = [1 , 2 , 3 , 4 , 5 , 6 , 7] represent terminals.

4 Table 9 is the history of parsing with the toy grammar. It shows the currently popped state, the visiting order, the parsed prefix, the previous state, and which states are currently in the queue. The three consecutive numbers in the column pop, from, and queue indicate m , n , and d of the state. The column p represents the prefix probability excluding the frame-wise probability, which can be considered a parsing probability since all frame-wise probabilities are assumed to be the same. Note that the table includes some history after the parsed sequence satisfied the early stop constraint to illustrate how BEP prioritizes the states. Returning to the subject, the table shows BEP preferentially searches for states with a small depth. In order 14, even though the probability of state Q (1 , 1 , 3) is higher, BEP visits the state Q (3 , 0 , 2) with a lower depth.

Algorithm 1: Breadth-first Earley Parser (BEP)

→

·

C Comparison with the existing grammar induction algorithms

C.1 Existing grammar induction algorithms

Kuehne et al. [23] introduce a hierarchical context-free grammar induction algorithm. The root rule with the starting variable S is induced as S → V 1 | V 2 | ... | V N A , where the variable V i represents a single activity and N A is the number of activities from the dataset. Then each V i expands into action sequences from each activity, i.e. , the rule is formed as: V i →A i, 1 | A i, 2 | ... | A i, |A i | , where A i is a set of action sequences from the i -th activity and each A i,j represents a j -th action sequence in A i .

Richard et al. [35] propose a grammar induction method for a probabilistic right-regular grammar, where every rule has the form of ˜ H → c H . The algorithm is motivated by n-gram models [18] and finite grammars [14]. Specifically, the variable ˜ H represents an action sequence a 1: n , H represents an action sequence a 1: n -1 , and a terminal c is an action class of a n . The induced grammar can express the intermediate action sequences a 1: n and expands its rules based on the sequential order of actions.

Recently, Qi et al. [32] adopt the Automatic Distillation of Structure (ADIOS) [37] algorithm to induce a probabilistic context-free grammar. The ADIOS algorithm finds the significant patterns (AND rules) and equivalence action classes (OR rules) from the given action sequences. The algorithm identifies repetitive patterns in action sequences to minimize redundant sequences and find potential candidates for generalized action classes. Following the grammar induction methods of ADIOS, we

set a decreasing ratio of the motif extraction algorithm η to 1, a significance level for the decrease ratio γ to 0.1, and the context window size 1 for ADIOS-AND-induced grammar. For ADIOS-OR-induced grammar, we set η to 0.9, γ to 0.1, and the context window size to 4.

However, none of these approaches have managed to effectively integrate recursive rules, which are crucial for representing intricate and lifelike structures of action phrases and activities. The proposed KARI algorithm introduces a probabilistic context-free grammar that allows for the expression of complex activity structures, which captures a distinctive temporal structure based on key actions.

C.2 Performance on temporal action segmentation

In Table 10, we compare the performance of each grammar induction algorithm on refining temporal action segmentation models [45]. The overall results show that the KARI-induced grammar demonstrates the best refinement performance compared to the other grammar induction algorithms, showing a significant performance gap in both datasets. The induced grammar of Richard et al. also shows better performance than other grammar induction algorithms except for KARI, indicating that the ability to represent intermediate action sequences by production rules helps improve refinement performance. In conclusion, the generalization capabilities and variability of expressing action sequences are essential to guide the temporal action segmentation network to better refinement results.

D Qualitative results

We provide additional qualitative results for the Breakfast and 50Salads. Figure 7 shows examples of successful output refinements by the KARI-induced grammar, demonstrating its ability to cover various sequences comprising combinations of multiple actions. This is further evident in Figure 7a, where ADIOS-OR falls short in covering the 'add dressing' action following the 'serve salad' action, while KARI handles it proficiently.

We also show failure cases of the KARI-induced grammar in Figure 8, where further improvement is needed. We acknowledge that the KARI-induced grammar sometimes deletes certain actions. This deletion of actions, along with the challenges posed by inaccurate identification of actions by the segmentation model, show areas for improvement in the refinement process. We recognize these as opportunities for future work to enhance the performance of the grammar induction algorithm and address these limitations.

E Broader Impact

The research presented in this paper holds significant potential for impact across multiple domains. The development of efficient and effective grammar induction algorithms for activity grammar, coupled with the Breadth-first Earley parser, has the potential to greatly enhance human activity recognition and understanding systems. This, in turn, can have far-reaching implications in various

GT

ASFormer

ADIOS-OR

action start/end

cut lettuce

cut tomato

place lettuce

peel cucumber

cut cucumber

place cucumber

cut cheese

place cheese

mix ingredients

add oil

add vinegar

add salt

add pepper

mix dressing

add dressing

serve salad

(a) 50Salads

GT

ASFormer

ADIOS-OR

SIL

take bowl

pour cereals

pour milk

stir_cereals

(b) Breakfast (activity: cereals)

GT

ASFormer

ADIOS-OR

SIL

take cup

add_teabag

pour water

P07_webcam01_cereals

SILtake_bowlpour_cerealspour_milkstir_cerealsSIL

GT

ASFormer

ADIOS-OR

action start/end

add pepper

add salt

add oil

add vinegar

cut cheese

cut lettuce

mix dressing

peel cucumber

cut tomato

place cucumber into bowl

place cheese into bowl

cut cucumber

add dressing

mix ingredient

serve salad onto plate

place lettuce into bowl

(a) 50Salads

P07_webcam01_cereals

GT

ASFormer

ADIOS-OR

SIL

butter pan

crack egg

stirfry egg

add saltnpepper

take plate

put egg2plate

pour oil

(b) Breakfast (activity: scrambled egg)

SILtake_bowlpour_cerealspour_milkstir_cerealsSIL applications, such as video surveillance, human-computer interaction, robotics, and healthcare monitoring. By improving the accuracy and efficiency of activity recognition systems, our research contributes to advancements in these domains, enabling more robust and intelligent systems. The broader implications of this research extend beyond activity grammar induction itself, fostering innovation and enhancing the capabilities of intelligent systems in diverse fields.

Peer reviews:
comments: SUMMARY:
This paper proposes a grammar-based activity segmentation method. The authors proposed a grammar induction algorithm, as well as an improvement over the existing activity grammar parser, the Generalized Earley Parser (GEP). The proposed model shows improvements over prior works on both grammar induction and action segmentation.

STRENGTHS:
- The exploration of marrying neural-symbolic representations is an essential aspect of research, especially given the limited application of neural-symbolic methods for real-world problems.

- The proposed grammar induction method does improve existing prior works and could potentially be beneficial for future research.

WEAKNESSES:
- The experiments on activity segmentation are mainly compared with baselines that are not state-of-the-art. Though the ablations prove the improvement in grammar induction and parsing, the overall performance can not be justified by the current experiments (i.e., might need better results to justify the motivation of grammar-based activity understanding methods).

- Another concern is the design of grammar. Given recent advances in language modeling and unsupervised grammar induction, the motivation of symbolic methods is not very clear or not shown by the experiments.

- The overall formulation and algorithm design for the BEP follows from the GEP parser.

- The notation used in this paper is error-prone and needs to be clearer, especially given that the grammar-based method largely depends on these notations. (e.g. L.146, should it be $a_i^{M}$).

QUESTIONS:
See weakness section

LIMITATIONS:
The authors have properly addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new grammar induction algorithm, an effective parser, and a grammar evaluation framework. They improve temporal action segmentation by extracting and handling context-free grammar with recursive rules. The assessment presents good generalization and discrimination capabilities of induced grammar. \n\nAfter reading the rebuttal, my concern about the second and third points are resolved. For the first point, the authors provide the SOTA table as requested. However, the performance of the proposed method is not convincing.\n\nOverall, given that this paper introduces a new framework for temporal action segmentation, I still consider the pros outweigh the cons and will keep my initial recommendation.\n\nSTRENGTHS:\n1. KARI proposes a new grammar induction algorithm, an effective parser, and a grammar evaluation framework. They improve temporal action segmentation by extracting and handling context-free grammar with recursive rules. The assessment presents good generalization and discrimination capabilities of induced grammar. \n\n2. KARI outperforms state-of-the-art models on two benchmarks, Breakfast, and 50 Salads, for the temporal action segmentation task.\n\n3. The writing is good.\n\nWEAKNESSES:\n1. The evaluation of MIF is based on breakfast and 50salads. Can the author provide results on more benchmarks? The SOTA table is missing in the paper.\n2. The method is relatively new, but the details are mostly presented in the form of text. Some illustrations for the method's visual representation are missing. I think this is very beneficial to readers' understanding and the dissemination of the paper.\n3. For the first row of MS-TCN results in Table 2, is better than KARI, but the author did not bold it. Also, the author should give an explanation why KARI doesn't work here.\n\nQUESTIONS:\nSee weaknesses. \n\nIn the first row of Section 3.1, the citation is missing.\n\nLIMITATIONS:\nLimitations are not discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of temporal action segmentation by introducing an activity grammar to guide neural predictions. The proposed approach involves a grammar induction algorithm (KARI) to extract a powerful context-free grammar from action sequence data. Additionally, an efficient generalized parser (BEP) transforms frame-level probabilities into a reliable sequence of actions based on the induced grammar. Experimental results on benchmark datasets show significant improvements in both performance and interpretability of temporal action segmentation.\n\nSTRENGTHS:\nThis paper is well written which demonstrates its motivation, methodology and experiments. Especially, the method of this paper is easy to follow and the provided visualization is a plus to understand the proposed algorithm.\n\nThe idea of introducing the recursive rules is inspiring, which helps to identify the repetitions of actions and action phrases.  \n\nThe proposed grammar evaluation scheme demonstrates the effectiveness of the proposed KARI-induced activity grammar, achieving good recall while maintaining reasonable precision.\n\nThe experimental results look promising, achieving good performance on various benchmarks.\n\nWEAKNESSES:\nThe benchmark is relatively small, which generates concerns about the scalability of the proposed method. Also, it’s good to know the computational cost of the proposed method to evaluate its scalability. \n\nThe design of the proposed KARI is not well justified by using ablation study. It will be good to show the effectiveness of each component and its alternatives.\n\nMinor:\n\nMissing reference for ln 109 and 119.\n\nQUESTIONS:\nIs the proposed method limited by the action type? For example, it could only work for well-structured activities.\n\nLIMITATIONS:\nas is mentioned in previous sections.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a grammar induction algorithm that takes as input sequences of frame-level predictions and outputs structured sequences of actions. The advantage of their approach is that it allows recursive rules, which enhances its generalization abilities.\n\nSTRENGTHS:\n1. The method is more flexible than previously proposed grammar induction methods. \n2. The results are better compared to baselines.\n3. All the steps are explained in detail and most of them are well justified.\n\nWEAKNESSES:\n1. The grammar explanation (section 3.2) is confusing. Especially the large amount of superindices and subindices representing different concepts. For example, what does E^{M(m,n)} mean? Or, in \"t \\in {b, a, k(m, n)}\" what does the \"b\" represent? Probably simpler notation or a figure would help. I also did not understand what \"each sub-string a_i^M starts with a key action and includes all the key actions in K\" means. Do you start at a key action and do not close the sub-string until all key actions have been found? But then, I assume there will only be one such sub-string in the whole sequence. Is there any overlap between sub-string? Is it possible that a sequence does not contain any key action?\n\n2. I do not think the synthetic activity grammars are a good setting to judge the quality of the proposed approach. Without delving into the details of the generated random grammars, it is unclear whether the biases introduced in the generation may make them easier to deal with for some kinds of methods than others. Real-world grammars do not have that problem. \n\n3. There are no results showing the importance of the specific contribution. Did the authors identify cases where the inclusion of recursive rules made a difference?\n\nMinor weakness: a few typos and grammar mistakes: \"due to the reason\" -> \"due to this reason\" (lines 4, 22), \"and also be applied\" -> \"and can also be applied\" (line 29), \"we proceeds\" (line 137), \"has\" -> \"have\" (line 70), \"other researches\" (line 74), a few empty citations (lines 109, 119), etc.\n\nQUESTIONS:\n- Would it be possible to combine this approach to perform end-to-end training with the frame-level predictions Y? This is, train the grammar and the temporal action segmentation network simultaneously.\n\n- Accuracy does not improve when using grammars (it actually decreases). Why is that the case? I would assume that having contextual grammatical information about future and past actions would improve the accuracy of the current action prediction. Do you think the accuracy would improve with a flexible/good enough grammar, or this is a limitation of using action grammars?\n\nLIMITATIONS:\nLimitations and broader impact are discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}