SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dayoung Gong*
Joonseok Lee*
Deunsol Jung
Suha Kwak
Minsu Cho
Pohang University of Science and Technology (POSTECH)
{dayoung.gong, jameslee, deunsol.jung, suha.kwak, mscho}@postech.ac.kr
http://cvlab.postech.ac.kr/research/KARI
Sequence prediction on temporal data requires the ability to understand compositional structures of multi-level semantics beyond individual and contextual properties. The task of temporal action segmentation, which aims at translating an untrimmed activity video into a sequence of action segments, remains challenging for this reason. This paper addresses the problem by introducing an effective activity grammar to guide neural predictions for temporal action segmentation. We propose a novel grammar induction algorithm that extracts a powerful context-free grammar from action sequence data. We also develop an efficient generalized parser that transforms frame-level probability distributions into a reliable sequence of actions according to the induced grammar with recursive rules. Our approach can be combined with any neural network for temporal action segmentation to enhance the sequence prediction and discover its compositional structure. Experimental results demonstrate that our method significantly improves temporal action segmentation in terms of both performance and interpretability on two standard benchmarks, Breakfast and 50 Salads.
Human activities in videos do not proceed by accident; they are structured being subject to generative rules imposed by the goal of activities, the properties of individual actions, the physical environment, and so on. Comprehending such a compositional structure of multi-granular semantics in human activity poses a significant challenge in video understanding research. The task of temporal action segmentation, which aims at translating an untrimmed activity video into a sequence of action segments, remains challenging due to the reason. The recent methods based on deep neural networks [25, 9, 45, 2, 15, 16, 1] have shown remarkable improvement in learning temporal relations of actions in an implicit manner, but often face out-of-context errors that reveal the lack of capacity to capture the intricate structures of human activity, and the scarcity of annotated data exacerbates the issue in training. In this work, we address the problem by introducing an effective activity grammar to guide neural predictions for temporal action segmentation.
Grammar is a natural and powerful way of explicitly representing the hierarchical structure of languages [14] and can also be applied to express the structure of activities. Despite the extensive body of grammar-based research for video understanding [23, 24, 33, 35, 32], none of these approaches have successfully integrated recursive rules. Recursive rules are indispensable for expressing complex and realistic structures found in action phrases and activities. To achieve this, we introduce a novel activity grammar induction algorithm, Key-Action-based Recursive Induction (KARI), that extracts a powerful probabilistic context-free grammar while capturing the characteristics of the activity. Since an activity is composed of multiple actions, each activity exhibits a distinctive temporal structure based on pivotal actions, setting it apart from other activities. The proposed grammar induction
*Equal contribution.
enables recursive rules with flexible temporal orders, which leads to powerful generalization capability. We also propose a novel activity grammar evaluation framework to evaluate the generalization and discrimination power of the proposed grammar induction algorithm. To incorporate the induced activity grammar into temporal action segmentation, we develop an effective parser, dubbed BEP, which searches the optimal rules according to the classification outputs generated by an off-the-shelf action segmentation model. Our approach can be combined with any neural network for temporal action segmentation to enhance the sequence prediction and discover its compositional structure.
The main contribution of this paper can be summarized as follows:
- We introduce a novel grammar induction algorithm that extracts a powerful context-free grammar with recursive rules based on key actions and temporal dependencies.
- We develop an effective parser that efficiently handles recursive rules of context-free grammar by using Breadth-first search and pruning.
- We propose a new grammar evaluation framework to assess the generalization and discrimination capabilities of the induced activity grammars.
- We show that the proposed method significantly improves the performance of temporal action segmentation models, as demonstrated through a comprehensive evaluation on two benchmarks, Breakfast and 50 Salads.
Grammar for activity analysis. Grammar is an essential tool to represent the compositional structure of language [14] and has been mainly studied in the context of natural language processing (NLP) [21, 22, 20, 37]. Grammar has been extensively studied in various research areas [28, 29, 32, 8, 43, 13, 42, 11, 27, 12]. Similarly, a grammatical framework can be used to express the structure of activities. Several work [23, 24, 33, 35] have defined context-free grammars based on possible temporal transitions between actions for action detection and recognition. Vo and Bovick [41] propose a stochastic grammar to model a hierarchical representation of activity based on AND-rules and OR-rules. Richard et al. [34] propose a context-free grammar defined on action sequences for weakly-supervised temporal action segmentation. Qi et al. [30, 32, 31] utilize a grammar induction algorithm named ADIOS [37] to induce grammar from action corpus. However, none of the proposed grammar for activity analysis includes recursive rules, which are a fundamental factor in expressing repetitions of actions or action phrases. In this paper, we propose a novel action grammar for temporal action segmentation based on key action and temporal dependency between actions considering recursive temporal structure.
Temporal action segmentation (TAS). Various methods have been proposed to address the task. Early work utilizes temporal sliding windows [36, 19] to detect action segments, and language-based methods [24, 23] has been proposed to utilize a temporal hierarchy of actions during segmentation. Recently, a deep-learning-based model named the temporal convolutional networks (TCN) has been proposed with an encoder-decoder architecture [25, 9]. Moreover, transformer-based models [45, 2] are recently introduced to leverage global temporal relations between actions based on self-attention and cross-attention mechanisms [40]. Other researches have been proposed to improve the accuracy of temporal action segmentation based on existing models [9, 45]. Huang et al. [15] introduce a network module named Graph-based Temporal Reasoning Module (GTRM) that is applied on top of baseline models to learn temporal relations of action segments. Ishikawa et al. [16] suggest an action segment refinement framework (ASRF) dividing a task into frame-wise action segmentation and boundary regression. They refine frame-level classification results with the predicted action boundaries. Gao et al. [10] propose a global-to-local search scheme to find appropriate receptive field combinations instead of heuristic respective fields. Ahn and Lee [1] recently propose a hierarchical action segmentation refiner (HASR), which refines segmentation results by applying multi-granular context information from videos. A fast approximate inference method named FIFA for temporal action segmentation and alignment instead of dynamic programming is proposed by Souri et al. [38]. Other researches [5, 6] reformulate TAS as a cross-domain problem with different domains of spatio-temporal variations, introducing self-supervised temporal domain adaptation. Xu et al. [44] proposes differentiable temporal logic (DTL), which is a model-agnostic framework to give temporal constraints to neural networks. In this paper, we propose a neuro-symbolic approach where the activity grammar induced by the proposed grammar induction algorithm guides a temporal action segmentation model to refine segmental errors through parsing.
Figure 1: Overall pipeline of the proposed method. (a) KARI induces an activity grammar  $G$  from action sequences in the training data, (b) BEP parses neural predictions  $Y$  from the off-the-shelf temporal action segmentation model given a video  $F$  by using the KARI-induced grammar  $G$ , and (c) the final output of optimal action sequences and lengths  $(a^{*}, l^{*})$  is achieved through segmentation optimization. It is best viewed in color.
3 Our approach
Given a video of  $T$  frames  $\mathbf{F} = [F_{1}, F_{2}, \dots, F_{T}]$  and a predefined set of action classes  $\mathcal{A}$ , the goal of temporal action segmentation is to translate the video into a sequence of actions  $\mathbf{a} = [a_{1}, a_{2}, \dots, a_{N}]$  and their associated frame lengths  $l = [l_{1}, l_{2}, \dots, l_{N}]$  where  $N$  is unknown,  $a_{i} \in \mathcal{A}$  for  $1 \leq i \leq N$ ,  $a_{i} \neq a_{i+1}$  for  $1 \leq i \leq N-1$ , and  $\sum_{i=1}^{N} l_{i} = T$ . The resultant output of  $\mathbf{a}$  and  $l$  indicates that the video consists of  $N$  segments and each pair  $(a_{i}, l_{i})$  represents the action and length of  $i_{\mathrm{th}}$  segment.
In this work, we introduce an activity grammar that guides neural predictions for temporal action segmentation through parsing. We propose a novel activity grammar induction algorithm named KARI and an efficient parser called BEP. The overall pipeline of the proposed method consists of three steps, as illustrated in Fig. 1. First of all, KARI induces an activity grammar from action sequences in the training data. Using the KARI-induced grammar, BEP then takes the frame-level class prediction  $\mathbf{Y} \in \mathbb{R}^{T \times |\mathcal{A}|}$  from the off-the-shelf temporal action segmentation model [45, 9] and produces a grammar-consistent action sequence  $\mathbf{a}^*$ . Finally, segmentation optimization is performed to obtain optimal action lengths  $l^*$  based on  $\mathbf{a}^*$  and  $\mathbf{Y}$ . In the following, we introduce the activity grammar as a probabilistic context-free grammar (Section 3.1), present KARI (Section 3.2) and BEP (Section 3.3), and describe a segmentation optimization method for final outputs (Section 3.4).
3.1 Activity grammar
We define the activity grammar as a probabilistic context-free grammar (PCFG) [17], designed to derive diverse action sequences pertaining to a specific activity class. The activity grammar, denoted as  $G = (\mathcal{V}, \Sigma, \mathcal{P}, S)$ , follows the conventional PCFG which consists of four components: a finite set of variables  $\mathcal{V}$ , a finite set of terminals  $\Sigma$ , a finite set of production rules  $\mathcal{P}$ , and the start symbol  $S \in \mathcal{V}$ . In our context, the set of terminals  $\Sigma$  becomes the set of action classes  $\mathcal{A}$ , and the production rules  $\mathcal{P}$  are used to generate action sequences from the start variable  $S$ . We use two types of production rules, 'AND' and 'OR', defined as follows:
$$
\text {A N D}: V \rightarrow \alpha \quad \text {w h e r e} V \in \mathcal {V} \text {a n d} \alpha \in (\Sigma \cup \mathcal {V}) ^ {*}, \tag {1}
$$
$$
\mathrm {O R}: \quad V \rightarrow V _ {1} [ p _ {1} ] \mid V _ {2} [ p _ {2} ] \mid \dots \mid V _ {n} [ p _ {n} ] \quad \text {w h e r e} V, V _ {1}, \dots , V _ {n} \in \mathcal {V}. \tag {2}
$$
The AND rule replaces a head variable  $V$  with a sequence of variables and terminals  $\alpha$ , determining the order of the terminals and variables. In contrast, the OR rule converts a head variable  $V$  to a sub-variable  $V_{i}$  with the probability  $p_i$ , providing multiple alternatives for replacement; 'l' denotes 'OR' operation. These two types of rules allow us to generate action sequences hierarchically.
3.2 Grammar induction: Key-Action-based Recursive Induction (KARI)
Grammar induction refers to the process of learning grammars from data [37]. In our context, it takes action sequences of a specific activity in the training set and produces an activity grammar that is
2In fact, this form of output is equivalent to that of frame-level action classification, which predict an action class for each frame, and the sequence of frame-level actions is easily converted to  $(\pmb{a},\pmb{l})$  and vice versa.
-  $a_1 = [\text{pour coffee, pour milk}]$
-  $a_2 = [\text{pour coffee, spoon sugar, pour milk}]$
-  $a_3 =$  [take cup, pour coffee, pour milk, spoon sugar, stir coffee]
-  $a_4 =$  [take cup, pour coffee, spoon sugar, stir coffee]
(a) Action sequences in the training set  $D$
(b) Sub-sequence partitioning based on the key action
[ D^{\mathrm{L}} = \{[\epsilon], [\text{take cup}]\}, D^{\mathrm{M}} = \{[\text{pour coffee}]\}, ]
$D^{\mathrm{R}} = \{$  [pour milk], [spoon sugar, pour milk], [pour milk, spoon sugar, stir coffee], [spoon sugar, stir coffee]
(c) The set of sub-sequences  $D^{\mathrm{L}}, D^{\mathrm{M}}$ , and  $D^{\mathrm{R}}$
-  $A^{\mathrm{L}} = \{\text{take cup}\}$ ,  $A^{\mathrm{M}} = \{\text{pour coffee}\}$ ,  $A^{\mathrm{R}} = \{\text{pour milk, spoon sugar, stir coffee}\}$
(d) The action set  $A^{\mathrm{L}}, A^{\mathrm{M}}$ , and  $A^{\mathrm{R}}$
-  $\pmb{d}^{\mathrm{L}} = [\{\mathrm{take~cup}\} ],\pmb{d}^{\mathrm{M}} = [\{\mathrm{pour~coffee}\} ],\pmb{d}^{\mathrm{R}} = [\{\mathrm{pour~milk},$  spoon sugar}, \{stir coffee\}]
(e) The dependency sequence  $\pmb{d}^{\mathrm{L}}, \pmb{d}^{\mathrm{M}}$ , and  $\pmb{d}^{\mathrm{R}}$
$S\to V^{\mathrm{L}}V^{\mathrm{M}}V^{\mathrm{R}}$
-  $V^{\mathrm{L}} \rightarrow \text{take cup}'| \epsilon$
-  $V^{\mathrm{M}} \rightarrow$  'pour coffee'
-  $V^{\mathrm{R}}\to V_{1}^{\mathrm{R}}V_{2}^{\mathrm{R}}$
-  $V_{1}^{R} \rightarrow {}^{\prime}$  pour milk'  $V_{1}^{R}|$  spoon sugar'  $V_{1}^{R}|\epsilon$
-  $V_{2}^{R} \rightarrow$  'stir coffee'  $V_{2}^{R}|$ $\epsilon$
(f) KARI-induced activity grammar
Figure 2: Example of activity grammar induction of KARI. (a) Example action sequences are provided with 'pour coffee' as the key action with  $N^{\mathrm{key}}$  set to 1. (b) Action sequence, e.g.,  $a_3$ , is segmented into sub-sequences based on key actions. (c) All action sub-sequences  $a^\Omega$  consist in a set of sub-sequences  $\mathcal{D}^\Omega$ . (d) The action set  $\mathcal{A}^\Omega$  contains all the actions occurring in  $\mathcal{D}^\Omega$ . (e) Temporally independent actions are grouped, where each action group is temporally dependent in the action group sequence  $d^\Omega$ . (f) The resultant KARI-induced activity grammar is shown. For simplicity, we omit the probability, and it is best viewed in color.
able to parse action sequences of the activity; the induced grammar should be able to parse unseen sequences of the activity as well as the sequences in the training set for generalization. To obtain an effective activity grammar avoiding under-/over-generalization, we introduce two main concepts for grammar induction: key action and temporal dependency.
The key actions for a specific activity are those consistently present in every action sequence from the training dataset. Specifically, the top  $N^{\text{key}}$  most frequently occurring actions among these are selected as the key actions. The hyperparameters of the number of key actions  $N^{\text{key}}$  affects the degree of generalization achieved by the induced grammar. The temporal dependency refers to the relevance of temporal orders across actions. Temporally independent actions do not occur in a specific temporal order. This concept of temporal dependency can also be extended to groups of actions, meaning that some groups of actions can be temporally dependent on others.
We induce an activity grammar based on the key actions and the temporal dependency. Action sequences are divided into sub-sequences using the key actions as reference points, and the temporal dependencies between actions within the sub-sequences are established; temporally dependent actions are represented using AND rules (Eq. 1), while temporally independent actions are expressed with OR rules (Eq. 2). We give an example of grammar induction in Fig. 2; four action sequences are given in Fig. 2a, where the action class 'pour coffee' is chosen as the key action with the number of key actions  $N^{\mathrm{key}}$  set to 1.
Given the action sequences from the training dataset  $\mathcal{D}$ , we begin grammar induction by identifying a set of key actions  $\mathcal{K} \subset \mathcal{A}$  with the pre-defined hyperparameter  $N^{\mathrm{key}}$ . Using the key actions, each action sequence  $\pmb{a} \in \mathcal{D}$  is divided into three parts:  $\pmb{a} = [a^{\mathrm{L}}, a^{\mathrm{M}}, a^{\mathrm{R}}]$ . The sub-sequences  $\pmb{a}^{\mathrm{L}}, \pmb{a}^{\mathrm{M}}$  and  $\pmb{a}^{\mathrm{R}}$  denote the portions of the original action sequence that occurred before, between, and after the key actions, respectively; the sub-sequence  $\pmb{a}^{\mathrm{M}}$  starts from the first key action and includes up to the last key action in  $\mathcal{K}$ . An example in Fig. 2b shows that the action sequence  $\pmb{a}_3$  is divided into three sub-sequences using the key actions. For notational convenience, we will use the superscript  $\Omega \in \{\mathrm{L}, \mathrm{M}, \mathrm{R}\}$  to denote one of the three parts. All action sub-sequences  $\pmb{a}^\Omega$  in a specific part  $\Omega$  are grouped to consist in a corresponding set of sub-sequences  $\mathcal{D}^\Omega$  (cf. Fig 2c). The action set  $\mathcal{A}^\Omega \subseteq \mathcal{A}$
is then defined to contain all the actions occurring in  $\mathcal{D}^{\Omega}$  (cf. Fig 2d). To determine the temporal dependencies among the actions of  $\mathcal{A}^{\Omega}$ , pairwise temporal orders are considered as follows. If one action always occurs before the other in  $\mathcal{D}^{\Omega}$ , then the two actions are temporally dependent and otherwise temporally independent. Based on the concepts, we construct the action group sequence  $d^{\Omega}$  by collecting the temporally independent actions as an action group and arranging such action groups according to their temporal dependencies (cf. Fig 2e).
In the following, we describe how to construct the production rules  $\mathcal{P}$  of the activity grammar  $G$ .
Start rule. We first create the rule for the start variable  $S$ :
$$
S \rightarrow V ^ {\mathrm {L}} V ^ {\mathrm {M}} V ^ {\mathrm {R}}, \tag {3}
$$
where  $V^{\mathrm{L}}$ ,  $V^{\mathrm{M}}$ , and  $V^{\mathrm{R}}$  are variables used to derive left, middle, and right parts of the action sequence, respectively.
Rule for the variable  $V^{\Omega}$ . For  $V^{\Omega}, \Omega \in \{\mathrm{L}, \mathrm{R}\}$ , we construct an AND rule of action groups based on action group sequence  $d^{\Omega}$ :
$$
V ^ {\Omega} \rightarrow V _ {1} ^ {\Omega} V _ {2} ^ {\Omega} \dots V _ {| \boldsymbol {d} ^ {\Omega} |} ^ {\Omega}, \tag {4}
$$
where the variable  $V_{i}^{\Omega}$  represents the  $i_{\mathrm{th}}$  action group in the action group sequence  $d_{i}^{\Omega}$ . Since actions in an action group are considered temporally independent, we construct an OR rule for each action group:
$$
V _ {i} ^ {\Omega} \rightarrow d _ {i, 1} ^ {\Omega} V _ {i} ^ {\Omega} [ p _ {i, 1} ^ {\Omega} ] \mid d _ {i, 2} ^ {\Omega} V _ {i} ^ {\Omega} [ p _ {i, 2} ^ {\Omega} ] \mid \dots \mid d _ {i, | \boldsymbol {d} ^ {\Omega} |} ^ {\Omega} V _ {i} [ p _ {i, | \boldsymbol {d} ^ {\Omega} |} ^ {\Omega} ] \mid \epsilon [ p _ {i, \epsilon} ^ {\Omega} ], \tag {5}
$$
where  $d_{i,j}^{\Omega}$  denotes the  $j_{\mathrm{th}}$  action from the action group  $\pmb{d}_i^\Omega$ . The variable  $V_{i}^{\Omega}$  yields  $d_{i,j}^{\Omega}V_{i}^{\Omega}$  with the probability  $p_{i,j}^{\Omega}$ . This rule can be recursively used to proceed to the variable  $V_{i}^{\Omega}$  in the next step. This recursive structure allows for repeated selection of actions within the same action group, leading to the generation of diverse action sequences, which is effective for generalization. To avoid an infinite loop of the recursion, the empty string  $\epsilon$  with the escape probability  $p_{i,\epsilon}^{\Omega}$  is added to Eq. 5. For the details, refer to the transition probability  $p_{i,j}^{\Omega}$  and the escape probability  $p_{i,\epsilon}^{\Omega}$  in Appendix A.1.
Rule for the middle variable  $V^{\mathrm{M}}$ . Since the temporal order of key actions might vary, we consider all the possible temporal orders between key actions in  $\mathcal{K}$ . A set of temporal permutations of actions is denoted as  $\Pi$ , where each possible temporal permutation is represented by the OR rule:
$$
V ^ {\mathrm {M}} \rightarrow V _ {1} ^ {\mathrm {M}} \left[ p _ {1} ^ {\mathrm {M}} \right] \mid V _ {2} ^ {\mathrm {M}} \left[ p _ {2} ^ {\mathrm {M}} \right] \mid \dots \mid V _ {| \Pi |} ^ {\mathrm {M}} \left[ p _ {\Pi} ^ {\mathrm {M}} \right] \mid \epsilon \left[ p _ {\epsilon} ^ {\mathrm {M}} \right]. \tag {6}
$$
The rule for the permutation variable  $V_{i}^{\mathrm{M}}$  is defined by the AND rule:
$$
V _ {i} ^ {\mathrm {M}} \rightarrow \pi_ {i, 1} V ^ {\mathrm {M} (i, 1)} \dots \pi_ {i, | \boldsymbol {\pi} _ {i} |} V ^ {\mathrm {M} (i, | \boldsymbol {\pi} _ {i} |)} V ^ {\mathrm {M}}, \tag {7}
$$
where all the key actions are included. Note that  $\pi_{i,j}$  represents the  $j_{\mathrm{th}}$  action of the permutation  $\pi_i\in \Pi$ , and the variable  $V^{\mathrm{M}(i,j)}$  derives action sub-sequences between actions  $\pi_{i,j}$  and  $\pi_{i,j + 1}$ . The production rule for  $V^{\mathrm{M}(i,j)}$  adheres to the rules specified in Eq. 4 and 5. The resultant KARI-induced grammar from the example is shown in Fig. 2f, highlighting the compositional structure of actions.
3.3 Parser: Breadth-first Earley Parser (BEP)
The goal of the parser is to identify the optimal action sequence  $\mathbf{a}^*$  by discovering the most likely grammatical structure based on the output of the action segmentation model [9, 45]. In other words, the parser examines the production rules of the activity grammar to determine whether the given neural prediction  $\mathbf{Y}$  can be parsed by the grammar  $G$ . However, when the grammar includes recursive rules, the existing parser struggles to complete the parsing within a reasonable time due to the significant increase in branches from the parse tree. To address this challenge, we introduce an effective parser dubbed BEP, integrating Breadth-first search (BFS) and a pruning technique into a generalized Earley parser (GEP) [32]. Since the BFS prioritizes production rules closer to the start variable, it helps the parser understand the entire context of the activity before branching to recursive iterations. Simultaneously, pruning effectively reduces the vast search space generated by OR nodes and recursion, enabling the parser to focus on more relevant rules for the activity.
For parsing, we employ two heuristic probabilities introduced in [32] to compute the probability of variables and terminals within the parse tree. Specifically, let  $\mathbf{Y}_{t,x}$  denote the probability of frame  $t$
being labeled as  $x$ . In this context, we denote the last action in the action sequence  $\mathbf{a}$  as  $x$ , i.e.  $x = a_{N}$  where  $\mathbf{a} = [a_1, a_2, \dots, a_N]$ , for simplicity. The transition probability  $g(x \mid \mathbf{a}_{1:N-1}, G)$  determines the probability of parsing action  $x$  given the  $\mathbf{a}_{1:N-1}$  and the grammar  $G$ .
The parsing probability  $p(\pmb{F}_{1:T} \to \pmb{a} \mid G)$  computes the probability of  $\pmb{a}$  being the action sequence for  $\pmb{F}_{1:T}$ . The probability at  $t = 1$  is initialized by:
$$
p \left(F _ {1} \rightarrow \boldsymbol {a} \mid G\right) = \left\{\begin{array}{l l}g (x \mid \epsilon , G) \mathbf {Y} _ {1, x}&\text {i f} \boldsymbol {a} \text {c o n t a i n s o n l y} x,\\0&\text {o t h e r w i s e},\end{array}\right. \tag {8}
$$
where  $\epsilon$  indicates an empty string.