SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since we assume that the last action of  $\pmb{a}$  is classified as  $x$ , the parsing probability  $p(\pmb{F}_{1:t} \to \pmb{a} \mid G)$  can be represented with the probability of the previous frames:
$$
p \left(\boldsymbol {F} _ {1: t} \rightarrow \boldsymbol {a} \mid G\right) = \boldsymbol {Y} _ {t, x} \left(p \left(\boldsymbol {F} _ {1: t - 1} \rightarrow \boldsymbol {a} \mid G\right) + g (x \mid \boldsymbol {a} _ {1: N - 1}, G) p \left(\boldsymbol {F} _ {1: t - 1} \rightarrow \boldsymbol {a} _ {1: N - 1} \mid G\right)\right). \tag {9}
$$
The prefix probability  $p(\pmb{F}_{1:T} \to \pmb{a} \dots | G)$  represents the probability of  $\pmb{a}$  being the prefix of  $\pmb{a}^*$ . This probability is computed by measuring the probability that  $\pmb{a}$  is the action sequence for the frame  $\pmb{F}_{1:t}$  with  $t$  in the range  $[1, T]$ :
$$
p \left(\boldsymbol {F} _ {1: T} \rightarrow \boldsymbol {a} \dots \mid G\right) = p \left(F _ {1} \rightarrow \boldsymbol {a} \mid G\right) + g (x \mid \boldsymbol {a} _ {1: N - 1}, G) \sum_ {t = 2} ^ {T} \boldsymbol {Y} _ {t, x} p \left(\boldsymbol {F} _ {1: t - 1} \rightarrow \boldsymbol {a} _ {1: N - 1} \mid G\right). \tag {10}
$$
The parsing operation is structured following the original Earley parser [7], consisting of three key operations: prediction, scanning, and completion. These operations involve the update and generation of states, where every state comprises the rule being processed, the parent state, the parsed action sequence denoted as  $\mathbf{a}$ , and the prefix probability denoted as  $p(\mathbf{a}\dots)$ . The states are enqueued and prioritized by their depth  $d$  within the parse tree.
- Prediction: for every state  $Q(m, n, d)$  of the form  $(A \to \alpha \cdot B\beta, Q(i, j, k), a, p(a...))$ , add  $(B \to \gamma \cdot \Gamma, Q(m, n, d), a, p(a...))$  to  $Q(m, n, d + 1)$  for every production rule in the grammar with  $B$  on the left-hand side.
- Scanning: for every state in  $Q(m, n, d)$  of the form  $(A \to \alpha \cdot w\beta, Q(i, j, k), \pmb{a}, p(\pmb{a}...))$ , append the new terminal  $w$  to  $\pmb{a}$  and compute the probability  $p((\pmb{a} + w)...)$ . Create a new set  $Q(m + 1, n', d)$  where  $n'$  is the current size of  $Q(m + 1)$ . Add  $(A \to \alpha w \cdot \beta, Q(i, j, k), \pmb{a} + w, p((\pmb{a} + w)...))$  to  $Q(m + 1, n', d)$ .
- Completion: for every state in  $Q(m, n, d)$  of the form  $(A \to \Gamma \cdot, Q(i, j, k), a, p(a \ldots))$ , find states in  $Q(i, j, k)$  of the form  $(B \to \alpha \cdot A\beta, Q(i', j', k'), a', p(a' \ldots))$  and add  $(B \to \alpha A \cdot \beta, Q(i', j', k'), a, p(a \ldots))$  to  $Q(m, n, d - 1)$ .
The symbols  $\alpha, \beta$ , and  $\Gamma$  represent arbitrary strings consisting of terminals and variables, i.e.  $\alpha, \beta, \Gamma \in (\Sigma \cup V)^*$ . The symbols  $A$  and  $B$  refer to the variables, while  $w$  denotes a single terminal. The symbol  $Q$  represents the set of states, and the dot  $(\cdot)$  denotes the current position of the parser within the production rule.
Additionally, we introduce a pruning technique of limiting the queue size to reduce the vast search space in the parse tree, similar to the beam search. Specifically, the parser preserves only the top  $N^{\mathrm{queue}}$  elements from the queue in order of the parsing probability of each state. The parsing process terminates when the parser identifies that the parsed action sequence  $\pmb{a}^{*}$  has a higher parsing probability than the prefix probabilities of any other states in the queue. For the further details, refer to Appendix B.
3.4 Segmentation optimization
The objective of segmentation optimization is to determine the optimal alignment between the input classification probability matrix  $\mathbf{Y}$  and the action sequence  $\mathbf{a}^*$ . In other words, the entire frames are allocated within the action sequences  $\mathbf{a}^* = [a_1^*, a_2^*, \dots, a_N^*]$ , obtained from the parser, to determine the optimal action lengths  $l^* = [l_1^*, l_2^*, \dots, l_N^*]$ . In this work, we utilize dynamic programming-based Viterbi-like algorithm [35] for activity parsing. Similar to [26, 32], the optimizer explores all possible allocations and selects the one with the maximum product of probabilities:
$$
\boldsymbol {l} ^ {*} = \underset {\boldsymbol {l}} {\arg \max } (p (\boldsymbol {l} \mid \boldsymbol {a} ^ {*}, \boldsymbol {Y} _ {1: T})), \tag {11}
$$
$$
p (\boldsymbol {l} \mid \boldsymbol {a}, \boldsymbol {Y} _ {1: t}) = \max  _ {i <   t} (p \left(\boldsymbol {l} _ {1: N - 1} \mid \boldsymbol {a} _ {1: N - 1}, \boldsymbol {Y} _ {1: i}\right) \prod_ {j = i} ^ {t} \boldsymbol {Y} _ {j, a _ {N}}). \tag {12}
$$
Figure 3: Grammar evaluation
Figure 4: Confusion matrix of activity grammars. The results of KARI-induced grammar are similar to the synthetic grammar, showing high recall with comparable precision.
Table 1: Synthetic  $G\mathbf{I}$
grammar	precision	recall
ADIOS-AND	0.97	0.08
ADIOS-OR	0.99	0.25
KARI (ours)	0.93	0.98
Table 2: Synthetic  $G$  II
grammar	precision	recall
ADIOS-AND	0.98	0.06
ADIOS-OR	1.00	0.33
KARI (ours)	0.92	0.96
4 Experimental evaluation and analysis
4.1 Datasets and evaluation metrics
Datasets. We conduct experiments on two widely used benchmark datasets for temporal action segmentation: Breakfast [23] and 50 Salads [39]. The Breakfast dataset, consisting of 1,712 videos, involves 52 individuals preparing 10 different breakfast activities comprised of 48 actions in 18 different kitchens. Similarly, the 50 Salads dataset comprises 50 egocentric videos of people preparing salads of a single activity with 17 fine-grained actions from 25 people. We used I3D [4] features provided by [9].
Evaluation metrics. For evaluation metrics, we report edit score, F1@{10,25,50} scores, and frame-wise accuracy following the previous work [9, 45].
4.2 Implementation details
For KARI, we set the hyperparameters of the number of key actions  $N^{\mathrm{key}}$  to 4 for Breakfast, and 3 for 50 Salads. We individually induce separate activity grammar for the ten activity classes within Breakfast and subsequently merge them into a unified grammar. For the comparison with the existing grammar used in the previous work [32, 31], we induce activity grammars of ADIOS [37] provided by [31]. Two types of ADIOS-induced grammar are induced: ADIOS-AND-induced grammar, primarily composed of AND rules with limited generalization capabilities, and ADIOS-OR-induced grammar, predominantly incorporating OR rules, offering improved generalization. Please refer to Appendix C.1 for grammar induction details.
For BEP, we configured the queue size  $N^{\text{queue}}$  to be 20. For efficiency, we adjust the sampling rate of the input video features to 50 for Breakfast and 100 for 50 Salads. We use two widely used models for the temporal action segmentation: ASFormer [45] based on Transformer and MS-TCN [9] based on CNNs. Since we apply the proposed method to the reproduced temporal action segmentation models, we directly compare and evaluate the performance based on the reproduced results.
4.3 Evaluation framework for activity grammar
We propose a novel evaluation framework to assess the generalization and discrimination capabilities of the activity grammar. Figure 3 shows the overall process of the grammar evaluation framework. We first generate a set of synthetic activity grammars  $\mathcal{G}^{\mathrm{S}}$  randomly. Action sequences  $a\in \mathcal{D}_i^{\mathrm{all}}$  are generated from each synthetic grammar  $G_{i}^{\mathrm{S}}\in \mathcal{G}^{\mathrm{S}}$ , and these sequences are randomly divided into two sets: seen seen and unseen. For each seen set, a grammar induction algorithm is applied, resulting in the induced grammar  $G_{i}^{\mathrm{I}}$  consisting in a corresponding set of induced grammars  $\mathcal{G}^{\mathrm{I}}$ . For grammar evaluation, the induced grammar  $G_{i}^{\mathrm{I}}\in \mathcal{G}^{\mathrm{I}}$  parses action sequences from the entire unseen sets. The induced grammar should accurately parse the action sequences generated by the original synthetic grammar from which it was induced, while also effectively discriminating those generated by other synthetic grammars.
To simulate real-world video action sequences, we generate the synthetic activity grammars assuming temporal dependencies across actions. This indicates that certain actions follow a temporal order
Table 3: The performance comparison on 50Salads
model	re-prod.	refine-ment	grammar induction	edit	F1@10	F1@25	F1@50	acc.
ASFormer [45]	-	-	-	75.0	76.0	70.6	57.4	73.5
✓	-	-	76.5	83.8	81.7	74.8	86.1
✓	✓	ADIOS-AND	58.3	70.0	68.0	59.4	76.2
✓	✓	ADIOS-OR	61.1	72.0	70.1	62.4	78.9
✓	✓	KARI	79.9	85.4	83.8	77.4	85.3
MS-TCN [9]	-	-	-	67.9	76.3	74.0	64.5	80.7
✓	-	-	62.4	69.5	65.3	55.7	75.2
✓	✓	ADIOS-AND	56.8	66.4	63.8	52.9	72.5
✓	✓	ADIOS-OR	61.9	69.1	66.9	57.2	74.2
✓	✓	KARI	66.7	75.1	73.2	60.8	76.7
Table 4: The performance comparison on Breakfast
model	re-prod.	refine-ment	grammar induction	edit	F1@10	F1@25	F1@50	acc.
ASFormer [45]	-	-	-	75.0	76.0	70.6	57.4	73.5
✓	-	-	75.6	77.3	72.0	59.4	74.3
✓	✓	ADIOS-AND	69.2	69.8	64.9	52.2	72.4
✓	✓	ADIOS-OR	70.3	71.8	66.8	54.2	71.8
✓	✓	KARI	77.8	78.8	73.7	60.8	74.0
MS-TCN [9]	-	-	-	61.7	52.6	48.1	37.9	66.3
✓	-	-	69.7	70.7	65.1	52.6	69.4
✓	✓	ADIOS-AND	68.0	66.7	61.0	48.0	68.4
✓	✓	ADIOS-OR	69.6	69.2	63.3	50.3	68.2
✓	✓	KARI	74.9	74.6	68.7	55.1	68.8
while others do not adhere to such dependencies. To prevent parsing failures arising from uncovered terminals, we maintain a consistent set of terminals throughout the entire grammar while randomly assigning key actions to these terminals. The number of variables is randomly determined for each synthetic grammar. As evaluation metrics, we use precision and recall similar to the previous work [37, 3]. For the induced grammar  $G_{i}^{\mathrm{I}}$ , action sequences successfully parsed from the synthetic grammar  $G_{i}^{\mathrm{S}}$  are classified as positive samples from the entire unseen sets, otherwise considered negative samples.
Details. In our experiment, we generate a total of 100 grammars, each consisting of 20 variables and 20 terminals. We have developed two types of synthetic grammars that differ in terms of temporal hierarchical difficulty. In synthetic grammar I, each terminal is allocated to a single variable, while in synthetic grammar II, terminals are randomly assigned multiple times to different variables. Three types of grammars are evaluated: induced by ADIOS-AND, ADIOS-OR, and proposed KARI.
Results. Table 1 and Table 2 show the results of grammar evaluation by using synthetic grammar I and II, respectively. Our activity grammar demonstrates robust generalization performances, achieving a recall of approximately 1.0 on unseen action sequences compared to others, maintaining comparable precision. The ADIOS-OR-induced grammar shows better generalization ability compared to the ADIOS-AND-induced grammar. We visualize a confusion matrix of the three types of grammar: synthetic grammar, KARI-induced grammar, and ADIOS-OR-induced grammar, as shown in Fig. 4. The confusion matrix shows the parsing accuracy of each unseen set over the synthetic grammar II. Higher accuracy is represented by brighter cells in the matrix. KARI-induced grammars demonstrate similar patterns in their confusion matrix compared to the synthetic grammars. This similarity indicates their capacity to generalize to unseen sets from which each grammar is induced, allowing effective discrimination of action sequences from other synthetic grammars.
4.4 Effects of the grammar-based refinement on temporal action segmentation
Table 3 and Table 4 show the performance of applying the proposed method to temporal action segmentation models [9, 45] across two benchmark datasets. The first row in Table 3 and Table 4 indicates the performance from the original paper [9, 45], whereas the second row represents the
Table 5: Ablation study of KARI on 50 Salads. Using both key action and recursive rules is effective for refining neural predictions from the temporal action segmentation models.
refinement	key actions	recursive rules	edit	F1@10	F1@25	F1@50	acc.
✓	✓	✓	79.9	85.4	83.8	77.4	85.3
✓	✓	-	69.2 (10.7↓)	77.1 (8.3↓)	74.9 (8.9↓)	67.4 (10.0↓)	80.9 (4.4↓)
✓	-	-	62.6 (17.3↓)	72.9 (12.5↓)	70.5 (13.3↓)	63.0 (14.1↓)	78.8 (6.5↓)
Table 6: BEP vs. GEP. BEP is effective under a fair comparison to GEP.
parser	Nqueue	edit	F1@10	F1@25	F1@50	acc.
GEP	10	73.3	81.1	79.1	72.9	84.0
20	72.1	80.5	79.1	72.3	83.9
30	72.3	79.8	78.1	71.4	84.2
BEP	10	78.3	84.9	83.2	76.9	84.9
20	79.9	85.4	83.8	77.4	85.3
30	78.9	85.5	83.8	77.3	85.1
Table 7: Ablation on  $N^{\mathrm{key}}$ . Using proper number of key actions matters.
Nkey	edit	F1@10	F1@25	F1@50	acc.
1	73.0	81.5	79.9	72.6	83.5
2	77.8	85.1	83.5	77.1	85.8
3	79.9	85.4	83.8	77.4	85.3
4	74.3	82.2	80.5	73.5	83.3
5	71.2	78.6	76.3	68.6	80.4
6	68.4	75.2	73.3	64.2	77.9
reproduced performance obtained using official codes. The comparison between the second and the last row of each compartment in each table reveals significant improvements in both edit scores and F1 scores. This result validates the effectiveness of leveraging activity grammars to refine segment-wise classification. Remarkably, the KARI-induced grammar shows great performance compared to both ADIOS-induced grammars, demonstrating the importance of generalizing the grammar to cover unseen action sequences during inference effectively.
4.5 Analysis
Ablation studies of KARI. Ablation studies of KARI are conducted on the 50 Salads dataset using ASFormer [45], as shown in Table 5 to demonstrate the effectiveness of each component, including key actions and temporal dependency. The results show that both key actions and recursive rules contribute to the significant improvement of grammar-based refinement. In particular, using recursive rules is essential for the activity grammar to be generalized to the unseen action sequences.
BEP vs. GEP. Table 6 presents the performance comparison of GEP and BEP using KARI-induced grammar. We limit the queue size of both parsers, as the parser without the limitation fails to complete parsing within a reasonable time. The results indicate that our BEP outperforms GEP under the same condition. This is attributed to GEP prioritizing states based on the highest probability, which increases the risk of getting trapped in local optima when performing selective pruning within specific branches. In contrast, BEP, which prioritizes low-depth states, allows for easier escape from cycles and OR nodes, contributing to improved overall performance.
The number of key actions. Table 7 shows the results by adjusting the number of key actions  $N^{\text{key}}$  of KARI on 50 Salads. We set the value of  $N^{\text{key}}$  ranging from 1 to 6, where the induced grammar with a smaller value generates the larger activity corpus. We find that setting  $N^{\text{key}}$  to 3 outperforms the others, demonstrating the importance of achieving an appropriate level of generalization for effective refinement. Both excessive and insufficient generalization can negatively impact performance, highlighting the need to strike a balance in the generalization ability of the activity grammar.
Grammar evaluation on real data. We evaluate the parsing recall on the unseen action sequences of the Breakfast dataset. The results present the average recall across all splits for each activity. The number inside brackets indicates the average length of action sequences of each activity in  $\mathcal{D}$ . Table 8 compares the generalization capability of the five grammar induction algorithms [23, 35, 37], including KARI (details in Appendix C.1). The result demonstrates that KARI-induced grammar shows better generalization ability on real data compared to others. Remarkably, the KARI-induced grammar shows robust performance with the extended average length of the action sequences, whereas other algorithms exhibit poor generalization.
4.6 Qualitative results
Figure 5 presents a visual representation of the refined segmentation results on benchmark datasets. The proposed method successfully parses and identifies the actions 'pour oil' (red bar in Fig. 5a)
Table 8: Grammar evaluation on real data. We evaluate the proposed KARI-induced-grammar on Breakfast, demonstrating the superior high recall on unseen action sequences from each activity. The average length of action sequences of each activity is shown in parentheses.
Grammar induction	scrambled egg (11.9)	pancake (11.1)	salad (9.9)	fried egg (9.5)	juice (7.2)	coffee (6.7)	sandwich (6.0)	cereal (5.1)	milk (5.0)	tea (5.0)	total (7.7)
Kuehne et al. [23]	0.25	0.24	0.0	0.32	0.53	0.80	0.63	0.96	0.78	0.91	0.53
Richard et al. [35]	0.25	0.24	0.0	0.32	0.53	0.80	0.63	0.96	0.78	0.91	0.54
ADIOS-AND [37]	0.25	0.24	0.0	0.32	0.53	0.80	0.63	0.96	0.78	0.91	0.54
ADIOS-OR [37]	0.39	0.30	0.37	0.53	0.55	0.80	0.73	0.96	0.78	0.92	0.63
KARI	0.84	0.71	0.90	0.70	0.77	1.00	0.91	0.96	0.90	0.98	0.87
(a) Breakfast
(b) 50 Salads
Figure 5: Qualitative results. KARI-induced grammar efficiently insert missing actions and removes out-of-context actions in ASFormer [45].
and 'add saltnpepper', (blue bar in Fig. 5a), which are omitted in the results obtained by using the ADIOS-OR induced grammar. The results show that KARI-induced grammar allows a more flexible temporal structure between actions. Furthermore, our method effectively removes actions such as 'put pancake2plate' that do not correspond to the intended activity. Similarly, qualitative results on 50 Salads in Fig. 5b show the effectiveness of the proposed method with complex action sequences. The overall results show that activity grammar-based refinement for the temporal action segmentation model is effective for correcting the neural predictions by using the grammar as a guide.
We have shown that the proposed approach enhances the sequence prediction and discovers its compositional structure, significantly improving temporal action segmentation in terms of both performance and interpretability. However, the improvement is limited by the initial output of the action segmentation network, which remains further research in the future. We believe that the grammar induction and parsing methods can be easily applied to other sequence prediction tasks.
6 Acknowledgements
This work was supported by the IITP grants (2022-0-00264: Comprehensive video understanding and generation with knowledge-based deep logic  $(50\%)$ , 2022-0-00290: Visual intelligence for space-time understanding and generation based on multi-layered visual common sense  $(20\%)$ , 2022-0-00959: Few-shot learning of causal inference in vision and language  $(20\%)$ , and 2019-0-01906: AI graduate school program at POSTECH  $(10\%))$  funded by the Korea government (MSIT).
[1] H. Ahn and D. Lee. Refining action segmentation with hierarchical video representations. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 16302-16310, 2021. 1, 2
[2] N. Behrmann, S. A. Golestaneh, Z. Kolter, J. Gall, and M. Noroozi. Unified fully and times-tamp supervised temporal action segmentation via sequence to sequence translation. In Proc. European Conference on Computer Vision (ECCV), pages 52-68. Springer, 2022. 1, 2
[3] P. Belcák, D. Hofer, and R. Wattenhofer. A neural model for regular grammar induction. arXiv preprint arXiv:2209.11628, 2022. 8
[4] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6299-6308, 2017. 7
[5] M.-H. Chen, B. Li, Y. Bao, and G. AlRegib. Action segmentation with mixed temporal domain adaptation. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV), pages 605-614, 2020. 2
[6] M.-H. Chen, B. Li, Y. Bao, G. AlRegib, and Z. Kira. Action segmentation with joint self-supervised temporal domain adaptation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9454-9463, 2020. 2
[7] J. Earley. An efficient context-free parsing algorithm. Communications of the ACM, 13(2):94-102, 1970. 6, 15
[8] H.-S. Fang, Y. Xu, W. Wang, X. Liu, and S.-C. Zhu. Learning pose grammar to encode human body configuration for 3d pose estimation. In Proc. AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018. 2
[9] Y. A. Farha and J. Gall. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3575-3584, 2019. 1, 2, 3, 5, 7, 8
[10] S.-H. Gao, Q. Han, Z.-Y. Li, P. Peng, L. Wang, and M.-M. Cheng. Global2local: Efficient structure search for video action segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16805-16814, 2021. 2
[11] M. Guo, V. Thost, B. Li, P. Das, J. Chen, and W. Matusik. Data-efficient graph grammar learning for molecular generation. In Proc. International Conference on Learning Representations (ICLR), 2021. 2
[12] Y. Hong, Q. Li, R. Gong, D. Ciao, S. Huang, and S.-C. Zhu. Smart: A situation model for algebra story problems via attributed grammar. In Proc. AAAI Conference on Artificial Intelligence (AAAI), pages 13009-13017, 2021. 2
[13] Y. Hong, Q. Li, S.-C. Zhu, and S. Huang. VLgrammer: Grounded grammar induction of vision and language. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 1665-1674, 2021. 2
[14] J. E. Hopcroft, R. Motwani, and J. D. Ullman. Introduction to automata theory, languages, and computation. *Acm Sigact News*, 32(1):60–65, 2001. 1, 2, 18