SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[33] A. Richard, H. Kuehne, and J. Gall. Weakly supervised action learning with rnn based fine-to-coarse modeling. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 754-763, 2017. 1, 2
[34] A. Richard, H. Kuehne, and J. Gall. Action sets: Weakly supervised action segmentation without ordering constraints. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5996, 2018. 2
[35] A. Richard, H. Kuehne, A. Iqbal, and J. Gall. Neuralnetwork-viterbi: A framework for weakly supervised video learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7386-7395, 2018. 1, 2, 6, 9, 10, 18, 19
[36] M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele. A database for fine grained activity detection of cooking activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1194-1201. IEEE, 2012. 2
[37] Z. Solan, D. Horn, E. Ruppin, and S. Edelman. Unsupervised learning of natural languages. Proceedings of the National Academy of Sciences, 102(33):11629-11634, 2005. 2, 3, 7, 8, 9, 10, 18
[38] Y. Souri, Y. A. Farha, F. Despinoy, G. Francesca, and J. Gall. Fifa: Fast inference approximation for action segmentation. In Pattern Recognition: 43rd DAGM German Conference, DAGM GCPR 2021, Bonn, Germany, September 28–October 1, 2021, Proceedings, pages 282–296. Springer, 2022. 2
[39] S. Stein and S. J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 729-738, 2013. 7, 19
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Proc. Neural Information Processing Systems (NeurIPS), 30, 2017. 2
[41] N. N. Vo and A. F. Bobick. From stochastic grammar to bayes network: Probabilistic parsing of complex activity. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2641-2648, 2014. 2
[42] B. Wan, W. Han, Z. Zheng, and T. Tuytelaars. Unsupervised vision-language grammar induction with shared structure modeling. In Proc. International Conference on Learning Representations (ICLR), 2021. 2
[43] Y. Xu, W. Wang, T. Liu, X. Liu, J. Xie, and S.-C. Zhu. Monocular 3d pose estimation via pose grammar and data augmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021. 2
[44] Z. Xu, Y. S. Rawat, Y. Wong, M. Kankanhalli, and M. Shah. Don't pour cereal into coffee: Differentiable temporal logic for temporal action segmentation. In Advances in Neural Information Processing Systems (NeurIPS). 2
[45] F. Yi, H. Wen, and T. Jiang. Asformer: Transformer for action segmentation. In Proc. British Machine Vision Conference (BMVC), 2021. 1, 2, 3, 5, 7, 8, 9, 10, 19
In this supplement, we provide detailed descriptions of the proposed method and additional results, which are omitted in the main paper due to the lack of space. In Section A, we will describe the formulation of the probability of activity grammar. Algorithmic details of BEP is included in Section B. Section C compares KARI with the existing grammar induction algorithms for activity grammar and Section D presents additional qualitative results. We conclude this Appendix by discussing the broader impact of our research in Section E.
A Formulation of the probabilities in KARI
In this section, we describe the formulation of transition probability  $p_{i,j}^{\Omega}$  and the escape probability  $p_{i,\epsilon}^{\Omega}$  and  $p_{\epsilon}^{\mathrm{M}}$  in Eq. 5 and 6 in Section A.1. In the following, the derivation of the expectation of the escape probability is described in Section A.2.
A.1 Formulation of the escape and transition probability
We first introduce and escape probabilities and the transition probabilities introduced in Eq. 5.
Pre-processing. Let  $h_i^\Omega$  represent a list of action sub-sequences, where the sub-sequence from  $h_i^\Omega$  removes actions that does not exist in the action group  $d_i^\Omega$  from the action sub-sequence in  $\mathcal{D}^\Omega$ . The empty string  $\epsilon$  remains when the action sub-sequence does not include actions within the action group  $d_i^\Omega$ . For example in Fig. 2, a list of sub-sequences  $h_1^{\mathrm{R}}$  can be structured as  $h_1^{\mathrm{R}} = [[\mathrm{pour~milk}], [\mathrm{spoon~sugar}, \mathrm{pour~milk}], [\mathrm{pour~milk}, \mathrm{spoon~sugar}], [\mathrm{spoon~sugar}]]$  with the corresponding action group  $d_1^{\mathrm{R}} = \{\mathrm{pour~milk}, \mathrm{spoon~sugar}\}$ . Similar, a list of sub-sequences  $h_2^{\mathrm{R}}$  is structured as  $h_2^{\mathrm{R}} = [\epsilon, \epsilon, [\mathrm{stir~coffee}], [\mathrm{stir~coffee}]]$  with the action group  $d_2^{\mathrm{R}} = \{\mathrm{stir~coffee}\}$ . This pre-processing step of generating  $h_i^\Omega$  enables us to consider the statistical probabilities associated with actions.
Formulation of the escape probability. The escape probability  $p_{i,\epsilon}^{\Omega}$  and the transition probability  $p_{i,j}^{\Omega}$  are both defined based on the number of recursion  $n^{\mathrm{rec}}$  of the current timestep; thereby these probabilities are represented as functions of  $n^{\mathrm{rec}}$ . We first define the escape probability function:
$$
p _ {i, \epsilon} ^ {\Omega} \left(n ^ {\text {r e c}}\right) = \left\{ \begin{array}{l l} \frac {\left| [ \boldsymbol {a} \in \boldsymbol {h} _ {i} ^ {\Omega} \mid \boldsymbol {a} = \epsilon ] \right|}{| \boldsymbol {h} _ {i} ^ {\Omega} |} & \text {i f n ^ {\text {r e c}} = 1 ,} \\ \frac {1}{\bar {N} h _ {i} ^ {\Omega}} & \text {o t h e r w i s e ,} \end{array} \right. \tag {13}
$$
where  $\bar{N}^{h_i^\Omega}$  is the average length of the sub-sequences in  $h_i^\Omega$ . In the first recursion, i.e.,  $n^{\mathrm{rec}} = 1$ , the probability calculation solely considers statistics of the actions. Otherwise, the probability is calculated based on the expected number of recursions, which will be introduced in Appendix A.2. In Fig. 2,  $p_{1,\epsilon}^{\mathrm{R}}(1) = 0$ , since none of the action sub-sequence  $\mathbf{a}$  from  $h_1^{\mathrm{R}}$  is equal to the empty sequence, and  $p_{1,\epsilon}^{\mathrm{R}}(n^{\mathrm{rec}} > 1) = \frac{2}{3}$ , since the average length of sub-strings in  $h_1^{\mathrm{R}}$  is 1.5.
Formulation of the transition probability. The action sequence  $\mathbf{a} = [a_1, a_2, \dots, a_N]$  represents the distinct action labels for the video segments, where  $a_i \neq a_{i+1}$  as described in Section 3. In order to prevent the repetition of the same action in Eq. 5, we introduce an additional input  $q$  when defining the transition probability. Here,  $q$  refers to the index of the actions selected by the rule in the previous step, specifically at  $(n^{\mathrm{rec}} - 1)_{\mathrm{th}}$  step where  $n^{\mathrm{rec}} > 1$ . We simply put  $q$  to 0 in the first recursion, i.e.,  $n^{\mathrm{rec}} = 1$ , which does not affect the results. The transition probability  $p_{i,j}^{\Omega}$  is defined by:
$$
p _ {i, j} ^ {\Omega} \left(n ^ {\text {r e c}}, q\right) = \left\{ \begin{array}{l l} \frac {\left| \left[ \boldsymbol {a} \in \boldsymbol {h} _ {i} ^ {\Omega} \mid a _ {1} = d _ {i , j} ^ {\Omega} \right] \right|}{\left| \boldsymbol {h} _ {i} ^ {\Omega} \right|} & \text {i f n ^ {\text {r e c}} = 1 ,} \\ 0 & \text {i f n ^ {\text {r e c}} > 1 a n d j = q ,} \\ \frac {p _ {i , j} ^ {\Omega} (1 , 0) \left(1 - p _ {i , \epsilon} ^ {\Omega} \left(n ^ {\text {r e c}}\right)\right)}{\sum_ {l \neq q} p _ {i , l} ^ {\Omega} (1 , 0)} & \text {o t h e r w i s e .} \end{array} \right. \tag {14}
$$
The escape probability  $p_{\epsilon}^{\mathrm{M}}$  and the transition probability  $p_{i,j}^{\mathrm{M}}$  for the middle variable  $V^{\mathrm{M}}$  in Eq. 6 is defined in the same way as Eq. 13 and Eq. 14, respectively.
A.2 Derivation of the escape probability
We introduce the formulation of the escape probability  $p_{i,\epsilon}^{\Omega}$  in Appendix A.1. The escape probability is required to avoid an infinite loop of the rules and guarantee the length of sequences from the recursive rules in Eq. 5. Since the number of recursions directly determines the sequence lengths, we determine the escape probability regarding the length of action sequences. For notational simplicity, we denote the escape probabilities  $p_{i,\epsilon}^{\Omega}$  by  $p$ , omitting superscripts and subscripts. The expectation of the number of recursions is calculated by:
$$
\begin{array}{l} \lim  _ {n \rightarrow \infty} \sum_ {k = 1} ^ {n} k p (1 - p) ^ {k} = \lim  _ {n \rightarrow \infty} \frac {(1 - p) (1 + (1 - p) ^ {n} - n p (1 - p) ^ {n})}{p}, (15) \\ = \frac {1 - p}{p}. (16) \\ \end{array}
$$
Since we derive the escape probability when  $n^{\mathrm{rec}} > 1$ , the expected number of recursions is equal to  $\bar{N} - 1$ :
$$
\frac {1 - p}{p} = \bar {N} - 1, \tag {17}
$$
, where  $\bar{N}$  is the average length of action sequences. Finally, we obtain the escape probability by
$$
p = \frac {1}{\bar {N}}, \tag {18}
$$
where this equation is used in Eq. 13 when  $n^{\mathrm{rec}} > 1$ . The derivation of the escape probability  $p_{\epsilon}^{\mathrm{M}}$  of the middle variable  $V^{\mathrm{M}}$  in Eq. 6 is also formulated as the same.
B Breadth-first Earley Parser (BEP)
B.1 Earley parser
The Earley parser [7] is a classic algorithm that efficiently parses strings for context-free grammar. It operates by maintaining a set of states of the parsing process. Each state consists of a production rule, a position within that rule, and a position in the input string. The parser builds a parse tree for the input string, which records the structure of parsing. The Earley parser is commonly used for natural language processing tasks, such as syntactic analysis and semantic parsing.
The Earley parser consists of three main operations: scanning, prediction, and completion.
- Scanning: The parser matches a terminal symbol in the input string with the current position in the production rule. This operation moves the parser forward in the input string.
- Prediction: The parser expands a variable in the production rule based on the current position. It adds new states to the set of states for possible future matches.
- Completion: When the parser reaches the end of the production rule, it searches for other states predicting the head variable of the current rule. Subsequently, the parser update the positions within the rule of the searched states.
By iterating these three operations, the Earley parser builds a parse chart that represents all possible parse trees for the input string.
B.2 Implementation details
The parsing probability  $p(\pmb{F}_{1:t} \to \pmb{a} \mid G)$  (Eq. 9) can suffer from numerical underflow due to its exponential decrease as  $t$  increases. To overcome the issue, we compute the probabilities in logarithmic space, following [31]. For simplicity, we denote  $\log(p(\pmb{F}_{1:t-1} \to \pmb{a} \mid G))$  as  $P_N$  and  $\log(p(\pmb{F}_{1:t-1} \to \pmb{a}_{1:N-1} \mid G))$  as  $P_{N-1}$  below:
$$
P _ {N - 1} ^ {\prime} = \log (g (x | \boldsymbol {a} _ {1: N - 1}, G)) + P _ {N - 1}, \tag {19}
$$
$$
z = \max  \left(P _ {N}, P _ {N - 1} ^ {\prime}\right), \tag {20}
$$
$$
\log (p \left(\boldsymbol {F} _ {1: t} \rightarrow \boldsymbol {a} \mid G\right)) = \log \left(\boldsymbol {Y} _ {t, x}\right) + z + \log \left(\exp \left(P _ {N} - z\right) + \exp \left(P _ {N - 1} ^ {\prime} - z\right)\right). \tag {21}
$$
$$
S \rightarrow A B C
$$
$$
A \to A _ {1} A _ {2}
$$
$$
A _ {1} \rightarrow x _ {1} [ 0. 7 ] \mid x _ {2} [ 0. 3 ]
$$
$$
A _ {2} \rightarrow A _ {3} A _ {4} [ 0. 5 ] | \epsilon [ 0. 5 ]
$$
$$
A _ {3} \rightarrow x _ {4} A _ {4} [ 0. 5 ] \mid \epsilon [ 0. 5 ]
$$
$$
A _ {4} \rightarrow x _ {3} A _ {3} [ 0. 5 ] \mid \epsilon [ 0. 5 ]
$$
$$
B \rightarrow x _ {5} [ 1. 0 ]
$$
$$
C \rightarrow x _ {6} [ 0. 7 ] \mid x _ {7} [ 0. 3 ]
$$
Figure 6: Toy grammar used for the example of the BEP parsing
For the computational efficiency, we set the sampling stride of input matrix  $\mathbf{Y}$  as 50 for Breakfast and 100 for 50Salads. Additionally, we set the maximum length of the refined action sequence as 20 for Breakfast and 25 for 50Salads.
B.3 Parsing algorithm
Algorithm 1 shows the parsing procedure of BEP. We utilize a priority queue that sorts the elements in ascending order. The currentSet stores multiple states with the same  $m$ ,  $n$ , and  $d$ . See B.4 for the examples. BEP stops parsing when the probability of  $a^*$  has the highest probability compared to states in the queue while ensuring the current state can reach depth 1 with only completions.
B.4 Parsing example
In this section, we provide an example to help understand how BEP works. For simplicity, we assume that frame-wise class probabilities from the segmentation model are identical across all action classes. First of all, we define toy grammar as shown in Figure 6. In the context of grammar,  $S$  indicates the starting variable.  $A, B, C$ , and  $A_{i}$  for  $i = [1,2,3,4]$  represent the variables, while  $x_{j}$  for  $j = [1,2,3,4,5,6,7]$  represent terminals.
Table 9 is the history of parsing with the toy grammar. It shows the currently popped state, the visiting order, the parsed prefix, the previous state, and which states are currently in the queue. The three consecutive numbers in the column pop, from, and queue indicate  $m$ ,  $n$ , and  $d$  of the state. The column  $p$  represents the prefix probability excluding the frame-wise probability, which can be considered a parsing probability since all frame-wise probabilities are assumed to be the same. Note that the table includes some history after the parsed sequence satisfied the early stop constraint to illustrate how BEP prioritizes the states. Returning to the subject, the table shows BEP preferentially searches for states with a small depth. In order 14, even though the probability of state  $Q(1,1,3)$  is higher, BEP visits the state  $Q(3,0,2)$  with a lower depth.
Algorithm 1: Breadth-first Earley Parser (BEP)
Input: probability matrix  $Y$ , grammar  $G$ , queue size  $N^{\text{queue}}$   
Output: Best parsed sequence  $a^{*}$   
function Breadth-first Earley Parser  
q  $\leftarrow$  priorityQueue(); // init priority queue  
Q(0,0,0)  $\leftarrow$  ( $\Gamma \rightarrow R$ , Q(0,0,0),  $\epsilon, 1.0$ ); // set initial state  
q.push(0, (1.0,0,0,  $\epsilon$ , Q(0,0,0))); // push initial state to queue  
 $a^{*} \leftarrow \epsilon$ ; // init  $a^{*}$   
while  $(d, (p(a_{1:|a|-1}), m, n, a_{1:|a|-1}, \text{currentSet})) \leftarrow q.pop()$  do  
for  $(r, Q(i,j,k), a, p(a...)) \in \text{currentSet}$  do  
// update  $a^{*}$  when  $a$  has higher probability  
if  $p(a) > p(a^{*})$  then  
 $|a^{*} \leftarrow a$   
end if  
// prediction  
if  $r$  is  $(A \rightarrow \alpha \cdot B\beta)$  then  
for each  $(B \rightarrow \Gamma)$  in  $G$  do  
 $\begin{array}{c} r' \leftarrow (B \rightarrow \Gamma) \\ Q' \leftarrow (r', Q(m,n,d), a, p(a...)) \\ Q(m,n,d+1).add(Q') q.push(d+1, (p(a...), m,n,a, Q(m,n,d+1))) \end{array}$  end for  
end if  
// scanning  
if  $r$  is  $(A \rightarrow \alpha \cdot x\beta)$  then  
 $\begin{array}{c} r' \leftarrow (A \rightarrow \alpha x \cdot \beta) \\ n' \leftarrow |Q(m+1)| \\ Q' \leftarrow (r', Q(i,j,k), a + x, p((a+x)...)) \\ Q(m+1,n',d).add(Q') \\ q.push(d, (p(a+x)...m+1,n',d, Q(i,j,k))) \end{array}$  end for  
// completion  
if  $r$  is  $(B \rightarrow \Gamma)$  then  
for each  $(A \rightarrow \alpha \cdot B\beta)$ ,  $Q(i',j',k')$ ,  $a, p(a...))$  in  $Q(i,j,k)$  do  
 $\begin{array}{c} r' \leftarrow (A \rightarrow \alpha B \cdot \beta) \\ Q' \leftarrow (r', Q(i',j',k'), a, p(a...)) \\ Q(m,n,d-1).add(Q') \\ q.push(d-1, (p(a...), m,n,a, Q(m,n,d-1))) \end{array}$  end for  
end if  
// early stop when  $a^{*}$  has the highest probability and finished parsing  
if  $p(a^{*}) > p(a')$  for all  $a'$  in  $q$  then  
if  $a^{*}$  has parsed then  
return  $a^{*}$   
end if  
// Queue pruning  
if  $|q| > N^{\text{queue}}$  then  
// sort  $q$  in probability descending order  
 $q' \leftarrow$  sorted(q,key=p(a...), reverse=True)  
q.clear()  
for  $i \leftarrow 1$  to  $N^{\text{queue}}$  do  
q.push(q'.pop())  
end for  
end if  
end while  
return  $a^{*}$
Table 9: Parsing log for the given toy grammar through BEP.
pop	order	m	n	d	rule	prefix	operation	from	p	queue
-	1	0	0	0	Γ → ·S	-	ROOT	-	1	000
000	2	0	0	1	S → ·ABC	-	PRED	000	1	001
001	3	0	0	2	A → ·A1A2	-	PRED	001	1	002
002	4	0	0	3	A1 → ·x1	-	PRED	002	0.7	003
0	0	3	A1 → ·x2	-	PRED	002	0.3
003	5	1	0	3	A1 → x1·	x1	SCAN	003	0.7	103, 113
19	1	1	3	A1 → x2·	x2	SCAN	003	0.3
103	6	1	0	2	A → A1·A2	x1	COMP	103	0.7	113, 102
102	7	1	0	3	A2 → ·A3A4	x1	PRED	102	0.35	103, 113
1	0	3	A2 → ·e	x1	PRED	102	0.35
103	-	1	0	4	A3 → ·a4A4	x1	PRED	103	0.175	203, 113, 104
1	0	4	A3 → ·e	x1	PRED	103	0.175
8	2	0	3	A2 → e·	x1	SCAN	103	0.35
203	9	2	0	2	A → A1A2·	x1	COMP	203	0.35	202, 113, 104
202	10	2	0	1	S → A·BC	x1	COMP	202	0.35	201, 113, 104
201	11	2	0	2	B → x5	x1	PRED	201	0.35	202, 113, 104
202	12	3	0	2	B → x5·	x1 x5	SCAN	202	0.35	302, 113, 104
302	13	3	0	1	S → AB·C	x1 x5	COMP	302	0.35	301, 113, 104
301	14	3	0	2	C → x6	x1 x5	PRED	301	0.245	302, 113, 104
3	0	2	C → x7	x1 x5	PRED	301	0.105
302	15	4	0	2	C → x6·	x1 x5 x6	SCAN	302	0.245	402, 412, 113, 104
17	4	1	2	C → x7·	x1 x5 x7	SCAN	302	0.105
402	16	4	0	1	S → ABC·	x1 x5 x6	COMP	402	0.245	401, 412, 113, 104
401	-	-	-	-	Γ → S·	x1 x5 x6	COMP	401	0.245	412, 113, 104
412	18	4	1	1	S → ABC·	x1 x5 x7	COMP	412	0.105	411, 113, 104
411	-	-	-	-	Γ → S·	x1 x5 x7	COMP	411	0.105	113, 104
113	-	1	1	2	A → A1·A2	x2	COMP	113	0.3	112, 104
C Comparison with the existing grammar induction algorithms
C.1 Existing grammar induction algorithms
Kuehne et al. [23] introduce a hierarchical context-free grammar induction algorithm. The root rule with the starting variable  $S$  is induced as  $S \to V_1 \mid V_2 \mid \ldots \mid V_{N^{\mathrm{A}}}$ , where the variable  $V_i$  represents a single activity and  $N^{\mathrm{A}}$  is the number of activities from the dataset. Then each  $V_i$  expands into action sequences from each activity, i.e., the rule is formed as:  $V_i \to \mathcal{A}_{i,1} \mid \mathcal{A}_{i,2} \mid \ldots \mid \mathcal{A}_{i,|\mathcal{A}_i|}$ , where  $\mathcal{A}_i$  is a set of action sequences from the  $i$ -th activity and each  $\mathcal{A}_{i,j}$  represents a  $j$ -th action sequence in  $\mathcal{A}_i$ .
Richard et al. [35] propose a grammar induction method for a probabilistic right-regular grammar, where every rule has the form of  $\tilde{H} \rightarrow cH$ . The algorithm is motivated by n-gram models [18] and finite grammars [14]. Specifically, the variable  $\tilde{H}$  represents an action sequence  $a_{1:n}$ ,  $H$  represents an action sequence  $a_{1:n-1}$ , and a terminal  $c$  is an action class of  $a_n$ . The induced grammar can express the intermediate action sequences  $a_{1:n}$  and expands its rules based on the sequential order of actions.
Recently, Qi et al. [32] adopt the Automatic Distillation of Structure (ADIOS) [37] algorithm to induce a probabilistic context-free grammar. The ADIOS algorithm finds the significant patterns (AND rules) and equivalence action classes (OR rules) from the given action sequences. The algorithm identifies repetitive patterns in action sequences to minimize redundant sequences and find potential candidates for generalized action classes. Following the grammar induction methods of ADIOS, we
Table 10: The performance comparison with other grammar induction algorithms on two benchmark datasets.
dataset	reprod.	refinement	grammar induction	edit	F1@10	F1@25	F1@50	acc.
50Salads [39]	✓	-	-	76.5	83.8	81.7	74.8	86.1
✓	✓	Kuehne et al. [23]	62.9	73.0	70.6	63.0	78.6
✓	✓	Richard et al.[35]	63.1	73.1	70.5	62.9	78.7
✓	✓	ADIOS-AND	58.3	70.0	68.0	59.4	76.2
✓	✓	ADIOS-OR	61.1	72.0	70.1	62.4	78.9
✓	✓	KARI	79.9	85.4	83.8	77.4	85.3
Breakfast [23]	✓	-	-	75.6	77.3	72.0	59.4	74.3
✓	✓	Kuehne et al. [23]	72.8	74.0	69.1	55.5	72.9
✓	✓	Richard et al.[35]	77.3	77.2	72.2	59.4	74.1
✓	✓	ADIOS-AND	69.2	69.8	64.9	52.2	72.4
✓	✓	ADIOS-OR	70.3	71.8	66.8	54.2	71.8
✓	✓	KARI	77.8	78.8	73.7	60.8	74.0
set a decreasing ratio of the motif extraction algorithm  $\eta$  to 1, a significance level for the decrease ratio  $\gamma$  to 0.1, and the context window size 1 for ADIOS-AND-induced grammar. For ADIOS-OR-induced grammar, we set  $\eta$  to 0.9,  $\gamma$  to 0.1, and the context window size to 4.
However, none of these approaches have managed to effectively integrate recursive rules, which are crucial for representing intricate and lifelike structures of action phrases and activities. The proposed KARI algorithm introduces a probabilistic context-free grammar that allows for the expression of complex activity structures, which captures a distinctive temporal structure based on key actions.
C.2 Performance on temporal action segmentation
In Table 10, we compare the performance of each grammar induction algorithm on refining temporal action segmentation models [45]. The overall results show that the KARI-induced grammar demonstrates the best refinement performance compared to the other grammar induction algorithms, showing a significant performance gap in both datasets. The induced grammar of Richard et al. also shows better performance than other grammar induction algorithms except for KARI, indicating that the ability to represent intermediate action sequences by production rules helps improve refinement performance. In conclusion, the generalization capabilities and variability of expressing action sequences are essential to guide the temporal action segmentation network to better refinement results.
D Qualitative results
We provide additional qualitative results for the Breakfast and 50Salads. Figure 7 shows examples of successful output refinements by the KARI-induced grammar, demonstrating its ability to cover various sequences comprising combinations of multiple actions. This is further evident in Figure 7a, where ADIOS-OR falls short in covering the 'add dressing' action following the 'serve salad' action, while KARI handles it proficiently.
We also show failure cases of the KARI-induced grammar in Figure 8, where further improvement is needed. We acknowledge that the KARI-induced grammar sometimes deletes certain actions. This deletion of actions, along with the challenges posed by inaccurate identification of actions by the segmentation model, show areas for improvement in the refinement process. We recognize these as opportunities for future work to enhance the performance of the grammar induction algorithm and address these limitations.
The research presented in this paper holds significant potential for impact across multiple domains. The development of efficient and effective grammar induction algorithms for activity grammar, coupled with the Breadth-first Earley parser, has the potential to greatly enhance human activity recognition and understanding systems. This, in turn, can have far-reaching implications in various
(a) 50Salads
(b) Breakfast (activity: cereals)
(c) Breakfast (activity: tea)
Figure 7: Qualitative results on successful cases
Figure 8: Qualitative results on failure cases
applications, such as video surveillance, human-computer interaction, robotics, and healthcare monitoring. By improving the accuracy and efficiency of activity recognition systems, our research contributes to advancements in these domains, enabling more robust and intelligent systems. The broader implications of this research extend beyond activity grammar induction itself, fostering innovation and enhancing the capabilities of intelligent systems in diverse fields.