## 3 The Proposed Encoder-Decoder Network
### Decoding Search Space Reduction
#### 3.4.1 Example  
Figure 6 illustrates how Equation (6) re-weights a ground-truth path and a degenerated path. In Figure 6, there is a ground-truth path \(\mathbf{\pi}^{1}\) and another path \(\mathbf{\pi}^{2}\) that produces the same action order with \(\mathbf{\pi}^{1}\) \(\left(\mathcal{B}(\mathbf{\pi}^{1})=\mathcal{B}(\mathbf{\pi}^{2})=\mathbf{\ell}\right)\), but yields different frame-wise label sequences:  
\(\mathcal{A}\ =\{Action1,\ Action2,\ Action3\}\), represented by green, yellow, and orange nodes, respectively. \(\mathbf{\ell}\ =\ green\to yellow\to orange\), which is supposed to be already known during training. \(\mathbf{\pi}^{1}\ =\ green\to green\to green\to yellow\to orange\). \(\mathbf{\pi}^{2}\ =\ green\to yellow\to orange\to orange\to orange\). Figure 6: For illustration, an input with \(T=6\) frames and \(|\mathcal{A}|=3\) annotated actions. High similarity is indicated by thick lines between frames. Note that \(\pi_{1}\) stays at the same action prediction across similar frames. Consequently, Equation (6) weights \(\pi_{1}\) higher over \(\pi_{2}\). In contrast, \(\pi_{1}\) and \(\pi_{2}\) are equally treated in original CTC. In fact, for \(T=6\), \(|\boldsymbol{\ell}|=3\), there are altogether \( \begin{pmatrix}T-2\\ |\boldsymbol{\ell}|-1\end{pmatrix} = \begin{pmatrix}4\\ 2\end{pmatrix} =\frac{4!}{22! }=6\) distinct paths \(\boldsymbol{\pi}^{i}\) satisfying the supervised constraints \(\mathcal{B}(\boldsymbol{\pi}^{i})=\boldsymbol{\ell}\), because:  
* \(\pi_{1}\) has to be \(\boldsymbol{\ell}(1)=\textit{green}\);
* \(\pi_{T}\) has to be \(\boldsymbol{\ell}(end)=\textit{orange}\); and
* The difference among different \(\boldsymbol{\pi}^{i}\)s is to choose at which \(\pi_{t},t=2,\cdots,T-1\) to transit the node label from \(\boldsymbol{\ell}(j)\) to \(\boldsymbol{\ell}(j+1)\), with \(j=1,2,\cdots,|\boldsymbol{\ell}|-1\). To summarize, the necessary and sufficient condition for path \(\boldsymbol{\pi}\) to be consistent with the supervised action order \(\boldsymbol{\ell}\), if and only if:
* \(\pi_{1}=\boldsymbol{\ell}(1)\). * \(\pi_{T}=\boldsymbol{\ell}(end)\). * For each middle node \(\pi_{t},t=2,\cdots,T-1\), there are only two possible options: (1) Stay the same as the previous node, which means if \(\pi_{t-1}=\boldsymbol{\ell}(j)\), \(\pi_{t}=\boldsymbol{\ell}(j)\) as well. Whenever this case holds true, a'repetition' happens in \(\boldsymbol{\pi}\). (2) Transit from \(\pi_{t-1}=\boldsymbol{\ell}(j)\) to the next label in \(\boldsymbol{\ell}\), which means \(\pi_{t}=\boldsymbol{\ell}(j+1)\). Any other label assignment will cause \(\mathcal{B}(\boldsymbol{\pi})=\boldsymbol{\ell}\) holding false. So far we can draw the conclusion that the only difference between valid paths \(\boldsymbol{\pi}\) and supervised action order \(\boldsymbol{\ell}\) is that \(\boldsymbol{\pi}\) contains'repetitions'; by inserting and removing'repetitions', \(\boldsymbol{\pi}\) and \(\boldsymbol{\ell}\) can be converted to each other. In this example, \(\pi_{t}=\pi_{t+1}\) holds true at \(t=1,\ 2,\ 4\) in path \(\boldsymbol{\pi}^{1}\), and \(t=3,\ 4,\ 5\) in path \(\boldsymbol{\pi}^{2}\). \(\pi_{t}=\pi_{t+1}\) means that a'repetition' happens in path \(\boldsymbol{\pi}\) at frame \(t+1\), but there are two cases to judge whether such a'repetition' should be encouraged or not if taking into account its consistency with the ground truth alignment. Case 1: \(s_{t}^{t+1}>\theta\), which means that apart from the supervised order information \(\boldsymbol{\ell}\), it is also unsupervisedly observed that frame \(t+1\) is visually in great similarity with frame \(t\), which suggests such a'repetition' should be additionally encouraged. At \(t=1\), \(\pi_{t}^{1}=\pi_{t+1}^{1}=\textit{green}\) holds true for \(\boldsymbol{\pi}^{1}\) and false for \(\boldsymbol{\pi}^{2}\), where \(\pi_{t}^{2}=\textit{green}\
eq\pi_{t+1}^{2}=\textit{yellow}\). \(\boldsymbol{\pi}^{1}\) is consistent with ground truth while \(\boldsymbol{\pi}^{2}\) is not, since \(s_{t}^{t+1}=10>\theta\).