# Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models
## 5 Methodology
### Action Segmentation Models
Like the dilated residual block, it also includes a dropout layer and a skip connection. The operations are summarized as follows:  
\[\widetilde{\mathbf{H}}_{l,1}=\widetilde{\mathbf{W}}_{l,1}\ast\mathbf{H}_{l-1}+\widetilde{\mathbf{ 1}}_{l,1}^{T} \tag{9}\]  
\[\widetilde{\mathbf{H}}_{l,2}=\widetilde{\mathbf{W}}_{l,2}\ast\mathbf{H}_{l-1}+\widetilde\mathbf{{1}}_{l,2}^{T} \tag{10}\]  
\[\widetilde{\mathbf{H}}_{l}=ReLU\left(\left[\widetilde{\mathbf{H}}_{l,1},\widetilde\mathbf{{H}}_{l,2}\right]\right) \tag{11}\]  
\[\mathbf{H}_{l}=\mathbf{H}_{l-1}+\mathbf{W}_{l}\ast\widetilde{\mathbf{H}}_{l}+\mathbf{1}\mathbf{b}_{l}^{ T} \tag{12}\]  
In Equations (9)-(12) \(\widetilde{\mathbf{W}}_{l,1}\in\mathbb{R}^{3\times H\times H}\) and \(\widetilde{\mathbf{W}}_{l,2}\in\mathbb{R}^{3\times H\times H}\) are the parameters of the dilated convolution kernels in the branches with increasing and decreasing dilation factors. \(\mathbf{W}_{l}\in\mathbb{R}^{1\times 2H\times H}\) is the parameter of the convolution with a kernel size of 1. The architecture of the convolutional model containing these two types of blocks (see Figure 12b) can be divided into two parts, which in the literature are called the prediction generation stage and the prediction refinement stage. Preliminary experiments indicated that only one stage of prediction generation was sufficient for models developed on the collected sample. The number of refinement stages was one of the hyperparameters adjusted in the experiments (see Table 2). **Transformer models:** The Transformer architecture offers a distinct approach to handling sequential data. Unlike RNN-based models, the Transformer operates without recurrence, relying on self-attention mechanisms to process entire sequences simultaneously. This design allows for the efficient modeling of long-range dependencies and parallel processing, especially in cases where temporal dynamics span over various lengths. Convolutional architectures, despite being parallelizable, struggle to model long-term contexts without the stacking of multiple layers, which can increase complexity and computational load. As the Transformer lacks recurrent mechanisms, it employs positional encodings to incorporate sequence order. Positional encodings \(\mathbf{P}\in\mathbb{R}^{T\times D}\) are added to the input sequence to provide the model with information about the relative or absolute position of the elements in the sequence, and we us those proposed in the original Transformer paper [46] based on sine and cosine functions. The core of the Transformer is the multihead attention mechanism, which allows the model to focus on different parts of the sequence simultaneously. The attention mechanism is applied independently in parallel heads, allowing the model to capture different aspects of the input sequence. In a single head attention, the input \(\mathbf{X}\in\mathbb{R}^{T\times D}\) is transformed into three matrices called queries (\(\mathbf{Q}\in\mathbb{R}^{T\times D_{q}}\)), keys (\(\mathbf{K}\in\mathbb{R}^{T\times D_{k}}\)), and values (\(\mathbf{V}\in\mathbb{R}^{T\times D_{v}}\)), and the attention mechanism can be mathematically described as:  
\[\mathbf{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{T}} {\sqrt{D_{k}}}\right)\mathbf{V} \tag{13}\]  
where \(\mathbf{Q}\mathbf{K}^{T}\) is the dot product similarity score for each query-key pair, which is calculated to determine how closely related each query (representing a specific segment or focus in \begin{table}
\begin{tabular}{c c c} \hline \hline
**Label 1** & **Hyperparameters** & **Set of Values Used** \\ \hline D & Number of dilated residual layers & \{1, 5, 10\} \\ DD & Number of dual dilated layers & \{1, 5, 10\} \\ R & Number of refinement stages & \{1, 2, 3, 4, 5\} \\ CK & Number of kernels in each CNN layer & \{64, 128\} \\ DR & Dropout rate & \{0, 0.1, 0.2, 0.5\} \\ \hline \hline \end{tabular} 1  
\end{table} 
Table 2: Hyperparameters tuned in experiments with 1D CNN architecture.