# Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models
## 5 Methodology
### Action Segmentation Models  
The literature review pointed to three types of architectures that dominate in the action segmentation research. These are architectures based on LSTM/biLSTM, 1D CNN, and Transformers (see Figure 12). The developed models received input data in the form of a sequence \(\mathbf{X}_{1}^{\text{T}}=\left(\mathbf{x}^{(1)},\mathbf{x}^{(2)},\dots,\mathbf{x}^{(\text{T} )}\right)\) of total length \(T\). The task of the models was to make a prediction of the activity class occurring at each timestep in the output sequence \(\hat{\mathbf{Y}}_{1}^{\text{T}}=\left(\hat{\mathbf{y}}^{(1)},\hat{\mathbf{y}}^{(2)},\dots,\hat{\mathbf{y}}^{(\text{T})}\right)\), where the member of the prediction in the output sequence at timestep \(t\) was a vector \(\hat{\mathbf{y}}^{(t)}\in\mathbb{R}^{10}\), for which the sum \(\sum_{i=1}^{10}y_{i}^{(t)}=1\). Figure 11: Custom 2D CNN model used for feature extraction. We used the random search procedure [44] for the selection of optimal hyperparameters for all the models. The cross-entropy loss function was used for model training. The stochastic gradient descent optimization method with a momentum of 0.9 and a learning rate of \(1\times 10^{-2}\) was used in the case of the biLSTM and 1D CNN models, while ADAM, with a learning rate of \(1\times 10^{-3}\), was used in the case of the Transformer model. The learning rate was selected using the procedure described in paper [45]. The mini-batch size in the experiments was 32. **biLSTM models**: Bidirectional Long Short-Term Memory networks are an advanced variant of recurrent neural networks (RNN), adept at capturing long-range dependencies in sequential data. Their design addresses the limitations of traditional RNNs, particularly the vanishing gradient problem, by introducing a gated mechanism with additional paths for gradient flow and learning. Another limitation of recurrent neural networks is that they make predictions at the current timestep based on the current element of the input sequence and all previous hidden states. biLSTM allows for the use of information from the entire sequence at each timestep, a useful property if the application of the model has no limitations in terms of the availability of future timesteps. biLSTM can be implemented using two parallel LSTM layers, each processing the sequence in a different direction. The operations within the biLSTM unit can be mathematically described as follows:  
\[\mathbf{f}^{(t)}=\sigma\Big{(}\mathbf{W}_{f}\mathbf{x}^{(t)}+\mathbf{U}_{f}h^{(t-1)}+\mathbf{b}_{f }\Big{)} \tag{1}\]  
\[\mathbf{i}^{(t)}=\sigma\Big{(}\mathbf{W}_{i}\mathbf{x}^{(t)}+\mathbf{U}_{i}\mathbf{h}^{(t-1)}+\mathbf{ b}_{i}\Big{)} \tag{2}\]  
\[\widetilde{\mathbf{c}}^{(t)}=tanh\Big{(}\mathbf{W}_{\widetilde{c}}\mathbf{x}^{(t)}+\mathbf{U}_ {\widetilde{c}}\mathbf{h}^{(t-1)}+\mathbf{b}_{\widetilde{c}}\Big{)} \tag{3}\]  
\[\mathbf{o}^{(t)}=\sigma\Big{(}\mathbf{W}_{o}\mathbf{x}^{(t)}+\mathbf{U}_{o}\mathbf{h}^{(t-1)}+\mathbf{ b}_{o}\Big{)} \tag{4}\]  
\[\mathbf{c}^{(t)}=\mathbf{f}^{(t)}\bigodot\mathbf{c}^{(t-1)}+\mathbf{i}^{(t)}\bigodot\widetilde {\mathbf{c}}^{(t)} \tag{5}\]  
\[\mathbf{h}^{(t)}=\mathbf{o}^{(t)}\bigodot tanh\Big{(}\mathbf{c}^{(t)}\Big{)} \tag{6}\]  
Figure 12: Architectures used for action segmentation: (**a**) biLSTM, (**b**) 1D CNN, and (**c**) Transformer encoder (hyperparameters adjusted in experiments are shown in parentheses). In Equations (1)-(6), \(\sigma(x)=\frac{1}{1+e^{-x}}\) is a sigmoid function that maps inputs to the range [0, 1], and \(tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}\) is the hyperbolic tangent function that outputs values in the range [\(-1\), 1]. \(\textbf{W}_{(\cdot)}\in\mathbb{R}^{H\times D}\), \(\textbf{U}_{(\cdot)}\in\mathbb{R}^{H\times D}\), and \(\textbf{b}_{(\cdot)}\in\mathbb{R}^{H}\) are model parameters and \(\bigcirc\) is an element-wise multiplication operator. \(\textbf{f}^{(t)}\in\mathbb{R}^{H}\) is a forget gate that decides which information to discard from the cell state \(\textbf{c}^{(t)}\in\mathbb{R}^{H}\), guided by the sigmoid function.