# Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models
## 5 Methodology
### Feature Extraction Methods  
The purpose of using feature extraction models is to reduce the dimensionality of input data and create descriptive features. The practice of using transfer learning for feature extraction is very common in the context of action segmentation, where all studies from the literature review use some form of transfer learning. Three different approaches were used for feature extraction:  
* Feat. extraction from a pretrained model without fine-tuning (FE);
* Feat. extraction from a pretrained model with fine-tuning on our dataset (FT);
* Feat. extraction from a new 2D CNN model trained only on our dataset (TN). The initial assumption was that transfer learning without fine-tuning (the FE approach) allows for quick extraction of lower-quality features, in the sense that the final models learned on them will be less effective compared to models that will use features obtained by some of the other two approaches. In all experiments related to feature extraction methods, the input data was preprocessed in such a way that the dimensions of the input video frames were \(\mathbf{X}^{(i)}\in\mathbb{R}^{224\times 224\times 3}\). Also, for experiments related to the FE and FT methods, pixel values in individual channels of images were centered by subtracting the arithmetic means calculated on the ImageNet dataset. The result of feature extraction from image \(\mathbf{X}^{(i)}\) in each approach was a feature vector \(\mathbf{x}^{(i)}\in\mathbb{R}^{2048}\). We conducted the experiment with the FE method using the original ResNet50 model described in [41], which was trained on the ImageNet dataset. Feature extraction was performed by removing the classification layer from the ResNet50 model and making a forward pass through the model for each frame from both viewpoints. Feature extraction from the pretrained model with fine-tuning on our dataset was carried out by removing the classification layer from the ResNet50 model. We added three new fully connected layers with 2048, 512, and 10 neurons to the ResNet50 base model, with dropout regularization applied to the output vectors of the first and second fully  
Figure 10: Methodology resulting in 27 different approaches. connected layers, with a neuron dropout rate of 50%. After that, models were fine-tuned on our dataset. We trained two models, one on images from the videos recorded from the OH viewpoint and the other on images from the HF viewpoint. The number of parameters in these models was 28,785,162, of which 5,250,570 belonged to the newly added layers. Models were trained using the cross-entropy loss function, and ADAM [42] was chosen as an optimization method, with a learning rate of \(1\times 10^{-4}\) and a batch size of 64 images. Only newly added layers were trained. Both models were trained until convergence, which was determined by the criteria of early stopping in the case of an increase in the ratio of loss on the validation set and loss on the training set above the defined threshold, or stagnation in a decrease of loss on the validation set in three consecutive epochs. After that, we removed the last two layers and extracted features in a similar way as in the FE method. The last approach to feature extraction (TN) was based on a model that was trained only on images from our dataset. The developed model (see Figure 11) uses a type of residual block in its architecture that was introduced in study [43]. In developing the model, good practices observed in the ResNet50 architecture were used. The model contains 8,040,662 parameters, which is about three times less than ResNet50. This makes sense, considering that this model was trained on hundreds of thousands of images compared to ResNet50, which was trained on the ImageNet dataset. Again, we trained two models: one for OH images and the other for HF images. Models were trained using cross-entropy as the loss function and stochastic gradient descent, with a momentum factor of 0.9 and a batch size of 64 images.