# Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models
## 2 Related Work  
The problem of the simultaneous recognition and time segmentation of activities from video input, known as action segmentation or action detection, remains a critical area in computer vision research. The objective is to construct models capable of temporally segmenting and recognizing activities in untrimmed videos containing multiple temporal segments. \"Action detection\" typically refers to processing videos consisting of a predominantly background class (unlabeled activities) to detect sparse segments with labeled activities [11]. In contrast, \"action segmentation\" involves processing densely labeled videos and predicting action at every frame [12], a common scenario in manufacturing processes. This review focuses on deep learning-based action segmentation models and approaches used in a manufacturing context. Early attempts to diminish the dimensionality of input data and define descriptive features commonly employed 2D convolutional neural networks (CNNs) such as VGG16 [12; 13; 14; 15; 16], VGG19 [17], AlexNet [18], and ResNet50 [19; 20]. These models, pretrained on datasets like ImageNet, were fine-tuned for specific tasks. Since 2D CNNs can only capture spatial features from individual images, features derived from optical flow computations [14; 15; 16; 20] or similar, less computationally demanding algorithms [12; 13; 21] were often used to incorporate a limited temporal context. Some researchers have employed 3D CNNs like C3D [22; 23], which consider both spatial and short-term temporal components. Papers [11; 24] combined 3D CNNs and optical flow for enhanced temporal feature representation. However, the main drawback of 3D CNNs is their computational cost compared to 2D CNNs. Long Short-Term Memory (LSTM) and 1D CNN models have been popular for classification in action segmentation. For example, paper [13] utilized bidirectional LSTMs (biLSTM) to leverage information from both ends of a time series. This study introduced an approach using a combination of four distinct 2D CNN models, two focusing on full-image dimension data and the other two trained on person-centric data, to capture both location-dependent and -independent activity information fed into the biLSTM model. Paper [12] proposed an encoder-decoder architecture based on 1D convolutional layers hierarchy, designed to capture long-term temporal regularities. This model, compared to [13], showed much faster learning (30 times faster) on the same dataset, with better or comparable efficiency. Research study [14] combined ideas from [12; 13], employing the architecture from [12] but replacing the encoder's 1D CNN with biLSTM. These modifications resulted in slightly higher accuracy but slower learning. Study [19] suggested the simultaneous training of feature extraction and classification models, with ResNet50 for feature extraction and LSTM for classification, albeit requiring the processing of 2 s video clips. The results were marginally better than training the models separately but with a longer training time for the integrated model. In paper [16], improvements were proposed over similar research like [12; 14] by designing a model with two parallel temporal branches. The branch responsible for analyzing videos at full temporal resolution for fine-grained action segmentation was named the residual branch. The other branch used downsampling and upsampling to process videos at various temporal resolutions, improving frame-by-frame classification accuracy. These branches were linked using a deformable temporal residual module employing deformable convolutional filters. However, this model struggled with accurately segmenting shorter actions between longer segments. [PERSON] et al. [25] introduced an ASRF framework to address excessive segmentation in existing models like MS-TCN [24]. This framework had three phases, using features extracted via models like I3D [26] and refined via MS-TCN.