# Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models
## 4 Dataset  
We collected a dataset from the manual assembly process of metal grills in a heating, ventilation, and air conditioning system (HVAC). Data collection spanned seven working days, with recordings conducted in manufacturing conditions (see Figure 1). This setting differentiates our dataset from those commonly found in action segmentation literature, which are usually collected in controlled environments or include videos of everyday human activities. The collected videos contain interactions between the worker, product parts, and tools. With minimal differences between individual activity classes, the identification of certain activities relies on models capable of capturing temporal context. Additionally, the background's appearance remains constant throughout the videos, offering no information about the activity class. The assembly activities were conducted on three different product types of varying dimensions by four workers, each with a unique ID (O1, O2, O3, O4), as illustrated in Figure 2. This variability captures the human factor in our dataset. The products within the dataset are labeled as T1, T2, and T3. The videos were recorded from two viewpoints: an overhead (OH) viewpoint and a hand-focused (HF) viewpoint, as shown in Figure 3. This multi-viewpoint approach was driven by the objective of assessing the impact of viewpoint on model performance. The use of OH and HF viewpoints was inspired by research in [40], which suggested that using lower resolution images could accelerate the learning process but at the cost of losing vital visual details. Therefore, we opted for different viewpoints with lower resolutions focused on the work object. The recordings were made using two cameras at a resolution of 1920 \(\times\) 1080 and 30 frames per second in MP4 format. We performed subsampling of the video frames at a rate of 5 frames per second. This formatting aimed to expedite model training, enable learning with the existing infrastructure, and address the redundancy in adjacent video frames. Subsequently, videos were segmented into shorter clips based on specific criteria: each clip had to encompass all assembly process activities, including background activities,  
Figure 1: Manufacturing location. without exceeding two minutes. This process yielded 620 clips with a total duration of 10 h.  
Activity labeling and duration in the video clips were crucial for employing supervised machine learning algorithms. The assembly process included nine work activities for assembling the product and placing it on the shelf, as demonstrated in Figure 4.  
We developed precise labeling rules to define the start and end moments of each activity, ensuring consistent data labeling, as shown in Figure 5, for activity 1. Each frame was labeled with the corresponding activity, including a background activity label (label 0), with a time resolution of 0.2 s between frames implicitly indicating activity duration. Figure 4: Assembly work activities. Figure 3: Different viewpoints of the scene. Figure 2: Four workers in the sample videos. For data preparation, we split the dataset into training, validation, and test sets: 480 videos for training and 70 videos each for validation and testing. The split was conducted through random sampling with stratification based on worker ID and product type to ensure that the dataset's population strata were reflected in all subsets, aiming for better model generalization. The dataset was designed to have a uniform number of samples per worker and product type, which can be seen from Figure 6.  
The typical video duration (see Figure 7), with a median value of 52.8 s, varied depending on the worker and product type, ranging from 36.4 to 90.8 s. Yellow highlight indicates position of a median value. However, total assembly time, measured from the start of the first to the end of the last activity, often differed from video duration due to the presence of background activities. Figure 5: Example of criteria for labeling activities in a video clip.