## 3 Methodology
### Data Collection and Work Type Definition  
The database developed for excavator activity recognition is collected from five types of earthwork operations: Approach, Digging, Dumping, Idle, and Leveling. The source data, including video, image, and point cloud data, are collected at a rate of five frames per second. The video/image data are collected in Audio Video Interleave (AVI) or Joint Photographic Experts Group (JPG) format, and the point cloud data are collected in Compressed LiDAR (LAZ) format. An external/surveillance camera (Hanwha Vision Co., Ltd., Seongnam-si, Gyeonggi-do, Republic of Korea) is placed at the construction site to monitor the excavator's activities. The LiDAR (Light Detection and Ranging) also captured the point cloud at specific intervals during the earthwork operations, as shown in Figure 2. The LiDAR system consists of a laser scanner, GPS, and an IMU sensor that measures distances and creates 3D maps of objects and environments using remote sensing technology scanner (Leica Geosystems AG, Heerbrugg, Switzerland). The point cloud data are processed to curate an image dataset representing the point cloud data. The image dataset of both the external camera and the lidar is synchronized. A total of 495,000 labeled datasets, comprising video frames and point cloud data, were utilized to facilitate the learning process of the two-stream CNN-LSTM network. A high-quality, synchronized database has been developed, which facilitates the autonomous identification of equipment activity and the development of autonomous construction sites. Figure 1: Overview of the proposed method based on a two-stream deep learning architecture (CNN-LSTM) for excavator activity recognition. Hydraulic excavators are one of the most essential equipment in the construction industry for performing various tasks at the construction site, playing a pivotal role in earthwork operations [33; 76]. In this research, a medium-sized hydraulic crawler excavator manufactured by Doosan Infracore Co., Ltd., Incheon, Republic of Korea, is used. The work of excavators is distinguished based on the movement of the excavator body and categorized into four types: \"Excavation and Loading\", \"Leveling\", \"Foundation and Trench Excavation\", and \"Slope Excavation\" (Figure 3). Excavation and Loading deals with the excavator-dump truck interaction for digging the earthen material and dumping it into the excavator; Foundation and Trench Excavation deals with the trenches for the underground utilities or foundation works for the preparation of the underground infrastructure installation; and Slope Excavation deals with the slope or hillside work to create a flat and stable surface. These works are further divided into activities by the excavators. Idle is the state of the excavator when it is either stopped or waiting for the other equipment to perform its task with the bucket empty or filled with material. The digging activity starts when the bucket touches the ground for excavation and ends when it leaves the ground with earthen material in the bucket (reducing the total amount of earthen material at the earthwork workspace). Dumping refers to transferring earthen material from the bucket to the surroundings or a dump truck by closing and unfolding the bucket joints until the bucket is empty of earthen material. Leveling refers to flattening the earthen material from the front, rear, left, and right sides with the tip of the bucket without changing the total amount of earthen material in the earthworkspace and any earthen material in the bucket. Approach, digging, and leveling are also known as productive activities that directly contribute to the progress of the construction process and are called digging, dumping, and leveling. The approach refers to moving excavators from one place to another or swinging to assist in productive activities. The approach includes moving and swinging.