## 3 Methodology
### CNN Network for Activity Recognition Through Optical Flow  
In this research, a multi-stream input data framework is developed to enhance the accuracy of excavator activity recognition, particularly under occlusion conditions. The multi-stream input data are collected through an external camera and a lidar at the construction site. The external camera records the videos of excavator movements at the construction site, which are later converted into video frames. The lidar generates the point cloud of the construction site and focuses the movement of the excavator body parts at a specific frame rate. The point cloud data are processed using an open-source 3D point cloud processing software to construct an image dataset that represents the point cloud data. The image dataset of both the external camera and the lidar is synchronized. The synchronized data are annotated manually based on the type of activity of the excavator during that timestamp of the video stream. Then, the labeling process follows the specific labeling format to label the image dataset systematically. The multi-stream image-based labeled dataset serves as input for the activity recognition method, utilizing the two-stream CNN-LSTM deep learning algorithm, as illustrated in Figure 5. The RGB image data provide the spatial characteristics of the input data. In contrast, the temporal information is estimated through  
Figure 5: Two-stream CNN-LSTM network for activity recognition of excavator activities. the optical flows of these RGB images. Optical flow is computed using the Farneback algorithm, a dense optical flow estimation method that effectively captures motion patterns between consecutive frames to improve motion representation. The Farneback algorithm uses polynomial expansion to approximate pixel intensities within a local neighborhood, enabling efficient calculation of dense motion fields. The resulting optical flow representations are then combined with RGB images to create a comprehensive spatiotemporal feature set for activity recognition. The Farneback algorithm's ability to efficiently compute dense motion fields improves motion representation, assisting the model in tracking excavator activities. This method balances computational efficiency and accuracy by applying polynomial expansion for motion estimation. Optical flow is calculated by tracking and analyzing the motion patterns of objects' edges and surfaces between consecutive grayscale frames. The motion vectors of the video frames are computed to evaluate the dense optical flow. This study implemented the Farneback method to assess dense optical flows, which is computationally efficient and handles various types of motion. The proposed research identifies the various activity types of excavators (approach, digging, dumping, idle, and leveling) by analyzing the optical flows and spatial information of RGB images. After evaluating the dense optical flows, the two-stream CNN-LSTM processes an integrated input stream of labelled RGB images collected through an external camera and lidar. A two-stream architecture incorporating spatial and temporal information has been introduced using RGB and optical flow data [30]. The architecture of the two-stream CNN-LSTM, comprising 14 layers, is illustrated in Figure 5. It consists of two parallel branches, each designed to extract spatiotemporal features from the synchronized RGB and point cloud input streams. Each branch includes two convolutional layers, where the first uses 32 filters with a 3 \(\times\) 3 kernel and the second uses 64 filters with a 5 \(\times\) 5 kernel, followed by Rectified Linear Unit (ReLU) activations and 2 \(\times\) 2 max-pooling layers with a stride of 2 to reduce spatial dimensionality. Padding is applied (1 pixel in the first stream and 2 pixels in the second) to preserve spatial resolution. The outputs of both branches are concatenated along the channel dimension to maintain spatial and channel-wise integrity before being passed to the LSTM, which has 64 hidden units to capture sequential dependencies.