## 1 Introduction
To address these limitations and enable robust activity recognition, there is a crucial need for methodologies that combine multi-stream data inputs from different sources, providing a more comprehensive understanding of the excavator's operational context. Several other fields have implemented multi-stream input data for diagnosing health conditions [27; 28; 29]. A two-stream architecture incorporates spatial and temporal information using RGB and optical flow data [30]. However, the construction industry has not yet explored the use of multi-stream video data input, which combines external camera images and point cloud images, for activity recognition. Furthermore, point clouds are commonly used in the construction industry for tasks such as 3D site reconstruction, quality assessment, and progress tracking. Despite their widespread use, they have not been fully exploited for activity recognition. Due to their spatial richness, point clouds offer unique advantages, such as depth perception and robustness to environmental lighting conditions. Therefore, they provide a valuable input stream that complements traditional RGB video data, especially for capturing excavator geometry and movement in three-dimensional space. As a result, the potential of a two-stream deep learning architecture that simultaneously utilizes RGB frames and point cloud inputs needs to be explored, as it has the potential to improve the robustness and generalizability of activity recognition models significantly. This paper presents a novel methodology for activity recognition of excavators through an integrated input stream of external camera images and point cloud images, utilizing a two-stream CNN-LSTM deep learning pipeline. This approach represents a significant advancement over single-source methods. This study simultaneously collects data from the external camera and lidar scanner to train a two-stream CNN-LSTM. The Farneback method analyzes visual information and estimates dense optical flows based on temporal sequences of images. A two-stream CNN pipeline processes an integrated input stream, including camera images, point cloud images, and motion optical flow. This architecture evaluates the sequential pattern of video frames and processes it using the LSTM architecture to enable the recognition of various excavator activities. The excavator activities, including Approach, Digging, Dumping, Idle, and Leveling, were chosen for activity recognition. The proposed deep learning model accurately categorizes excavator activities with a 94.67% accuracy rate. When compared to the single stream on the same dataset for each input separately, the RGB-based model achieves an accuracy of 90.67%, while the point cloud-based model achieves an accuracy of 92.00%. This method outperforms existing algorithms for recognizing excavator activities due to its high accuracy. Additionally, a standard data labeling format is being proposed for labeling the data gathered based on excavator activities at a specific timestamp. Moreover, a database containing 495,000 video frames of external camera images and point cloud images from various construction sites with different conditions has been developed. The main academic contributions of this paper are as follows:  
* A novel multi-stream input data-based methodology for excavators' activity recognition using a two-stream CNN-LSTM DL algorithm. * Improvement of activity recognition through a two-stream CNN-LSTM algorithm by effectively combining spatial (RGB frames) and temporal (optical flow) data from RGB camera and Lidar streams under occlusion. * A distinct dataset has been developed with two different input sources for excavator activities during the earthmoving process at various site conditions. A database containing 495,000 video frames of RGB external camera images and point cloud images from various construction sites with different conditions has been developed. * A standard data labelling format is proposed for labelling the data gathered based on the excavator activities at a specific timestamp.