## 3 Methodology
### Data Fusion  
The image data samples of both streams are labeled and time-synchronized with each other to develop two different spatial characteristics for one instant of the construction equipment at a specific time. The external camera records the movement of the excavator and its company in the surrounding environment. The standalone external camera view limits monitoring the excavator's performance due to the occlusions, camera angle, or position. A second stream based on the point cloud data sample would provide additional information where the external camera stream is limited. The point cloud data stream mainly focused on the boom, arm, and bucket movement of the excavator to assist in learning the first stream for various activities. A customized data loader is developed to achieve a single synchronized input for the two-stream CNN pipeline. External camera image and point cloud image form a single unit of the multi-stream input data, as shown in Figure 5. Each branch of the two-stream CNN architecture is linked with only one input stream to extract the spatiotemporal features of the input data streams. Additionally, the optical flows of the input data stream are estimated simultaneously. Before training, dense optical flow is calculated for consecutive frames using the Farneback algorithm to capture detailed motion dynamics. This optical flow estimation enhances the ability to recognize motion variations, particularly when static RGB frames may not provide sufficient distinctive features. After extracting the spatiotemporal features and optical flows of the multi-stream input data through two branches of the CNN, the sequential information of both streams is constructed and concatenated for further learning. During the concatenation operation, the spatial features of both streams are horizontally stacked along the feature dimensions, and spatial and channel-wise information of both streams is preserved through the concatenation tensor to enhance the learning of the LSTM model. The LSTM layer receives the single concatenated input of both streams for processing the spatiotemporal features and optical flows, followed by a Multi-Layer Perceptron (MLP) head that constructs a two-stream CNN-LSTM model. The MLP head is responsible for making the final prediction based on learning the model and evaluating the performance predictors and loss values. The multi-stream input data consisting of five activities, approach, digging, dumping, idle, and leveling, is used to train the two-stream CNN-LSTM model.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Components of the** & \multicolumn{3}{c}{**File Naming Standard**} \\
**Proposed Standard** & \multicolumn{3}{c}{**E\_230905\_0907\_CE\_000750\_SFSD.jpg**} \\ \hline Equipment type & \multicolumn{3}{c}{Excavator} \\ \hline Acquisition date & Year month date & & \\ \hline Acquisition time & Hour min & & \\ \hline Source sensor type & Camera External (CE) & & Point Cloud (PC) \\ \hline Frame number & xxxxxx & 000000 & \(\sim\) & 999999 \\ \hline Work type & \begin{tabular}{c} Excavating \\ and Loading \\ \end{tabular} & \begin{tabular}{c} Grading \\ (Leveling) \\ \end{tabular} & \
\begin{tabular}{c} Foundation and Trenching \\ \end{tabular} &
\begin{tabular}{c} Slope \\ Digging \\ \end{tabular} \\ \hline Activity type & **A**Approach & **Digging** & **DUmping** & **Idle** & **Leveling** \\ \hline \hline \end{tabular}
\end{table}
Table 2: File naming rules for the standard labeling format of input data.