## 3 Theoretical Background
### Custom CNN-GRU-3 Model  
In this study, a hybrid CNN-GRU-3 architecture was selected to utilize the strengths of both convolutional and recurrent layers. The CNN layers effectively understand local patterns and spatial features from the API-DLL embeddings, while the stacked GRU layers are well-suited for modeling sequential dependencies across time steps. This combination enables the model to learn low-level patterns and long-term behavioral dependencies, which is essential to identifying complex malware behaviors accurately. The model has three convolutional layers, three grouping layers, three GRU layers, and one fully connected layer. The size of the convolutional layer used for feature extraction is 3 \(\times\) 3. We use the Exponential Linear Unit (ELU) activation function [29]. Max-pooling layers with 2 \(\times\) 2 kernels reduce the feature map dimensions, followed by a smoothing layer that transforms the output into a one-dimensional vector. The model architecture includes one GRU layer with 512, 256, and 128 neurons, a depth of 3, and a dropout rate of 0.3. Finally, a fully connected dense layer with 64 neurons and a Rectified Linear Unit (ReLU) activation function is added, followed by a dense layer with a single neuron and a Sigmoid activation function. This GRU component increases the ability of the model to distinguish complex patterns in sequential data. The proposed architecture that mixes two DL models is shown in Figure 2. The optimization process for the model included 150 epochs during training and a batch size of 64. The Adam optimizer with a learning rate of 0.001 was preferred for adaptive learning  
Figure 2: Overview of the GraphSAGE node embedding process. (**a**) A node’s local neighborhood is sampled up to depth \(k\), (**b**) feature information from neighbors is aggregated using learnable functions, and (**c**) the resulting embedding is used for prediction [28].  
rate computations, making it a suitable choice for efficient optimization. The dataset is split into 80% training and 20% test sets and trained with 5-fold cross-validation. The pseudocode for the custom CNN-GRU-3 model is given in Algorithm 2.  
```
0:\(X_{\text{train\_input}}\)\(X_{\text{train\_output}}\)
0: Trained CNN-GRU-3 model and classification performance metrics
1:functionModelTraining(X_train_input, X_train_output)
2: Apply convolutional layers with ELI activation:
3:Conv1D(filters=64, kernel_size=3), activation=ELU
4:MaxPooling1D (pool_size=2)
5:Conv1D(filters=64, kernel_size=3), activation=ELU
6:MaxPooling1D (pool_size=2)
7:Conv1D(filters=64, kernel_size=3), activation=ELU
8:MaxPooling1D (pool_size=2)
9: Apply a Flatten layer
10:GRU Layers:
11:GRU Layer 1:512 units, dropout=0.3
12:GRU Layer 2:256 units, dropout=0.3
13:GRU Layer 3:128 units, dropout=0.3
14:Fully Connected Output Layers:
15:Dense: 64 units, dropout=0.3, activation=ReLU
16:Dense: 1 unit, activation=Sigmoid
17:Output layer with sigmoid activation for 2 classes
18:Training and Evaluation:
19:Train model using Adam optimizer and binary_crossentropy loss
20:Evaluate model performance
21:endfunction
```  
**Algorithm 2** Pseudocode for Custom CNN-GRU-3 Model  
The embeddings developed by GAT are arranged in a time series-like sequence to provide data input to the CNN-GRU model. The embedding of each node is placed in the sequence according to the original order of the API calls. This arrangement allows CNN to understand spatial relationships between adjacent nodes and GRU to understand sequential patterns. For example, if the API call order in a software is [\"CreateFile\", \"WriteFile\", \"CloseHandle\"], the embeddings of these calls are fed into the model in the same order. This approach allows graph data to be aligned with time-series models.