## 5 Experiments
### Baselines and Evaluation Criteria  
To validate the effectiveness of HertDroid, we compare it with the following state-of-the-art baselines using the evaluation criteria shown in Table 4. In the table, TP (i.e., True Positive) and FP (i.e., False Positive) represent the number of applications predicted to be malicious accurately and inaccurately, respectively, while TN (i.e., True Negative) and FN (i.e., False Negative) represent the number of applications predicted to be benign accurately and inaccurately, respectively.  
To test the effectiveness of AndroAT, we employ several generic GNNs as our baselines, including:  
1. GCN [14] aggregates graph structure and node features through graph convolutions for homogeneous graphs to generate node embeddings;
2. GAT [18] treats all nodes as nodes of the same type and computes attention for each node's neighbor node.
3. GATv2 [36] is an improvement of GAT that replaces the static attention mechanism with dynamic attention. This means that GATv2 is capable of changing the ranking of attention weights based on the query node.
4. GraphSAGE [37] learns node representations by aggregating information from the node's neighbors and recursively aggregates information from the node's k-hop neighbors until a fixed-depth is reached. Similar to GAT, it is also a node generation method designed for homogeneous graphs.
5. HAN [38] is a heterogeneous graph network utilizing meta-paths and attention mechanism. It first performs attention computation on the connected nodes of the target node and then performs attention computation on each meta-path after computing all nodes.
6. Metapath2 vec [39] is a node representation generation method for HG. Metapath2 vec first specifies a meta-path and follows that path to randomly wander, constructs the heterogeneous neighborhood of each vertex, and finally computes the node embedding using the Skip-Gram model.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Measurement** & **Description** \\ \hline Params & Number of parameters in a model \\ Accuracy & (TP + TN)/(TP + TN + FP + FN) \\ Precision & TP/(TP + FP) \\ Recall & TP/(TP + FN) \\ F1 & 2 \(\times\) Precision \(\times\) Recall/(Precision + Recall) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Measurements of HertDroid and baselines.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Datasets** & **Number of Apps** & **Sources** & **Years** \\ \hline Benign set & 10,361 & Google Play Store, AppChina & 2018–2022 \\ Malicious set & 11,043 & VirusShare & 2018–2022 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Statistics of the datasets.  
To further evaluate whether HertDroid performs well among the models based on heterogeneous GNNs, we convert GAT and GraphSAGE into heterogeneous GNNs. This involves duplicating the convolutional layers for each edge type and summing up the results to form the output embeddings, following the method proposed by [PERSON] et al. (2018). We refer to these two heterogeneous models as GAT-hete and GraphSAGE-hete. To evaluate these baselines, we replace these neural networks with AndroAT.  
The second type of baseline consists of the methods specific for the area of Android malware detection, including:  
1. [noitemsep,topsep=0 pt]
2. HinDroid (Droid, 2018) utilizes API calls and three types of relations to construct heterogeneous graphs and classify based on multi-kernel support vector machines.
3. DroidEvolver (DroidEvolver, 2018) extracts features based on API occurrence and determines whether the application is benign or malicious by implementing a model pool such as Passive Aggressive and Online Gradient Descent.