## 5 Experiments
### Binary Clone Detection  
In this experiment, we evaluate whether GeneDiff can detect the similarity of assembly functions. First, we compare GeneDiff with some popular mono-architecture detection methods (i.e., BinDiff [12], Graphlet [32], n-gram and LSH-S [33]) on binaries targeted to the same ISA but with different compiler optimization options. Then, we evaluate GeneDiff in cross-ISAs with different compiler optimization \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Similarity** & **ADD RAX, 1** & **ADD EAX, 1** & **ADD RBX, [RAX+0x10]** & **ADD REX, [RSP+0x158]** \\ \hline ADD X20, X20, \#0x18 & 0.9992 & 0.9631 & 0.9925 & 0.9925 \\ \hline ADD X1, X1, \#1 & 0.9992 & 0.9631 & 0.9925 & 0.9925 \\ \hline ADD X0, X0, \#0x21 & 0.9992 & 0.9631 & 0.9925 & 0.9925 \\ \hline ADDS X0, X21, \#1 & 0.9540 & 0.9271 & 0.9528 & 0.9528 \\ \hline ADD W19, W19, \#1 & 0.9631 & 0.9992 & 0.9416 & 0.9416 \\ \hline ADD X2, X29, \#0x60 & 0.9992 & 0.9631 & 0.9925 & 0.9925 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The similarity of the _add_ family instructions. Figure 7: A visualization of assembly instructions from different instruction set architectures. options (O0-O3) and compare it with the symbolic execution approach, which is widely used by other binary clone detection approaches [9; 10; 13]. In our experiments, the dataset included binaries that were compiled from OpenSSL-1.0.1f, curl-7.52.1, zlib-1.2.11, libpng-1.6.33, binutils-2.32 and coreutils-8.30, all of which are commonly used by high-power IoT devices. We divided the dataset into two parts: 80% of the binaries were used for training, and the remaining 20% were used for testing. In training mode, we used the binaries in the database to train the representation model to generate numerical vectors for every instruction. In testing, GeneDiff used these instruction vectors to generate semantic vectors for assembly functions and detects the similarity. The control group, which utilized the symbolic execution approach, converts test functions into VEX and uses randomly sampled values to compare I/O values for the functions, where the sampling bound is 2000. Table 4 shows the experimental results from the same ISA. The results indicate that the average AUC-ROC of GeneDiff is higher than those of the mono-architecture detection methods. In addition, GeneDiff can also detect binary clones in cross-ISAs. Figure 8 shows the experimental results, where the red line is GeneDiff. Figure 8a shows the ROC curve of the function similarity detection in the same ISA with different compiler optimization options. The two ROC curves are close to the top-left border, which means that GeneDiff and BinDiff achieve good accuracy when comparing binaries targeted to the same ISA. The ROC curve of GeneDiff is closer to the top-left border. Figure 8b-e shows the ROC curves of similarity detection for functions in different ISAs with the same compiler optimization options (O0-O3). Note that the ROC curve of GeneDiff in Figure 8b is very similar to the ROC curve of Figure 8a, but the curve of BinDiff is far from the top-left border; it indicates that the differences between the different ISAs have little effect on GeneDiff's similarity detection performance, but that of BinDiff is seriously affected. Figure 8f shows the ROC curves for detecting the similarities of functions from different ISAs and with different compiler optimization options: the red line is GeneDiff, the green line is BinDiff, and the orange line is the control group. The ROC curve of GeneDiff is significantly closer to the top-left border than the control group, and the AUC (=92.35%) of GeneDiff is greater than that of the control group's AUC (=74.47%) and BinDiff's AUC (=68.71%), which means that GeneDiff is better than the control group and BinDiff at detecting function clones across ISAs and across compiler optimization options.