# APIASO: A Novel API Call Obfuscation Technique Based on Address Space Obscurity
## 3 API Address Space obscurity Model
### API Call Space Obfuscation  
The goal of API call space obfuscation is to obscure the boundary between API and user functions so that the analyst fails to obtain information associated with known APIs. APIs typically have complex internal call relationships. As a result, obfuscation of key functions can effectively protect API calls, while obfuscation of non-key functions can result in higher overhead. The following chapter provides an analysis of the API's internal call relationships and discusses the API call space obfuscation strategy. **Definition 1**. : _Define the \(DLL\) address space as \(DASpace=\left\{API_{Entry},API_{Internal},G_{DLL}\right\}\), which comprises three parts: \(API_{Entry}=\left\{f_{e_{1}},f_{e_{2}},\cdots,f_{e_{m}}\right\}\), includes the set of API entry functions called directly by the user; \(API_{Internal}=\left\{f_{m+1},f_{m+2},\cdots,f_{n}\right\}\), includes the set of API internal functions called indirectly by the user; and \(m\), which refers to the number of functions inside the DLL space that can be called directly by the user. \(n\) denotes the total number of functions inside the DLL space. \(G_{DLL}\) represents the function call graph within the DLL address space, and is denoted as \(G_{DLL}=\left\{F_{DLL},E_{DLL}\right\}\). \(F_{DLL}\) in \(G_{DLL}\) refers to the set of functions inside the DLL address space, i.e., \(F_{DLL}=API_{Entry}+API_{Internal}\)._  
**Definition 2**. : _Define each API address space as \(AASpace_{API_{i}}=\left\{\{f_{e_{i}}\},\left\{f_{f_{i_{1}}},\cdots,f_{i_{s}}, \cdots f_{i_{k}}\right\},\right.\)\(\left.G_{API_{i}}\right\}\), \(\left(1\leq i\leq m,m+1\leq i_{x}\leq n\right)\). \(f_{e_{i}}\in API_{Entry}\) indicates the functions that can be called directly by the user inside the API address space, \(\left\{f_{i_{1}},f_{i_{2}},\cdots,f_{i_{k}}\right\}\subset API_{Internal}\) indicates the set of internal call functions, and there is a direct or indirect call relationship between \(f_{e_{i}}\) and \(f_{i_{x}}\). \(G_{API_{i}}\) denotes the \(API_{i}\) internal function call graph, denoted as \(G_{API_{i}}=\left\{F_{API_{i}},F_{API_{i}}\right\}\), \(\left(1\leq i\leq m\right)\), where \(F_{API_{i}}\) denotes the set of functions, i.e., \(F_{API_{i}}=\left\{\left\{f_{e_{i}}\right\},\left\{f_{i_{1}},\cdots,f_{i_{s}}, \cdots f_{i_{k}}\right\}\right\}\)._  
According to each API function call graph, \(G_{API_{i}}\), \(\left(1\leq i\leq m\right)\) can calculate \(G_{DLL}\), the calculation process is as follows:  
\[G_{DLL}=G_{API_{1}}\cup G_{API_{2}}\cup\cdots G_{API_{n}} \tag{1}\]  
The adjacency matrix can be used to describe the relationship between function calls. Based on the function call graph \(G_{DLL}\) inside the DLL address space generates the adjacency matrix as \(A\). If there exists a function \(f_{p}\) calling a function \(f_{q}(1\leq p\
eq q\leq n)\) inside the DLL address space, then \(A_{pq}=1\), otherwise \(A_{pq}=0\). Obviously, the matrix \(A\) is a square matrix and the number of matrix ranks is equal to the number of DLL space functions \(n\). The reachable matrix \(P\) and the adjacency matrix \(A\) are both Boolean matrices. \(P_{pq}=1\) means that there is a call path from function \(f_{p}\) to \(f_{q}\). The reachable matrix \(P\) can be obtained from the adjacency matrix \(A\), which is calculated as follows:  
\[P=A^{1}\lor A^{2}\lor A^{3}\lor\ldots\lor A^{n},n=r(A)=c(A) \tag{2}\] \begin{table}
\begin{tabular}{c c} \hline \hline
**Notation** & **Description** \\ \hline \(DASpace\) & DLL address space \\ \(API_{Entry}=\left\{f_{e_{i}},f_{e_{2}},\cdots,f_{e_{m}}\right\}\) & API entry functions \\ \(API_{Internal}=\left\{f_{m+1},f_{m+2},\cdots,f_{n}\right\}\) & API internal functions \\ \(G\) & Function call graph \\ \(AASpace_{API_{k}}\) & API address space \\ \(A\) & Adjacency matrix \\ \(P\) & Reachable matrix \\ \(Level(f_{q})\) & Level of functions in the DLL address space \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Explanation of notations. **Definition 3**.