# GoaTracer: An Open Service for Advanced PE Tracing

Romain Guittienne, Quentin Jacqmin, Jean-Yves Marion, Pierre Marty,
Fabrice Sabatier

<figure><img src="image_1.png" /></figure>

## To cite this version:

Romain Guittienne, Quentin Jacqmin, Jean-Yves Marion, Pierre Marty, Fabrice Sabatier. GoaTracer: An Open Service for Advanced PE Tracing. Botconf 2025 - 12th Edition, May 2025, Angers, France. [hal-05086241](https://hal.science/hal-05086241v1)

HAL Id: hal-05086241

https://hal.science/hal-05086241v1

Submitted on 27 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# GoaTracer: An Open Service for Advanced PE Tracing

Romain GUITTIENNE<fnref n="1" />, Quentin JACQMIN<fnref n="1" />, Jean-Yves MARION<fnref n="1" />, Pierre MARTY<fnref n="1" />, Fabrice SABATIER<fnref n="2" />

Corresponding author : jean-yves.marion@loria.fr

## Abstract

We introduce GoaTracer, a hybrid dynamic binary analysis platform combining instrumentation and introspection to efficiently reconstruct Control Flow Graphs and Call Graphs of Windows Portable Executable files. GoaTracer minimizes execution slowdowns, tracks obfuscated and self-modifying code, and bypasses anti-analysis measures, offering a comprehensive view of malware behavior.

## Introduction

Malware analysis requires both specialized knowledge and software tools, making access to this field particularly challenging for research and teaching purposes. To address this need, we have developed GoaTracer (Goat-Tracer), an open service for dynamic analysis.

GoaTracer executes programs in PE format within a secure environment hosted at the High Security Lab (LHS) of LORIA [1], a joint computer science laboratory of the Université de Lorraine and CNRS. The service generates both the control flow graph and the function call graph, capturing internal calls as well as external API calls based on the program's execution.

GoaTracer is capable of performing dynamic analysis on packers and obfuscated malware. For GoaTracer, a packer mechanism is represented as a sequence of "executable waves." Starting with the first wave, each subsequent wave is defined as the set of run instructions, at least one of which was generated by the previous wave. A formal definition of the packer mechanism is provided in [2].

Another key capability of GoaTracer is its effectiveness in thwarting anti-dynamic-analysis tricks, such as virtual machine detection techniques. This makes GoaTracer highly stealthy and ²reliable for advanced malware analysis.

<footnote n="1">¹Université de Lorraine, CNRS, LORIA, 54000 Nancy, France.</footnote>

<footnote n="2">²CNRS, Université de Lorraine, LORIA, 54000 Nancy, France.</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: GoaTracer architecture</figcaption></figure>

## Usage

The GoaTracer service is freely available at https://goatracer.lhs.loria.fr/. GoaTracer supports Portable Executable (PE) files, both x86 and x64, that can run on Windows 7. For a Dynamic Link Library (DLL), a function to analyze must be selected first.

The output is a ZIP file containing a Call Graph (CG) and a Control Flow Graph (CFG) for each process spawned by the program being analyzed. The graphs are encoded in JSON format, following a "nodes/edges" structure, where "nodes" and "edges" are lists of elements, each representing a node or an edge, respectively.

Two Python scripts are also available at Gitlab - Reader-CFG-CG-JSON, allowing users to efficiently read and extract information from the graphs more efficiently. They include pre-built functions to extract critical CFG data, making them ideal for seamless integration into workflows and batch processing. Users can import these functions as a library to automate tasks, or leverage them for educational purposes to better understand and approach the data. The readers can also serve as simple parsers to extract targeted information. They provide multiple functionalities, such as dumping assembly instructions for a file, generating Control Flow Graphs images, or listing all functions called. Additionally, users can specify waves or basic blocks to target their analysis.

## GoaTracer's outputs

### Control Flow Graph (CFG)

Figure 2 illustrates the Control Flow Graph (CFG) of the snippet below.

<table><tr><td>01_0x0078182e:</td><td>8b75fc (3)</td><td>mov esi, dword ptr [ebp - 4]</td></tr><tr><td>01_0x00781831:</td><td>3375f8 (3)</td><td>xor esi, dword ptr [ebp - 8]</td></tr><tr><td>01_0x00781834:</td><td>ff1528107800 (6)</td><td>call dword ptr [0x781028]</td></tr></table>
<figure><img src="image_4.png" /><figcaption>Figure 2: CFG with nodes represented in JSON format</figcaption></figure>

Nodes represent basic blocks. The key fields of the nodes are:

* **"wave"**: A number representing the unpacking layer of the instruction

* **"instructions"**: A list of the instructions contained in the basic block. Each instruction is represented by its offset from the start of the basic block, its opcode, and its mnemonics.

* **"registers"**: The values of the registers at the beginning of the basic block, as observed the first time it is encountered.
* **"syscalls"**: A list of the system calls made by the basic block. Each syscall is bF2bieBQEBvgQpUVzJUtfC2QJ6rAyJKmzxhwjwypumprepresented by its name, arguments, output, and return value.

* **"exceptions"**: A list of the exceptions caused by the basic block, along with the associated handler.

* **"function_identifier"**: If the node represents a function from a DLL, this field contains the name of the function.

Edges connect two basic blocks and are defined by:

* **"src"**: The parent block.

* **"dest"**: The child block.

* **"type"**: The type of relation (e.g., child, exception, or wave transition).

## Call Graph (CG)

Figure 3 illustrates the call of two API functions from the main procedure.

<figure><img src="image_5.png" /><figcaption>Figure 3: CG with nodes represented in JSON format</figcaption></figure>

Nodes in the Call Graph (CG) represent function calls, either internal or external. A node in the CG contains the following fields:

* **"index"**: A unique natural number identifying the node.

* **"address"**: The hexadecimal address of the function. If the function is from a DLL, this field contains the name of the function instead.

* **"type"**: The type of function represented. This can be either **"function"** if the function originates from a DLL or **"subroutine"** if it is part of the binary itself.

## GoaTracer design in a nutshell

GoaTracer combines both **Introspection** and **Instrumentation** techniques. Before delving further, let us explain these approaches.

**Instrumentation** tools, such as **PIN** [3] and **DynamoRIO** [4] insert custom analysis code into an application during its execution, without modifying the original application binary.
**Introspection**, as implemented in tools like **libVMI** [5] (Virtual Machine Introspection), allows the analysis and interaction with the state of a virtual machine (VM) from the hypervisor (e.g., Xen [6]). This method enables monitoring the behavior of a running system, even at the kernel level, without requiring direct interaction with the guest operating system.

<table><caption>Table 1: Comparative analysis of the characteristics of the following technologies: "Instrumentation," "Introspection," and "GoaTracer."</caption><thead><tr><th></th><th>Instrumentation (DynamoRIO)</th><th>Introspection (libVMI)</th><th>GoaTracer</th></tr></thead><tbody><tr><th>Placement</th><td>Inside the program</td><td>Outside the VM</td><td>Inside/Outside</td></tr><tr><th>Scope</th><td>Program-level analysis</td><td>System-level analysis</td><td>Program and system level analysis</td></tr><tr><th>Impact</th><td>Hooks into execution</td><td>Non-intrusive (external monitoring)</td><td>Hooks into execution</td></tr><tr><th>Pro</th><td>Fine grained analysis at the instruction level</td><td>Stealthy and not easily detectable</td><td>Instruction level analysis<br>Reasonably stealthy and difficult to detect</td></tr><tr><th>Cons</th><td>Intrusive and easily detectable</td><td>Higher grained analysis and not able to output instruction traces</td><td></td></tr></tbody></table>

GoaTracer is built on the Xen hypervisor[7], which runs virtual machines (VMs). Within these VMs, the program being analyzed is instrumented using **DynamoRIO**. To achieve this, we developed two key components: **DynTracer**, a DynamoRIO-based tool, and **DynHybridVMI**, operating at the hypervisor level.

## DynTracer

DynTracer is hooked to the program or DLL being analyzed. It monitors events such as:

* The execution of a new basic block,

* The loading or unloading of a system library,

* The creation or destruction of a thread,

* The occurrence of an exception

* The execution of call and ret instructions

DynTracer leverages DynamoRIO's cache mechanisms to minimize performance overhead during the analysis.

Whenever an event of interest occurs, DynTracer triggers a type-1 interrupt after storing the necessary arguments in the virtual processor (vCPU) registers. This interrupt passes control to **DynHybridVMI**, operating at the hypervisor level.
## DynHybridVMI

A convention for argument passing through registers defines the **RAX** register as the key field for transferring information between DynTracer and DynHybridVMI. For instance:

* If DynHybridVMI is notified of a system library being loaded, it sets the RAX register to 1, signaling DynTracer to whitelist the address space of the library. This ensures that assembly instructions within the library are not unnecessarily tracked.

At the end of processing, DynHybridVMI returns information to DynTracer via the registers, enabling seamless collaboration between the two components.

## Anti-Analysis Countermeasures

An important task of DynHybridVMI is to counter virtual environment detection mechanisms implemented in malware[8]. To achieve this, DynHybridVMI operates at two levels during the execution of a binary:

1. **System Call Alteration**

It modifies the results of certain system calls to conceal the presence of the hypervisor and DynTracer.

2. **Dynamic Modification of Assembly Instruction Sequences**

It alters the output of specific assembly instruction sequences, such as the timing analysis sequence `rdtsc; cpuid; rdtsc`.

The modification of system function behavior is highly transparent to the binary being analyzed in the virtual machine. This is because the functions themselves are not hooked; instead, only their input or output data is modified.

Similarly, hot patching is completely undetectable due to the use of **shadow page technology**. With this technique, a physical memory page is mapped to a virtual memory page in the VM differently depending on its purpose:

* A specific physical page containing the patched code of the program is used for execution.

* A different page is presented when the virtual memory page is accessed for read/write operations.

This ensures that the patched code is executed without being revealed to the program, effectively bypassing anti-analysis techniques.

## Benchmarks on packers

We submitted approximately thirty different versions of the hostname command to GoaTracer, packaged using well-known packers. This allowed us to verify that the execution was carried out correctly and that GoaTracer successfully captured all the system calls made by the hostname code.
<table><caption>Table 2: hostname packed with several packers</caption><thead><tr><th>Packer name</th><th>Execution time</th><th># Basic blocks</th><th># Waves</th><th>Packer name</th><th>Execution time</th><th># Basic blocks</th><th># Waves</th></tr></thead><tbody><tr><td>acpacker</td><td>00h03m29s</td><td>13864</td><td>100</td><td>neolite</td><td>00h02m31s</td><td>322</td><td>1</td></tr><tr><td>acprotect</td><td>00h26m32s</td><td>14250</td><td>99</td><td>npack</td><td>00h00m56s</td><td>227</td><td>1</td></tr><tr><td>aspack</td><td>00h00m20s</td><td>337</td><td>2</td><td>packman</td><td>00h00m25s</td><td>125</td><td>1</td></tr><tr><td>boxedApp</td><td>00h56m52s</td><td>51</td><td>1</td><td>pecompact</td><td>00h00m34s</td><td>62</td><td>2</td></tr><tr><td>epprotect</td><td>00h00m23s</td><td>57</td><td>1</td><td>pespin</td><td>00h01m16s</td><td>2985</td><td>63</td></tr><tr><td>expressor</td><td>00h01m16s</td><td>5</td><td>0</td><td>rlpack</td><td>00h00m41s</td><td>170</td><td>1</td></tr><tr><td>fsg</td><td>00h00m41s</td><td>100</td><td>2</td><td>telock</td><td>00h00m56s</td><td>543</td><td>16</td></tr><tr><td>jdpack</td><td>00h51m48s</td><td>210</td><td>3</td><td>upx</td><td>00h00m25s</td><td>106</td><td>1</td></tr><tr><td>mew</td><td>00h00m40s</td><td>109</td><td>2</td><td>vmprotect</td><td>00h00m32s</td><td>68</td><td>0</td></tr><tr><td>mystic</td><td>00h00m31s</td><td>409</td><td>1</td><td>yodaprotect</td><td>00h02m55s</td><td>650</td><td>5</td></tr></tbody></table>

For this benchmark, no timeout was set, allowing each version to run to completion. However, in some cases, we had to manually interrupt the analysis after a period deemed excessively long.

## Related Works

Well-known instrumentation tools are Pin[3], DynamoRIO[4], Frida[9], and Qiling[10] that are excellent at tracking detailed execution paths.

Cuckoo[11] is one of the best-known sandboxes, designed with two primary components: a host-side part and an agent running within the malware environment. However, the Cuckoo project is currently discontinued. Other sandboxes, such as Malwr[12] and CAPE[13], are based on Cuckoo, although CAPE now appears to have independent development.

Drakvuf[14] employs virtual machine introspection (VMI), allowing it to analyze interactions at the kernel level. Drakvuf is resistant to detection by malware.

In addition to these two open-source solutions, there are older sandboxes like BitBlaze[15] and CWSandbox[16], as well as newer tools such as SecBox, which adopts a contained approach.

On the commercial side, details about implementation are scarce. The most notable products include:

* Any.run[17]

* VMray[18]

* Hybrid Analysis[19]

* JoeSandbox[20]

* VirusTotal[21]
## Limitations and Conclusion

GoaTracer is still under development and has several limitations. Currently, it does not generate thread-specific Control Flow Graphs (CFGs) and does not detect process injection. Additionally, the virtual machines run on Windows 7, and it would be beneficial to support newer versions of Windows. The user interface and the analysis reports also require improvement.

Despite these limitations, GoaTracer is a free tool that enables the analysis of Portable Executable (PE) files at the instruction level. It can output CFGs for processes, API calls, and the sequence of waves in the case of packed or self-modifying applications.

## Acknowledgements

The authors would like to thank Ms. de Freitas Pereira for her insightful comments and the entire Carbone team for their efforts in testing the platform. Then, this work was supported by the French National Research Agency (ANR) under the Plan France 2030, bearing the reference ANR-22-PECY-0007.

## References

[1] High Security Laboratory url: https://lhs.loria.fr/

[2] G. Bonfante, J. M. Fernandez, J.-Y. Marion, B. Rouxel, F. Sabatier, and A. Thierry. "CoDisasm: Medium Scale Concatic Disassembly of Self-Modifying Binaries with Overlapping Instructions". In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA, October 12-6, 2015.

[3] C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: building customized program analysis tools with dynamic instrumentation. In PLDI '05: Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, pages 190–200, New York, NY, USA, 2005. ACM

[4] Derek L. Bruening. "Efficient, Transparent, and Comprehensive Runtime Code Manipulation". AAI0807735. PhD thesis. Cambridge, MA, USA, 2004.

[5] Bryan D Payne. 2012. Simplifying virtual machine introspection using LibVMI. Technical Report. Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA.

[6] Xen Wiki url: https://wiki.xenproject.org/wiki/Virtual_Machine_Introspection

[7] Xen project url: https://xenproject.org/

[8] Unprotect project url: https://unprotect.it/about/

[9] Frida url. https://frida.re/

[10] Qiling framework url: https://qiling.io/

[11] D. Oktavianto and I. Muhardianto, Cuckoo Malware Analysis. Packt Publishing, 2013

[12] Malwr url : https://www.malwr.ee/

[13] Cape github url : https://github.com/kevoreilly/CAPEv2

[14] Tamas K. Lengyel, Steve Maresca, Bryan D. Payne, George D. Webster, Sebastian Vogl, and Aggelos Kiayias. 2014. Scalability, fidelity and stealth in the DRAKVUF dynamic malware analysis system. In Proceedings of the 30th Annual Computer Security Applications
Conference, ACSAC 2014 (New Orleans, LA, USA), Charles N. Payne Jr., Adam Hahn, Kevin R. B. Butler, and Micah Sherr (Eds.). ACM,

[15] BitBlaze url: https://bitblaze.cs.berkeley.edu/

[16] C. Willems, T. Holz, and F. Freiling. Toward automated dynamic malware analysis using cwsandbox. Security and Privacy, IEEE, 5(2):32–39, March-April 2007.

[17] Any.run url : https://any.run/

[18]VMRay url : https://www.vmray.com/

[19] Hybrid Analysis url: https://www.hybrid-analysis.com/

[20] Joe Sandbox url : https://www.joesandbox.com/

[21] VirusTotal url : https://www.virustotal.com/gui/home/search