DOI: 10.25549/usctheses-c40-376998
title: Supporting faithful and safe live malware analysis

description: Current malware analysis relies heavily on the use of virtual machines and debuggers for safety and functionality. However, these analysis tools produce a variety of artifacts, which can be detected by malware and thus hinder malware analysis. First, a virtual machine usually executes multiple host instructions to simulate a single guest instruction. This inevitably introduces delay and often deviations from the instruction’s semantics defined in the instruction set manual. Existing research approaches to uncover differences between VMs and physical machines use randomized testing, and thus cannot completely enumerate these differences. Second, debuggers modify malware code and handle it in specific ways. These artifacts can be used by malware to detect debuggers. When malware detects the presence of such entities by using anti-VM and anti-debugging techniques, it may hide its original, malicious purposes. For example, malware may behave like normal programs, exit prematurely, escape from the analysis environment, or even crash the system. Third, current malware analysis usually analyze malware in an isolated system without network access. However, recent malware tends to rely on networking to function properly. For example, bot clients have to fetch commands and payload from bot masters on the Internet. Therefore, current isolated analysis setting runs the risk of incomplete analysis. There are some other research efforts that adopt no network restrictions at all, which allow malware to contact Internet hosts freely. These approaches may cause substantial damage to others if malware launches an attack, and thus are not encouraged due to legal and ethical considerations. ❧ In this dissertation, we propose three complete pieces of work to address the above challenges. First, we propose cardinal pill testing, which aims to enumerate the differences between a given VM and a physical machine, through carefully designed tests. Cardinal pill testing finds five times more pills by running fifteen times fewer tests than previous approaches. ❧ Second, we propose Apate, a debugger plug-in, which systematically hides the debugger from malware. In this piece of work, we enumerate anti-debugging attacks and classify them into 6 categories and 16 sub-categories. We then develop techniques that detect all these anti-debugging attacks and defeat them by hiding debugger artifacts. We develop novel techniques for handling attacks in two out of our six attack categories. For three out of the remaining four categories, prior research has sketched ideas for attack handling, but we implement and evaluate them. ❧ Apate can also be used to detect and defeat cardinal pills, thanks to its extensible design. Based on Apate, we propose a framework, called VM Cloak, to handle the anti-VM attacks in malware. This is achieved by monitoring each executed malware command, detecting potential pills, and modifying at run time the command’s outcomes to match those that a physical machine would generate. ❧ Finally, we propose a malware analysis framework, Pandora, to support live and safe malware experiment. Pandora enforces a mixed containment policy for malware’s network traffic, which can provide necessary network input for malware’s normal execution and limit potential damage to Internet hosts to the minimal level. Depending on the nature of network flows, we may forward them to Internet hosts, redirect to our service impersonators, fake replies, rate limit, or drop them. Pandora executes malware using a set of machines, which can be either physical or virtual machines, with or without Apate and VM Cloak. In order to help defenders better understand malware’s network behaviors, we propose a model, called NetDigest, to extract the session dynamics from network packets.