Robust and efficient malware analysis and host-based monitoring

Over the last few years, a tremendous increase has occurred in the rate in which new malware is appearing in the Internet. Today, organized cybercriminals are using malware as their primary vehicle for carrying out various cyberattacks on computers for huge financial gains. On the defense side, host-based malware detection approaches such as antivirus programs are severely lagging. Industrial average detection rates range from 18% for zero day to 60% for one month old malware samples. The two important aspects that the overall effectiveness of malware detection depend on are the success of extracting information from malware using malware analysis to generate signatures, and then the success of utilizing these signatures on target hosts with appropriate system monitoring techniques. Today's malware employ a vast array of anti-analysis and anti-monitoring techniques to deter analysis and to neutralize antivirus programs, reducing the overall success of malware detection. In this dissertation, we present a set of practical approaches of robust and efficient malware analysis and system monitoring that can help make malware detection on hosts become more effective. The contributions are summarized below: (1) Efficient Methods for Enabling Static Malware Analysis: Static malware analysis suffers greatly for its susceptibility to obfuscations employed by malware. However, it can provide complementary insight to dynamic analysis in those cases where obfuscations can be sufficiently overcome. We present Eureka, a framework that efficiently deobfuscates single-pass and multi-pass packed binaries and restores obfuscated API calls, providing a basis for extracting comprehensive information from the malware using further static analysis. (2) Making Dynamic Analysis Approaches more Robust: While dynamic analysis techniques provide better resilience to malware obfuscations than static analysis, it is susceptible to evasion attacks that detect the presence of a run-time analysis environment. We present the formal framework of transparent malware analysis and Ether, a dynamic malware analysis environment based on this framework that provides transparent fine-(single instruction) and coarse-(system call) granularity tracing. (3) Anticipating Obfuscations that Hide Trigger-based Behavior: Multipath exploring dynamic analysis overcomes the limitation of straightforward dynamic analysis, which may miss trigger-based behavior for its limited view of execution paths. We introduce an input-based obfuscation technique that hides trigger-based behavior from any input-oblivious analyzer. We present the analysis of strengths and weaknesses of this obfuscation and explain how such a technique can impact the efficiency and effectiveness of malware analysis. (4) Reversing Emulator based Obfuscation: Recently, malware authors have adopted emulation as a forms of fine-grained obfuscation that can affect the robustness of both white-box and gray-box analysis techniques. We present an approach that automatically reverse-engineers the emulator and extracts the syntax and semantics of the bytecode language, which helps constructing control-flow graphs of the bytecode program and enables further analysis on the malicious code. (5) Robust and Efficient System Monitoring Techniques: Antivirus programs require monitoring of the target host for code and events that are matched with the signatures or behavior models that they employ, overcoming disabling attacks used by malware. We present Secure In-VM Monitoring, an approach of efficiently monitoring a target host while being robust against unknown malware that may attempt to neutralize security tools.