Static Binary Instrumentation with Applications to COTS Software Security
Authors: Zhang, Mingwei
Published in: Academic Commons (Stony Brook University) (2015-05-01)

Binary instrumentation has assumed an important role in software security, as well as related areas such as debugging and monitoring. Binary instrumentation can be performed statically or dynamically. Static binary instrumentation (SBI) is attractive because of its simplicity and efficiency. However, none of the previous SBI systems support secure instrumentation of COTS binaries. This is because of several challenges including: (a) static binary code disassembly errors, (b) difficulty of handling indirect control flow transfers, (c) ensuring completeness of instrumentation, i.e. | instrumenting all of the code, including code contained in system libraries and compiler-generated stubs, and (d) maintaining compatibility with complex code, i.e. | ensuring that the instrumentation does not break any existing code. We have developed a new static binary instrumentation approach, and present a software platform called PSI that implements this approach. PSI integrates a coarse grained control flow integrity (CFI) property as the basis of secure, non-bypassable instrumentation. PSI scales to large and complex stripped binaries, including low-level system libraries. It provides a powerful API that simplifies the development of custom instrumentations. We describe our approach, present several interesting security instrumentations, and analyze the performance of PSI. Our experiments on several real-world applications demonstrate that PSI’s runtime overheads are about an order of magnitude smaller than that of the most popular platforms available today, such as DynamoRIO and Pin. (Both these platforms rely on dynamic instrumentation.) PSI has been tested on over 300 MB of binaries. In addition to our platform PSI, we describe two novel security applications developed using PSI. First, we present a comprehensive defense against injected code attacks that ensures code integrity at all times, even against very powerful adversaries. Second, we present a defense against code reuse attacks such as return-oriented programming (ROP) that is effective against adversaries possessing a wide range of capabilities. | Binary instrumentation has assumed an important role in software security, as well as related areas such as debugging and monitoring. Binary instrumentation can be performed statically or dynamically. Static binary instrumentation (SBI) is attractive because of its simplicity and efficiency. However, none of the previous SBI systems support secure instrumentation of COTS binaries. This is because of several challenges including: (a) static binary code disassembly errors, (b) difficulty of handling indirect control flow transfers, (c) ensuring completeness of instrumentation, i.e. | instrumenting all of the code, including code contained in system libraries and compiler-generated stubs, and (d) maintaining compatibility with complex code, i.e. | ensuring that the instrumentation does not break any existing code. We have developed a new static binary instrumentation approach, and present a software platform called PSI that implements this approach. PSI integrates a coarse grained control flow integrity (CFI) property as the basis of secure, non-bypassable instrumentation. PSI scales to large and complex stripped binaries, including low-level system libraries. It provides a powerful API that simplifies the development of custom instrumentations. We describe our approach, present several interesting security instrumentations, and analyze the performance of PSI. Our experiments on several real-world applications demonstrate that PSIâ€™s runtime overheads are about an order of magnitude smaller than that of the most popular platforms available today, such as DynamoRIO and Pin. (Both these platforms rely on dynamic instrumentation.) PSI has been tested on over 300 MB of binaries. In addition to our platform PSI, we describe two novel security applications developed using PSI. First, we present a comprehensive defense against injected code attacks that ensures code integrity at all times, even against very powerful adversaries. Second, we present a defense against code reuse attacks such as return-oriented programming (ROP) that is effective against adversaries possessing a wide range of capabilities. | 157 pages