Strengthening the Evidence that Attack Surfaces Can Be Approximated with Stack Traces

Proactive security review and test efforts are a necessary component of the software development lifecycle. Resource limitations often preclude reviewing the entire code base. Making informed decisions on what code to review can improve a team’s ability to find and remove more vulnerabilities. Automated attack surface approximation is a technique that uses crash dump stack traces to predict what code may contain exploitable vulnerabilities. The goal of this research is to help software development teams prioritize security efforts by approximating the attack surface of a software system via stack trace analysis. We explore the attack surface approximation approach using Firefox stack traces. We also generate a vulnerability prediction model using metrics such as frequency of appearance, code churn, and unique authors. We create an attack surface approximation at the file level, in which the 8.4% of the files that were part of the attack surface approximation contained 72.1% of the vulnerabilities seen for the Firefox product. We observed a recall for vulnerability prediction of 0.8 and a precision of 0.04. We generate a decision tree based on churn, author count, and our attack surface approximation metric to guide practitioners in prioritizing code for security review. These results corroborate previous work that showed crash dump stack traces can be used as a metric to prioritize security efforts. Categories and Subject Descriptors D.2. 8 [Software Engineering]: Metrics – Process Metrics, Product Metrics. General Terms Management, Measurement, Documentation, Experimentation, Security. Keywords stack traces, attack surface, prediction models 1. INTRODUCTION The attack surface of a system can be used for determining which parts of a system could have exploitable vulnerabilities. If data cannot be passed to a vulnerability from outside a system, perhaps precious engineering hours working on finding and fixing vulnerabilities should be spent elsewhere. The Open Web Application Security Project (OWASP) defines the attack surface of a system as the paths in and out of a system, the data that travels those paths, and the code that protects the paths and the data. Items not on the attack surface of a system are unreachable by outside input, and therefore less likely to be exploited [1]. Vulnerability detection and removal techniques, such as security reviews and penetration testing, can then be prioritized to the attack surface rather than the entire codebase. Reducing the amount of code to be inspected may help improve the economics of security assessments and allow for more proactive reviews of potentially vulnerable code. Previous efforts to determine the attack surface of a system have used API scanning techniques [41], but these techniques require a large amount of manual effort from practitioners. Automated attack surface approximation (ASA) [1] is an approach to identifying code in a software system that is contained on the attack surface through crash dump stack trace analysis. Depending upon the granularity of the analysis, these units could be binaries, files, or methods. In ASA, all code found in the stack traces from crash dumps is classified as being on the attack surfaced of a system. By definition, code that appears in stack traces caused by outside activity is on the attack surface. Crash dumps may be a useful metric for determining where security vulnerabilities might be in code, because they represent code that has been involved in a failure. Crash dumps are sometimes used by attackers to determine where security vulnerabilities might be and/or to create a denial of service. Using crash dump stack traces as a metric for proactive security hardening efforts may help security professionals find them before the attackers do. The goal of this research is to help software development teams prioritize security efforts by approximating the attack surface of a software system via stack trace analysis. In this work, we look at two methods of approximating the attack surface of a system. The first is a binary classification (on/off) as performed in previous work [1]. The second counts the number of occurrences of code in stack traces, allowing for a priority system for performing security reviews. Furthermore, we also developed a Vulnerability Prediction Model (VPM) to guide practitioners in these prioritization efforts. Finally, we generate a decision tree based on code metrics and stack trace appearances from ASA to help guide prioritization efforts when deciding what code to review. We explore the following research questions: RQ1: How effective is the attack surface approximation approach in predicting the location of vulnerabilities? https://www.owasp. org/index.php? title=Attack_Surface_Analysis_ Cheat_Sheet&oldid=156006 RQ2 How can we improve Vulnerability Prediction Model performance with attack surface approximation? RQ3: How could a decision tree with attack surface approximation be used to prioritize security efforts? In this paper, we build an ASA for Firefox, based on stack traces from the Firefox historical crash dump stack trace defect repository. A set of scripts parses the information in the traces. To assess ASA, we compared the set of known security vulnerabilities from Mozilla’s bug database during the same time period against the files identified as part of the approximation. We replicate the Windows 8 ASA [1] and apply it to Mozilla Firefox. We include the following as contributions in this paper: • An evaluation of the effectiveness of attack surface approximation for an open source application that corroborates with an earlier attack surface approximation study on a proprietary product. • A vulnerability prediction model that uses the attack surface approximation • A decision tree approach for aiding in the selection of which files to inspect and/or test. The rest of the paper is organized as follows: Section 2 discusses background and related work, Section 3 presents our research methodology, Section 4 presents our case study, Section 5 discusses the results and the meaning of each, Section 6 presents limitations and threats to validity, and Section 7 summarizes and discusses future work. 2. BACKGROUND AND RELATED WORK In this section, we provide a brief overview of related work and the previous study done in the area of attack surfaces and defect prediction. 2.1 Attack Surface As mentioned previously, The Open Web Application Security Project (OWASP) defines the attack surface of a system as the paths into and out of a system, the data that travels those paths, and the code that protects the paths and the data. The attack surface definition also includes “the sum of all paths for data/commands into and out of the application. ” Howard et al. [17] provided a definition of attack surface using three dimensions: 1) targets and enablers, 2) channels and protocols, and 3) access rights. Not all areas of a system may be directly or indirectly exposed to the outside. Some parts of a complex software system, e.g. Windows OS, may not be reachable or exploitable by an attacker. In Figure 1, we present a graphical representation of what the attack surface of a system is. The nodes with the thick dark arrows pointing at them are the entry points into a system, showing where an outsider can pass input into a system. The remaining shaded nodes and arrows represent the path outside input takes through the system, with data eventually terminating in the center of the system. These central nodes are typically crash handlers in our analysis due to inspecting stack traces. Manadhata et al. [32] describe how an attack surface might be approximated by looking at Application Program Interface (API) 2 https://crash-analysis.mozilla. com/crash_analysis 3 https://bugzilla.mozilla. org/ entry points. However, the Manadhata approach does not cover all exposed code, as the authors mention. Specifically, internal flow of data through a system could not be identified. While the external points of a system are a useful place to start, they do not encompass the entirety of exposed code in the system. Internal points within the system could also contain security vulnerabilities that the reviewer should be aware of. Further, their approach to measuring attack surfaces required expert judgment of security professionals to determine if code is security relevant. In a previous ASA study [1], researchers found a correlation between binaries that appear on stack traces generated by the system and historical vulnerabilities discovered by security professionals have been fixed in code. The correlation could be useful to security professionals when targeting security reviews of codebases. By targeting security efforts to binaries in the ASA instead of the entire codebase, security professionals could save engineering hours. The researchers created the ASA by parsing stack traces from Windows 8 OS, and including any binaries involved in a stack trace in their approximation. They evaluated the effectiveness of their approach by comparing the approximation against the location of historical vulnerabilities in Windows 8 OS. In that study, 48.4% of shipped binaries seen in at least one crash dump stack trace in Windows 8 OS contained 94.8% of the vulnerabilities seen over the same time period [1]. However, the industrial study has a few limitations. First, the approximation was only performed at the binary level of the application. A single binary could contain thousands of files, making the metric difficult to act on. Second, The industrial study only looked at an operating system. To address these concerns, we construct an ASA using stack traces at the file level instead of the binary level. We also perform the approximation on an open source application instead of an operating system. Figure 1. A visual representation of what an attack surface is for a system. The shaded area is the attack surface, where input flows through the sy