Software Source Code Testing

Submier: Eric Brewer, on behalf of Google, LLC

Topic: (4) Initial minimum requirements for testing soware source code

Speakers: Oliver Chang, Sta Soware Engineer, ochang@google.com

Jonathan Metzman, Senior Soware Engineer, metzman@google.com

Continuous fuzzing

Fuzzing is the process of feeding automatically generated data to a soware program to nd

unexpected bugs, and it has evolved from a tool used in one-o eos by security researchers into an

integral pa of the soware development process. Since soware is developed continuously and

changes over time, fuzzing must also be continuous and work seamlessly alongside traditional soware

testing practices such as unit or integration testing.

By incorporating continuous fuzzing in development workows, many security issues can be caught

early in the soware development lifecycle before any production releases are made. Pre-release

fuzzing is especially impoant because fuzzing is a method also employed by malicious paies; early

fuzzing by soware developers helps identify vulnerabilities before they are found by others.

Google does fuzzing at scale to great success for almost all soware projects. For instance, continuous

fuzzing accounts for almost half of all security vulnerabilities repoed in the Chromium1 browser.

Beyond Google, services such as OSS-Fuzz2 also provide continuous fuzzing to the maintainers of over

500 critical open source projects. To date, OSS-Fuzz has resulted in over 5,000 xed vulnerabilities

across projects such as OpenSSL and systemd.

1 Chromium Browser - hps://www.chromium.org 2 OSS-Fuzz - continuous fuzzing for open source soware - hps://github.com/google/oss-fuzz
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The continuous fuzzing workow is demonstrated in the diagram above. A soware developer writes a

small fuzzing harness, akin to a traditional unit test. Then, automation will manage the rest of the

process end-to-end. This involves continuously building the latest source, to nding crashes via fuzzing,

to deduplication and issue ling. Then, aer testing shows that the bug has been xed, the issue is

closed automatically, and the cycle repeats itself as the soware is fuher developed. The result is that

fuzzing becomes a natural pa of all developer workows.

Throughout this process, aifacts are generated that allow evaluation of the overall success. For

example, coverage repos provide a quantitative measure of how much of the soware is tested.

Fuzzing also continuously builds a corpus of inputs and regression tests that provide another indication

of the vulnerabilities that were found and resolved. Turnaround time for bug xes is another useful

measure.

Securing dependencies

Simply testing a soware’s primary codebase is not enough, as most soware relies on a large number

of third-pay libraries. In addition to fuzzing the codebase, we advocate automatically checking these

dependencies for adherence to security best practices and scanning them for known vulnerabilities.

The OpenSSF3 foundation created the Scorecard4 tool to provide security health metrics for any open

source library. This tool can be used to generate an automated repo of the security posture of a

project and all dependencies that it uses.

For vulnerability scanning and dependency updates, tools such as depandabot and renovatebot provide

varying degrees of automation. However, these tools must be accompanied by good automated test

coverage to ensure that automated updates can be done reliably without breaking the primary use case.

More development is needed in this space. One key problem today is that matching vulnerabilities to a

soware’s dependencies is dicult to do in an automated way. Many existing standards do not track

package names and precise version ranges in a way that is consistent with what’s actually used, leading

to missed vulnerabilities. There are also many vulnerability databases with dierent format standards,

requiring dierent tooling and parsers for each. Google has staed the OSV5 eo in collaboration with

open source communities to improve automated vulnerability matching and sharing, including a schema

format to enable interchange6

.

Summary

To conclude, continuous fuzzing is a very eective way of preventing vulnerabilities from being

introduced, and should be integrated into the development process of all soware projects. In addition

to this, all dependencies of a soware project should be regularly checked by automated tools to make

sure they follow good security practices and are free of known vulnerabilities.

3 OpenSSF - open source security foundation - hps://openssf.org/ 4 Scorecard - security health metrics for open source - hps://github.com/ossf/scorecard 5 OSV - vulnerability database and triage service for open source - hps://github.com/google/osv 6 Proposed vulnerability interchange format for triage automation - hps://tinyurl.com/vuln-json