Project: Envoy (Orchestration & Management / Service Proxy)
File: audit_fuzzer_adalogics_2021.pdf

Envoy fuzzing improvements
,
April 2021
Envoy fuzzing improvements
presented by
in collaboration with
A u t h o r s
David Korczynski
<
david@adalogics.com
>
Adam Korczynski
<
adam@adalogics.com
>
Date: 29th april, 2021
This report is licensed under Creative Commons 4.0
(CC BY  4.0)
Ada Logics
London, United Kingdom
1
Envoy fuzzing improvements
,
April 2021
Table of Contents
Executive summary
3
1 Envoy fuzzing optimisations
4
1.1 Background on how coverage collection in libFuzzer
works.
4
1.1 is it possible to disable certain 8-bit counters?
6
1.2 How does coverage instrumentation impact fuzzer
execution speed in Envoy?
7
1.2.1 Execution speed of an empty fuzzer.
7
1.2.2 Using Prodfiler to observe the impact.
9
1.2.3 Using fuzzer instrumentation to assess the overall
time spent with fuzzing.
10
1.2.4 Counting the number of 8-bit inline counters
in each Envoy fuzzer.
11
1.2.5 Conclusions
12
1.3 Improving Envoy fuzzer execution by reducing instrumentation
13
1.3.1 Disabling instrumentation
13
1.3.2 Results from disabling instrumentation
14
Measuring different configurations
14
Deploying on OSS-Fuzz.
15
2 Fuzzing the Envoy UdpListener
16
3 Future advice
18
4 Conclusions
19
5 Appendix
20
A.0 Disassembly of empty fuzzers
20
A.1 Build configurations
22
A.2 Envoy UDP fuzzer coverage
26
Ada Logics
London, United Kingdom
2
Envoy fuzzing improvements
,
April 2021
Executive summary
In this engagement we performed two tasks. The first
was optimising the performance of
end-to-end fuzzers and the second was developing a
fuzzer for the UDP listener code of
Envoy.
Regarding optimisation, we find that the cause of
performance issues in the end-to-end
fuzzers is due to the large amount of code that is
instrumented for coverage-guiding the
Envoy fuzzers. Specifically, the Envoy fuzzers run
with a large amount of inline 8-bit
counters which is a counter inserted by
SanitizerCoverage
on every edge of the target
application. libFuzzer uses the counters by iterating
through all of the counters after each
fuzz iteration in order to measure code coverage.
Thus, the more counters there are, the
more effort libFuzzer has to spend iterating through
all counters on each fuzz iteration. We
show how this impacts Envoy performance in four different
ways:
1.
The exact same empty fuzzer compiled with fuzzer instrumentation
runs 419 times
slower when also compiled with Envoy instrumentation.
2.
The h2_capture_persistent_fuzz_test fuzzer spends
only 20% of the actual execution
inside LLVMFuzzerTestOneInput.
3.
Profilers reveal an excessive amount of time is spent
inside coverage-collection
code.
4.
The end-to-end fuzzers contain around 1.2 million
inline 8-bit counters, which is huge
in comparison to other fuzzers, e.g. Lua fuzzers contain
6000 inline 8-bit counters.
We show how limiting the coverage instrumentation
improves fuzzer performance and
observe the same effect on OSS-Fuzz statistics.
Regarding UDP fuzzing, we develop a UDP fuzzer that
targets the code requested by the
Envoy team, namely handleReadCallback. We demonstrate
code coverage by observing the
target code in OSS-Fuzz reports, and also demonstrates
that the fuzzer catches the DOS
issue previously reported.
In this report we go through both of these tasks and
spend considerable detail on the
coverage instrumentation part.
Ada Logics
London, United Kingdom
3
Envoy fuzzing improvements
,
April 2021
1 Envoy fuzzing optimisations
The goal of this task was to improve performance of
the end-to-end fuzzers of Envoy. In this
section we go through how we achieved this by first giving background information on how
libFuzzer uses coverage instrumentation, then we proceed
to show how this instrumentation
affects the Envoy fuzzers, and finally we show how
to improve the performance of the
fuzzers by reducing instrumentation.
Executive summary:
The instrumentation of Envoy has a significant slowdown
of fuzzer execution. For example, a
fuzzer with a single branch runs 419 times slower
with Envoy instrumentation than without.
Limiting the amount of instrumentation used in the
Envoy project shows to increase
performance of the fuzzers. This has been added to
the OSS-Fuzz build of Envoy, which has
shown to increase performance on the OSS-Fuzz logs.
There is future work for the Envoy
maintainers on refining the instrumentation of the
fuzzers.
1.1 Background on how coverage collection in libFuzzer
works.
In this section we outline how libFuzzer uses coverage
instrumentation to track fuzzer
progress. This is not related to Envoy explicitly,
however, the content described in this
section gives an improved understanding as to why
the Envoy fuzzers lack performance.
In the LLVM source code, the file llvm-project/compiler-rt/lib/fuzzer/FuzzerTracePC.cpp
contains the logic for tracing coverage. At the bottom
of this file you find a set of functions:
void__sanitizer_cov_trace_pc_guard(uint32_t*Guard)void__sanitizer_cov_trace_pc()void__sanitizer_cov_trace_pc_guard_init(uint32_t*Start,uint32_t*Stop)void__sanitizer_cov_8bit_counters_init(uint8_t*Start,uint8_t*Stop)void__sanitizer_cov_pcs_init(constuintptr_t*pcs_beg,constuintptr_t*pcs_end)void__sanitizer_cov_trace_pc_indir(uintptr_tCallee)void__sanitizer_cov_trace_cmp8(uint64_tArg1,uint64_tArg2)void__sanitizer_cov_trace_const_cmp8(uint64_tArg1,uint64_tArg2)void__sanitizer_cov_trace_cmp4(uint32_tArg1,uint32_tArg2)void__sanitizer_cov_trace_const_cmp4(uint32_tArg1,uint32_tArg2)void__sanitizer_cov_trace_cmp2(uint16_tArg1,uint16_tArg2)void__sanitizer_cov_trace_const_cmp2(uint16_tArg1,uint16_tArg2)void__sanitizer_cov_trace_cmp1(uint8_tArg1,uint8_tArg2)void__sanitizer_cov_trace_const_cmp1(uint8_tArg1,uint8_tArg2)void__sanitizer_cov_trace_switch(uint64_tVal,uint64_t*Cases)void__sanitizer_cov_trace_div4(uint32_tVal)void__sanitizer_cov_trace_div8(uint64_tVal)void__sanitizer_cov_trace_gep(uintptr_tIdx)
These are all callbacks that SanitizerCoverage uses
(
https://clang.llvm.org/docs/SanitizerCoverage.html#id2
)
and from a high level perspective
the callbacks are used to track execution of code,
e.g. basic blocks, in the target. One of the
Ada Logics
London, United Kingdom
4
Envoy fuzzing improvements
,
April 2021
important callback functions is__sanitizer_cov_8bit_counters_init
which is described
here
https://clang.llvm.org/docs/SanitizerCoverage.html#inline-8bit-counters
.
This callback is
used during fuzzing compilation and instructs the
compiler to insert inline counter increments
on every code edge. The idea is then that these counters
can be used to track how many
times an edge was hit during execution.
The way libFuzzer uses these 8bit-counters is by iterating
through all of them, i.e. iteration
through a sequence of counters corresponding to the
number of edges in the target
application, after each fuzz iteration in order to
determine if new coverage has been
achieved.
In theRunOne
function inside the FuzzerLoop.cpp
(
https://github.com/llvm/llvm-project/blob/28ab7ff2d732fb0580486baa02b1383a72cec0cb/co
mpiler-rt/lib/fuzzer/FuzzerLoop.cpp#L506
) source we
observe the following code:
ExecuteCallback(Data, Size);autoTimeOfUnit = duration_cast<microseconds>(UnitStopTime- UnitStartTime);
UniqFeatureSetTmp.clear();size_tFoundUniqFeaturesOfII =0;size_tNumUpdatesBefore = Corpus.NumFeatureUpdates();TPC.CollectFeatures([&](uint32_tFeature) {if(Corpus.AddFeature(Feature,static_cast<uint32_t>(Size),Options.Shrink))UniqFeatureSetTmp.push_back(Feature);if(Options.Entropic)Corpus.UpdateFeatureFrequency(II, Feature);if(Options.ReduceInputs && II && !II->NeverReduce)if(std::binary_search(II->UniqFeatureSet.begin(),II->UniqFeatureSet.end(),Feature))FoundUniqFeaturesOfII++;});
This code snippet has two purposes. First, to runExecuteCallback
which calls into the
fuzzing entry point function (LLVMFuzzerTestOneInput
).
Second,
to gather the code coverage of the target follow the
fuzzer’s single execution, which
is done byTPC.CollectFeatures
. TheTPC.CollectFeatures
code begins as follows:
template<classCallback> //voidCallback(uint32_tFeature)ATTRIBUTE_NO_SANITIZE_ADDRESSATTRIBUTE_NOINLINEsize_tTracePC::CollectFeatures(Callback HandleFeature)const{autoHandle8bitCounter = [&](size_tFirstFeature,size_tIdx,uint8_tCounter) {if(UseCounters)HandleFeature(static_cast<uint32_t>(FirstFeature+ Idx *8+CounterToFeature(Counter)));elseHandleFeature(static_cast<uint32_t>(FirstFeature+ Idx));};
Ada Logics
London, United Kingdom
5
Envoy fuzzing improvements
,
April 2021
size_tFirstFeature =0;
for(size_ti =0; i < NumModules; i++) {for(size_tr =0; r < Modules[i].NumRegions;r++) {if(!Modules[i].Regions[r].Enabled)continue;FirstFeature +=8* ForEachNonZeroByte(Modules[i].Regions[r].Start,Modules[i].Regions[r].Stop,FirstFeature,Handle8bitCounter);}}
The nested for loop calls intoForEachNonZeroByte
with two pointers as the first
two arguments. These arguments point to the areas
in the code where a page of
inline 8-bit counters exist, namely, each region in
theModules
vector encapsulates
a page of 8bit counters. The implementation ofForEachNonZeroByte
is defined here
https://github.com/llvm/llvm-project/blob/e60d6e91e196d91a1b9bfcc93d9f43946ea29299/co
mpiler-rt/lib/fuzzer/FuzzerTracePC.h#L184
and essentially
the function will call the callback
(Handle8bitCounter
) for each nonzero byte in the memory
region. In the case of theRunOne
function the result is to call into the function
provided by the call toTPC.CollectFeatures
, which eventually calls  into
various functions related to the corpus,
e.g.Corpus.AddFeature
andCorpus.UpdateFeatureFrequency
which will trigger
changes in the Corpus representation if the features
indicate that a new piece of code has
been executed.
The main point to get across in the above section
is that nested for-loop inCollectFeatures
iterates through the regions with
inline 8-bit counters, and what we will
observe in the Envoy fuzzers is that the number of
inline 8-bit counters is significantly large.
1.1 is it possible to disable certain 8-bit counters?
A question that is relevant about how libFuzzer uses
the instrumentation is whether you can
tell libFuzzer to disable checking of inline counters.
Indeed, in theTPC.CollectFeatures
function the loop that is extensive (shown above),
namely:
for(size_ti =0; i < NumModules; i++) {for(size_tr =0; r < Modules[i].NumRegions;r++) {if(!Modules[i].Regions[r].Enabled)continue;FirstFeature +=8* ForEachNonZeroByte(Modules[i].Regions[r].Start,Modules[i].Regions[r].Stop,FirstFeature,Handle8bitCounter);}}
only iterates through the inline 8-bit counters if
a given region is “Enabled
”. Unfortunately,
theEnabled
boolean will always be true for all regions
as all regions are initialised to true
Ada Logics
London, United Kingdom
6
Envoy fuzzing improvements
,
April 2021
(
https://github.com/llvm/llvm-project/blob/ab5823867c4aee7f3e02ddfaa217905c87471bf9/co
mpiler-rt/lib/fuzzer/FuzzerTracePC.cpp#L59
) and never
set to false.
1.2 How does coverage instrumentation impact fuzzer
execution speed in Envoy?
In this section we show three different ways of observing
the impact of the current
instrumentation set up in Envoy.
1.2.1 Execution speed of an empty fuzzer.
We assess the slowdown of the instrumentation by creating
the simplest fuzzer we can. We
do this by having an empty fuzzer and compiling it
in two different ways. First, we compile it
without linking it to any Envoy code. Second, we compile
it in the exact same way ash2_capture_fuzz
namely by linking it to the exact
same code in the Envoy code base,
which corresponds to essentially all of the Envoy
code. We define the new fuzzer as follows:
#include "test/integration/h2_fuzz.h"DEFINE_FUZZER(constuint8_t* buf,size_tlen) {if(len ==1234123412) {std::cout<<"hello "<< buf <<"\n";}}
and enable it by extending test/integration/BUILD
with the following
envoy_cc_fuzz_test(name ="h2_empty_fuzz_test",srcs = ["h2_empty_fuzz_test.cc"],copts = ["-DPERSISTENT_FUZZER"],corpus ="h2_corpus",deps = [":h2_fuzz_persistent_lib"],)
We then compile and run the fuzzer, and observe the
following output:
$ h2_empty_fuzz_testINFO: Running with entropic power schedule (0xFF,100).INFO: Seed: 2391912932INFO: Loaded 1 modules   (1282530 inline 8-bit counters):1282530 [0xa51f110,0xa6582f2),INFO: Loaded 1 PC tables (1282530 PCs): 1282530 [0xa6582f8,0xb9ea118),INFO: -max_len is not provided; libFuzzer will notgenerate inputs larger than 4096bytesINFO: A corpus is not provided, starting from an emptycorpus#2      INITED cov: 3 ft: 4 corp: 1/1b exec/s: 0 rss:170Mb
Ada Logics
London, United Kingdom
7
Envoy fuzzing improvements
,
April 2021
#8192   pulse  cov: 3 ft: 4 corp: 1/1b lim: 80 exec/s: 4096 rss: 170Mb#16384  pulse  cov: 3 ft: 4 corp: 1/1b lim: 163 exec/s:4096 rss: 171Mb#32768  pulse  cov: 3 ft: 4 corp: 1/1b lim: 325 exec/s:4096 rss: 172Mb#65536  pulse  cov: 3 ft: 4 corp: 1/1b lim: 652 exec/s:3855 rss: 175Mb
If we create the same empty fuzzer but outside the
envoy build environment but following the
Envoy fuzzer set up
https://github.com/envoyproxy/envoy/blob/1d1b708c7bf6efa02c41d9ce22cbf1e4a1aeec2c/te
st/fuzz/fuzz_runner.h#L71
:
#include <iostream>
#define DEFINE_TEST_ONE_INPUT_IMPL\extern "C" int LLVMFuzzerTestOneInput(const uint8_t*data, size_t size) {   \EnvoyTestOneInput(data, size);\return 0;\}
#define DEFINE_FUZZER\static void EnvoyTestOneInput(const uint8_t* buf,size_t len);              \DEFINE_TEST_ONE_INPUT_IMPL\static void EnvoyTestOneInput
DEFINE_FUZZER(const uint8_t* buf, size_t len) {if (len ==1234123412) {std::cout<<"hello "<< buf <<"\n";}}
and run the fuzzer, then we observe the output:
$ $CXX $CXXFLAGS $LIB_FUZZING_ENGINE ./empty_fuzzer.cc$ ./a.outINFO: Running with entropic power schedule (0xFF,100).INFO: Seed: 1703773772INFO: Loaded 1 modules   (126 inline 8-bit counters):126 [0x84c0f8, 0x84c176),INFO: Loaded 1 PC tables (126 PCs): 126 [0x5e6398,0x5e6b78),INFO: -max_len is not provided; libFuzzer will notgenerate inputs larger than 4096bytesINFO: A corpus is not provided, starting from an emptycorpus#2      INITED cov: 3 ft: 4 corp: 1/1b exec/s: 0 rss:29Mb#4194304        pulse  cov: 3 ft: 4 corp: 1/1b lim:4096 exec/s: 2097152 rss: 343Mb#8388608        pulse  cov: 3 ft: 4 corp: 1/1b lim:4096 exec/s: 1677721 rss: 611Mb#16777216       pulse  cov: 3 ft: 4 corp: 1/1b lim:4096 exec/s: 1677721 rss: 614Mb
We verify the disassembly ofLLVMFuzzerTestOneInput
is similar in both of the fuzzers (see
Appendix A.0), specifically that they each execute
the same amount of code in theLLVMFuzzerTestOneInput
. We also note here that the
coverage achieved by both fuzzers
is equal (namely 3) as shown by the output of libFuzzer.
Ada Logics
London, United Kingdom
8
Envoy fuzzing improvements
,
April 2021
The execution speed and the number of inline 8-bit
counters are very different in the two
cases. Specifically, we extract two important conclusions
from the above observations
1.
The fuzzer that is compiled outside of the Envoy fuzzer
environment executes with
roughly 1.67 million executions per second, whereas
the one in the Envoy
environment executes with roughly 4000 executions
per second. The envoy
instrumentation slows down the fuzzer by roughly 419x.
2.
The number of inline 8-bit counters in the Envoy fuzzer
is 1282530 whereas it is 126
in the empty fuzzer (these numbers can be observed
by the output from LibFuzzer
given above). This corresponds to iterating through
126 bytes versus 1282530 bytes
after each fuzz iteration.
1.2.2 Using Prodfiler to observe the impact.
Another way we tried to assess the execution speed
of the fuzzers was by using profiling
tools to tell us where execution is spent. To do this,
we relied on Prodfiler
https://github.com/optimyze/prodfiler-documentation
to profile the entire machine in which we
ran the fuzzer. Prodfiler reported that 25.8% of all
function samples observed happened
inside theForEachNonZeroByte
function
.
In addition
to this, looking at the flame graphs
provided by Prodfiler we see that around 19% of the
entire machine execution was spent
inside of theCollectFeatures
function.
Ada Logics
London, United Kingdom
9
Envoy fuzzing improvements
,
April 2021
1.2.3 Using fuzzer instrumentation to assess the overall
time spent with
fuzzing.
Another thing we were interested in understanding
was how much of the fuzzing execution
was actually spent in the target and how much was
spent in the fuzzing engine itself. To do
this, we added some simple instrumentation to the
h2_capture_fuzz.cc and h2_fuzz.cc files,
such that the h2_capture_fuzz fuzzer will log timestamps
as follows:
DEFINE_PROTO_FUZZER(..H2CaptureFuzzTestCase input)ADD_TIME_STAMP(F1)H2FuzzIntegrationTest h2_var;h2_var.replay(input)--> H2FuzzIntegrationTest::replay(... H2CaptureFuzzTestCase&input,)...ADD_TIME_STAMP(R4)IntegrationTcpClientPtr tcp_client = makeTcpConnection(lookupPort("http"));...ADD_TIME_STAMP(R5)for (int i =0; i < input.events().size();++i) {if (stop_further_inputs) {break;}INC_GLOBAL(PACKET_COUNT)// send packet}// endloopADD_TIME_STAMP(R6)...<--ADD_TIME_STAMP(F2)<--
With these timestamps available we set up an experiment
to measure the following metrics:
Ada Logics
London, United Kingdom
10
Envoy fuzzing improvements
,
April 2021
-
Total time spent fuzzing
-
Total time spent in code between the following timestamps:
-
F1-F2: This is the total time spent in the fuzzer
code.
-
R4-R6: This is the total time spent in the replay
function.
-
R5-R6: This is the total time spent in the loop sending
fuzz inputs to the client
-
Number of total fuzz inputs to the client
-
Various proportionate metrics, e.g. inputs sent to
the client.
We executed
h2_capture_persistent_fuzz
with the seeds
provided, and observed the
following results:
Metric
Result
Total execution time
1200 sec
F1-F2
244 sec
R4-R6
239 sec
R5-R6
226 sec
F1-F2/Total time
20%
R4-R6/Total time
19.9%
R5-R6/Total time
18.8%
Total number of client fuzz inputs
82797
Total fuzz inputs to client / (F1-F2)
339 inputs/sec
Total fuzz inputs to client / (R4-R6)
346 inputs/sec
Total fuzz inputs to client / (R5-R6)
366 inputs/sec
Total number of fuzz iterations
27761
We observe that only 20% of the fuzz time is spent
in the actual fuzzing code, and thus 80%
of the time is spent in the libFuzzer engine.
1.2.4 Counting the number of 8-bit inline counters
in each Envoy fuzzer.
Finally, we wanted to understand how many 8-bit inline
counters are actually used by the
Envoy fuzzers. This number is reported by the fuzzers
when they are initiated, so the only
thing we have to do in this case is build the fuzzers,
launch them and then count the number
of inline 8-bit counters. We do this using a build
with AddressSanitizer, and we see the
following results:
Fuzzer name
inline 8-bit counters
h2_capture_persistent_fuzz_test
1281655
h2_capture_fuzz_test
1281649
Ada Logics
London, United Kingdom
11
Envoy fuzzing improvements
,
April 2021
h1_capture_fuzz_test
1268160
h1_capture_persistent_fuzz_test
1268166
codec_fuzz_test
1030067
hash_fuzz_test
570594
get_sha_256_digest_fuzz_test
579995
evaluator_fuzz_test
1100639
dns_filter_fuzz_test
1139928
For comparison purposes the following table gives
examples of 8-bit counters in various
open source projects that we have fuzzed.
Project fuzzer targets
Inline 8-bit counters
Lua
6322
PugiXML
4871
LevelDB
5948
RocksDB
137913
It is clear that the number of inline 8-bit counters
in Envoy is significantly larger than any of
the other projects. For example, our fuzzer targeted
the core library (liblua.a) for the Lua
programming language has a total of 6322 inline 8-bit
counters whereas the h2 end-to-end
Envoy fuzzer has a total of 1.2 million.
1.2.5 Conclusions
The instrumentation set up in Envoy is a significant
performance bottleneck for the fuzzers.
The main reason for the performance is the post-processing
that occurs after each fuzz
execution in terms of collecting coverage features
to assess whether the execution
discovered new code execution. An empty fuzzer without
Envoy instrumentation runs 419x
faster than an empty fuzzer with Envoy instrumentation.
We want to emphasize here that the fuzzing infrastructure
of Envoy is of high quality and the
team behind it has shown high competencies in fuzzing
Envoy. In fact, from a certain
perspective it is positive that Envoy builds all code
related to Envoy with sanitizers and this
should be considered a great achievement as well as
having the right intentions. It is due to
the sheer amount of code in the resulting Envoy binaries,
which is due to linking of many
third-party libraries in addition to the large Envoy
code base in and of itself, that the
coverage collection ends up having a significant slowdown.
In the vast majority of projects
we would not advise limiting the coverage instrumentation
across a fuzzing set up.
Ada Logics
London, United Kingdom
12
Envoy fuzzing improvements
,
April 2021
Furthermore, the observations we made in this section are not common knowledge and it is
not to be expected from non-fuzzing experts to consider
this during fuzzer development.
1.3 Improving Envoy fuzzer execution by reducing
instrumentation
1.3.1 Disabling instrumentation
The crucial task for improving fuzzing performance
in Envoy is to reduce the amount of
coverage instrumentation in the target application.
Specifically, the goal is to reduce the
amount of inline 8-bit counters. The main way to this
is by including the compilation flags:
-fsanitize-coverage=0-fno-sanitize=all
The first flag-fsanitize-coverage=0
disablescoverage
instrumentation, whereas the
second flag also disables bug-finding sanitizers,
e.g. AddressSanitizer. The first flag should
be removed from code that can avoid being explored
or are potentially fuzzed elsewhere.
The second flag, namely disabling bug-finding sanitizers,
should be disabled with more care.
In order to disable instrumentation, we tried three
things:
Bazel builds:
First, we tried controlling instrumentation
details inside of the Envoy BUILD
scripts. In particular, adding `copts` definitions
to various places in the BUILD scripts of
ENVOY. This works for the majority of bazel rules,
however, it does not work for any rule
defined by way ofproto_library
. This is becauseproto_library
does not allow us to
propagate compilation options to the protobuf compiler,
and thus it does not allow us to
include the flags to disable instrumentation on a
per-target basis. We reference to issues in
the bazelbuild/rules_proto repository here:
https://github.com/bazelbuild/rules_proto/issues/85
https://github.com/bazelbuild/rules_proto/issues/41
Passing flags to clang:
Second, we tried using the
`--per_file_copt
` command line
option for bazel builds. This option accepts a regular
expression that encapsulates a set of
files used during compilation as well as a string
that will be passed to the compiler during
compilation of files that match the regular expression.
For example, the following line:
--per_file_copt=^.*\.pb\.cc$@-fsanitize-coverage=0,-fno-sanitize=all
will pass the flags-fsanitizer-coverage=0
and-fno-sanitize=all
to all files compiled
in the bazel build with.pb.cc
extensions. This option
worked well.
Ada Logics
London, United Kingdom
13
Envoy fuzzing improvements
,
April 2021
Using a blocklist for SanitizerCoverage
: Third, using the-fsanitize-coverage-blocklist=/src/block_list.txt
also by way ofper_file_copt
. This option disables coverage instrumentation
based on contents of the file/src/block_list.txt
in which it is possible to disable
instrumentation based on file names
or (mangled) function names.
1.3.2 Results from disabling instrumentation
Measuring dif ferent configurations
The next was to identify which parts of the code yields
best results when disabling coverage
instrumentation. To do this, we set up a set of different
build configurations following the
approaches (2 and 3) described above and measured
the following data:
1.
The number of 8-bit inline counters in the h1 e2e
and h2 2e2 fuzzers;
2.
The total number of fuzz iterations achieved over
a 5 minute run with the seeds from
the OSS-Fuzz repository.
In total, we compiled the Envoy fuzzers with 6 different
configurations and the configurations
are shown in Appendix A.1. The following table shows
the results.
Fuzzer
Setting
inline 8-bit 
counters
Total execs 
(300 sec)
Exec/sec
http2 e2e
Configuration 1 
(regular build)
1281922
4527
15.1
http1 e2e
Configuration 1 
(regular build)
1268433
7123
23.7
http2 e2e
Configuration 2
271239
7372
24.6
http1 e2e
Configuration 2
272107
25370
84.6
http2 e2e
Configuration 3
313473
6864
22.9
http1 e2e
Configuration 3
314363
17640
58.9
http2 e2e
Configuration 4
313545
5291
17.6
http1 e2e
Configuration 4
314438
16530
55.1
http2 e2e
Configuration 5
258955
6522
21.7
http1 e2e
Configuration 5
259216
20639
68.8
http2 e2e
Configuration 6 
(includes 
blocked list of 
functions)
273673
8055
26.85
http1 e2e
Configuration 6
274564
19932
66.4
Ada Logics
London, United Kingdom
14
Envoy fuzzing improvements
,
April 2021
(includes 
blocked list of 
functions)
Finally, we note that the executions per second increases
if the fuzzer is run for a longer
period of time. For example, in our experiments we
have observed above 100 exec/sec in
the http2 end-to-end fuzzer with seeds in longer fuzz
runs. The data of these runs have been
shared with the Envoy team internally.
Deploying on OSS-Fuzz.
We integrated some of the above efforts into the OSS-Fuzz
build script of Envoy and the
following figure shows impact.
We can see here that a performance improvement happened
on the 16th April 2021.
Inspecting the logs, we notice on the 15th April the
amount of inline 8-bit counters was
~600,000
(
https://console.cloud.google.com/storage/browser/_details/envoy-logs.clusterfuzz-external.a
ppspot.com/libFuzzer_envoy_h2_capture_persistent_fuzz_test/libfuzzer_asan_envoy/2021-
04-15/00:06:53:330737.log
) whereas on the 16th April
it was ~300,000
(
https://console.cloud.google.com/storage/browser/_details/envoy-logs.clusterfuzz-external.a
ppspot.com/libFuzzer_envoy_h2_capture_persistent_fuzz_test/libfuzzer_asan_envoy/2021-
04-16/01:54:36:943426.log
). This decrease in inline
8-bit counters resulted in a 3x
improvement of the fuzzer execution. We note here
that in addition to the decrease in inline
8-bit counters the stability of the fuzzer was also
improved on the same day through bug
fixing, which may have impacted the performance as
well.
Ada Logics
London, United Kingdom
15
Envoy fuzzing improvements
,
April 2021
2 Fuzzing the Envoy UdpListener
The second task of the Engagement was to create a
new fuzzer that targets the UDP
listener of Envoy. The goal of this fuzzer was to
enable catching the bug described in this
Github issue
https://github.com/envoyproxy/envoy/issues/14113
.
The fuzzer we created was
merged into Envoy and is available at this location
in the repository:
https://github.com/envoyproxy/envoy/blob/main/test/common/network/udp_fuzz.cc
The goal of the fuzzer is to send an arbitrary number
of arbitrary packets to the Envoy UDP
listener. To do this we rely on the primitives provided
byNetwork::Test::UdpSyncPeer
and
the core of the fuzzer is encapsulated by the following
loop
(
https://github.com/envoyproxy/envoy/blob/58a13570f3f4dea9bad8b8fa5e1221d7ed5056de/t
est/common/network/udp_fuzz.cc#L92
):
// Now do all of the fuzzingstaticconstintMaxPackets =15;total_packets_ = provider.ConsumeIntegralInRange<uint16_t>(1,MaxPackets);Network::Test::UdpSyncPeerclient_(ip_version_);for(uint16_ti =0; i < total_packets_; i++){std::stringpacket_ =provider.ConsumeBytesAsString(provider.ConsumeIntegralInRange<uint32_t>(1,3000));if(packet_.empty()) {packet_ ="EMPTY_PACKET";}client_.write(packet_, *send_to_addr_);}dispatcher_->run(Event::Dispatcher::RunType::Block);
The fuzzer supports communicating by way ofGRO
,MMSG
as well as regularrecvmsg
, which
is controlled by the following code
(
https://github.com/envoyproxy/envoy/blob/58a13570f3f4dea9bad8b8fa5e1221d7ed5056de/t
est/common/network/udp_fuzz.cc#L73
):
FuzzedDataProviderprovider(buf, len);uint16_tSocketType = provider.ConsumeIntegralInRange<uint16_t>(0,2);if(SocketType ==0) {config.mutable_prefer_gro()->set_value(true);ON_CALL(override_syscall_, supportsUdpGro()).WillByDefault(Return(true));}elseif(SocketType ==1) {ON_CALL(override_syscall_, supportsMmsg()).WillByDefault(Return(true));}else{ON_CALL(override_syscall_, supportsMmsg()).WillByDefault(Return(false));ON_CALL(override_syscall_, supportsUdpGro()).WillByDefault(Return(false));}
In order to verify the fuzzer finds the original crash
of the code we ran the fuzzer
Ada Logics
London, United Kingdom
16
Envoy fuzzing improvements
,
April 2021
against the Envoy codebase with the changes applied by the commit that fixed the
UDP bug (
https://github.com/envoyproxy/envoy/pull/14122/files
).
More specifically,
if we avoid the use of the condition
if(output.msg_[i].truncated_and_dropped_) {continue;}
in this commit
https://github.com/envoyproxy/envoy/pull/14122/files#diff-c9702469f313c70572e261f4af736
87873ab8247264eb47fd05300098067c5abR632
then we get
the following result:
$ /out/udp_fuzz ./seeds/INFO: Running with entropic power schedule (0xFF,100).INFO: Seed: 185475315INFO: Loaded 1 modules   (1087563 inline 8-bit counters):1087563 [0x861fcb0,0x87294fb),INFO: Loaded 1 PC tables (1087563 PCs): 1087563 [0x8729500,0x97c19b0),INFO:       55 files foundin./seeds/INFO: -max_len is not provided; libFuzzer will notgenerate inputs larger than 4096bytesINFO: seed corpus: files: 55 min: 1b max: 2251b total:4519b rss: 150MbAddressSanitizer:DEADLYSIGNAL===================================================================28367==ERROR: AddressSanitizer: SEGV on unknownaddress 0x000000000000 (pc0x000005872691 bp 0x7fff415ade70 sp 0x7fff415ad980T0)==28367==The signal is caused by a READ memory access.==28367==Hint: address points to the zero page.#0 0x5872691 in Envoy::Network::passPayloadToProcessor(unsignedlong,std::__1::unique_ptr<Envoy::Buffer::Instance,std::__1::default_delete<Envoy::Buffer::Instance>>,std::__1::shared_ptr<Envoy::Network::Address::Instanceconst>,std::__1::shared_ptr<Envoy::Network::Address::Instanceconst>,Envoy::Network::UdpPacketProcessor&,std::__1::chrono::time_point<std::__1::chrono::steady_clock,std::__1::chrono::duration<long long, std::__1::ratio<1l,1000000000l> > >)/proc/self/cwd/source/common/network/utility.cc:562:3#1 0x587501e in Envoy::Network::Utility::readFromSocket(Envoy::Network::IoHandle&,Envoy::Network::Address::Instance const&, Envoy::Network::UdpPacketProcessor&,std::__1::chrono::time_point<std::__1::chrono::steady_clock,std::__1::chrono::duration<long long, std::__1::ratio<1l,1000000000l> > >, bool,unsigned int*) /proc/self/cwd/source/common/network/utility.cc:669:7#2 0x5877d3a inEnvoy::Network::Utility::readPacketsFromSocket(Envoy::Network::IoHandle&,Envoy::Network::Address::Instance const&, Envoy::Network::UdpPacketProcessor&,Envoy::TimeSource&, bool, unsigned int&)/proc/self/cwd/source/common/network/utility.cc:702:38#3 0x552b14d in Envoy::Network::UdpListenerImpl::handleReadCallback()/proc/self/cwd/source/common/network/udp_listener_impl.cc:75:34#4 0x552a15f in Envoy::Network::UdpListenerImpl::onSocketEvent(short)/proc/self/cwd/source/common/network/udp_listener_impl.cc:64:5
Ada Logics
London, United Kingdom
17
Envoy fuzzing improvements
,
April 2021
#5 0x5531e26 inEnvoy::Network::UdpListenerImpl::UdpListenerImpl(Envoy::Event::DispatcherImpl&,std::__1::shared_ptr<Envoy::Network::Socket>, Envoy::Network::UdpListenerCallbacks&,Envoy::TimeSource&, envoy::config::core::v3::UdpSocketConfigconst&)::$_0::operator()(unsigned int) const/proc/self/cwd/source/common/network/udp_listener_impl.cc:38:53
…
.
…
.
To confirm the fuzzer targets the code that was asked
for by the Envoy team we observe
coverage of the fuzzer in OSS-Fuzz. Consider the following
links showing the UDP Listener
code is being analysed:
●Envoy::Network::UdpListenerImpl::handleReadCallback
https://storage.googleapis.com/oss-fuzz-coverage/envoy/report
s/20210423/linux/proc/self/cwd/source/common/network/udp_list
ener_impl.cc.html#L72
●Envoy::Network::Utility::readFromSocket
https://storage.googleapis.com/oss-fuzz-coverage/envoy/reports/20210423/linux/proc
/self/cwd/source/common/network/utility.cc.html#L576
In addition to this, consider the figures in Appendix
A.2 showing the coverage is being
achieved.
3 Future advice
We consider there to be three problems that need solving
for the Envoy team.
First
, enable the ability to disable instrumentation
of protobuf code included in header files,
which in themselves can have a huge amount of code.
This is because a lot of protobuf code
is included by way of header files in the Envoy source
code:
envoy/source/common$ grep -rn"pb.h"./ | wc -l388
The unfortunate consequence of this is that the code
in the header file could avoid being
instrumented but because it is included by important
code that has to be instrumented, the
protobuf code consequently will also be instrumented.
Some refactoring that makes it easy
to control the instrumentation parameters of the protobuf
code would likely have a large
impact on the execution speed.
An option for solving the protobuf problems above
is to disable instrumentation on a
namespace or function level. We can do this by performing
partial instrumentation with
SanitizerCoverage
as described here:
https://clang.llvm.org/docs/SanitizerCoverage.html#partially-disabling-instrumentation
Ada Logics
London, United Kingdom
18
Envoy fuzzing improvements
,
April 2021
Specifically, consider the file
b l o c k _ l i s t . t x t
with the following contents:
fun:*nocoveragepls*
This will disable coverage instrumentation for any
function that containsnocoveragepls
in
its function name, which includes the namespace name.
As such, a way to disable
instrumentation in a smart manner is to include a
file
b l o c k e d _ l i s t . t x t
in the Envoy repository,
which we will then use during Envoy fuzzing compilation
as follows:--per_file_copt="-fsanitize-coverage-blocklist=envoy_code/block_list.txt"
We did an initial experimentation of this by way of
configuration #6 discussed in section
1.3.2.
Second
, add the ability to do per-target instrumentation.
It is clear that the fuzz targets in the
Envoy code target separate parts of the code base.
Ideally instrumentation should be made
on a target-specific basis. The goal should be to
have less than 100,000 inline 8-bit counters
per fuzzer. Notice that the number of counters is
shown in the libFuzzer logs (as shown
above section 1.2.1) which are also available on OSS-Fuzz
for each of the OSS-Fuzz runs.
We leave the following instrumentation-specific advice:
●
It can be beneficial to reduce coverage instrumentation
also from a perspective of
letting the fuzzer focus on code that matters, since
the fuzzer will be guided by
improvements in relevant code and not get “lost” in
irrelevant code.
●
Large chunks of sequential code does not need to be
instrumented with coverage,
including Envoy critical code. We advise to keep bug-finding
sanitizers on
nonetheless.
Third
, during the engagement we observed that several
of the fuzzers had a high crashing
percentage (above 90%) for several months. In fact,
the http2 end-to-end fuzzer had an
input in its corpus that caused the fuzzer to crash.
Effectively, the fuzzer had not explored
new code for months because this input blocked the
fuzzer from continuing. We think it is
crucial for the Envoy team to ensure the fuzzers of
Envoy are running properly and this
should be considered higher priority than ensuring
the fuzzers run fast.
4 Conclusions
In this engagement we improved Envoy fuzzing by identifying
the performance bottleneck in
the Envoy fuzzing set up and creating a new UDP fuzzer
that targets previously buggy code.
Our findings identify that the performance bottleneck
in Envoy is due to large amounts of
code instrumentation which causes LibFuzzer to spend
significant effort in determining if
each fuzz iteration caused new code to execute. We
show how this affects the Envoy
fuzzers, for example by highlighting an empty fuzzer
runs 419 times slower with Envoy’s
instrumentation approach. We show how to limit the
amount of instrumentation in Envoy and
how it improves performance.
Ada Logics
London, United Kingdom
19
Envoy fuzzing improvements
,
April 2021
5 Appendix
A.0 Disassembly of empty fuzzers
Appendix showing the disassembly of
L L V M F u z z e r T e s t O n e I n p u t
in the two empty fuzzers is
similar.
Disassembly of
h2_empty_fuzz_test
, namely the empty
fuzzer compiled with Envoy
instrumentation.
$ gdb /out/h2_empty_fuzz_testReading symbols from /out/h2_empty_fuzz_test...done.(gdb)setdisassembly-flavor intel(gdb) disass LLVMFuzzerTestOneInputDump of assembler codeforfunctionLLVMFuzzerTestOneInput:0x0000000003235e90 <+0>:     push   rbp0x0000000003235e91 <+1>:     mov    rbp,rsp0x0000000003235e94 <+4>:     add    BYTE PTR [rip+0x72d9dfc],0x1# 0xa50fc970x0000000003235e9b <+11>:    mov    rax,0xffffffffffffedc80x0000000003235ea2 <+18>:    cmp    rbp,QWORD PTRfs:[rax]0x0000000003235ea6 <+22>:    jae    0x3235eac <LLVMFuzzerTestOneInput+28>0x0000000003235ea8 <+24>:    mov    QWORD PTR fs:[rax],rbp0x0000000003235eac <+28>:    call   0x3235ec0 <_ZL17EnvoyTestOneInputPKhm>0x0000000003235eb1 <+33>:    xor    eax,eax0x0000000003235eb3 <+35>:    pop    rbp0x0000000003235eb4 <+36>:    retEnd of assembler dump.(gdb) disass _ZL17EnvoyTestOneInputPKhmDump of assembler codeforfunction_ZL17EnvoyTestOneInputPKhm:0x0000000003235ec0 <+0>:     push   rbp0x0000000003235ec1 <+1>:     mov    rbp,rsp0x0000000003235ec4 <+4>:     push   r140x0000000003235ec6 <+6>:     push   rbx0x0000000003235ec7 <+7>:     add    BYTE PTR [rip+0x72d9dca],0x1# 0xa50fc980x0000000003235ece <+14>:    mov    rbx,rsi0x0000000003235ed1 <+17>:    mov    r14,rdi0x0000000003235ed4 <+20>:    mov    rax,0xffffffffffffedc80x0000000003235edb <+27>:    cmp    rbp,QWORD PTRfs:[rax]0x0000000003235edf <+31>:    jae    0x3235ee5 <_ZL17EnvoyTestOneInputPKhm+37>0x0000000003235ee1 <+33>:    mov    QWORD PTR fs:[rax],rbp0x0000000003235ee5 <+37>:    mov    edi,0x498f3a940x0000000003235eea <+42>:    mov    rsi,rbx0x0000000003235eed <+45>:    call   0x3153860 <__sanitizer_cov_trace_const_cmp8()>0x0000000003235ef2 <+50>:    cmp    rbx,0x498f3a940x0000000003235ef9 <+57>:    jne    0x3235f31 <_ZL17EnvoyTestOneInputPKhm+113>0x0000000003235efb <+59>:    add    BYTE PTR [rip+0x72d9d98],0x1# 0xa50fc9a0x0000000003235f02 <+66>:    lea    rdi,[rip+0x908047f]# 0xc2b6388 <_ZNSt3__14coutE>0x0000000003235f09 <+73>:    lea    rsi,[rip+0xfffffffffd3ba150]# 0x5f0060 <.str.4>0x0000000003235f10 <+80>:    call   0x3235f40<_ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKc>0x0000000003235f15 <+85>:    mov    rdi,rax0x0000000003235f18 <+88>:    mov    rsi,r140x0000000003235f1b <+91>:    call   0x3235f80<_ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKh>0x0000000003235f20 <+96>:    lea    rsi,[rip+0xfffffffffd3ba179]# 0x5f00a0 <.str.5>0x0000000003235f27 <+103>:   mov    rdi,rax0x0000000003235f2a <+106>:   call   0x3235f40<_ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKc>
Ada Logics
London, United Kingdom
20
Envoy fuzzing improvements
,
April 2021
0x0000000003235f2f <+111>:   jmp    0x3235f38 <_ZL17EnvoyTestOneInputPKhm+120>0x0000000003235f31 <+113>:   add    BYTE PTR [rip+0x72d9d61],0x1# 0xa50fc990x0000000003235f38 <+120>:   pop    rbx0x0000000003235f39 <+121>:   pop    r140x0000000003235f3b <+123>:   pop    rbp0x0000000003235f3c <+124>:   retEnd of assembler dump.(gdb)
Disassembly of empty fuzz test compiled without instrumentation:
# gdb -q ./a.outReading symbols from ./a.out...done.(gdb)setdisassembly-flavor intel(gdb) disass LLVMFuzzerTestOneInputDump of assembler codeforfunctionLLVMFuzzerTestOneInput:0x0000000000557d40 <+0>:     push   rbp0x0000000000557d41 <+1>:     mov    rbp,rsp0x0000000000557d44 <+4>:     add    BYTE PTR [rip+0x2f43ad],0x1# 0x84c0f80x0000000000557d4b <+11>:    mov    rax,QWORD PTR[rip+0x2f1246]# 0x848f980x0000000000557d52 <+18>:    cmp    rbp,QWORD PTRfs:[rax]0x0000000000557d56 <+22>:    jae    0x557d5c <LLVMFuzzerTestOneInput+28>0x0000000000557d58 <+24>:    mov    QWORD PTR fs:[rax],rbp0x0000000000557d5c <+28>:    call   0x557d70 <_ZL17EnvoyTestOneInputPKhm>0x0000000000557d61 <+33>:    xor    eax,eax0x0000000000557d63 <+35>:    pop    rbp0x0000000000557d64 <+36>:    retEnd of assembler dump.(gdb) disass _ZL17EnvoyTestOneInputPKhmDump of assembler codeforfunction_ZL17EnvoyTestOneInputPKhm:0x0000000000557d70 <+0>:     push   rbp0x0000000000557d71 <+1>:     mov    rbp,rsp0x0000000000557d74 <+4>:     push   r140x0000000000557d76 <+6>:     push   rbx0x0000000000557d77 <+7>:     add    BYTE PTR [rip+0x2f437b],0x1# 0x84c0f90x0000000000557d7e <+14>:    mov    rbx,rsi0x0000000000557d81 <+17>:    mov    r14,rdi0x0000000000557d84 <+20>:    mov    rax,QWORD PTR[rip+0x2f120d]# 0x848f980x0000000000557d8b <+27>:    cmp    rbp,QWORD PTRfs:[rax]0x0000000000557d8f <+31>:    jae    0x557d95 <_ZL17EnvoyTestOneInputPKhm+37>0x0000000000557d91 <+33>:    mov    QWORD PTR fs:[rax],rbp0x0000000000557d95 <+37>:    mov    edi,0x498f3a940x0000000000557d9a <+42>:    mov    rsi,rbx0x0000000000557d9d <+45>:    call   0x473370 <__sanitizer_cov_trace_const_cmp8()>0x0000000000557da2 <+50>:    cmp    rbx,0x498f3a940x0000000000557da9 <+57>:    jne    0x557ddb <_ZL17EnvoyTestOneInputPKhm+107>0x0000000000557dab <+59>:    add    BYTE PTR [rip+0x2f4349],0x1# 0x84c0fb0x0000000000557db2 <+66>:    mov    edi,0x11a77e00x0000000000557db7 <+71>:    mov    esi,0x5e03a00x0000000000557dbc <+76>:    call   0x557df0<_ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKc>0x0000000000557dc1 <+81>:    mov    rdi,rax0x0000000000557dc4 <+84>:    mov    rsi,r140x0000000000557dc7 <+87>:    call   0x557e30<_ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKh>0x0000000000557dcc <+92>:    mov    esi,0x5e03e00x0000000000557dd1 <+97>:    mov    rdi,rax0x0000000000557dd4 <+100>:   call   0x557df0
Ada Logics
London, United Kingdom
21
Envoy fuzzing improvements
,
April 2021
<_ZNSt3__1lsINS_11char_traitsIcEEEERNS_13basic_ostreamIcT_EES6_PKc>0x0000000000557dd9 <+105>:   jmp    0x557de2 <_ZL17EnvoyTestOneInputPKhm+114>0x0000000000557ddb <+107>:   add    BYTE PTR [rip+0x2f4318],0x1# 0x84c0fa0x0000000000557de2 <+114>:   pop    rbx0x0000000000557de3 <+115>:   pop    r140x0000000000557de5 <+117>:   pop    rbp0x0000000000557de6 <+118>:   retEnd of assembler dump.
A.1 Build configurations
Configuration 1. Regular build
This is simply a regular build.
Configuration 2.
This configuration disables:
●
Coverage and bug-finding sanitizers in many of the
external libraries
●
Coverage instrumentation of various folders under
source/common and
source/extensions
●
Coverage instrumentation of code in test/ directory.
●
Coverage instrumentation of all .pb.cc files and all
(.cc) files in the bazel-out directory
declare -r DI="$(if [ "$SANITIZER" != "coverage" ]then# Envoy code. Disable coverage instrumentationecho " --per_file_copt=^.*source/extensions/access_loggers/.*\.cc\$@-fsanitize-coverage=0"echo " --per_file_copt=^.*source/extensions/filters/.*\.cc\$@-fsanitize-coverage=0"echo " --per_file_copt=^.*source/extensions/.*\.cc\$@-fsanitize-coverage=0"echo " --per_file_copt=^.*source/common/protobuf/.*\.cc\$@-fsanitize-coverage=0"echo " --per_file_copt=^.*source/common/network.*\.cc\$@-fsanitize-coverage=0"echo " --per_file_copt=^.*source/common/runtime.*\.cc\$@-fsanitize-coverage=0"
# Envoy test code. Disable coverage instrumentationecho " --per_file_copt=^.*test/.*\.cc\$@-fsanitize-coverage=0"
# Disable external libraries.echo " --per_file_copt=^.*antlr4_runtimes.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_alibaba_hessian2_codec.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*io_opencensus_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_protobuf.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_absl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*googletest.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_googlesource_code_re2.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*upb.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_jbeder_yaml_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*proxy_wasm_cpp_host/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_google_libprotobuf_mutator/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"
Ada Logics
London, United Kingdom
22
Envoy fuzzing improvements
,
April 2021
echo " --per_file_copt=^.*com_googlesource_googleurl/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_datadog_dd_opentracing_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"
# All protobuf code and code in bazel-outecho " --per_file_copt=^.*\.pb\.cc\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*bazel-out/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"fi)"
Configuration 3.
This configuration disables everything that configuration
2 disables
except
:
●
Code under source/common and source/extensions.
declare -r DI="$(if [ "$SANITIZER" != "coverage" ]then# Envoy code. Disable coverage instrumentation
# Envoy test code. Disable coverage instrumentationecho " --per_file_copt=^.*test/.*\.cc\$@-fsanitize-coverage=0"
# Disable celcpp and grpc correctly.echo " --per_file_copt=^.*antlr4_runtimes.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_alibaba_hessian2_codec.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*io_opencensus_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_protobuf.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_absl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*googletest.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_googlesource_code_re2.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*upb.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_jbeder_yaml_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*proxy_wasm_cpp_host/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_google_libprotobuf_mutator/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_googlesource_googleurl/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_datadog_dd_opentracing_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_envoyproxy_sqlparser.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*grpc_httpjson_transcoding.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*grpc_httpjson_transcoding.*\$@-fsanitize-coverage=0,-fno-sanitize=all"
# All protobuf code and code in bazel-outecho " --per_file_copt=^.*\.pb\.cc\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*bazel-out/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"fi)"
Configuration 4
This configuration disables everything that configuration
3 disables
and also
:
●
Disabling bug-finding instrumentation in files under
the test/ directory.
Ada Logics
London, United Kingdom
23
Envoy fuzzing improvements
,
April 2021
declare -r DI="$(if [ "$SANITIZER" != "coverage" ]then# Envoy code. Disable coverage instrumentation
# Envoy test code. Disable coverage instrumentationecho " --per_file_copt=^.*test/.*\.cc\$@-fsanitize-coverage=0,-fno-sanitize=all"
# Disable celcpp and grpc correctly.echo " --per_file_copt=^.*antlr4_runtimes.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_alibaba_hessian2_codec.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*io_opencensus_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_protobuf.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_absl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*googletest.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_googlesource_code_re2.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*upb.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_jbeder_yaml_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*proxy_wasm_cpp_host/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_google_libprotobuf_mutator/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_googlesource_googleurl/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_datadog_dd_opentracing_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_envoyproxy_sqlparser.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*grpc_httpjson_transcoding.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*grpc_httpjson_transcoding.*\$@-fsanitize-coverage=0,-fno-sanitize=all"
# All protobuf code and code in bazel-outecho " --per_file_copt=^.*\.pb\.cc\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*bazel-out/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"fi)"
Configuration 5
This configuration disables everything that configuration
4 disables
and also
:
●
Disabling coverage instrumentation of all .cc files
in source/server folder.
declare-r DI="$(if [ "$SANITIZER" != "coverage" ]then# Envoy code. Disable coverage instrumentationecho " --per_file_copt=^.*source/server.*\.cc\$@-fsanitize-coverage=0"# Envoy test code. Disable coverage instrumentationecho " --per_file_copt=^.*test/.*\.cc\$@-fsanitize-coverage=0,-fno-sanitize=all"
# Disable celcpp and grpc correctly.echo " --per_file_copt=^.*antlr4_runtimes.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_alibaba_hessian2_codec.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*io_opencensus_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_protobuf.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_absl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*googletest.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"
Ada Logics
London, United Kingdom
24
Envoy fuzzing improvements
,
April 2021
echo " --per_file_copt=^.*com_googlesource_code_re2.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*upb.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_jbeder_yaml_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*proxy_wasm_cpp_host/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_google_libprotobuf_mutator/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_googlesource_googleurl/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_datadog_dd_opentracing_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_envoyproxy_sqlparser.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*grpc_httpjson_transcoding.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*grpc_httpjson_transcoding.*\$@-fsanitize-coverage=0,-fno-sanitize=all"
# All protobuf code and code in bazel-outecho " --per_file_copt=^.*\.pb\.cc\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*bazel-out/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"fi)"
Configuration 6
This configuration is similar to configuration #4,
however, it also includes the use of a file
with a function name regex on the path /src/blog_list.txt.
This file is used by
SanitizerCoverage
to disable coverage instrumentation
in functions that match the regular
expression, the contents of this file is:
fun:*envoy*
The important part of the build.sh script is as follows:
declare-r DI="$(if [ "$SANITIZER" != "coverage" ]then# Envoy code. Disable coverage instrumentation
# Envoy test code. Disable coverage instrumentationecho " --per_file_copt=^.*test/.*\.cc\$@-fsanitize-coverage=0,-fno-sanitize=all"
# Disable celcpp and grpc correctly.echo " --per_file_copt=^.*antlr4_runtimes.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_alibaba_hessian2_codec.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*io_opencensus_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_protobuf.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_absl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*googletest.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_grpc_grpc.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*boringssl.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_googlesource_code_re2.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*upb.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*org_brotli.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_google_cel_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_jbeder_yaml_cpp.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*proxy_wasm_cpp_host/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_google_libprotobuf_mutator/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"
Ada Logics
London, United Kingdom
25
Envoy fuzzing improvements
,
April 2021
echo " --per_file_copt=^.*com_googlesource_googleurl/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_lightstep_tracer_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_datadog_dd_opentracing_cpp/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*com_github_envoyproxy_sqlparser.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*grpc_httpjson_transcoding.*\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*grpc_httpjson_transcoding.*\$@-fsanitize-coverage=0,-fno-sanitize=all"
# All protobuf code and code in bazel-outecho " --per_file_copt=^.*\.pb\.cc\$@-fsanitize-coverage=0,-fno-sanitize=all"echo " --per_file_copt=^.*bazel-out/.*\$@-fsanitize-coverage=0,-fno-sanitize=all"fi)"
# Benchmark about 3 GB per CPU (10 threads for 28.8GB RAM)# TODO(asraa): Remove deprecation warnings when Envoyand deps moves to C++17bazel build -s --verbose_failures --dynamic_mode=off${DI}\--per_file_copt="^.*\$"@"-Wno-error=unused-command-line-argument"\--per_file_copt="^.*\$"@"-fsanitize-coverage-blocklist=/src/block_list.txt"\
A.2 Envoy UDP fuzzer coverage
Envoy::Network::UdpListenerImpl::handleReadCallback
:
https://storage.googleapis.com/oss-fuzz-coverage/envoy/reports/202
10423/linux/proc/self/cwd/source/common/network/udp_listener_impl.
cc.html#L72
Ada Logics
London, United Kingdom
26
Envoy fuzzing improvements
,
April 2021
Envoy::Network::Utility::readFromSocket
https://storage.googleapis.com/oss-fu
zz-coverage/envoy/reports/20210423/linux/proc/self/cwd/source/common/network/utility.cc.ht
ml#L576
Ada Logics
London, United Kingdom
27
Envoy fuzzing improvements
,
April 2021
Ada Logics
London, United Kingdom
28