opened
MarcoFalke
username_0
WIP
fuzz: Add test/fuzz/test_runner.py and run it in travis
issue
created
practicalswift
username_1
Concept ACK. Very nice!

Related: #10364 ("Feature request: Make Bitcoin libFuzzer-friendly and consider integration into the OSS-Fuzz project"). Feel free to collect the $20k bounty :-)
comment
created
laanwj
username_2
Can you please explain what you're adding here—
Will adding fuzzing to travis make the tests non-deterministic, by randomizing it? Or does it only verify the current corpus?
comment
created
MarcoFalke
username_0
Indeed the script currently only supports running over all seeds exactly once
comment
created
Sjors
username_3
Concept ACK. I can't get fuzzing to work on macOS (not that I tried hard), so having it on Travis sounds like a safe idea.
comment
created
MarcoFalke
username_0
Thx, done
comment
created
practicalswift
username_1
utACK fa7ca8ef58bf3e3b91d1f5a67fa42008e63b1f7b

Let's get this merged :-)
comment
created
MarcoFalke
username_0
Would be nice if at least some person other than myself and travis could test this. Just copy-paste the commands in the readme on a linux machine and tell me your computer didn't crash or something.
comment
created
jamesob
username_4
Tested changes (https://github.com/bitcoin/bitcoin/pull/15295/commits/fa7ca8ef58bf3e3b91d1f5a67fa42008e63b1f7b) to `doc/fuzzing.md` on `Linux 4.15.0-43-generic #46-Ubuntu SMP x86_64` with `afl-fuzz 2.52b`. Installed afl from scratch, rebuilt Bitcoin with requisite flags, and confirmed that running `afl-fuzz` with the `transaction_deserialize` works as intended.

Used new test_runner script which looks like an easy way to delegate to calling `src/test/fuzz/*`. Have yet to get any meaningful output back, but maybe this is intended?
```
$ ./test/fuzz/test_runner.py -l DEBUG ~/src/qa-assets/fuzz_seed_corpus transaction_deserialize

Fuzz targets found: ['address_deserialize', 'addrman_deserialize', 'banentry_deserialize', 'block_deserialize', 'blockheader_deserialize', 'blocklocator_deserialize', 'blockmerkleroot', 'blocktransactions_deserialize', 'blocktransactionsrequest_deserialize', 'blockundo_deserialize', 'bloomfilter_deserialize', 'coins_deserialize', 'diskblockindex_deserialize', 'inv_deserialize', 'messageheader_deserialize', 'netaddr_deserialize', 'service_deserialize', 'transaction_deserialize', 'txoutcompressor_deserialize', 'txundo_deserialize']
Fuzz targets selected: ['transaction_deserialize']
```
comment
created
jamesob
username_4
Tested ACK https://github.com/bitcoin/bitcoin/pull/15295/commits/fa535af92c179b0ffb9280e0b2dc5acfeb80964a. Rebuilt with libfuzzer and tested the command that previously failed with afl:
```
$ ./test/fuzz/test_runner.py -l DEBUG ~/src/qa-assets/fuzz_seed_corpus transaction_deserialize

Fuzz targets found: ['address_deserialize', 'addrman_deserialize', 'banentry_deserialize', 'block_deserialize', 'blockheader_deserialize', 'blocklocator_deserialize', 'blockmerkleroot', 'blocktransactions_deserialize', 'blocktransactionsrequest_deserialize', 'blockundo_deserialize', 'bloomfilter_deserialize', 'coins_deserialize', 'diskblockindex_deserialize', 'inv_deserialize', 'messageheader_deserialize', 'netaddr_deserialize', 'service_deserialize', 'transaction_deserialize', 'txoutcompressor_deserialize', 'txundo_deserialize']
Fuzz targets selected: ['transaction_deserialize']
Run transaction_deserialize with args ['/home/james/src/bitcoin/src/test/fuzz/transaction_deserialize', '-runs=1', '/home/james/src/qa-assets/fuzz_seed_corpus/transaction_deserialize']
Output: INFO: Seed: 819372946
INFO: Loaded 1 modules   (2002 inline 8-bit counters): 2002 [0x55b9cec15620, 0x55b9cec15df2),
INFO: Loaded 1 PC tables (2002 PCs): 2002 [0x55b9cec15df8,0x55b9cec1db18),
INFO:      295 files found in /home/james/src/qa-assets/fuzz_seed_corpus/transaction_deserialize
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 52575 bytes
INFO: seed corpus: files: 295 min: 1b max: 52575b total: 491499b rss: 45Mb
#296    INITED cov: 240 ft: 938 corp: 277/475Kb exec/s: 0 rss: 106Mb
#296    DONE   cov: 240 ft: 938 corp: 277/475Kb exec/s: 0 rss: 106Mb
Done 296 runs in 0 second(s)
```

Worth noting that I had to bump clang from 4. to 6. in order for the `-fsanitize=fuzzer` flags to work.
comment