bitcoin/bitcoin — pull request #3514 by sipa (2014-01-11 18:21:17+00:00)
Title: Per-peer block tracking, stalled block download detection, orphan pool limiting

This pull request brings in several block downloaded related improvements. It's intended as a preparation for headers-first, but It's useful on itself.

Changes:
* Track blocks being downloaded per-peer, and limit the number in-flight.
* Never request the same block twice, so no more duplicate downloads during sync.
* In case a peer seems to be too slow to send blocks we need, they're disconnected and their in-flight blocks can be requested elsewhere (which saved both us and them bandwidth).
* Limit the number of orphans held in memory, as those still happen very frequently during sync (the result from accidentally downloading blocks from two peers at once).

This should result in generally better synchronization behaviour from random/multiple peers, but may slow down the first part of the process (due to small blocks) and slow down fetching over high-latency high-bandwidth connections (which is relatively rare).

Test plan:
* Delete blocks/ and chainstate/
* Start a download from random peers (no special options, or use -addnode=IP optionally).
* Verify that no "already have block ..." messages appear during sync.
* Check that synchronization completes (it may still be slow, as there is no optimization to select particularly good peers to download from, and orphans are expected).
* Delete blocks/ and chainstate/ again.
* Start a download from a single fast peer (-connect=IP)
* Verify that no "already have block ..." messages appear during sync.
* Check that synchronization completes and progresses quickly (doesn't get stuck; it may be a bit slower than before this patch, particularly in the initial part of the chain; very occasional orphans are possible)

Headers-first will remove a part of the logic in this pull request again, but as it's useful on itself, I prefer getting this reviewed and merged anyway:
* There won't be a per-peer to-be-downloaded queue of block hashes anymore; those will be decided globally based on good known (and verified) headers chains.
* There won't be any orphan processing in the block download mechanism anymore (so no need to limit orphans).
* Stalled download detection won't use as many hardcoded constants anymore (as the question becomes "is this peer slowing too slow compared to the others to keep up" rather than "is this peer slow according to some absolute measurement").

Tested by synchronizing while running in valgrind.