log: use single logging record for all logs.

The current logs are separate, ie. JSON-RPC, one per peer peer,
generator and core.  But the core log is usually related to what's
happening in the other logs, so it's hard to interpret them separately.

Use a single log backend for all of them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
log: add a per-line iterator.

This will be used for JSON log output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
jsonrpc: add getlog command.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
jsonrpc: dev-detachedblocks command

Useful for debugging: what blocks do we have where we don't know
the parent?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
json: vastly speed up json creation.

Now we dump large JSON (ie. logs) our multiple strlen() calls really
hurt us.  We could get even more efficient by not continually realloc()
to extend the buffer, but this patch abstracts the type so we could
do that in future with minimal disruption.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
pettycoin-query: speed up handling of large results.

The JSON parser is simply too slow, so add a hacky fastpath if it's
in the format we expect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
peer: continue to send blocks while syncing.

While tracking down bug #32 (where nodes fail to sync) I found that mostly
my peer failed to sync.  The reason is that:

1) By the time we sync up to the initial welcome block, there are other
   blocks which have been generated.
2) Newly generated blocks aren't sent to us while we're syncing.

This fixes the sending path, but I'm still investigating why we didn't
get the predecessors after syncing.     

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
todo: wake peer when we complete a todo.

I noticed that my peer got stuck syncing.  It had a huge TODO list, but it
wasn't sending them out.  This is because we hit MAX_REQUESTS and then don't
wake up again when we process a request.  Fix this.

We still have a problem where our todo list gets backlogged with requests
for the children of every single block, and takes quite a while to clear.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
welcome: use normal recv_block path for welcome block.

This puts all the logic in one place.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
welcome: use peer_read_packet().

This is one case where we read into something other than peer->incoming;
unify that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
welcome: duplicate the welcome packet.

If all goes well, the welcome packet gets owned by "state", which
outlives the peer.  But if it becomes a detached packet, it is owned
by that detached_block structure, and then is later discovered to be
bogus, we'll free it before peer.  In that case we'll queue an error
packet, but a correctly timed input packet could cause us to dereference
peer->welcome before that closes the connection.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
todo: fix uninitialized 'unused' field in PROTOCOL_PKT_GET_SHARD packets.

Valgrind says:

==2904== Syscall param write(buf) points to uninitialised byte(s)
==2904==    at 0x55053B0: __write_nocancel (syscall-template.S:81)
==2904==    by 0x432EA5: do_write (io.c:170)
==2904==    by 0x433523: do_plan (io.c:368)
==2904==    by 0x4335BD: io_ready (io.c:389)
==2904==    by 0x434814: io_loop (poll.c:295)
==2904==    by 0x408D0B: main (pettycoin.c:375)
==2904==  Address 0xaf1c41a is 74 bytes inside a block of size 104 alloc'd
==2904==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2904==    by 0x42AD59: allocate (tal.c:230)
==2904==    by 0x42B23A: tal_alloc_ (tal.c:390)
==2904==    by 0x42B444: tal_alloc_arr_ (tal.c:447)
==2904==    by 0x42C137: tal_dup_ (tal.c:806)
==2904==    by 0x41ADF4: get_todo_pkt (todo.c:356)
==2904==    by 0x40D04F: plan_output (peer.c:465)
==2904==    by 0x432ADD: next_plan (io.c:58)
==2904==    by 0x433611: io_do_always (io.c:398)
==2904==    by 0x4345B6: handle_always (poll.c:218)
==2904==    by 0x434651: io_loop (poll.c:244)
==2904==    by 0x408D0B: main (pettycoin.c:375)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
horizon: routines to check horizon.

Corner cases like < vs <= are important, so centralize the logic
inside a single helper.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
create_refs, inputs_refs: use horizon helpers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
recv_block: when syncing, don't ask about blocks below horizon.

This currently fails because it requires that we adjust the heuristic
for "known" blocks to only go back to the horizon, so we still ask for
the transactions (if any), just not the children.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>