Major rework: group transactions explicitly by shards, not batches.

We currently group transactions into 256-tx batches; each batch is
merkled together into a top-level hash.

Now we have up to 255 transactions in each shard, and each shard is
merkled into a top-level hash.  In fact, the term "batch" goes away.

So, to begin with, there are 4 transaction hashes in each block header
(they're there even if a shard is empty).  When the network grows
this will double, etc.  There's also an 8 bit counter for the number
of transactions in each shard.

This should make network transmission simpler: you know that you need
the transactions in shard N of a batch if you're interested in an
address in N.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
patch enhance-sizes.patch
Remove unused create_proof.c

This was from when we were going to create proofs of a transactions'
entire history.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Separate packet allocation functions from packet io functions.

Makes unit testing easier by untangling dependencies.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make minimal log do minimal helpers too.

Makes unit testing easier by untangling dependencies.  inject.c
needs to manually printf the hash now, but it should have anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Inline shard_full(), block_get_tx(), block_preceeds(), block_ancestor()

More dependency untangling, though I hate inlines.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fix shard_of()

If we want them to sort correctly with a simple memcmp, the most
obvious thing is to use the upper bits of an address as the shard.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make mkpriv generate by name.

For regenerating test/helper_key.c.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Compile ccan/err/err.c

This is a noop if you have <err.h>, but let's not break weird platforms
deliberately.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Refresh tests.

After so much churn, they're basically being rewritten.  Since I
broke everything with my recent rewrite, that was sufficient
motivation to blow the cobwebs off.

I also added one.  Many more to go!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make it easier to see which test fails
Added non-default makefile rule to just build binaries without testing
Speed up tests.

Make the difficulty stupidly easy, for valgrind is slow.
One in 16 blocks should pass this difficulty.

Time for make check on my laptop goes from 40-55 seconds (with
significant variance) to 10 seconds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Remove talv.

I've been moving away from structs with the last element dangling
(which doesn't work for variable length structures like transactions
anyway) to simply documenting the extra fields in comments.  The
two transaction types are the last holdovers.

So we create transaction.[ch] for transaction helper routines, and
put accessors there (it's also a much better place than check_transactions.h
for the helpers which already exist).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
test: make clean should remove .d files.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Shrink struct transaction_shard (part 1).

We're going to have a lot of these (4 per block, to start with), so
some optimization make sense: with 5 second block times we would have
over 500,000 of these within a 30-day horizon.

Since tx on the wire are always followed by the input_refs (for normal
txs) we don't need to have separate refs pointers.  We need to make
such linear structures for the case where we actually generate a block
ourselves though.

Before:
	sizeof(struct transaction_shard) = 4104
After:
	sizeof(struct transaction_shard) = 2056

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Shrink transaction_shard further.

It's worth using a dangling struct to save a pointer here: we're down
now to sizeof(ptr) * (num txs in shard + 1).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rename shard_full/batch_full to shard_all_known/batch_all_known.

This is what it really means.  Once we allow naked transaction hashes
in struct transaction_shard, we may be full, but not know all transactions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
patch protocol-send-two-hashes-per-tx.patch
Allow transaction hashes in struct transaction_shard.

The PROTOCOL_PKT_SHARD packet will give us tx hashes, not the
actual transactions, so we need to keep them until we get the actual
txs.

These hashes are still useful: in particular, they are sufficient
for merkle proofs.

We could save a few bytes by making the bitmap variable length,
but it's only 32 bytes so let's leave that for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>