Merge spec branch into master

This PR basically contains my last two weeks of work; I now consider this more or less complete (for the time being), and it's clear that fixes that now go into this branch should have gone to master, so I want to merge this.

Basic description at http://hashdist.readthedocs.org/en/spec/

I'll wait a day or so and then hit the button (provided Travis tests passes).