Added --redo option to spider.

A bit more convenient than manually deleting the relevant files.
Enforce document uniquity at the schema.

Basically a hack; what should happen is that we figure out why subject.txt
parsing seems to be occasionally broken. In the interim, though, this ought
to keep the index clean regardless, at the expense of a very significant
slowdown for large indexes.
This commit entails a schema change that will require rebuilding your indexes.
Hence the earlier addition of --redo.