Combine Hash2TokenStream and List2TokenStream
be more specific about required install of SCM environment
Make Iterable2TokenStream a Reader.

Iterable2TokenStream splits the input into tokens in a way similar to
PlainSymbolTokenizer, but not identical. If it is a Reader instead, the
input could be passed on to PlainSymbolTokenizer to produce tokens in
a consistent and uniform way.
Change the analyzers so that they don't keep the entire file in memory.

Instead of passing a stream to the analyze() methods, and have them
cache the entire contents in a byte or char array, open a new stream
each time the source file should be read.

Also, write the xref files from the analyze() methods. In many
analyzers, this avoids the need for building up a data structure that
holds the full xref output in memory.

This fixes #8.
fix duplicate documents and tests for add/remove files to index
this fixes #7