Ideas and tokenizer

This commit contains the file with our ideas for tokenizing and our
list of plausible features.  Also it has a functioning copy of the
tokenizer (unless there are unseen bugs in it).