syntax: shuffle some allocation out of binop_to_str
Add comments for the token table
syntax: methodify the lexer
syntax: use doc comments in the interner
auto merge of #14630 : cmr/rust/rewrite-lexer, r=alexcrichton

These are a pain to rebase, so I'm separating this from the rest of my work.
Nothing controversial here, just some simple refactoring and removal of an
unused entry in the token table. Brings the lexer into 2012 with methods!