syntax: doc comments all the things
token: replace ast::Ident with just Ident
testsuite: merge some lexer testcases

Now that the lexer is more robust, these tests don't need to be in separate
files. Yay!
str: use more helpful assertion failure messages
test: simplify numeric literal cfail tests
syntax: use a better Show impl for Ident

Rather than just dumping the id in the interner, which is useless, actually
print the interned string. Adjust the lexer logging to use Show instead of
Poly.
codemap: be less annoying in debug logging
lexer: shuffle around some functions
lexer: add ident_from and ident_from_to methods
ast: add an `as_str` method to Ident

This is technically unsafe but interned strings are considered immortal.
syntax: don't process string/char/byte/binary lits

This shuffles things around a bit so that LIT_CHAR and co store an Ident
which is the original, unaltered literal in the source. When creating the AST,
unescape and postprocess them.

This changes how syntax extensions can work, slightly, but otherwise poses no
visible changes. To get a useful value out of one of these tokens, call
`parse::{char_lit, byte_lit, bin_lit, str_lit}`

[breaking-change]
syntax: don't parse numeric literals in the lexer

This removes a bunch of token types. Tokens now store the original, unaltered
numeric literal (that is still checked for correctness), which is parsed into
an actual number later, as needed, when creating the AST.

This can change how syntax extensions work, but otherwise poses no visible
changes.

[breaking-change]
lexer: lex WS/COMMENT/SHEBANG rather than skipping

Now, the lexer will categorize every byte in its input according to the
grammar. The parser skips over these while parsing, thus avoiding their
presence in the input to syntax extensions.
auto merge of #15339 : cmr/rust/rewrite-lexer2, r=brson

Mostly minor things that rebasing is becoming painful.