Add backquote to punctuation character class.
Less fragile word-and-* emitters for tokenization.

Change the word-and-apostrophe and word-and-dash emitters used by the
tokenization state machine to emit dashes or apostrophes even if no word
is in the token, to emit a word even if no dash is in the token, and to
never emit nil or empty string tokens.
Slight refactor of the tokenize function.

Make the tokenization operation more clear by only using the if
expression to append whitespace to the input if necessary, and changing
the first statement in the threading macro to be `(token-machine input)`
rather than an entire if expression.
Fix some token state machine bugs.

Change the em-dash handling to only emit the word and dash once the run
of dashes ends, rather than always emitting after two dashes.

Jump directly to the running dash state when a dash is encountered in
the whitespace state.

Use only the match-all input predicate where redundant with the
whitespace predicate (so that unknown characters are treated as
whitespace).

Add a few more tests to make sure that punctuation after words and
longer em-dashes are correctly tokenized.