Trailing whitespace now leads to an empty token at the end of the line.
This empty token corresponds to the empty string that should be completed.
It's amazing how this simple change (moved one line in `bacom-tokenize')
 presents itself as "correct": the code of `bacom-process-tokens-1' is
now simpler and obvious.  The repetitions of `skip-chars-forward', for
skipping whitespace, are gone.  And the code works: "cd <tab>" shows
all subdirectories of the current directory.