LUCENE-5389: more analysis advice.

Before we change the protocol for tokenizer construction,
let's get plenty of explanation of the existing one, in case
of a 4.7.