Made abbrevation literals case insensitive. Added constant forms of the
punctuation normalisation strings. Added an ignore rule so that we can
account for the break flags in all cases. Improved the definition of the
default rule so that two hyphens cannot appear after one another.
Adding updated generated tokenizer source.
Added smart'ish single quote normalisation.