Tokenizer initial commit
John Bandela says to remove this file
VC++ fix from John
Add acknowledgements
John's fixes for BCC and GCC
John's fixes for BCC
added missing "typename"s
VC typename fixes by John
Fixed formatting
Fix formatting attemp2
Fixes by John for char_delimiters_separator constructor
Fix some broken links. Fix order of constructor initializers.
Fixes for some warnings -JRB
Fixed escape_list_separator bug with terminating empty field,
and enhanced it to allow strings as e,c,q
-jrb
iterator adaptor update and related changes
offset_separator and char_delimiters_separator now use
tok.assign(cur,next)
to assign the token sequence for all iterator types except input iterators where
tok+=*next
is used
There is a workaround for Borland C++ and strings (see code)

Thanks to Gennadiy E. Rozental for pointing this out

-John R. Bandela
Reverted to previous version
added new class char_separator with tests and doc
added new class char_separator
some more edits, with regard to the isspace and ispunct behaviour