Revert "Use a CharacterBuffer in CharacterReader"

This reverts commit e47d90818b1918b436d252212e90a03ddf503b59.

On further review this has a negative impact. Counter intuitive, need
to dig in more.