simonmar/alex — pull request #32 by yinguanhao (2013-10-29 07:38:59+00:00)
Title: Add alexScanB and alexScanUserB, fix ByteString wrappers w.r.t. unicode

The ByteString wrappers return truncated lexemes if there are non-ascii characters. See `tests/tokens_bytestring_unicode.x`.

This is due to `alexScan` returning length in characters.

I have added two functions `alexScanB` and `alexScanUserB` which return length in bytes, and modified
ByteString wrappers to use `alexScanB`.