westes/flex — pull request #3 by mplucinski (2014-06-20 23:22:55+00:00)
Title: Support for charsets and unify internal character storage type

This pull request introduces two changes:

* 2e6add4: in places where text characters are processed, replace `char` type by `YY_CHAR` macro. This introduces distinguishness between character as byte of file ("physical") and character as smallest part of text ("logical"). For logical one, `YY_CHAR` macro should be used. For now, they represent the same value, but in future `YY_CHAR` may be replaced by wider type. This is essential to introduce support for extended character sets (any set with number of code points bigger than 256).

To properly introduce `YY_CHAR` macro usage, some tests are changed to not use `str*` family of standard C functions, but instead versions that operate on `YY_CHAR` values. New functions are sitting in `tests/strutils.h` header, and test case for this file is located in `tests/test-tests_strutils` directory.

* f1456f9, 0d3107e: adds support for character sets. The first commit adds test case and the second actual support. This gives way to convert between "physical" bytes and "logical" characters with user-provided function. It allows input data to be encoded in different character set than the one that has been used to write input ".l" file.

Charset support is completely optional. It is activated by `%option charset`. When it is active, new variables are activated: `yycharset` and `yycharset_handler`. First one should be set to currently used character set before calling `yylex()`. Second one should be set to function that will be called to convert incoming bytes into characters. Additionally, `%option charset-source="ENCODING"` should be set to the name of encoding used to write ".l" file - it will be used as internal scanner's character encoding. In case when incoming data are in the same encoding as internal one, conversion function is not used.