JuliaLang/julia — pull request #5576 by stevengj (2014-01-27 18:06:09+00:00)
Title: RFC: export utf8proc Unicode transformation functionality in Julia

Following up to #5462 and #5434, this patch exposes a bunch of functionality from the bundled [utf8proc](http://www.public-software-group.org/utf8proc) library in Julia:

* `is_valid_char(s)` is replaced by the implementation in utf8proc, which detects a few more invalid codepoints
* `is_assigned_char(c)`: new function to return whether a code point is assigned
* `charcategory(c)`: new function to return a `Base.UnicodeCategory` describing the Unicode [general category](http://www.fileformat.info/info/unicode/category/index.htm) of the character. I defined a type for this to make it easy to get the general category or the subcategory parts, and for pretty printing, but I expect that most people will just compare these to strings (e.g. `"Lu"` for uppercase letter).
* `normalize_string(s::String, normalform::Symbol)`: new function to perform standard Unicode normalization, e.g. `normalize_string(s, :NFKC)` for normal form KC.
* `normalize_string(s::String; keywords...)`: new function to perform various normalization transformations, including case folding, diacritical stripping, newline conversion, etcetera.