xbmc/xbmc — pull request #5634 by Paxxi (2014-11-02 14:02:33+00:00)
Title: [WIP][RFC] replace iconv and fribidi with icu library

I started a discussion about this in PR #3903 and thought it might be a good idea to create this to have a place to discuss details. (and if anyone want to follow along on a rambling journey to learn Unicode, encoding, bidi and normalization... :)

The code is in prototyping phase, fairly ugly and lots of random changes I'm trying out, design tips are more than welcome but nitpicking over details is a waste of time currently :)
Todolist
* Fix mac utf-8 normalization
* Fix Bidi code
* Fix RTL conversion
* More test cases, especially with bad input

Thinking about API design and my current thinking is along the lines of removing the failOnBadChar switch and rather use two different functions, possible naming could be utf8ToUtf16 and utf8ToUtf16SystemSafe. First option would skip bad chars and be the mainly used one, second option would be for syscalls and could include mac specific utf-8 handling and other considerations needed to cater for system specific stuff.

Currently thinking that it's most logical to use platform defined endian encoding and provide LE/BE variants for special use needs. I welcome thoughts on this subject.

get rid of wstrings everywhere except Win32 and ignore the whole wchar_t size issue. Thoughts welcome