Adding 32bits UTF-8 reading with unicode entities conversion instead of question marks

background:
As most of you are aware, CHM does not support Unicode & so, UTF-8 (unicode encoding) files in DITA are converted by the org.dita.dost.util.ConvertLang ant task in the dita.htmlhelp.convertlang target of DITA in the correct non unicode format with a specific charset.

In consequence, the Java Charset decoder classes puts question marks on unknown/unmanaged/foreign characters when they cannot be found in the destination non unicode charset.

Potential workaround(s) found on Internet/or in this forum:
Lot of people complains around the net about this issue. Lot of them modified/tweaked/hijacked the codepages.xml file to force the output charset to be UTF-8. But CHM compiler does not really support unicode in reality.

Proposal defintive solution fix:
Goals:

    Having the org.dita.dost.util.ConvertLang be able to read true/full UTF-8 chars based on 32 bits code points (instead of 16 as previsouly)
    Having unmanaged/unknown/foreigen chars in the destination non unicode charset, put/converted into HTML entities instead of question marks to be still displayed correctly even if they will not be indexed or cannot be found through the index (that''s a limitation)


This fix works only for html, hhc, hhk files (so only for HTML files). Instead of having Java reading UTF-8 and using its old read() method returning only a 16 bit value, the new code read it in binary and convert the value itself into an UTF-8 32 bits code point.

As previously it will first search for an entity in the entities.properties. workflow:

    As in the past, If no corresponding forced entitiy is found in the entities.properties,
    the binary storage of the code point is next converted into Java String (UTF-16 internal java string format if I am right)
    it next tries to convert the Java String (UTF-16) into the destination output charset.
    If the converted can be converted back to the UTF-16 and is identical to the source, then the string/converted char is saved otherwise, its 32 bits code point is put in the html file as entity.

A:Byte->B:UTF-8->C:code point value of the UTF-8 code->D:Java String ->E:output encoding request->F:JavaString 
A is read from the file
B is the full UTF-8 sequence
C is the B sequence converted into an int code point
D is the UTF-8 Java decoding and stored into JavaString (same for E and as I said but maybe I am wrong, but Java Strings are stored internaly into UTF-16 ?)
E is the final output
By comparing D & F, you can check if you lost something in the destination which was replaced by a question mark & if lost, then put the Unicode value as HTML entity (C)!

Why using HTML entities for lost chars in the encoded output ?
looking to this doc http://en.wikipedia.org/wiki/Unicode_and_HTML
...
Numeric character references[edit source | editbeta]
Main article: Numeric character reference
In order to work around the limitations of legacy encodings, HTML is designed such that it is possible to represent characters from the whole of Unicode inside an HTML document by using a numeric character reference: a sequence of characters that explicitly spell out the Unicode code point of the character being represented. A character reference takes the form &#N;, where N is either a decimal number for the Unicode code point, or a hexadecimal number, in which case it must be prefixed by x. The characters that compose the numeric character reference are universally representable in every encoding approved for use on the Internet

More information can be found in the dita-ot forum here: http://groups.yahoo.com/neo/groups/dita-users/conversations/topics/32823