dita-ot/dita-ot — pull request #1564 by llecaroz (2013-09-17 10:51:20+00:00)
Title: Adding 32bits UTF-8 reading with unicode entities convertion

background:
As most of you are aware, CHM does not support UTF-8 & so, UTF-8 files in DITA are converted by the org.dita.dost.util.ConvertLang ant task in the dita.htmlhelp.convertlang target of DITA in the correct destination charset.

In consequence, the Java Charset decoder classes puts question marks on unknown/unmanaged/foreign characters when they cannot be found in the destination charset.

Potential workaround(s) found on Internet/or in this forum:
Lot of people complains around the net about this issue. Lot of them modified/tweaked/hijacked the codepages.xml file to force the output charset to be UTF-8. But CHM compiler does not really support UTF-8 in reality.

Proposal defintive solution fix:
Goals:

Having the org.dita.dost.util.ConvertLang be able to read true/full UTF-8 chars based on 32 bits code points (instead of 16 as previsouly)
Having unmanaged/unknown/foreigen chars in the destination charset, put/converted into HTML entities instead of question marks to be still displayed correctly

This fix works only for html, hhc, hhk files (so only for HTML files). Instead of having Java reading UTF-8 and using its old read() method returning only a 16 bit value, the new code read it in binary and convert the value itself into an UTF-8 32 bits code point.

As previously it will first search for an entity in the entities.properties. workflow:

As in the past, If no corresponding forced entitiy is found in the entities.properties,
the binary storage of the code point is next converted into Java String (UTF-16 internal format)
it next tries to convert the Java String (UTF-16) into the destination output charset.
If the converted can be converted back to the UTF-16 and is identical to the source, then the string/converted char is saved otherwise, its 32 bits code point is put in the html file as entity.