zotero/translators — pull request #694 by aurimasv (2014-03-12 02:59:00+00:00)
Title: [CrossRef] Try to fix encoding issues

Not sure if this has been reported, but I remember seeing this issue before.

E.g. see the title in [10.1057/9780230391116.0016](http://doi.crossref.org/servlet/query?usr=zter&pwd=zter321&format=unixref&qdata=%3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22UTF-8%22%3F%3E%3Cquery_batch%20version%3D%222.0%22%20xmlns%3D%22http%3A%2F%2Fwww.crossref.org%2Fqschema%2F2.0%22%20xmlns%3Axsi%3D%22http%3A%2F%2Fwww.w3.org%2F2001%2FXMLSchema-instance%22%3E%3Chead%3E%3Cemail_address%3Eckoscher%40crossref.org%3C%2Femail_address%3E%3Cdoi_batch_id%3Etrackingid1%3C%2Fdoi_batch_id%3E%3C%2Fhead%3E%3Cbody%3E%3Cquery%20key%3D%2210.1057%2F9780230391116.0016%22%20expanded-results%3D%22true%22%3E%3Cdoi%3E10.1057%2F9780230391116.0016%3C%2Fdoi%3E%3C%2Fquery%3E%3C%2Fbody%3E%3C%2Fquery_batch%3E) (notice the â instead of en dash in the title)

I think this happens when UTF-8 encoded string is treated as ISO-8859-1 (or something similar) and that is then transcoded into UTF-8. So en dash in UTF-8 (U+2013; hex E28093) treated as ISO-8859-1 (E2 80 93; 3 separate characters), transcoded into UTF-8 (U+00E2 U+0080 U+0093; hex C3A2C280C293) resulting in `â`

Here we check for some control characters that should never appear in strings (at least I don't expect them to) and reverse the above. This probably won't catch every case where the encoding is messed up, but it should also not break anything. The fixes are not applied to creators. I don't recall a case where that was an issue.