Repository: fonttools/fonttools

Issue:
[ttx] Non-ascii in CFF FullName string doesn't round-trip
A font mentioned in #2897 uses non-ascii in the CFF `FullName` field. This is somehow dumped to .ttx with an unclear encoding, and reading the dumped .ttx fails on the non-ascii chars:

```
Compiling "/Users/just/Downloads/LeeSeoyun.ttx" to "/Users/just/Downloads/LeeSeoyun#1.otf"...
Parsing 'GlyphOrder' table...
Parsing 'head' table...
Parsing 'hhea' table...
Parsing 'maxp' table...
Parsing 'OS/2' table...
Parsing 'name' table...
Parsing 'cmap' table...
Parsing 'post' table...
Parsing 'CFF ' table...
ERROR: Unhandled exception has occurred
Traceback (most recent call last):
  File "/Users/just/code/git/fonttools/Lib/fontTools/ttx.py", line 405, in main
    process(jobs, options)
  File "/Users/just/code/git/fonttools/Lib/fontTools/ttx.py", line 387, in process
    action(input, output, options)
  File "/Users/just/code/git/fonttools/Lib/fontTools/misc/loggingTools.py", line 372, in wrapper
    return func(*args, **kwds)
  File "/Users/just/code/git/fonttools/Lib/fontTools/ttx.py", line 298, in ttCompile
    ttf.importXML(input)
  File "/Users/just/code/git/fonttools/Lib/fontTools/ttLib/ttFont.py", line 349, in importXML
    reader.read()
  File "/Users/just/code/git/fonttools/Lib/fontTools/misc/xmlReader.py", line 47, in read
    self._parseFile(self.file)
  File "/Users/just/code/git/fonttools/Lib/fontTools/misc/xmlReader.py", line 72, in _parseFile
    parser.Parse(chunk, 0)
  File "/Users/sysadmin/build/v3.10.5/Modules/pyexpat.c", line 470, in EndElement
  File "/Users/just/code/git/fonttools/Lib/fontTools/misc/xmlReader.py", line 155, in _endElementHandler
    self.currentTable.fromXML(name, attrs, content, self.ttFont)
  File "/Users/just/code/git/fonttools/Lib/fontTools/ttLib/tables/C_F_F_.py", line 46, in fromXML
    self.cff.fromXML(name, attrs, content, otFont)
  File "/Users/just/code/git/fonttools/Lib/fontTools/cffLib/__init__.py", line 346, in fromXML
    topDict.fromXML(name, attrs, content)
  File "/Users/just/code/git/fonttools/Lib/fontTools/cffLib/__init__.py", line 2618, in fromXML
    value = conv.xmlRead(name, attrs, content, self)
  File "/Users/just/code/git/fonttools/Lib/fontTools/cffLib/__init__.py", line 1358, in xmlRead
    return tobytes(attrs["value"], encoding=("ascii"))
  File "/Users/just/code/git/fonttools/Lib/fontTools/misc/textTools.py", line 131, in tobytes
    return s.encode(encoding, errors)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-11: ordinal not in range(128)
```

OTMaster somehow manages to read this field as a Korean string (which makes sense, given the font):

<img width="1223" alt="image" src="https://user-images.githubusercontent.com/4246121/202894386-6386eac7-565d-49db-9f55-2faabf0782f0.png">

Yet fonttools reads it as gibberish: `'ì\x9d´ì\x84\x9cì\x9c¤ì²´'`.


Discussion:
The string appears to be utf-8:
```python
>>> 'ì\x9d´ì\x84\x9cì\x9c¤ì²´'.encode("latin-1").decode("utf-8")
'이서윤체'
```
Or at least that's how OTMaster is interpreting it.

Edit: fonttools assumes CFF strings are Latin-1:
https://github.com/fonttools/fonttools/blob/5a0dc4bc8dfaa0c7da146cf902395f748b3cebe5/Lib/fontTools/cffLib/__init__.py#L2814-L2817
The font in question can be downloaded here: https://www.heungkuklife.co.kr/jsps/front/company/ci/digit-hand-font-info.jsp (click the "OTF" button).
@schriftgestalt, can you advise? This font appears to have been built with Glyphs.
The current version should make sure that it contains ASCII only. 
Can we consider this font broken, or should we do something?

The code reads the FullName field as Latin-1, writes it as utf-8 to the .ttx file, but the XML parser chokes on non-ascii, so either way it is not a clean round-trip.
I think we should assume it is UTF-8 when reading and always write it as UTF-8. I don’t think the CFF spec specifies an encoding here, so anything outside of ASCII is undefined and UTF-8 is a good default.

Alternatively, we can treat it as byte string and do no encoding conversion at all, not sure if we can write this to the XML data or not.
A byte string is the safer option, but it might be a breaking change.
In TTX, it is written as an XML attribute, I don't believe we can round-trip arbitrary byte strings that way?

```xml
<FullName value="<byte-string-here?>"/>
```

Hm, maybe we should use `Latin1Converter` for all name strings, and not just for Notice and Copyright.

https://github.com/fonttools/fonttools/blob/5a0dc4bc8dfaa0c7da146cf902395f748b3cebe5/Lib/fontTools/cffLib/__init__.py#L2124-L2129

Then at least we can round-trip, even if the TTX will not show the intended string.
> maybe we should use Latin1Converter for all name strings, and not just for Notice and Copyright... Then at least we can round-trip, even if the TTX will not show the intended string.

but encoding a previously utf-8 string to latin1 could fail, or decoding a utf-8 string as latin-1 give you gibberish - how would that round-trip?

> do no encoding conversion at all, not sure if we can write this to the XML data or not.

XMLWriter has a `write8bit` method that we use to round-trip name strings that we fail to decode/encode from/to unicode, see

https://github.com/fonttools/fonttools/blob/5a0dc4bc8dfaa0c7da146cf902395f748b3cebe5/Lib/fontTools/misc/xmlWriter.py#L70-L75

maybe we could use that for these CFF strings when they contain non-ASCII so that we can still round-trip them.
> but encoding a previously utf-8 string to latin1 could fail, or decoding a utf-8 string as latin-1 give you gibberish - how would that round-trip?

We start with a byte string without knowing the encoding. Converting this to Latin-1 can't fail. Encoding the converted Latin-1 string as utf-8 can't fail either. This works backwards as well. Yes, the XML will contain gibberish, but it does round-trip:
```python
s = "".join(chr(i) for i in range(256))
b = s.encode("latin-1")
assert list(b) == list(range(256))
utf = s.encode("utf-8")
assert utf.decode("utf-8") == s

b2 = bytes(range(256))
s2 = b2.decode("latin-1")
assert s2 == s
```
Not saying this is a _nice_ solution, but given there is no encoding information at all, and given _any_ non-ascii in FullName can be considered broken, this at least would round-trip without error.

And I agree that if we can plug in that write8bit functionality here, it would be the better and cleaner solution.
Any update to this issue?
No, not yet. But PRs are welcome

Resolving patch:
diff --git a/Lib/fontTools/cffLib/__init__.py b/Lib/fontTools/cffLib/__init__.py
--- a/Lib/fontTools/cffLib/__init__.py
+++ b/Lib/fontTools/cffLib/__init__.py
@@ -2172,9 +2172,9 @@ def xmlRead(self, name, attrs, content, parent):
     (0, "version", "SID", None, None),
     (1, "Notice", "SID", None, Latin1Converter()),
     ((12, 0), "Copyright", "SID", None, Latin1Converter()),
-    (2, "FullName", "SID", None, None),
-    ((12, 38), "FontName", "SID", None, None),
-    (3, "FamilyName", "SID", None, None),
+    (2, "FullName", "SID", None, Latin1Converter()),
+    ((12, 38), "FontName", "SID", None, Latin1Converter()),
+    (3, "FamilyName", "SID", None, Latin1Converter()),
     (4, "Weight", "SID", None, None),
     ((12, 1), "isFixedPitch", "number", 0, None),
     ((12, 2), "ItalicAngle", "number", 0, None),


Test patch: