Repository: fonttools/fonttools

Issue:
Pyftsubset fails to include codepoints outside BMP
I seems `pyftsubset` cannot handle characters that fall outside of Unicode’s Basic Multilingual Plane (BMP), and/or are encoded in above 21 bits (?), such as those in the Supplementary Multilingual Plane and other supplementary planes.

For example, I have some text containing 🕂 (U+1F542, Cross Pommee) and 🛒 (U+1F6D2, Shopping Trolley). Feeding the text, and a font that _does contain the offending characters/glyphs_ to `pyftsubset`

    pyftsubset BigFont.ttf --text="ABçdé🕂🛒" --layout-features+="*" --output-file=SubsettedFont.ttf

gets me a glyph-subsetted font, containing all the glyphs that map to characters in my source text (A, B, ç, d, é, …) except for 🕂 and 🛒, i.e. those that are not in the BMP. `pyftsubset` is silent about this: it does not complain, the font is generated allright, the glyphs just won’t be there. You’ll notice this only when using the font…

Yet my source text is a valid UTF-8 string, which, as per `pyftsubset --help`, should parse allright:

    --text=<text>
        Specify characters to include in the subset, as UTF-8 string.
    --text-file=<path>
        Like --text but reads from a file. Newline character are not added the the subset.

I suppose `pyftsubset` is having difficulty with Unicode’s variable length encoding. When we run the above command with the `--verbose` flag, then we get the following (i.a.) printed to the console:

    Missing glyphs for requested Unicodes: ['U+DD42', 'U+DED2', 'U+D83D']

My guess: glyphs for those codepoints are missing from the input font indeed, because, obviously, such characters do not exist in Unicode. They are made up by `pyftsubset` parsing the bitstream of two characters (🕂🛒) encoded in 21 bits each, into three codepoints of 7 bits each.

Am I missing something? Or, is support for characters out of the BMP on the roadmap?

What can I do to circumvent this defect?


Discussion:
can you try using a UTF-8 encoded `--text-file=` ?
passing `--text=` option is problematic because the encoding of command line argv varies between platforms and locales; on top of that, on python 2 they are bytes, whereas on python 3 it is (unicode) str... 
The problem is that pyftsubset is trying to decode that byte-string as UTF-8, but your console may be passing to Python a string with a different encoding.
BTW, are you on Windows? Python 2 or 3?

```python
>>> import sys
>>> print(sys.stdin.encoding)
'UTF-8'
```


Thanks for the quick response! Unfortunately, no success with providing a `--text-file` either.

I am on Mac OSX 10.11.3 El Capitan. I created three files, each containing the same string (ABçdé🕂🛒): `default.txt` was saved with Sublime Text’s (build 3083) default ‘Save’: `UTF8.txt` with ‘Save with Encoding > UTF-8’; `UTF8BOM.txt` with ‘Save with Encoding > UTF-8 with BOM’ (BOM being bit-order-mark — the common Unicode encoding hack). None of those files, passed to `pyftsubset` resulted in a subsetted font, containg 🕂 and/or 🛒. But using `UTF8BOM.txt` did add two additional (useless) glyphs.

    $ pyftsubset Dodecaglotta-Regular.ttf --text-file="default.txt" --layout-features+="*" --output-file=Dodecaglotta-Regular-subset-dflt.ttf
    $ pyftsubset Dodecaglotta-Regular.ttf --text-file="UTF8.txt" --layout-features+="*" --output-file=Dodecaglotta-Regular-subset-utf.ttf
    $ pyftsubset Dodecaglotta-Regular.ttf --text-file="UTF8BOM.txt" --layout-features+="*" --output-file=Dodecaglotta-Regular-subset-utfbom.ttf

![`pyftsubset`’s output of the above inspected with Glyphs app](http://i.imgur.com/MUBHzBn.png)

could you please attach the text files here?
the fonts you are trying to subset, are they publicly available somewhere?
you're sure their cmap table contains U+1F542 and U+1F6D2? (sorry for asking these silly questions)
> BOM being bit-order-mark — the common Unicode encoding hack

Byte order mark is not a hack, it's part of Unicode standard. Necessary to decode foreign UTF-16/UTF-32.
@anthrotype Here are the text files:

[UTF8BOM.txt](https://github.com/fonttools/fonttools/files/619075/UTF8BOM.txt)
[UTF8.txt](https://github.com/fonttools/fonttools/files/619076/UTF8.txt)
[default.txt](https://github.com/fonttools/fonttools/files/619077/default.txt)

I’m pretty sure 🕂 (U+1F542) and 🛒 (U+1F6D2) are in my font: I drew them myself. Not sure how they are in the generated font’s CMAP table, though (`pyftinspect` is broke on my system). Since they’re in development still and I don’t want to disclose them (in full) yet (the whole point of subsetting), they’re not publicly available.
Could you dump the cmap table with ttx, and check that the mappings are in there?

E.g.
```shell
$ ttx -t cmap -o - MyFont.ttf | grep -E "0x1f524|0x1f6d2"
```
They’re in there:

    <map code="0x1f542" name="u1F542"/><!-- ???? -->
    <map code="0x1f6d2" name="uni1F6D2"/><!-- ???? -->

I generated a stripped-down version of my font (deleting glyphs manually in Glyphs App), which you may want to use to reproduce:

[pyftsubsettest.otf.zip](https://github.com/fonttools/fonttools/files/619177/pyftsubsettest.otf.zip)

Using this font (which contains the offending chars) as well, invariably does not yield the desired result: a subsetted font containing  🕂, 🛒, and alike.

I used the following commands:

    $ ttx -t cmap -o - pyftsubsettest.otf > cmap.txt
    $ pyftsubset pyftsubsettest.otf --text-file="UTF8BOM.txt" --layout-features+="*" --output-file=pyftsubsettest-subset-utfbom.ttf --name-IDs+=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14

(Appreciating so much your ad-rem follow-ups!)
It looks like an issue with "narrow" vs "wide" Python 2 builds.
You python binary was compiled with UCS-2 support (ie. a "narrow" build), instead of UCS-4 ("wide"). I guess you're using the default OSX Python, which is a "narrow" build.
This means that internally anything outside of the BMP (Basic Multilingual Plane) is represented using a surrogate pair, and hence show up as 2 characters instead of one (e.g. when asking for the length, or looping through the characters in the string).
I'll have to think about a solution that handles both wide and narrow builds.

For the meantime, maybe you can use Python 3 (you can use `brew install python3`) or simply pass the codepoint directly with the --unicodes option?

Thanks for bringing this up. It is definitely a bug in pyftsubset.
Thank you so much for clarifying! I’m a bit hesitant using Python 3, but I’ll try. I’ll try passing `--unicodes` directly, too, but the flow would be then quite cumbersome: instead of just providing a text file (could be thousands and thousands of characters), one would first need to pass such file through a library that would parse the raw UTF-8 string and write to hex-encoded ascii. And then again we must hope such library does not depend… on a ‘wide’ Python build either…

Resolving patch:
diff --git a/Lib/fontTools/subset/__init__.py b/Lib/fontTools/subset/__init__.py
--- a/Lib/fontTools/subset/__init__.py
+++ b/Lib/fontTools/subset/__init__.py
@@ -2588,8 +2588,10 @@ def populate(self, glyphs=[], gids=[], unicodes=[], text=""):
         self.unicodes_requested.update(unicodes)
         if isinstance(text, bytes):
             text = text.decode("utf_8")
-        for u in text:
-            self.unicodes_requested.add(ord(u))
+        text_utf32 = text.encode("utf-32-be")
+        nchars = len(text_utf32)//4
+        for u in struct.unpack('>%dL' % nchars, text_utf32):
+            self.unicodes_requested.add(u)
         self.glyph_names_requested.update(glyphs)
         self.glyph_ids_requested.update(gids)
 


Test patch:
diff --git a/Lib/fontTools/subset/subset_test.py b/Lib/fontTools/subset/subset_test.py
--- a/Lib/fontTools/subset/subset_test.py
+++ b/Lib/fontTools/subset/subset_test.py
@@ -195,6 +195,36 @@ def test_passthrough_tables(self):
         # unknown tables are kept if --passthrough-tables option is passed
         self.assertTrue(unknown_tag in subsetfont)
 
+    def test_non_BMP_text_arg_input(self):
+        _, fontpath = self.compile_font(
+            self.getpath("TestTTF-Regular_non_BMP_char.ttx"), ".ttf")
+        subsetpath = self.temp_path(".ttf")
+        text = tostr(u"A\U0001F6D2", encoding='utf-8')
+
+        subset.main([fontpath, "--text=%s" % text, "--output-file=%s" % subsetpath])
+        subsetfont = TTFont(subsetpath)
+
+        self.assertEqual(subsetfont['maxp'].numGlyphs, 3)
+        self.assertEqual(subsetfont.getGlyphOrder(), ['.notdef', 'A', 'u1F6D2'])
+
+    def test_non_BMP_text_file_input(self):
+        _, fontpath = self.compile_font(
+            self.getpath("TestTTF-Regular_non_BMP_char.ttx"), ".ttf")
+        subsetpath = self.temp_path(".ttf")
+        text = tobytes(u"A\U0001F6D2", encoding='utf-8')
+        with tempfile.NamedTemporaryFile(delete=False) as tmp:
+            tmp.write(text)
+
+        try:
+            subset.main([fontpath, "--text-file=%s" % tmp.name,
+                         "--output-file=%s" % subsetpath])
+            subsetfont = TTFont(subsetpath)
+        finally:
+            os.remove(tmp.name)
+
+        self.assertEqual(subsetfont['maxp'].numGlyphs, 3)
+        self.assertEqual(subsetfont.getGlyphOrder(), ['.notdef', 'A', 'u1F6D2'])
+
 
 if __name__ == "__main__":
     unittest.main()