Repository: fonttools/fonttools

Issue:
ヒラギノ角ゴシック W4.ttc (Hiragino Sans W4) decompilation fails
I tried to convert "ヒラギノ角ゴシック W4.ttc" version 11.0d7e4 (macOS Sierra system font), with this error:

```bash
$ ttx -y 0 hiragino.ttc 
Dumping "hiragino.ttc" to "hiragino.ttx"...
Dumping 'GlyphOrder' table...
Dumping 'head' table...
Dumping 'hhea' table...
Dumping 'maxp' table...
Dumping 'OS/2' table...
Dumping 'name' table...
Dumping 'cmap' table...
Dumping 'post' table...
Dumping 'CFF ' table...
ERROR: Unhandled exception has occurred
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/ttx.py", line 398, in main
    process(jobs, options)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/ttx.py", line 372, in process
    action(input, output, options)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/misc/loggingTools.py", line 372, in wrapper
    return func(*args, **kwds)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/ttx.py", line 272, in ttDump
    newlinestr=options.newlinestr)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/ttLib/__init__.py", line 308, in saveXML
    self._tableToXML(tableWriter, tag, progress)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/ttLib/__init__.py", line 344, in _tableToXML
    table.toXML(writer, self, progress)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/ttLib/tables/C_F_F_.py", line 42, in toXML
    self.cff.toXML(writer, progress)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/cffLib.py", line 90, in toXML
    font.toXML(xmlWriter, progress)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/cffLib.py", line 1609, in toXML
    self.decompileAllCharStrings(progress)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/cffLib.py", line 1622, in decompileAllCharStrings
    for charString in self.CharStrings.values():
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/cffLib.py", line 364, in __getitem__
    item = self.produceItem(index, data, file, offset, size)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/cffLib.py", line 395, in produceItem
    private = self.fdArray[self.fdSelect[index]].Private
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/cffLib.py", line 364, in __getitem__
    item = self.produceItem(index, data, file, offset, size)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/cffLib.py", line 456, in produceItem
    fontDict.decompile(data)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/cffLib.py", line 1547, in decompile
    dec.decompile(data)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/misc/psCharStrings.py", line 1127, in decompile
    value, index = handler(self, b0, data, index)
  File "/Library/Python/2.7/site-packages/fonttools-3.2.0+2.gdf967e7-py2.7.egg/fontTools/misc/psCharStrings.py", line 21, in read_operator
    operator = self.operators[op]
KeyError: (12, 5)
```

Not sure if it’s a bug in the font or in FontTools ...


Discussion:
I'm reproducing this with a font called "ヒラギノ角ゴ Pro W6.otf", and that certainly worked before, so this is a regression.
If crashing on a `PaintStyle` entry inside the `FontDict` entry inside the `FDArray` element:

```xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<ttFont sfntVersion="OTTO" ttLibVersion="2.1+">

  <CFF>

    <CFFFont name="HiraKakuPro-W6">
      <ROS Registry="Adobe" Order="Japan1" Supplement="5"/>
      <Notice value="Copyright (c) 1993-2002 Dainippon Screen Mfg. Co., Ltd. All Rights Reserved.Hiragino is a trademark of Dainippon Screen Mfg. Co., Ltd. registered in Japan."/>
      <FullName value="HiraKakuPro-W6"/>
      <isFixedPitch value="0"/>
      <ItalicAngle value="0"/>
      <UnderlineThickness value="50"/>
      <PaintType value="0"/>
      <CharstringType value="2"/>
      <FontMatrix value="0.001 0 0 0.001 0 0"/>
      <FontBBox value="-401 -447 1327 1281"/>
      <StrokeWidth value="0"/>
      <XUID value="205 711003 8560901"/>
      <CIDFontVersion value="7.11"/>
      <CIDFontRevision value="0"/>
      <CIDFontType value="0"/>
      <CIDCount value="20317"/>
      <!-- charset is dumped separately as the 'GlyphOrder' element -->
      <UIDBase value="8560901"/>
      <FDSelect format="3"/>
      <FDArray>
        <FontDict index="0">
          <FontName value="HiraKakuPro-W6-AlphaNum"/>
          <isFixedPitch value="0"/>
          <ItalicAngle value="0"/>
          <UnderlineThickness value="50"/>
          <PaintType value="0"/>
          <CharstringType value="2"/>
          <FontMatrix value="0.001 0 0 0.001 0 0"/>
          <FontBBox value="0 0 0 0"/>
          <StrokeWidth value="0"/>
          <Encoding name="StandardEncoding"/>
          <Private>
            <BlueValues value="-250 -250 1100 1100"/>
            <BlueScale value="0.049999"/>
            <BlueShift value="7"/>
            <BlueFuzz value="1"/>
            <StdHW value="117"/>
            <StdVW value="129"/>
            <StemSnapH value="117"/>
            <StemSnapV value="129"/>
            <ForceBold value="0"/>
            <LanguageGroup value="1"/>
            <ExpansionFactor value="0.06"/>
            <initialRandomSeed value="0"/>
            <defaultWidthX value="1000"/>
            <nominalWidthX value="0"/>
            <Subrs>
```
`FontDict` used to be a subclass of `TopDict`, but now it's a subclass of `BaseDict` and has its own very limited set of operators. I'm fairly sure it's caused by this change: https://github.com/fonttools/fonttools/commit/3063def35b7ad733bf584b45452f9ef8f6efbab4
@readroberts, can you tell whether the font is buggy or that there's a problem with cffLib.py?
Finding this comment in cffLib.py:

> """Since fonttools used to pass a lot of fields that are not relevant in the FDArray FontDict, there are 'ttx' files in the wild that contain all these. These got in the ttx files because fonttools writes explicit values for all the TopDict default values. These are not actually illegal in the context of an FDArray FontDict - you can legally, per spec, put any arbitrary key/value pair in a FontDict - but are useless since current major company CFF interpreters ignore anything but the set listed in this file. So, we just silently skip them. An exception is Weight: this is not used by any interepreter, but some foundries have asked that this be supported in FDArray FontDicts just to preserve information about the design when the font is being inspected."""


Well, the situation is obviously worse: there are OTFs out there in the wild that contain these values... I suppose the decompiler could *also* just silently ignore KeyErrors there?
I think I have an elegant solution. I'll clean it up for a PR tomorrow.

Resolving patch:
diff --git a/Lib/fontTools/cffLib.py b/Lib/fontTools/cffLib.py
--- a/Lib/fontTools/cffLib.py
+++ b/Lib/fontTools/cffLib.py
@@ -464,11 +464,7 @@ def fromXML(self, name, attrs, content):
 			if isinstance(element, basestring):
 				continue
 			name, attrs, content = element
-			try:
-				fontDict.fromXML(name, attrs, content)
-			except KeyError:
-				"""Since fonttools used to pass a lot of fields that are not relevant in the FDArray FontDict, there are 'ttx' files in the wild that contain all these. These got in the ttx files because fonttools writes explicit values for all the TopDict default values. These are not actually illegal in the context of an FDArray FontDict - you can legally, per spec, put any arbitrary key/value pair in a FontDict - but are useless since current major company CFF interpreters ignore anything but the set listed in this file. So, we just silently skip them. An exception is Weight: this is not used by any interepreter, but some foundries have asked that this be supported in FDArray FontDicts just to preserve information about the design when the font is being inspected."""
-				pass
+			fontDict.fromXML(name, attrs, content)
 		self.append(fontDict)
 
 
@@ -1302,15 +1298,6 @@ def xmlRead(self, name, attrs, content, parent):
 	(17,		'CharStrings',		'number',	None,	CharStringsConverter()),
 ]
 
-fontDictOperators = [
-#	opcode		name			argument type	default	converter
-	((12, 38),	'FontName',		'SID',		None,	None),
-	((12, 7),	'FontMatrix',		'array',	[0.001, 0, 0, 0.001, 0, 0],	None),
-	(4,		'Weight',		'SID',		None,	None),
-	(18,		'Private',	('number', 'number'),	None,	PrivateDictConverter()),
-]
-
-
 
 # Note! FDSelect and FDArray must both preceed CharStrings in the output XML build order,
 # in order for the font to compile back from xml.
@@ -1357,15 +1344,11 @@ def addConverters(table):
 
 addConverters(privateDictOperators)
 addConverters(topDictOperators)
-addConverters(fontDictOperators)
 
 
 class TopDictDecompiler(psCharStrings.DictDecompiler):
 	operators = buildOperatorDict(topDictOperators)
 
-class FontDictDecompiler(psCharStrings.DictDecompiler):
-	operators = buildOperatorDict(fontDictOperators)
-
 class PrivateDictDecompiler(psCharStrings.DictDecompiler):
 	operators = buildOperatorDict(privateDictOperators)
 
@@ -1501,17 +1484,37 @@ def getChildren(self, strings):
 		return children
 
 
-class FontDictCompiler(DictCompiler):
-
-	opcodes = buildOpcodeDict(fontDictOperators)
+class FontDictCompiler(TopDictCompiler):
 
-	def getChildren(self, strings):
-		children = []
-		if hasattr(self.dictObj, "Private"):
-			privComp = self.dictObj.Private.getCompiler(strings, self)
-			children.append(privComp)
-			children.extend(privComp.getChildren(strings))
-		return children
+	def __init__(self, dictObj, strings, parent):
+		super(FontDictCompiler, self).__init__(dictObj, strings, parent)
+		#
+		# We now take some effort to detect if there were any key/value pairs supplied
+		# that were ignored in the FontDict context, and issue a warning for those cases.
+		#
+		ignoredNames = []
+		dictObj = self.dictObj
+		for name in sorted(set(dictObj.converters) - set(dictObj.order)):
+			if name in dictObj.rawDict:
+				# The font was directly read from binary. In this
+				# case, we want to report *all* "useless" key/value
+				# pairs that are in the font, not just the ones that
+				# are different from the default.
+				ignoredNames.append(name)
+			else:
+				# The font was probably read from a TTX file. We only
+				# warn about keys whos value is not the default. The
+				# ones that have the default value will not be written
+				# to binary anyway.
+				default = dictObj.defaults.get(name)
+				if default is not None:
+					conv = dictObj.converters[name]
+					default = conv.read(dictObj, default)
+				if getattr(dictObj, name, None) != default:
+					ignoredNames.append(name)
+		if ignoredNames:
+			log.warning("Some CFF FDArray/FontDict keys were ignored upon compile: " +
+				" ".join(sorted(ignoredNames)))
 
 
 class PrivateDictCompiler(DictCompiler):
@@ -1574,6 +1577,10 @@ def toXML(self, xmlWriter, progress):
 				continue
 			conv = self.converters[name]
 			conv.xmlWrite(xmlWriter, name, value, progress)
+		ignoredNames = set(self.rawDict) - set(self.order)
+		if ignoredNames:
+			xmlWriter.comment("some keys were ignored: %s" % " ".join(sorted(ignoredNames)))
+			xmlWriter.newline()
 
 	def fromXML(self, name, attrs, content):
 		conv = self.converters[name]
@@ -1633,19 +1640,36 @@ def decompileAllCharStrings(self, progress):
 
 
 class FontDict(TopDict):
-
-	defaults = buildDefaults(fontDictOperators)
-	converters = buildConverters(fontDictOperators)
-	order = buildOrder(fontDictOperators)
-	decompilerClass = FontDictDecompiler
+	#
+	# Since fonttools used to pass a lot of fields that are not relevant in the FDArray
+	# FontDict, there are 'ttx' files in the wild that contain all these. These got in
+	# the ttx files because fonttools writes explicit values for all the TopDict default
+	# values. These are not actually illegal in the context of an FDArray FontDict - you
+	# can legally, per spec, put any arbitrary key/value pair in a FontDict - but are
+	# useless since current major company CFF interpreters ignore anything but the set
+	# listed in this file. So, we just silently skip them. An exception is Weight: this
+	# is not used by any interpreter, but some foundries have asked that this be
+	# supported in FDArray FontDicts just to preserve information about the design when
+	# the font is being inspected.
+	#
+	# On top of that, there are fonts out there that contain such useless FontDict values.
+	#
+	# By subclassing TopDict, we *allow* all key/values from TopDict, both when reading
+	# from binary or when reading from XML, but by overriding `order` with a limited
+	# list of names, we ensure that only the useful names ever get exported to XML and
+	# ever get compiled into the binary font.
+	#
+	# We override compilerClass so we can warn about "useless" key/value pairs, either
+	# from the original binary font or from TTX input.
+	#
+	# See:
+	# - https://github.com/fonttools/fonttools/issues/740
+	# - https://github.com/fonttools/fonttools/issues/601
+	# - https://github.com/adobe-type-tools/afdko/issues/137
+	#
+	order = ['FontName', 'FontMatrix', 'Weight', 'Private']
 	compilerClass = FontDictCompiler
 
-	def __init__(self, strings=None, file=None, offset=None, GlobalSubrs=None):
-		TopDict.__init__(self, strings, file, offset)
-
-	def toXML(self, xmlWriter, progress):
-		BaseDict.toXML(self, xmlWriter, progress)
-
 
 class PrivateDict(BaseDict):
 	defaults = buildDefaults(privateDictOperators)


Test patch: