Repository: fonttools/fonttools

Issue:
hb.repack failed to serialize 'GPOS', works with pure fonttools
I'm trying to build a variable font, and it fails with `hb.repack failed to serialize 'GPOS'` when uharfbuzz is installed, but works when `uharfbuzz` is not available (but then fontbakery fails ...).

I've seen some similar issues which were apparently resolved, so I’m wondering if this still happening is a bug?

FontTools’ successful fixing:
```
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 9, 'ItemName:', 'Coverage', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 9, 'ItemName:', 'Coverage', 'ItemIndex:', None)
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 9, 'ItemName:', '<none>', 'ItemIndex:', None)
```

Traceback with harfbuzz:

```
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
WARNING:fontTools.ttLib.tables.otBase:hb.repack failed to serialize 'GPOS', reverting to pure-python serializer; the error message was: RepackerError
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
DEBUG:fontTools.ttLib.tables.otBase:serializing 'GPOS' with hb.repack
INFO:fontTools.ttLib.tables.otBase:Attempting to fix OTLOffsetOverflowError ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
ERROR:root:In 'master_ufo/TestVariable.designspace': Generating fonts from Designspace failed: ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
Traceback (most recent call last):
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/tables/otBase.py", line 111, in compile
    return writer.getAllDataUsingHarfbuzz()
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/tables/otBase.py", line 555, in getAllDataUsingHarfbuzz
    return hb.repack(data, obj_list)
  File "src/uharfbuzz/_harfbuzz.pyx", line 1219, in uharfbuzz._harfbuzz.repack
uharfbuzz._harfbuzz.RepackerError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/tables/otBase.py", line 343, in getData
    items[i] = packUShort(item.pos - pos)
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/tables/otBase.py", line 730, in packUShort
    return struct.pack(">H", value)
struct.error: 'H' format requires 0 <= number <= 65535

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jens/Code/fontmake/Lib/fontmake/font_project.py", line 1057, in run_from_designspace
    self._run_from_designspace_interpolatable(
  File "/Users/jens/Code/fontmake/Lib/fontmake/font_project.py", line 1147, in _run_from_designspace_interpolatable
    self.build_variable_fonts(
  File "/Users/jens/Code/fontmake/Lib/fontmake/font_project.py", line 378, in build_variable_fonts
    fonts = ufo2ft.compileVariableTTFs(
  File "/Users/jens/Documents/Schriften/Variable-Fonts/venv/lib/python3.10/site-packages/ufo2ft/__init__.py", line 605, in compileVariableTTFs
    vfNameToTTFont[vfName] = call_postprocessor(
  File "/Users/jens/Documents/Schriften/Variable-Fonts/venv/lib/python3.10/site-packages/ufo2ft/__init__.py", line 77, in call_postprocessor
    postProcessor = postProcessorClass(otf, ufo, glyphSet=glyphSet)
  File "/Users/jens/Documents/Schriften/Variable-Fonts/venv/lib/python3.10/site-packages/ufo2ft/postProcessor.py", line 50, in __init__
    self.otf = _reloadFont(otf)
  File "/Users/jens/Documents/Schriften/Variable-Fonts/venv/lib/python3.10/site-packages/ufo2ft/postProcessor.py", line 396, in _reloadFont
    font.save(stream)
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/ttFont.py", line 184, in save
    writer_reordersTables = self._save(tmp)
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/ttFont.py", line 224, in _save
    self._writeTable(tag, writer, done, tableCache)
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/ttFont.py", line 647, in _writeTable
    tabledata = self.getTableData(tag)
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/ttFont.py", line 669, in getTableData
    return self.tables[tag].compile(self)
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/tables/otBase.py", line 128, in compile
    return writer.getAllData(remove_duplicate=False)
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/tables/otBase.py", line 585, in getAllData
    tableData = table.getData()
  File "/Users/jens/Code/fonttools/Lib/fontTools/ttLib/tables/otBase.py", line 348, in getData
    raise OTLOffsetOverflowError(overflowErrorRecord)
fontTools.ttLib.tables.otBase.OTLOffsetOverflowError: ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jens/Code/fontmake/Lib/fontmake/__main__.py", line 627, in main
    project.run_from_designspace(inputs.designspace_path, **args)
  File "/Users/jens/Code/fontmake/Lib/fontmake/font_project.py", line 1072, in run_from_designspace
    raise FontmakeError(
fontmake.errors.FontmakeError: In 'master_ufo/TestVariable.designspace': Generating fonts from Designspace failed: ('GPOS', 'LookupIndex:', 3, 'SubTableIndex:', 0, 'ItemName:', 'PairSet.<none>', 'ItemIndex:', None)
make: *** [variable_ttf/TestVariable.ttf] Error 1
```


Discussion:
@garretrieger 
I'll take a look
@garretrieger here’s a designspace to reproduce the traceback.

```bash
fontmake -m DebugFont.ds4.designspace -o variable --keep-overlaps --keep-direction --output-dir .
```
[debug.zip](https://github.com/fonttools/fonttools/files/8956647/debug.zip)


Here's a better version to reproduce that doesn’t require fontmake:

```bash
fonttools varLib DebugFont.ds4.designspace
```
[debug_ttf.zip](https://github.com/fonttools/fonttools/files/8957081/debug_ttf.zip)


Thanks for the test case, I was able to reproduce the error. Looking into it now.
It must have something to do with this https://github.com/fonttools/fonttools/pull/2626 because if I disable `shareExtension`, then building the test case manages to complete without error...

```diff
diff --git a/Lib/fontTools/ttLib/tables/otBase.py b/Lib/fontTools/ttLib/tables/otBase.py
index 514ff8b9..ee96ffdf 100644
--- a/Lib/fontTools/ttLib/tables/otBase.py
+++ b/Lib/fontTools/ttLib/tables/otBase.py
@@ -534,7 +534,7 @@ class OTTableWriter(object):
                 https://github.com/harfbuzz/uharfbuzz/blob/main/src/uharfbuzz/_harfbuzz.pyx#L1149
                 """
                internedTables = {}
-               self._doneWriting(internedTables, shareExtension=True)
+               self._doneWriting(internedTables, shareExtension=False)
                tables = []
                obj_list = []
                done = {}

```
Yes that's right. When extension sharing is enabled it also enables sharing between extensions during the fallback to fontTools packing. In some cases this can lead fontTools to hit one of the cases where it gives up and bails out of repacking. Even though it could still make progress by splitting lookups. The immediate fix is to revert the changes from #2626. I'm looking into how to restructure the fallback from harfbuzz to fontTools to prevent this issue while allowing the harfbuzz side to keep sharing between extensions.
Sounds good, thanks for looking into this Garret!

Resolving patch:
diff --git a/Lib/fontTools/ttLib/tables/otBase.py b/Lib/fontTools/ttLib/tables/otBase.py
--- a/Lib/fontTools/ttLib/tables/otBase.py
+++ b/Lib/fontTools/ttLib/tables/otBase.py
@@ -1,6 +1,7 @@
 from fontTools.config import OPTIONS
 from fontTools.misc.textTools import Tag, bytesjoin
 from .DefaultTable import DefaultTable
+from enum import IntEnum
 import sys
 import array
 import struct
@@ -36,6 +37,25 @@ def __init__(self, overflowErrorRecord):
 	def __str__(self):
 		return repr(self.value)
 
+class RepackerState(IntEnum):
+	# Repacking control flow is implemnted using a state machine. The state machine table:
+	#
+	# State       | Packing Success | Packing Failed | Exception Raised |
+	# ------------+-----------------+----------------+------------------+
+	# PURE_FT     | Return result   | PURE_FT        | Return failure   |
+	# HB_FT       | Return result   | HB_FT          | FT_FALLBACK      |
+	# FT_FALLBACK | HB_FT           | FT_FALLBACK    | Return failure   |
+
+	# Pack only with fontTools, don't allow sharing between extensions.
+	PURE_FT = 1
+
+	# Attempt to pack with harfbuzz (allowing sharing between extensions)
+	# use fontTools to attempt overflow resolution.
+	HB_FT = 2
+
+	# Fallback if HB/FT packing gets stuck. Pack only with fontTools, don't allow sharing between
+	# extensions.
+	FT_FALLBACK = 3
 
 class BaseTTXConverter(DefaultTable):
 
@@ -96,62 +116,98 @@ def compile(self, font):
 						self.tableTag,
 					)
 
+		if (use_hb_repack in (None, True)
+				and have_uharfbuzz
+				and self.tableTag in ("GSUB", "GPOS")):
+			state = RepackerState.HB_FT
+		else:
+			state = RepackerState.PURE_FT
+
 		hb_first_error_logged = False
+		lastOverflowRecord = None
 		while True:
 			try:
 				writer = OTTableWriter(tableTag=self.tableTag)
 				self.table.compile(writer, font)
-				if (
-					use_hb_repack in (None, True)
-					and have_uharfbuzz
-					and self.tableTag in ("GSUB", "GPOS")
-				):
-					try:
-						log.debug("serializing '%s' with hb.repack", self.tableTag)
-						return writer.getAllDataUsingHarfbuzz()
-					except (ValueError, MemoryError, hb.RepackerError) as e:
-						# Only log hb repacker errors the first time they occur in
-						# the offset-overflow resolution loop, they are just noisy.
-						# Maybe we can revisit this if/when uharfbuzz actually gives
-						# us more info as to why hb.repack failed...
-						if not hb_first_error_logged:
-							error_msg = f"{type(e).__name__}"
-							if str(e) != "":
-								error_msg += f": {e}"
-							log.warning(
-								"hb.repack failed to serialize '%s', reverting to "
-								"pure-python serializer; the error message was: %s",
-								self.tableTag,
-								error_msg,
-							)
-							hb_first_error_logged = True
-						return writer.getAllData(remove_duplicate=False)
-				return writer.getAllData()
+				if state == RepackerState.HB_FT:
+					return self.tryPackingHarfbuzz(writer, hb_first_error_logged)
+				elif state == RepackerState.PURE_FT:
+					return self.tryPackingFontTools(writer)
+				elif state == RepackerState.FT_FALLBACK:
+					# Run packing with FontTools only, but don't return the result as it will
+					# not be optimally packed. Once a successful packing has been found, state is
+					# changed back to harfbuzz packing to produce the final, optimal, packing.
+					self.tryPackingFontTools(writer)
+					log.debug("Re-enabling sharing between extensions and switching back to "
+										"harfbuzz+fontTools packing.")
+					state = RepackerState.HB_FT
 
 			except OTLOffsetOverflowError as e:
+				hb_first_error_logged = True
+				ok = self.tryResolveOverflow(font, e, lastOverflowRecord)
+				lastOverflowRecord = e.value
 
-				if overflowRecord == e.value:
-					raise # Oh well...
-
-				overflowRecord = e.value
-				log.info("Attempting to fix OTLOffsetOverflowError %s", e)
-				lastItem = overflowRecord
+				if ok:
+					continue
 
-				ok = 0
-				if overflowRecord.itemName is None:
-					from .otTables import fixLookupOverFlows
-					ok = fixLookupOverFlows(font, overflowRecord)
+				if state is RepackerState.HB_FT:
+					log.debug("Harfbuzz packing out of resolutions, disabling sharing between extensions and "
+									 "switching to fontTools only packing.")
+					state = RepackerState.FT_FALLBACK
 				else:
-					from .otTables import fixSubTableOverFlows
-					ok = fixSubTableOverFlows(font, overflowRecord)
-				if not ok:
-					# Try upgrading lookup to Extension and hope
-					# that cross-lookup sharing not happening would
-					# fix overflow...
-					from .otTables import fixLookupOverFlows
-					ok = fixLookupOverFlows(font, overflowRecord)
-					if not ok:
-						raise
+					raise
+
+	def tryPackingHarfbuzz(self, writer, hb_first_error_logged):
+		try:
+			log.debug("serializing '%s' with hb.repack", self.tableTag)
+			return writer.getAllDataUsingHarfbuzz()
+		except (ValueError, MemoryError, hb.RepackerError) as e:
+			# Only log hb repacker errors the first time they occur in
+			# the offset-overflow resolution loop, they are just noisy.
+			# Maybe we can revisit this if/when uharfbuzz actually gives
+			# us more info as to why hb.repack failed...
+			if not hb_first_error_logged:
+				error_msg = f"{type(e).__name__}"
+				if str(e) != "":
+					error_msg += f": {e}"
+				log.warning(
+						"hb.repack failed to serialize '%s', attempting fonttools resolutions "
+						"; the error message was: %s",
+						self.tableTag,
+						error_msg,
+				)
+				hb_first_error_logged = True
+			return writer.getAllData(remove_duplicate=False)
+
+
+	def tryPackingFontTools(self, writer):
+		return writer.getAllData()
+
+
+	def tryResolveOverflow(self, font, e, lastOverflowRecord):
+		ok = 0
+		if lastOverflowRecord == e.value:
+			# Oh well...
+			return ok
+
+		overflowRecord = e.value
+		log.info("Attempting to fix OTLOffsetOverflowError %s", e)
+
+		if overflowRecord.itemName is None:
+			from .otTables import fixLookupOverFlows
+			ok = fixLookupOverFlows(font, overflowRecord)
+		else:
+			from .otTables import fixSubTableOverFlows
+			ok = fixSubTableOverFlows(font, overflowRecord)
+
+		if ok:
+			return ok
+
+		# Try upgrading lookup to Extension and hope
+		# that cross-lookup sharing not happening would
+		# fix overflow...
+		from .otTables import fixLookupOverFlows
+		return fixLookupOverFlows(font, overflowRecord)
 
 	def toXML(self, writer, font):
 		self.table.toXML2(writer, font)
@@ -533,9 +589,7 @@ def getAllDataUsingHarfbuzz(self):
                 https://github.com/harfbuzz/uharfbuzz/blob/main/src/uharfbuzz/_harfbuzz.pyx#L1149
                 """
 		internedTables = {}
-		# TODO: Restore shareExtension=True after we fix
-		# https://github.com/fonttools/fonttools/issues/2661
-		self._doneWriting(internedTables, shareExtension=False)
+		self._doneWriting(internedTables, shareExtension=True)
 		tables = []
 		obj_list = []
 		done = {}


Test patch:
diff --git a/Tests/subset/subset_test.py b/Tests/subset/subset_test.py
--- a/Tests/subset/subset_test.py
+++ b/Tests/subset/subset_test.py
@@ -879,8 +879,8 @@ def mock_repack(data, obj_list):
         # test we emit a log.error if hb.repack fails (and we don't if successful)
         assert (
                 (
-                "hb.repack failed to serialize 'GSUB', reverting to "
-                "pure-python serializer; the error message was: RepackerError: mocking"
+                "hb.repack failed to serialize 'GSUB', attempting fonttools resolutions "
+                "; the error message was: RepackerError: mocking"
             ) in caplog.text
         ) ^ ok