Repository: fonttools/fonttools

Issue:
Use Zopfli compression for WOFF files
Creating a WOFF1 font with ttx/fonttools results in a larger file than when you create a TTF file and then compress it with tools like [sfnt2woff-zopfli](https://github.com/bramstein/sfnt2woff-zopfli) or [ttf2woff](http://wizard.ae.krakow.pl/~jb/ttf2woff/), which use Zopfli compression. It would be nice to directly output an optimal compressed font.



Discussion:
That's a good idea! I'll find a good zopfli Python binding and use that instead of zlib when the user has it installed. Would that be ok?

Sounds perect! :)

Keep in mind that Zopfli is about 20-40x slower than zlib, and gives 3-8% better compression. Zopfli, just like Brotli that's used in WOFF2, is really slow when compressing. Depending on the particular usecase, I'd prefer to keep the ability (option) to use zlib (for example when fontTools is used for mass processing or subsetting). 

Hmm I see. I'll test this and see what comes up.
Would you prefer to have an option like `--with-zopfli` and keep zlib as the default, or instead a `--with-zlib` option use zopfli as default -- provided of course it's installed (otherwise fall back to zlib)?

I would prefer the default to be fast and independent of external packages. 
FWIW, in the AFDKO the extra "release" compression is optional as well. 

Erik

> On 28 Jan 2016, at 10:10, Cosimo Lupo notifications@github.com wrote:
> 
> Hmm I see. I'll test this and see what comes up.
> Would you prefer to have an option lik like --with-zopfli and keep zlib as the default, or instead a --with-zlib option use zopfli as default -- provided of course it's installed (otherwise fall back to zlib)?
> 
> —
> Reply to this email directly or view it on GitHub https://github.com/behdad/fonttools/issues/491#issuecomment-176076177.

ok then, we shall keep it optional.

Given that the majority of mass-scale uses of fontTools still probably is on raw SFNT, and WOFF[2] operations are still relatively rare, I’d say Zopfli is OK as default, with a way to explicitly ask for zlib. "Good but slow" WOFF creation will probably be more useful to most people while "fast but less good" is limited to more specialized cases. 

On second thoughts, Erik makes a good point. 

So, I reviewed the existing Python bindings for Zopfli. There are two alternative packages, both available on PyPI. Neither is compatible with Python 3... :(
- [zopfli](https://github.com/wnyc/py-zopfli): simple wrapper around zopfli's `ZlibCompress` and `GzipCompress` methods.
- [pyzopfli](https://github.com/Scondo/py-zopfli): this seems more promising; it originated as a fork of the above, and provides the same interface as the standard zlib module so it can be used as a drop-in replacement.

Both were last edited around 3 years ago and never updated since. They embed copies of the Zopfli C source files, checked out at the time the bindings were originally developed. On the other hand, the google/zopfli upstream repository has commits from few days ago.

Implementing support for Zopfli in fonttools would require:
- making the extension module compatible with Python 3
- updating them to use the the latest zopfli C source files (preferably as git submodule).
- deciding how to pass the command line option down to the SFNTWriter instance, so it uses zopfli instead of zlib.

Finally, I run some tests compressing WOFFs in fonttools with these zopfli extensions, and indeed, like Adam said, it is _extremely_ slow, and the gain is relatively small.

Oh, I forgot to mention: neither of the existing zopfli extension modules seems to compile on Windows.

I made the zopfli bindings work with both Python 2 and 3, as well as on Windows.

I make it build from the latest C sources from google/zopfli, pulled in as git submodule.
I also submitted a [pull request](https://github.com/wnyc/py-zopfli/pull/9) to the original maintainer.

My fork lives at https://github.com/anthrotype/py-zopfli.
You can install the package with pip:

```
pip install -v git+https://github.com/anthrotype/py-zopfli#egg=zopfli
```

And you use it like that:

``` python
>>> from zopfli.zlib import compress
>>> from zlib import decompress
>>> decompress(compress(b"hello"))
b'hello'
```

Tomorrow I'll add the option to use it when compressing WOFF 1.0 with fonttools.

Fantastic work, @anthrotype! Thanks for picking this up, and doing this so quickly. Much appreciated!


Resolving patch:
diff --git a/Lib/fontTools/subset/__init__.py b/Lib/fontTools/subset/__init__.py
--- a/Lib/fontTools/subset/__init__.py
+++ b/Lib/fontTools/subset/__init__.py
@@ -112,6 +112,11 @@
       Specify flavor of output font file. May be 'woff' or 'woff2'.
       Note that WOFF2 requires the Brotli Python extension, available
       at https://github.com/google/brotli
+  --with-zopfli
+      Use the Google Zopfli algorithm to compress WOFF. The output is 3-8 %
+      smaller than pure zlib, but the compression speed is much slower.
+      The Zopfli Python bindings are available at:
+      https://github.com/anthrotype/py-zopfli
 
 Glyph set expansion:
   These options control how additional glyphs are added to the subset.
@@ -2424,8 +2429,9 @@ def __init__(self, **kwargs):
         self.recalc_bounds = False # Recalculate font bounding boxes
         self.recalc_timestamp = False # Recalculate font modified timestamp
         self.prune_unicode_ranges = True  # Clear unused 'ulUnicodeRange' bits
-        self.canonical_order = False # Order tables as recommended
+        self.canonical_order = None # Order tables as recommended
         self.flavor = None  # May be 'woff' or 'woff2'
+        self.with_zopfli = False  # use zopfli instead of zlib for WOFF 1.0
         self.desubroutinize = False # Desubroutinize CFF CharStrings
         self.verbose = False
         self.timing = False
@@ -2452,7 +2458,12 @@ def parse_opts(self, argv, ignore_unknown=False):
             if i == -1:
                 if a.startswith("no-"):
                     k = a[3:]
-                    v = False
+                    if k == "canonical-order":
+                        # reorderTables=None is faster than False (the latter
+                        # still reorders to "keep" the original table order)
+                        v = None
+                    else:
+                        v = False
                 else:
                     k = a
                     v = True
@@ -2756,6 +2767,9 @@ def load_font(fontFile,
 def save_font(font, outfile, options):
     if options.flavor and not hasattr(font, 'flavor'):
         raise Exception("fonttools version does not support flavors.")
+    if options.with_zopfli and options.flavor == "woff":
+        from fontTools.ttLib import sfnt
+        sfnt.USE_ZOPFLI = True
     font.flavor = options.flavor
     font.save(outfile, reorderTables=options.canonical_order)
 
diff --git a/Lib/fontTools/ttLib/sfnt.py b/Lib/fontTools/ttLib/sfnt.py
--- a/Lib/fontTools/ttLib/sfnt.py
+++ b/Lib/fontTools/ttLib/sfnt.py
@@ -133,6 +133,46 @@ def close(self):
 		self.file.close()
 
 
+# default compression level for WOFF 1.0 tables and metadata
+ZLIB_COMPRESSION_LEVEL = 6
+
+# if set to True, use zopfli instead of zlib for compressing WOFF 1.0.
+# The Python bindings are available at https://github.com/anthrotype/py-zopfli
+USE_ZOPFLI = False
+
+# mapping between zlib's compression levels and zopfli's 'numiterations'.
+# Use lower values for files over several MB in size or it will be too slow
+ZOPFLI_LEVELS = {
+	# 0: 0,  # can't do 0 iterations...
+	1: 1,
+	2: 3,
+	3: 5,
+	4: 8,
+	5: 10,
+	6: 15,
+	7: 25,
+	8: 50,
+	9: 100,
+}
+
+
+def compress(data, level=ZLIB_COMPRESSION_LEVEL):
+	""" Compress 'data' to Zlib format. If 'USE_ZOPFLI' variable is True,
+	zopfli is used instead of the zlib module.
+	The compression 'level' must be between 0 and 9. 1 gives best speed,
+	9 gives best compression (0 gives no compression at all).
+	The default value is a compromise between speed and compression (6).
+	"""
+	if not (0 <= level <= 9):
+		raise ValueError('Bad compression level: %s' % level)
+	if not USE_ZOPFLI or level == 0:
+		from zlib import compress
+		return compress(data, level)
+	else:
+		from zopfli.zlib import compress
+		return compress(data, numiterations=ZOPFLI_LEVELS[level])
+
+
 class SFNTWriter(object):
 
 	def __new__(cls, *args, **kwargs):
@@ -246,8 +286,7 @@ def close(self):
 				self.metaOrigLength = len(data.metaData)
 				self.file.seek(0,2)
 				self.metaOffset = self.file.tell()
-				import zlib
-				compressedMetaData = zlib.compress(data.metaData)
+				compressedMetaData = compress(data.metaData)
 				self.metaLength = len(compressedMetaData)
 				self.file.write(compressedMetaData)
 			else:
@@ -439,7 +478,7 @@ class WOFFDirectoryEntry(DirectoryEntry):
 
 	format = woffDirectoryEntryFormat
 	formatSize = woffDirectoryEntrySize
-	zlibCompressionLevel = 6
+	zlibCompressionLevel = ZLIB_COMPRESSION_LEVEL
 
 	def decodeData(self, rawData):
 		import zlib
@@ -448,14 +487,13 @@ def decodeData(self, rawData):
 		else:
 			assert self.length < self.origLength
 			data = zlib.decompress(rawData)
-			assert len (data) == self.origLength
+			assert len(data) == self.origLength
 		return data
 
 	def encodeData(self, data):
-		import zlib
 		self.origLength = len(data)
 		if not self.uncompressed:
-			compressedData = zlib.compress(data, self.zlibCompressionLevel)
+			compressedData = compress(data, self.zlibCompressionLevel)
 		if self.uncompressed or len(compressedData) >= self.origLength:
 			# Encode uncompressed
 			rawData = data
diff --git a/Lib/fontTools/ttx.py b/Lib/fontTools/ttx.py
--- a/Lib/fontTools/ttx.py
+++ b/Lib/fontTools/ttx.py
@@ -72,6 +72,8 @@
     --flavor <type> Specify flavor of output font file. May be 'woff'
       or 'woff2'. Note that WOFF2 requires the Brotli Python extension,
       available at https://github.com/google/brotli
+    --with-zopfli Use Zopfli instead of Zlib to compress WOFF. The Python
+      extension is available at https://github.com/anthrotype/py-zopfli
 """
 
 
@@ -133,6 +135,7 @@ class Options(object):
 	unicodedata = None
 	recalcTimestamp = False
 	flavor = None
+	useZopfli = False
 
 	def __init__(self, rawOptions, numFiles):
 		self.onlyTables = []
@@ -191,6 +194,8 @@ def __init__(self, rawOptions, numFiles):
 				self.recalcTimestamp = True
 			elif option == "--flavor":
 				self.flavor = value
+			elif option == "--with-zopfli":
+				self.useZopfli = True
 		if self.verbose and self.quiet:
 			raise getopt.GetoptError("-q and -v options are mutually exclusive")
 		if self.verbose:
@@ -206,6 +211,8 @@ def __init__(self, rawOptions, numFiles):
 			raise getopt.GetoptError("-t and -x options are mutually exclusive")
 		if self.mergeFile and numFiles > 1:
 			raise getopt.GetoptError("Must specify exactly one TTX source file when using -m")
+		if self.flavor != 'woff' and self.useZopfli:
+			raise getopt.GetoptError("--with-zopfli option requires --flavor 'woff'")
 
 
 def ttList(input, output, options):
@@ -253,6 +260,9 @@ def ttDump(input, output, options):
 @Timer(log, 'Done compiling TTX in %(time).3f seconds')
 def ttCompile(input, output, options):
 	log.info('Compiling "%s" to "%s"...' % (input, output))
+	if options.useZopfli:
+		from fontTools.ttLib import sfnt
+		sfnt.USE_ZOPFLI = True
 	ttf = TTFont(options.mergeFile, flavor=options.flavor,
 			recalcBBoxes=options.recalcBBoxes,
 			recalcTimestamp=options.recalcTimestamp,
@@ -302,7 +312,8 @@ def guessFileType(fileName):
 
 def parseOptions(args):
 	rawOptions, files = getopt.getopt(args, "ld:o:fvqht:x:sim:z:baey:",
-			['unicodedata=', "recalc-timestamp", 'flavor='])
+			['unicodedata=', "recalc-timestamp", 'flavor=',
+			 'with-zopfli'])
 
 	options = Options(rawOptions, len(files))
 	jobs = []


Test patch: