Repository: fonttools/fonttools

Issue:
[feaLib] possibly suboptimal compilation of inline multiple substitution rules
I believe this is similar to https://github.com/fonttools/fonttools/issues/2150, but for multiple substitution instead of single substitution.

Given the following input,

```fea
# reduced from the ccmp feature in oswald
feature ccmp {
    lookup ccmp_Other_1 {
        @CombiningTopAccents = [acutecomb brevecomb];
        @CombiningNonTopAccents = [dotbelowcomb ogonekcomb];
        lookupflag UseMarkFilteringSet @CombiningTopAccents;
        sub idotbelow' @CombiningTopAccents by idotless dotbelowcomb;
        sub iogonek' @CombiningTopAccents by idotless ogonekcomb;
        sub idotbelow' @CombiningNonTopAccents @CombiningTopAccents by idotless dotbelowcomb;
        sub iogonek' @CombiningNonTopAccents @CombiningTopAccents by idotless ogonekcomb;
    } ccmp_Other_1;
} ccmp;
```

`fontmake` generates five lookups; one contextual lookup, and then one lookup for each inline sub statement. This means that there are two sets of duplicate lookups, i.e. two copies of `sub iodotbelow by idotless dotbelowcomb;` and two copies of `sub iogonek by idotless ogonekcomb;`.


`fontc`, for this input, generates two identical lookups, where each lookup contains both of the substitutions.

I am not sure that either is correct; I think we should be able to get away with a single lookup, containing both substitutions.


_This has unexpected implications!_ specifically, the presence of duplicate identical lookups (which will be deduplicated during table packing) has unexpectedly (to me) severe impact on binary size, because it means that we can not efficiently deduplicate various `ChainedClassSequenceRule` and `ChainedClassSequenceRuleSet` tables, since even though they are referencing semantically identical lookups those lookups have distinct lookup ids, which is the difference between 140 bytes for the fontmake table and 102 bytes for `fontc`.


Does my reasoning make sense? can we be more aggressive in reusing lookups in this case?


Discussion:
> Does my reasoning make sense? can we be more aggressive in reusing lookups in this case?

I think so. @simoncozens 
In theory this can be done with one lookup, yes. (Or fewer! It's possible to combine all non-overlapping "auxiliary" lookups which share the same lookup flags into a single lookup across multiple chaining rules.)

And yes, it makes sense that, because multiple `Rule` tables get generated, the impact of deduplication is limited to "leaf nodes".
> Or fewer! It's possible to combine all non-overlapping "auxiliary" lookups which share the same lookup flags into a single lookup across multiple chaining rules

I believe AFDKO does that? I thought feaLib does as well.
looks like feaLib builder always creates a new anonymous lookup for inline chained substitution rules, this method `get_chained_lookup_` always make a new one (unlike `get_lookup_` which attempts to reuse the current one where possible):

https://github.com/fonttools/fonttools/blob/308b449edc940b9e9bb2368ac5c4afcf92e911e8/Lib/fontTools/feaLib/builder.py#L246-L251

https://github.com/fonttools/fonttools/blob/308b449edc940b9e9bb2368ac5c4afcf92e911e8/Lib/fontTools/feaLib/builder.py#L1285-L1293

fea-rs has a `find_or_create_anon_lookup` method that takes two closures, one to check if a given existing lookup can be reused, another in case no suitable lookup is found to make a new one

https://github.com/googlefonts/fontc/blob/7511532553457fba90c4555ae111f5235446f399/fea-rs/src/compile/lookups/contextual.rs#L129

maybe we could take inspiration from that




Patch:
diff --git a/Lib/fontTools/feaLib/builder.py b/Lib/fontTools/feaLib/builder.py
index a91381ddc1..bda855e1e9 100644
--- a/Lib/fontTools/feaLib/builder.py
+++ b/Lib/fontTools/feaLib/builder.py
@@ -1286,10 +1286,7 @@ def add_multiple_subst(
         self, location, prefix, glyph, suffix, replacements, forceChain=False
     ):
         if prefix or suffix or forceChain:
-            chain = self.get_lookup_(location, ChainContextSubstBuilder)
-            sub = self.get_chained_lookup_(location, MultipleSubstBuilder)
-            sub.mapping[glyph] = replacements
-            chain.rules.append(ChainContextualRule(prefix, [{glyph}], suffix, [sub]))
+            self.add_multi_subst_chained_(location, prefix, glyph, suffix, replacements)
             return
         lookup = self.get_lookup_(location, MultipleSubstBuilder)
         if glyph in lookup.mapping:
@@ -1369,7 +1366,7 @@ def add_single_subst_chained_(self, location, prefix, suffix, mapping):
         # https://github.com/fonttools/fonttools/issues/512
         # https://github.com/fonttools/fonttools/issues/2150
         chain = self.get_lookup_(location, ChainContextSubstBuilder)
-        sub = chain.find_chainable_single_subst(mapping)
+        sub = chain.find_chainable_subst(mapping, SingleSubstBuilder)
         if sub is None:
             sub = self.get_chained_lookup_(location, SingleSubstBuilder)
         sub.mapping.update(mapping)
@@ -1377,6 +1374,19 @@ def add_single_subst_chained_(self, location, prefix, suffix, mapping):
             ChainContextualRule(prefix, [list(mapping.keys())], suffix, [sub])
         )
 
+    def add_multi_subst_chained_(self, location, prefix, glyph, suffix, replacements):
+        if not all(prefix) or not all(suffix):
+            raise FeatureLibError(
+                "Empty glyph class in contextual substitution", location
+            )
+        # https://github.com/fonttools/fonttools/issues/3551
+        chain = self.get_lookup_(location, ChainContextSubstBuilder)
+        sub = chain.find_chainable_subst({glyph: replacements}, MultipleSubstBuilder)
+        if sub is None:
+            sub = self.get_chained_lookup_(location, MultipleSubstBuilder)
+        sub.mapping[glyph] = replacements
+        chain.rules.append(ChainContextualRule(prefix, [{glyph}], suffix, [sub]))
+
     # GSUB 8
     def add_reverse_chain_single_subst(self, location, old_prefix, old_suffix, mapping):
         if not mapping:
diff --git a/Lib/fontTools/otlLib/builder.py b/Lib/fontTools/otlLib/builder.py
index 70fd87ab57..8fc685683a 100644
--- a/Lib/fontTools/otlLib/builder.py
+++ b/Lib/fontTools/otlLib/builder.py
@@ -544,6 +544,10 @@ def buildFormat2Subtable(self, ruleset, classdefs, chaining=True):
                 f"{classRuleAttr}Count",
                 getattr(setForThisRule, f"{classRuleAttr}Count") + 1,
             )
+        for i, classSet in enumerate(classSets):
+            if not getattr(classSet, classRuleAttr):
+                # class sets can be null so replace nop sets with None
+                classSets[i] = None
         setattr(st, self.ruleSetAttr_(format=2, chaining=chaining), classSets)
         setattr(
             st, self.ruleSetAttr_(format=2, chaining=chaining) + "Count", len(classSets)
@@ -781,14 +785,14 @@ def getAlternateGlyphs(self):
                             )
         return result
 
-    def find_chainable_single_subst(self, mapping):
-        """Helper for add_single_subst_chained_()"""
+    def find_chainable_subst(self, mapping, builder_class):
+        """Helper for add_{single,multi}_subst_chained_()"""
         res = None
         for rule in self.rules[::-1]:
             if rule.is_subtable_break:
                 return res
             for sub in rule.lookups:
-                if isinstance(sub, SingleSubstBuilder) and not any(
+                if isinstance(sub, builder_class) and not any(
                     g in mapping and mapping[g] != sub.mapping[g] for g in sub.mapping
                 ):
                     res = sub
diff --git a/Tests/feaLib/data/contextual_inline_multi_sub_format_2.fea b/Tests/feaLib/data/contextual_inline_multi_sub_format_2.fea
new file mode 100644
index 0000000000..9a9bfe20de
--- /dev/null
+++ b/Tests/feaLib/data/contextual_inline_multi_sub_format_2.fea
@@ -0,0 +1,17 @@
+# reduced from the ccmp feature in Oswald
+feature ccmp {
+    lookup ccmp_Other_1 {
+        @CombiningTopAccents = [acutecomb brevecomb];
+        @CombiningNonTopAccents = [dotbelowcomb ogonekcomb];
+        lookupflag UseMarkFilteringSet @CombiningTopAccents;
+        # we should only generate two lookups; one contextual and one multiple sub,
+        # containing 'sub idotbelow by idotless dotbelowcomb' and
+        # 'sub iogonek by idotless ogonekcomb'
+        sub idotbelow' @CombiningTopAccents by idotless dotbelowcomb;
+        sub iogonek' @CombiningTopAccents by idotless ogonekcomb;
+        sub idotbelow' @CombiningNonTopAccents @CombiningTopAccents by idotless dotbelowcomb;
+        sub iogonek' @CombiningNonTopAccents @CombiningTopAccents by idotless ogonekcomb;
+    } ccmp_Other_1;
+} ccmp;
+
+
diff --git a/Tests/feaLib/data/contextual_inline_multi_sub_format_2.ttx b/Tests/feaLib/data/contextual_inline_multi_sub_format_2.ttx
new file mode 100644
index 0000000000..d09431fa49
--- /dev/null
+++ b/Tests/feaLib/data/contextual_inline_multi_sub_format_2.ttx
@@ -0,0 +1,135 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<ttFont sfntVersion="\x00\x01\x00\x00" ttLibVersion="4.53">
+
+  <GDEF>
+    <Version value="0x00010002"/>
+    <MarkGlyphSetsDef>
+      <MarkSetTableFormat value="1"/>
+      <!-- MarkSetCount=1 -->
+      <Coverage index="0">
+        <Glyph value="acutecomb"/>
+        <Glyph value="brevecomb"/>
+      </Coverage>
+    </MarkGlyphSetsDef>
+  </GDEF>
+
+  <GSUB>
+    <Version value="0x00010000"/>
+    <ScriptList>
+      <!-- ScriptCount=1 -->
+      <ScriptRecord index="0">
+        <ScriptTag value="DFLT"/>
+        <Script>
+          <DefaultLangSys>
+            <ReqFeatureIndex value="65535"/>
+            <!-- FeatureCount=1 -->
+            <FeatureIndex index="0" value="0"/>
+          </DefaultLangSys>
+          <!-- LangSysCount=0 -->
+        </Script>
+      </ScriptRecord>
+    </ScriptList>
+    <FeatureList>
+      <!-- FeatureCount=1 -->
+      <FeatureRecord index="0">
+        <FeatureTag value="ccmp"/>
+        <Feature>
+          <!-- LookupCount=1 -->
+          <LookupListIndex index="0" value="0"/>
+        </Feature>
+      </FeatureRecord>
+    </FeatureList>
+    <LookupList>
+      <!-- LookupCount=2 -->
+      <Lookup index="0">
+        <LookupType value="6"/>
+        <LookupFlag value="16"/><!-- useMarkFilteringSet -->
+        <!-- SubTableCount=1 -->
+        <ChainContextSubst index="0" Format="2">
+          <Coverage>
+            <Glyph value="idotbelow"/>
+            <Glyph value="iogonek"/>
+          </Coverage>
+          <BacktrackClassDef>
+          </BacktrackClassDef>
+          <InputClassDef>
+            <ClassDef glyph="idotbelow" class="1"/>
+            <ClassDef glyph="iogonek" class="2"/>
+          </InputClassDef>
+          <LookAheadClassDef>
+            <ClassDef glyph="acutecomb" class="1"/>
+            <ClassDef glyph="brevecomb" class="1"/>
+            <ClassDef glyph="dotbelowcomb" class="2"/>
+            <ClassDef glyph="ogonekcomb" class="2"/>
+          </LookAheadClassDef>
+          <!-- ChainSubClassSetCount=3 -->
+          <ChainSubClassSet index="0" empty="1"/>
+          <ChainSubClassSet index="1">
+            <!-- ChainSubClassRuleCount=2 -->
+            <ChainSubClassRule index="0">
+              <!-- BacktrackGlyphCount=0 -->
+              <!-- InputGlyphCount=1 -->
+              <!-- LookAheadGlyphCount=1 -->
+              <LookAhead index="0" value="1"/>
+              <!-- SubstCount=1 -->
+              <SubstLookupRecord index="0">
+                <SequenceIndex value="0"/>
+                <LookupListIndex value="1"/>
+              </SubstLookupRecord>
+            </ChainSubClassRule>
+            <ChainSubClassRule index="1">
+              <!-- BacktrackGlyphCount=0 -->
+              <!-- InputGlyphCount=1 -->
+              <!-- LookAheadGlyphCount=2 -->
+              <LookAhead index="0" value="2"/>
+              <LookAhead index="1" value="1"/>
+              <!-- SubstCount=1 -->
+              <SubstLookupRecord index="0">
+                <SequenceIndex value="0"/>
+                <LookupListIndex value="1"/>
+              </SubstLookupRecord>
+            </ChainSubClassRule>
+          </ChainSubClassSet>
+          <ChainSubClassSet index="2">
+            <!-- ChainSubClassRuleCount=2 -->
+            <ChainSubClassRule index="0">
+              <!-- BacktrackGlyphCount=0 -->
+              <!-- InputGlyphCount=1 -->
+              <!-- LookAheadGlyphCount=1 -->
+              <LookAhead index="0" value="1"/>
+              <!-- SubstCount=1 -->
+              <SubstLookupRecord index="0">
+                <SequenceIndex value="0"/>
+                <LookupListIndex value="1"/>
+              </SubstLookupRecord>
+            </ChainSubClassRule>
+            <ChainSubClassRule index="1">
+              <!-- BacktrackGlyphCount=0 -->
+              <!-- InputGlyphCount=1 -->
+              <!-- LookAheadGlyphCount=2 -->
+              <LookAhead index="0" value="2"/>
+              <LookAhead index="1" value="1"/>
+              <!-- SubstCount=1 -->
+              <SubstLookupRecord index="0">
+                <SequenceIndex value="0"/>
+                <LookupListIndex value="1"/>
+              </SubstLookupRecord>
+            </ChainSubClassRule>
+          </ChainSubClassSet>
+        </ChainContextSubst>
+        <MarkFilteringSet value="0"/>
+      </Lookup>
+      <Lookup index="1">
+        <LookupType value="2"/>
+        <LookupFlag value="16"/><!-- useMarkFilteringSet -->
+        <!-- SubTableCount=1 -->
+        <MultipleSubst index="0">
+          <Substitution in="idotbelow" out="idotless,dotbelowcomb"/>
+          <Substitution in="iogonek" out="idotless,ogonekcomb"/>
+        </MultipleSubst>
+        <MarkFilteringSet value="0"/>
+      </Lookup>
+    </LookupList>
+  </GSUB>
+
+</ttFont>


Test patch:
diff --git a/Tests/feaLib/builder_test.py b/Tests/feaLib/builder_test.py
index 4cb9b4a1d2..e2f6a1d4d9 100644
--- a/Tests/feaLib/builder_test.py
+++ b/Tests/feaLib/builder_test.py
@@ -48,6 +48,7 @@ def makeTTFont():
         grave acute dieresis macron circumflex cedilla umlaut ogonek caron
         damma hamza sukun kasratan lam_meem_jeem noon.final noon.initial
         by feature lookup sub table uni0327 uni0328 e.fina
+        idotbelow idotless iogonek acutecomb brevecomb ogonekcomb dotbelowcomb
     """.split()
     glyphs.extend("cid{:05d}".format(cid) for cid in range(800, 1001 + 1))
     font = TTFont()
@@ -82,6 +83,7 @@ class BuilderTest(unittest.TestCase):
         GSUB_5_formats delete_glyph STAT_test STAT_test_elidedFallbackNameID
         variable_scalar_valuerecord variable_scalar_anchor variable_conditionset
         variable_mark_anchor duplicate_lookup_reference
+        contextual_inline_multi_sub_format_2
     """.split()
 
     VARFONT_AXES = [