Repository: fonttools/fonttools (2024-05-23 ~ 2024-07-03)

problem_statement: feaLib: handling of duplicate lookup references
Given the following FEA:

```fea
languagesystem DFLT dflt;
languagesystem latn dflt;

feature test {
    lookup alt1 {
	sub a by A;
    } alt1;
    lookup alt2 {
        sub b by B;
    } alt2;


    script latn;
    lookup alt1;

} test;
```

because `latn dflt` is one of the declared language systems, it inherits the two lookups that are declared at the top of the feature. The reference to `lookup alt1` under `latn` is redundant.

Currently, feaLib adds an extra lookup for this feature, even though it occurs earlier in the feature list. `makeotf` ignores the duplicate lookup, and the `latn/dflt` feature includes the two lookups inherited from `DFLT/dflt`.

I think that feaLib is incorrect here, although I'm not sure if this results in just slightly larger files, slightly less efficient shaping, or possibly errors in shaping?

(this test case is a reduction of an issue I found in about a hundred sources used by google fonts, so it may be worth some attention)

date: 2024-05-23
patch: diff --git a/Lib/fontTools/feaLib/builder.py b/Lib/fontTools/feaLib/builder.py
index 7921a3f179..a91381ddc1 100644
--- a/Lib/fontTools/feaLib/builder.py
+++ b/Lib/fontTools/feaLib/builder.py
@@ -880,8 +880,13 @@ def makeTable(self, tag):
             # l.lookup_index will be None when a lookup is not needed
             # for the table under construction. For example, substitution
             # rules will have no lookup_index while building GPOS tables.
+            # We also deduplicate lookup indices, as they only get applied once
+            # within a given feature:
+            # https://github.com/fonttools/fonttools/issues/2946
             lookup_indices = tuple(
-                [l.lookup_index for l in lookups if l.lookup_index is not None]
+                dict.fromkeys(
+                    l.lookup_index for l in lookups if l.lookup_index is not None
+                )
             )
 
             size_feature = tag == "GPOS" and feature_tag == "size"
diff --git a/Tests/feaLib/data/duplicate_lookup_reference.fea b/Tests/feaLib/data/duplicate_lookup_reference.fea
new file mode 100644
index 0000000000..f2746520e0
--- /dev/null
+++ b/Tests/feaLib/data/duplicate_lookup_reference.fea
@@ -0,0 +1,18 @@
+# https://github.com/fonttools/fonttools/issues/2946
+
+languagesystem DFLT dflt;
+languagesystem latn dflt;
+
+feature test {
+    lookup alt1 {
+	sub a by A;
+    } alt1;
+    lookup alt2 {
+        sub b by B;
+    } alt2;
+
+
+    script latn;
+    lookup alt1;
+
+} test;
diff --git a/Tests/feaLib/data/duplicate_lookup_reference.ttx b/Tests/feaLib/data/duplicate_lookup_reference.ttx
new file mode 100644
index 0000000000..e8be775c53
--- /dev/null
+++ b/Tests/feaLib/data/duplicate_lookup_reference.ttx
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<ttFont>
+
+  <GSUB>
+    <Version value="0x00010000"/>
+    <ScriptList>
+      <!-- ScriptCount=2 -->
+      <ScriptRecord index="0">
+        <ScriptTag value="DFLT"/>
+        <Script>
+          <DefaultLangSys>
+            <ReqFeatureIndex value="65535"/>
+            <!-- FeatureCount=1 -->
+            <FeatureIndex index="0" value="0"/>
+          </DefaultLangSys>
+          <!-- LangSysCount=0 -->
+        </Script>
+      </ScriptRecord>
+      <ScriptRecord index="1">
+        <ScriptTag value="latn"/>
+        <Script>
+          <DefaultLangSys>
+            <ReqFeatureIndex value="65535"/>
+            <!-- FeatureCount=1 -->
+            <FeatureIndex index="0" value="0"/>
+          </DefaultLangSys>
+          <!-- LangSysCount=0 -->
+        </Script>
+      </ScriptRecord>
+    </ScriptList>
+    <FeatureList>
+      <!-- FeatureCount=1 -->
+      <FeatureRecord index="0">
+        <FeatureTag value="test"/>
+        <Feature>
+          <!-- LookupCount=2 -->
+          <LookupListIndex index="0" value="0"/>
+          <LookupListIndex index="1" value="1"/>
+        </Feature>
+      </FeatureRecord>
+    </FeatureList>
+    <LookupList>
+      <!-- LookupCount=2 -->
+      <Lookup index="0">
+        <LookupType value="1"/>
+        <LookupFlag value="0"/>
+        <!-- SubTableCount=1 -->
+        <SingleSubst index="0">
+          <Substitution in="a" out="A"/>
+        </SingleSubst>
+      </Lookup>
+      <Lookup index="1">
+        <LookupType value="1"/>
+        <LookupFlag value="0"/>
+        <!-- SubTableCount=1 -->
+        <SingleSubst index="0">
+          <Substitution in="b" out="B"/>
+        </SingleSubst>
+      </Lookup>
+    </LookupList>
+  </GSUB>
+
+</ttFont>

test_patch: diff --git a/Tests/feaLib/builder_test.py b/Tests/feaLib/builder_test.py
index 875223af74..4cb9b4a1d2 100644
--- a/Tests/feaLib/builder_test.py
+++ b/Tests/feaLib/builder_test.py
@@ -81,7 +81,7 @@ class BuilderTest(unittest.TestCase):
         MultipleLookupsPerGlyph MultipleLookupsPerGlyph2 GSUB_6_formats
         GSUB_5_formats delete_glyph STAT_test STAT_test_elidedFallbackNameID
         variable_scalar_valuerecord variable_scalar_anchor variable_conditionset
-        variable_mark_anchor
+        variable_mark_anchor duplicate_lookup_reference
     """.split()
 
     VARFONT_AXES = [


----

problem_statement: [feaLib] possibly suboptimal compilation of inline multiple substitution rules
I believe this is similar to https://github.com/fonttools/fonttools/issues/2150, but for multiple substitution instead of single substitution.

Given the following input,

```fea
# reduced from the ccmp feature in oswald
feature ccmp {
    lookup ccmp_Other_1 {
        @CombiningTopAccents = [acutecomb brevecomb];
        @CombiningNonTopAccents = [dotbelowcomb ogonekcomb];
        lookupflag UseMarkFilteringSet @CombiningTopAccents;
        sub idotbelow' @CombiningTopAccents by idotless dotbelowcomb;
        sub iogonek' @CombiningTopAccents by idotless ogonekcomb;
        sub idotbelow' @CombiningNonTopAccents @CombiningTopAccents by idotless dotbelowcomb;
        sub iogonek' @CombiningNonTopAccents @CombiningTopAccents by idotless ogonekcomb;
    } ccmp_Other_1;
} ccmp;
```

`fontmake` generates five lookups; one contextual lookup, and then one lookup for each inline sub statement. This means that there are two sets of duplicate lookups, i.e. two copies of `sub iodotbelow by idotless dotbelowcomb;` and two copies of `sub iogonek by idotless ogonekcomb;`.


`fontc`, for this input, generates two identical lookups, where each lookup contains both of the substitutions.

I am not sure that either is correct; I think we should be able to get away with a single lookup, containing both substitutions.


_This has unexpected implications!_ specifically, the presence of duplicate identical lookups (which will be deduplicated during table packing) has unexpectedly (to me) severe impact on binary size, because it means that we can not efficiently deduplicate various `ChainedClassSequenceRule` and `ChainedClassSequenceRuleSet` tables, since even though they are referencing semantically identical lookups those lookups have distinct lookup ids, which is the difference between 140 bytes for the fontmake table and 102 bytes for `fontc`.


Does my reasoning make sense? can we be more aggressive in reusing lookups in this case?

date: 2024-06-03
patch: diff --git a/Lib/fontTools/feaLib/builder.py b/Lib/fontTools/feaLib/builder.py
index a91381ddc1..bda855e1e9 100644
--- a/Lib/fontTools/feaLib/builder.py
+++ b/Lib/fontTools/feaLib/builder.py
@@ -1286,10 +1286,7 @@ def add_multiple_subst(
         self, location, prefix, glyph, suffix, replacements, forceChain=False
     ):
         if prefix or suffix or forceChain:
-            chain = self.get_lookup_(location, ChainContextSubstBuilder)
-            sub = self.get_chained_lookup_(location, MultipleSubstBuilder)
-            sub.mapping[glyph] = replacements
-            chain.rules.append(ChainContextualRule(prefix, [{glyph}], suffix, [sub]))
+            self.add_multi_subst_chained_(location, prefix, glyph, suffix, replacements)
             return
         lookup = self.get_lookup_(location, MultipleSubstBuilder)
         if glyph in lookup.mapping:
@@ -1369,7 +1366,7 @@ def add_single_subst_chained_(self, location, prefix, suffix, mapping):
         # https://github.com/fonttools/fonttools/issues/512
         # https://github.com/fonttools/fonttools/issues/2150
         chain = self.get_lookup_(location, ChainContextSubstBuilder)
-        sub = chain.find_chainable_single_subst(mapping)
+        sub = chain.find_chainable_subst(mapping, SingleSubstBuilder)
         if sub is None:
             sub = self.get_chained_lookup_(location, SingleSubstBuilder)
         sub.mapping.update(mapping)
@@ -1377,6 +1374,19 @@ def add_single_subst_chained_(self, location, prefix, suffix, mapping):
             ChainContextualRule(prefix, [list(mapping.keys())], suffix, [sub])
         )
 
+    def add_multi_subst_chained_(self, location, prefix, glyph, suffix, replacements):
+        if not all(prefix) or not all(suffix):
+            raise FeatureLibError(
+                "Empty glyph class in contextual substitution", location
+            )
+        # https://github.com/fonttools/fonttools/issues/3551
+        chain = self.get_lookup_(location, ChainContextSubstBuilder)
+        sub = chain.find_chainable_subst({glyph: replacements}, MultipleSubstBuilder)
+        if sub is None:
+            sub = self.get_chained_lookup_(location, MultipleSubstBuilder)
+        sub.mapping[glyph] = replacements
+        chain.rules.append(ChainContextualRule(prefix, [{glyph}], suffix, [sub]))
+
     # GSUB 8
     def add_reverse_chain_single_subst(self, location, old_prefix, old_suffix, mapping):
         if not mapping:
diff --git a/Lib/fontTools/otlLib/builder.py b/Lib/fontTools/otlLib/builder.py
index 70fd87ab57..8fc685683a 100644
--- a/Lib/fontTools/otlLib/builder.py
+++ b/Lib/fontTools/otlLib/builder.py
@@ -544,6 +544,10 @@ def buildFormat2Subtable(self, ruleset, classdefs, chaining=True):
                 f"{classRuleAttr}Count",
                 getattr(setForThisRule, f"{classRuleAttr}Count") + 1,
             )
+        for i, classSet in enumerate(classSets):
+            if not getattr(classSet, classRuleAttr):
+                # class sets can be null so replace nop sets with None
+                classSets[i] = None
         setattr(st, self.ruleSetAttr_(format=2, chaining=chaining), classSets)
         setattr(
             st, self.ruleSetAttr_(format=2, chaining=chaining) + "Count", len(classSets)
@@ -781,14 +785,14 @@ def getAlternateGlyphs(self):
                             )
         return result
 
-    def find_chainable_single_subst(self, mapping):
-        """Helper for add_single_subst_chained_()"""
+    def find_chainable_subst(self, mapping, builder_class):
+        """Helper for add_{single,multi}_subst_chained_()"""
         res = None
         for rule in self.rules[::-1]:
             if rule.is_subtable_break:
                 return res
             for sub in rule.lookups:
-                if isinstance(sub, SingleSubstBuilder) and not any(
+                if isinstance(sub, builder_class) and not any(
                     g in mapping and mapping[g] != sub.mapping[g] for g in sub.mapping
                 ):
                     res = sub
diff --git a/Tests/feaLib/data/contextual_inline_multi_sub_format_2.fea b/Tests/feaLib/data/contextual_inline_multi_sub_format_2.fea
new file mode 100644
index 0000000000..9a9bfe20de
--- /dev/null
+++ b/Tests/feaLib/data/contextual_inline_multi_sub_format_2.fea
@@ -0,0 +1,17 @@
+# reduced from the ccmp feature in Oswald
+feature ccmp {
+    lookup ccmp_Other_1 {
+        @CombiningTopAccents = [acutecomb brevecomb];
+        @CombiningNonTopAccents = [dotbelowcomb ogonekcomb];
+        lookupflag UseMarkFilteringSet @CombiningTopAccents;
+        # we should only generate two lookups; one contextual and one multiple sub,
+        # containing 'sub idotbelow by idotless dotbelowcomb' and
+        # 'sub iogonek by idotless ogonekcomb'
+        sub idotbelow' @CombiningTopAccents by idotless dotbelowcomb;
+        sub iogonek' @CombiningTopAccents by idotless ogonekcomb;
+        sub idotbelow' @CombiningNonTopAccents @CombiningTopAccents by idotless dotbelowcomb;
+        sub iogonek' @CombiningNonTopAccents @CombiningTopAccents by idotless ogonekcomb;
+    } ccmp_Other_1;
+} ccmp;
+
+
diff --git a/Tests/feaLib/data/contextual_inline_multi_sub_format_2.ttx b/Tests/feaLib/data/contextual_inline_multi_sub_format_2.ttx
new file mode 100644
index 0000000000..d09431fa49
--- /dev/null
+++ b/Tests/feaLib/data/contextual_inline_multi_sub_format_2.ttx
@@ -0,0 +1,135 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<ttFont sfntVersion="\x00\x01\x00\x00" ttLibVersion="4.53">
+
+  <GDEF>
+    <Version value="0x00010002"/>
+    <MarkGlyphSetsDef>
+      <MarkSetTableFormat value="1"/>
+      <!-- MarkSetCount=1 -->
+      <Coverage index="0">
+        <Glyph value="acutecomb"/>
+        <Glyph value="brevecomb"/>
+      </Coverage>
+    </MarkGlyphSetsDef>
+  </GDEF>
+
+  <GSUB>
+    <Version value="0x00010000"/>
+    <ScriptList>
+      <!-- ScriptCount=1 -->
+      <ScriptRecord index="0">
+        <ScriptTag value="DFLT"/>
+        <Script>
+          <DefaultLangSys>
+            <ReqFeatureIndex value="65535"/>
+            <!-- FeatureCount=1 -->
+            <FeatureIndex index="0" value="0"/>
+          </DefaultLangSys>
+          <!-- LangSysCount=0 -->
+        </Script>
+      </ScriptRecord>
+    </ScriptList>
+    <FeatureList>
+      <!-- FeatureCount=1 -->
+      <FeatureRecord index="0">
+        <FeatureTag value="ccmp"/>
+        <Feature>
+          <!-- LookupCount=1 -->
+          <LookupListIndex index="0" value="0"/>
+        </Feature>
+      </FeatureRecord>
+    </FeatureList>
+    <LookupList>
+      <!-- LookupCount=2 -->
+      <Lookup index="0">
+        <LookupType value="6"/>
+        <LookupFlag value="16"/><!-- useMarkFilteringSet -->
+        <!-- SubTableCount=1 -->
+        <ChainContextSubst index="0" Format="2">
+          <Coverage>
+            <Glyph value="idotbelow"/>
+            <Glyph value="iogonek"/>
+          </Coverage>
+          <BacktrackClassDef>
+          </BacktrackClassDef>
+          <InputClassDef>
+            <ClassDef glyph="idotbelow" class="1"/>
+            <ClassDef glyph="iogonek" class="2"/>
+          </InputClassDef>
+          <LookAheadClassDef>
+            <ClassDef glyph="acutecomb" class="1"/>
+            <ClassDef glyph="brevecomb" class="1"/>
+            <ClassDef glyph="dotbelowcomb" class="2"/>
+            <ClassDef glyph="ogonekcomb" class="2"/>
+          </LookAheadClassDef>
+          <!-- ChainSubClassSetCount=3 -->
+          <ChainSubClassSet index="0" empty="1"/>
+          <ChainSubClassSet index="1">
+            <!-- ChainSubClassRuleCount=2 -->
+            <ChainSubClassRule index="0">
+              <!-- BacktrackGlyphCount=0 -->
+              <!-- InputGlyphCount=1 -->
+              <!-- LookAheadGlyphCount=1 -->
+              <LookAhead index="0" value="1"/>
+              <!-- SubstCount=1 -->
+              <SubstLookupRecord index="0">
+                <SequenceIndex value="0"/>
+                <LookupListIndex value="1"/>
+              </SubstLookupRecord>
+            </ChainSubClassRule>
+            <ChainSubClassRule index="1">
+              <!-- BacktrackGlyphCount=0 -->
+              <!-- InputGlyphCount=1 -->
+              <!-- LookAheadGlyphCount=2 -->
+              <LookAhead index="0" value="2"/>
+              <LookAhead index="1" value="1"/>
+              <!-- SubstCount=1 -->
+              <SubstLookupRecord index="0">
+                <SequenceIndex value="0"/>
+                <LookupListIndex value="1"/>
+              </SubstLookupRecord>
+            </ChainSubClassRule>
+          </ChainSubClassSet>
+          <ChainSubClassSet index="2">
+            <!-- ChainSubClassRuleCount=2 -->
+            <ChainSubClassRule index="0">
+              <!-- BacktrackGlyphCount=0 -->
+              <!-- InputGlyphCount=1 -->
+              <!-- LookAheadGlyphCount=1 -->
+              <LookAhead index="0" value="1"/>
+              <!-- SubstCount=1 -->
+              <SubstLookupRecord index="0">
+                <SequenceIndex value="0"/>
+                <LookupListIndex value="1"/>
+              </SubstLookupRecord>
+            </ChainSubClassRule>
+            <ChainSubClassRule index="1">
+              <!-- BacktrackGlyphCount=0 -->
+              <!-- InputGlyphCount=1 -->
+              <!-- LookAheadGlyphCount=2 -->
+              <LookAhead index="0" value="2"/>
+              <LookAhead index="1" value="1"/>
+              <!-- SubstCount=1 -->
+              <SubstLookupRecord index="0">
+                <SequenceIndex value="0"/>
+                <LookupListIndex value="1"/>
+              </SubstLookupRecord>
+            </ChainSubClassRule>
+          </ChainSubClassSet>
+        </ChainContextSubst>
+        <MarkFilteringSet value="0"/>
+      </Lookup>
+      <Lookup index="1">
+        <LookupType value="2"/>
+        <LookupFlag value="16"/><!-- useMarkFilteringSet -->
+        <!-- SubTableCount=1 -->
+        <MultipleSubst index="0">
+          <Substitution in="idotbelow" out="idotless,dotbelowcomb"/>
+          <Substitution in="iogonek" out="idotless,ogonekcomb"/>
+        </MultipleSubst>
+        <MarkFilteringSet value="0"/>
+      </Lookup>
+    </LookupList>
+  </GSUB>
+
+</ttFont>

test_patch: diff --git a/Tests/feaLib/builder_test.py b/Tests/feaLib/builder_test.py
index 4cb9b4a1d2..e2f6a1d4d9 100644
--- a/Tests/feaLib/builder_test.py
+++ b/Tests/feaLib/builder_test.py
@@ -48,6 +48,7 @@ def makeTTFont():
         grave acute dieresis macron circumflex cedilla umlaut ogonek caron
         damma hamza sukun kasratan lam_meem_jeem noon.final noon.initial
         by feature lookup sub table uni0327 uni0328 e.fina
+        idotbelow idotless iogonek acutecomb brevecomb ogonekcomb dotbelowcomb
     """.split()
     glyphs.extend("cid{:05d}".format(cid) for cid in range(800, 1001 + 1))
     font = TTFont()
@@ -82,6 +83,7 @@ class BuilderTest(unittest.TestCase):
         GSUB_5_formats delete_glyph STAT_test STAT_test_elidedFallbackNameID
         variable_scalar_valuerecord variable_scalar_anchor variable_conditionset
         variable_mark_anchor duplicate_lookup_reference
+        contextual_inline_multi_sub_format_2
     """.split()
 
     VARFONT_AXES = [


----

problem_statement: Visitor pattern does not search inheritance chain
Unfortunately, I can't do this:

```python
from fontTools.feaLib import ast

@GlyphFilterVisitor.register(ast.Statement)
def visit(visitor, statement, *args, **kwargs):
    print("This never fires")
```

This is because the visitor looks for a registered `visitor` function for the exact class name of the object, and does not search the class hierarchy to find any superclass visitors.

date: 2024-07-03
patch: diff --git a/Lib/fontTools/misc/visitor.py b/Lib/fontTools/misc/visitor.py
index d289895467..6de432ef93 100644
--- a/Lib/fontTools/misc/visitor.py
+++ b/Lib/fontTools/misc/visitor.py
@@ -61,9 +61,10 @@ def _visitorsFor(celf, thing, _default={}):
             if _visitors is None:
                 break
 
-            m = celf._visitors.get(typ, None)
-            if m is not None:
-                return m
+            for base in typ.mro():
+                m = celf._visitors.get(base, None)
+                if m is not None:
+                    return m
 
         return _default
 

test_patch: diff --git a/Tests/misc/visitor_test.py b/Tests/misc/visitor_test.py
index 268cc716d7..eb1969ea86 100644
--- a/Tests/misc/visitor_test.py
+++ b/Tests/misc/visitor_test.py
@@ -24,6 +24,10 @@ def __init__(self):
         self.a = A()
 
 
+class C(B):
+    pass
+
+
 class TestVisitor(Visitor):
     def __init__(self):
         self.value = []
@@ -71,3 +75,9 @@ def test_visitor(self):
         visitor.defaultStop = True
         visitor.visit(b)
         assert visitor.value == ["B", "B a"]
+
+    def test_visitor_inheritance(self):
+        b = C()  # Should behave just like a B()
+        visitor = TestVisitor()
+        visitor.visit(b)
+        assert visitor.value == ["B", "B a", "A", 1, 2, 3, 5, 7, "e", E.E2, 10]