LexSym: Compositionality as Lexical Symmetry
Year: 2023

Abstract:
In tasks like semantic parsing, instruction following, and question answering, standard deep networks fail to generalize compositionally from small datasets. Many existing approaches overcome this limitation with model architectures that enforce a compositional process of sentence interpretation. In this paper, we present a domain-general and model-agnostic formulation of compositionality as a constraint on symmetries of data distributions rather than models. Informally, we prove that whenever a task can be solved by a compositional model, there is a corresponding data augmentation scheme—a procedure for transforming examples into other well-formed examples—that imparts compositional inductive bias on any model trained to solve the same task. We describe a procedure called LEXSYM that discovers these transformations automatically, then applies them to training data for ordinary neural sequence models. Unlike existing compositional data augmentation procedures, LEXSYM can be deployed agnostically across text, structured data, and even images. It matches or surpasses state-of-the-art, task-specific models on COGS semantic parsing, SCAN and ALCHEMY instruction following, and CLEVR-COGENT visual question answering datasets.

heading: 1 Introduction
text: A central challenge in natural language processing is the design of models and learning algorithms that are simultaneously flexible enough to capture the variability of human language and structured enough to generalize in predictable and humanlike ways. One important source of structure is the principle of compositionality, which (in one formulation) states that sentence meanings can be computed from a lexicon of word meanings and a set of composition rules governing how meanings combine (Montague, 1970b). A long line of language processing research has operationalized the principle of compositionality as a constraint on
model architectures, via independence assumptions or parameter tying schemes that ensure a compositional process of sentence interpretation (Lewis and Stearns, 1968; Andreas et al., 2016). Compositional models enjoy sample-efficient learning and strong generalization in tasks from machine translation to question answering (McCoy et al., 2020).
But much of human language is not (or at least not straightforwardly) compositional. Idioms, disfluencies, and context-sensitive meanings present major challenges to models in which all predictions must derive from a sequence of local composition operations. In recent years, more generic model architectures such as recurrent neural networks (RNNs) and transformers, with no explicit compositional scaffolding, have consistently outperformed compositional models in language processing tasks with natural data (Wu et al., 2016). However, these models capture linguistic regularities only when trained on enormous amounts of data, and make surprising or problematic predictions when presented with novel word collocations or syntactic structures (Lake and Baroni, 2018).
How can we train unstructured neural sequence models that generalize compositionally? Recent work has introduced several compositional data augmentation schemes: rule-based procedures or learned models that synthesize artificial training examples to promote generalization (Andreas, 2020; Shaw et al., 2021; Akyürek et al., 2021; Zhang et al., 2022, inter alia). While often effective, existing methods are specialized to specific data modalities or datasets. The conditions under which they succeed, and their relationships to the formal principle of compositionality, have remained unclear.
This paper presents a framework for understanding and improving such data-centric approaches to compositional modeling. We first provide a mathematical characterization of the principle of compositionality as a constraint on data distributions rather than model architectures. Intuitively,
639
we show that whenever a language understanding task can be solved compositionally, that task’s data distribution is guaranteed to exhibit specific symmetries. These symmetries are functions that modify data points while preserving semantic acceptability. Fig. 1c gives an example of a symmetry in a visual question answering problem: in any wellformed (image, question, answer) triple, swapping the words yellow and green and their associated pixel values yields a valid new triple. Such symmetries exist even in complex tasks like instruction following (Fig. 1a), where they may depend not only on word-to-meaning mappings but relations between meanings (like the fact that red and green mix to produce brown).
Building on this formal link between compositionality and symmetry, we introduce a procedure called LEXSYM that discovers symmetries automatically, then uses them to synthesize new training examples guaranteed to be correct and informative. Crucially, LEXSYM does not require a complete compositional theory for a given problem domain—only a lexicon of word meanings. These lexicons may themselves be automatically derived for most tasks. This makes LEXSYM very flexible: it requires little or no task-specific engineering, can be combined with any predictor, and unlike other compositional data augmentation schemes does not require tree-structured or even sequential data.
Applied to ordinary neural sequence models, LEXSYM outperforms state-of-the-art models on the CLEVR COGENT visual question answering benchmark (Johnson et al., 2017) by a wide margin. LEXSYM is general, and matches or outperforms some specialized data augmentation schemes and models on the COGS semantic parsing task (Kim
and Linzen, 2020; Kim et al., 2022), and the SCAN and ALCHEMY instruction following tasks (Lake and Baroni, 2018; Long et al., 2016).
This paper thus offers two contributions: a theoretical contribution, in the form of a new lens on the principle of compositionality via symmetries of data distributions; and an empirical contribution, in the form of a data augmentation scheme that improves generalization on diverse language understanding tasks. The recent success of data augmentation approaches highlight the fact that compositional inductive bias need not require compositional models. Our work formalizes and generalizes this “data-centric” account of compositionality.1

heading: 2 Background & Approach
text: We begin with a discussion on the more general role of symmetry in machine learning applications.
Definition 1. A symmetry of a set X is a function f satisfying:
{f(x) : x ∈ X} = X (1)
That is, applying f to each element of X leaves X unchanged.
A familiar example from computer vision is reflection symmetry: in object recognition problems, image classes are generally invariant under reflection (a zebra seen in a mirror is still a zebra). The set of (image, class) pairs thus has as a symmetry the function (x, y) 7→ (reflect(x), y). In many domains, especially those (like computer vision and computational chemistry) that are constrained by physical laws, knowledge of the symmetries
1Code will be released after the anonymity period.
exhibited by a problem domain can dramatically reduce the difficulty of learning (Batzner et al., 2022; Simeonov et al., 2022).
Past work has incorporated symmetry into machine learning problems in two ways. Invariant and equivariant modeling approaches structurally enforce symmetries via specialized architectures (improving generalization by decreasing the size of the hypothesis class; Cohen and Welling, 2016). Data augmentation approaches generate new training examples by applying known symmetries like reflections directly to training data (improving generalization by increasing dataset size; Shorten and Khoshgoftaar, 2019). Data augmentation, the focus of this paper, is model-agnostic, and can be used in conjunction with pre-training while producing the same asymptotic effects as specialized model architectures (Chen et al., 2020).
The question this paper aims to answer is whether compositionality, like other domainspecific constraints, can be formalized in the language of symmetry. We are not the first to consider this question: Kiddon and Domingos (2015) define a theory of semantic equivalence in terms of symmetries of the set of natural language sentences, and Gordon et al. (2020) propose a model architecture for compositional semantic parsing via a symmetry that enforces permutation invariance of lexicon entries. LEXSYM also derives symmetries from lexicons. It builds on past work by (1) characterizing the algebraic relationship between compositionality and symmetry, explaining the effectiveness of both Gordon et al. (2020)’s approach as well as other data augmentation schemes based on token and phrase substitution (Andreas, 2020; Wang et al., 2018); (2) discovering symmetries automatically, and (3) showing how to leverage them in a model- and modality-agnostic way. Additional related work is discussed in Sec. 6.

heading: 3 Compositionality as Lexical Symmetry
text: Our main theoretical result, and the foundation of our modeling approach, can be stated as follows: in any language understanding task that can be modeled compositionally, data for the task exhibits symmetries in the sense of Definition 1. We explain, formalize, and prove this statement below.
We consider tasks defined by a space of possible examples X , of which a subset of examples X are well-formed. We assume each example x ∈ X is a discrete sequence [x1, . . . , xn], with xi
drawn from a vocabulary Σ. Finally, we assume that well-formedness can be computed by a a binary interpretation function I : X → {0, 1} with I(x) = 1 iff x ∈ X . A wide variety of language understanding problems, from very simple to very complex, may be defined in this way:
Example 1a: Arithmetic Language Modeling. Examples x are true sentences of the form a plus b is c, where a, b and c are numbers: I(one plus two is three) = 1 but I(two plus two is five) = 0. Example 1b: Semantic Parsing. Examples x are pairs (xNL,xLF), where xNL is an sentence, xLF is a logical form, and I(xNL,xLF) = 1 iff xLF represents a possible meaning of xNL (Fig. 1b).
Example 1c: Visual Question Answering. Examples x are triples (xQ,xI,xA), where xQ is a question, xI is a (rasterized) image, xA is an answer, and I(xQ,xI,xA) = 1 iff xA is the answer to xQ in xI (Fig. 1c).
Notice that the vocabulary Σ contains not just natural language words, but other kinds of data: logical symbols (1b) or even image patches (1c).
“Language understanding” in each of these tasks is encapsulated by the function I. What does it mean for I to be compositional? Under most definitions, a compositional language understanding procedure should factorize into a lexicon, which captures meanings of words, and a composition procedure, which derives example-level interpretations from these meanings. We model word meanings in terms of relations between items in Σ. In arithmetic, to know the meaning of the word five is to know that it is a number, less than seven, the successor of four, etc. In semantic parsing, the meaning of the word cat is encapsulated by the fact that it is of the same type as dog, and translatable into the logical symbol cat′. We model this notion of word meaning by equipping Σ with extra structure describing these relations:
Definition 2. A lexical algebra is a collection of relations r1, . . . , rn between vocabulary items, where each r : Σp → {0, 1}. A lexical algebra can represent type information, like “dog is a noun”, as a unary relation; semantic correspondence, like “sings maps to sing′”, as a binary relation; and richer semantic knowledge, like “three is the sum of one and two”, with higher-order relations.
We may then represent individual examples in purely relational terms:
Definition 3. Denote the lexical representation L(x) = (R1(x), . . . , Rn(x)). R(x) is an orderp tensor whose (i, . . . , j)th entry is equal to r(xi, . . . , xj). (If r is a binary relation, R(x) is an |x| × |x| matrix and R(x)ij specifies whether r holds between xi and xj .) See Fig. 2 for examples.
Finally, we use this relational representation to define compositionality of interpretation functions:
Definition 4. X is L-compositional if I(x) = C(L(x)) for some composition procedure C. In other words, X is compositional if it compute the well-formedness of x from word-level meanings and a generic composition procedure.2
This definition makes no assumptions about C beyond the fact that it can be defined purely in terms of L(x). It can be applied to many tasks: Example 2a: Arithmetic Language Modeling. Define r1 to be the ternary relation (a, b, c) 7→ 1[a+b=c]. Then C takes an example and checks whether the index corresponding to its three number words is true in R1.
Example 2b: Semantic Parsing. A sketch of a 2Every I is trivially L-compositional with respect to an L that assigns every vocabulary item to a unique unary relation.
semantic parser factorizable into a lexicon and an abstract composition function is depicted in Fig. 2. As a real-world example, in the factored CCG semantic parser of Kwiatkowski et al. (2011), words are assigned types and logical forms via a lexicon. These logical fragments are then composed by a parsing algorithm that depends only their types.
Example 2c: Natural Language Inference. MacCartney and Manning (2014)’s Natural Logic framework provides a procedure for determining entailment relations between sentences via a set of sentence rewriting operations that use only wordlevel information about entailment relations.
Under Definition 4, a sentence interpretation procedure is compositional if the meaning of a sentence can be derived in a generic way (C) from the meanings of its lexical items (L).3 We remark, finally, that the parsing procedure depicted in Fig. 2 is an idealization used to motivate our approach; our experiments use more flexible models.
We are now ready to describe how, for compositional I, structure in L translates into structure in the set of well-formed examples X . Definition 5. A function f is a homomorphism of (Σ,L) (an “L-homomorphism”) if:
∀r ∈ L, ∀x1 . . . xp ∈ Σ : r(x1, . . . , xp) = r(f(x1), . . . , f(xp)) (2)
f “preserves the structure” of L, ensuring that pairwise relationships are preserved among symbols. Fig. 1 shows examples: in (c), for instance, the words yellow and green and the corresponding colors must be swapped to satisfy Eq. 2.
Finally, we may state our main result: Theorem 1. If X is L-compositional, f is an L-homomorphism, and x ∈ X , then f(x) = [f(x1), . . . , f(xn)] ∈ X . Thus every homomorphism of L well-formed examples ∈ X . Proof. From Definition 3 and 5, Ri(f(x)) = Ri(x) ∀i. Then,
1[f(x)∈X] = I(f(x)) = C(L(f(x))) = C(R1(f(x)), . . . , Rn(f(x))) = C(R1(x), . . . , Rn(x)) = I(x) = 1[x∈X]
3As shown in Example 2b, it can be used to implement a language-to-logical form mapping, and thus generalizes the Montagovian definition of compositionality as a homomorphism from sentences to meanings (Montague, 1970a).
Corollary 1. With the additional constraint that f is an L-isomorphism (i.e., has an inverse), then f is a symmetry of X in the sense of Eq. 1.
Here it suffices to show that the preimage of every x ∈ X is also in X; the proof is the same as Theorem 1 with f−1 in place of f .
Despite their simplicity, Theorem 1 and its corollary have an important consequence: if we can identify candidate entries in L, even if C is unknown, we can construct new examples x ∈ X that respect, and provide evidence for, the compositional structure of X . There is an intriguing (if inexact) structural similarity between Corollary 1 and Noether’s theorem (Noether, 1918), which establishes an equivalence between symmetries of physical systems and their conserved quantities. Here, such symmetries imply constraints not on conservation laws but interpretation functions.

heading: 4 LEXSYM: Data Augmentation with
text: L-homomorphisms
Given a lexicon describing symbols and their relations, we have shown how to turn homomorphisms of the lexicon into transformations of a dataset. Each such function f that takes an example x as input, replaces each token xi ∈ x with a new one, and returns a well-formed example x′ as output. Every L-homomorphism may thus be viewed as a recipe for synthesizing training examples from a small initial training set (Japkowicz et al., 2000). However, to make this a practical modeling tool, we need some way of constucting L-homomorphisms for a task of interest. Below, we describe how to do so automatically: first, starting with only a taskspecific lexicon L (Sec. 4.1); next, starting with only a dataset and no initial lexicon (Sec. 4.2). We term the resulting approach LEXSYM.

heading: 4.1 Deriving Homomorphisms from Lexicons
text: Even in complex sequence modeling problems, useful lexicons are often simple enough that they can be specified by hand (Jones et al., 2012; Gordon et al., 2020). Given a pre-specified algebraic L, there is a straightforward procedure for generating the associated symmetries by enumerating all functions Σ → Σ and testing which ones satisfy Eq. 2. (See Algorithm 1 in Appendix B.) This algorithm is inefficient, but simple and practical for small |L|.

heading: 4.2 Deriving Lexicons from Datasets
text: For some tasks, it may be difficult to manually specify an algebraic lexicon. We next describe how to infer one automatically. We focus on an important and extremely common class of language understanding problems with special structure. In semantic parsing and instruction following, examples x consist of (input, output) pairs in which inputs are sentences, outputs are meaning representations, and word meaning is characterized by a lexicon with two components. First, a set of unary type predicates {rτ} that assign words to types (like ENTITY in semantic parsing). Second, a semantic correspondence relation rϵ that specifies which actions or logical symbols can be derived from words (like sings → sing′). With n types, the lexicon required for these problems is L = (rτ1 , . . . , rτn , rϵ), which we abbreviate ({rτk}, rϵ) below. We now show how to improve upon the procedure in Sec. 4.1 by deriving L from data and sampling L-homomorphisms in constant time.
Learning L We build on past work noting that dictionaries of semantic correspondences can be constructed using alignment algorithms (Brown et al., 1993). Given an input x consisting of a pair (xtext,xmeaning), we use existing algorithms to align tokens in individual training examples. Finally, we identify the most frequently occurring alignments and add these to the semantic correspondence relation. We may similarly use existing procedures to infer types by deriving them from part-of-speech tags or distributional patterns. See Appendix D for details of the alignment and type inference algorithms used in our experiments. These algorithms produce lexicons with three properties that are useful for the sampling scheme we describe next: types are disjoint, and semantic correspondences are oneto-many and type-preserving (if two words are of the same type, so are their translations).
Sampling L-homomorphisms Once we have identified types and semantic correspondences, sampling L-homomorphisms is straightforward:
Theorem 2. Let xi and xj ∈ Σ have the same type rτ (xi) = rτ (xj) = 1. For convenience, let Ei = {x : rϵ(xi, x) = 1} denote possible translations of
xi. The f is an L-homomorphism:
f(x) =    xj if x = xi xi if x = xj x′ ∈ Ej if x ∈ Ei x′ ∈ Ei if x ∈ Ej x otherwise
(3)
Proof is given in Appendix A. Theorem 2 yields an intuitive data augmentation procedure: select two (input, output) pairs of the same type, and swap them and any of their meanings wherever they occur. Fig. 1b shows an example. Eq. 3 is related to data augmentation schemes described by Andreas (2020) and Liu et al. (2021b), which synchronously substitute words or phrases (equivalent to removing cases 2 and 4). Unlike LEXSYM, these methods cannot guarantee correctness: in Fig. 1c, substituting green in place of yellow yields an image with two green objects and an incorrect answer.

heading: 5 Experiments
text: Our experiments aim to evaluate whether LEXSYM can improve compositional generalization in downstream models. The main goal of these experiments is to evaluate generality across tasks and data modalities. Evaluation focuses on three diverse classes of language understanding problems: complex, context-dependent computations (Sec. 5.1), large, automatically derived lexicons (Sec. 5.2), and multi-modal data (Sec. 5.3).

heading: 5.1 Complex computations
text: We first test LEXSYM on the ALCHEMY task from the SCONE benchmark (Long et al., 2016)—a problem involving a complex sentence interpretation procedure that makes it challenging to apply existing data augmentation schemes.
Data In ALCHEMY (Fig. 1a), models must execute a sequence of human-written English instructions x1:Nins , on an initial state x 0 state consisting of beakers of colored liquids (textually represented as sequence of symbols “1: g g , 2: ...”), to predict the final state xNstate. Initial and final states are encoded as sequences of color tokens. Predicting final states requires both grounding colors in state variables (brown → b , red → g ) and modeling what happens when colors are combined (e.g. mixing g and r yields b ).
LEXSYM We manually construct a lexicon to showcase how to inject prior knowledge into LEXSYM. We encode word meaning in two relations: a semantic equivalence relation between color words and colors:
rϵ(c1, c2) =    1 c1 = brown, c2 = b 1 c1 = red, c2 = r 1 c1 = green, c2 = g ... 0 otherwise
and a ternary relation that encodes the result of mixing colors:4
rmix(c1, c2, c3) =    1 c1 = c2 = c3
1 c1 ̸= c2 ∧ c3 = b 0 otherwise
Together, (rϵ, rmix, {rτk}), where {rτk} assigns different types to color words, colors, and remaining tokens. The homomorphic transformations of this lexicon exchange color words and colors but preserve mixing relations.
Models and Training We train an LSTM (Hochreiter and Schmidhuber, 1997) and finetune a T5 transformer (Raffel et al., 2020) on the sequence-to-sequence prediction problem (x1:Nins ,x 0 state) → xNstate Training details may be found in Appendix C. We compare these baseline models to their LEXSYM-augmented versions as well as the existing compositional data augmentation scheme of Liu et al. (2021b).
Results See Table 1. LSTM+LEXSYM improves substantially over an LSTM. Preserving the homomorphism condition in Eq. 2 is extremely important: the procedure of Liu et al. (2021b), which naively substitutes aligned color pairs, actually hurts performance. Pre-trained models achieve strong initial results; combining pre-training with LEXSYM gives additional improvements.

heading: 5.2 Learned lexicons
text: We next show that for more conventional sequenceto-sequence problems, we may apply LEXSYM with automatically derived lexicons.
4In ALCHEMY, mixing non-identical colors produces b .
Model ALCHEMY SCAN (jump) SCAN (around right) COGS COGS (nonce)
Previous Work on COGS & SCAN GECA (Andreas, 2020) – 99.94 ±0.10 98.50 ±1.90 47.74 ±4.52 – LeAR (Liu et al., 2021a) – – – 97.70 ±0.70 – LexLSTM (Akyurek and Andreas, 2021) 36.80 ±1.96 99.14 ±1.55 88.41 ±7.35 82.17 ±0.72 81.40 ±0.40
No Pre-training LSTM 41.72 ±1.15 000.41 ±0.34 08.65 ±4.52 61.13 ±4.12 61.13 ±4.12
Data We study two standard compositional generalization benchmarks: the SCAN (Lake and Baroni, 2018) instruction following and COGS (Kim and Linzen, 2020, Fig. 1b) semantic parsing datasets. SCAN consists of simple instruction following tasks in which strings are translated into sequences of actions. We focus on the jump split, which measures models’ ability to compose words that only appeared in isolation during training, and the around right split, which measures generalization to novel collocations. The COGS dataset tests compositional generalization in semantic parsing. The dataset includes English (sentence, logical form) pairs, with systematic differences between train and test set sentence structure. We include a variant containing nonce words (Kim
et al., 2022) to disentangle general compositional skills from lexical knowledge acquired during pretraining. See Appendix G for dataset statistics.
LEXSYM We use automatic lexicon extraction to find semantic correspondence relations (rϵ) and types ({rτk}) as described in Appendix D. Next, we apply swap-based augmentation (Eq. 3).
Models We use the same models as Sec. 5.1, along with a strong semi-structured model, LeAR (Liu et al., 2021a) tailored for COGS, and another substitution based augmentation (Andreas, 2020) tailored for SCAN. Following Akyurek and Andreas (2021), we equip the LSTM for COGS with a copy mechanism as it achieves significantly better results than Kim and Linzen (2020)’s baseline.
Results On SCAN, LEXSYM obtains near-perfect accuracy in both jump and around right splits. On the original COGS datasets, LEXSYM substantially outperforms the LSTM model and GECA augmentation, and is comparable to a neural sequence model specialized for lexical generalization (LexLSTM). Stronger results can be achieved with models specifically tailored toward semantic parsing tasks (LeAR). In both tasks, LEXSYM also improves upon large-scale pre-training.

heading: 5.3 Multi-modal data
text: Finally, we combine learned lexicons with nonsequential data to advance the state of the art on a long-standing visual question answering challenge.
Data The CLEVR dataset (Johnson et al., 2017, Fig. 1c) contains English-language questions about generated 3D scenes containing multiple objects.
Questions involve complex computational operations including quantification, comparison, and spatial reasoning. CLEVR has been a popular testbed for evaluating composition in visual question answering models. Our main experiment uses the COGENT split of the dataset, which focuses on compositional generalization. In the CLEVRCOGENT training set (Split A), which contains roughly 700K (question, image, answer) triples, all cubes are gray, blue, brown or yellow, while all cylinders are red, green, purple or cyan. In the test set (validation set of Split B), these are reversed.
LEXSYM In VQA and other multi-modal tasks, part of the input is continuous (e.g. images and videos). Recent work has shown that it is possible to learn high-quality discrete representations of continuous input data. For example, in the VQVAE model of van den Oord et al. (2017), a continuous image is transformed into a grid of categorical codes, with individual codes representing color, and in some cases materials and illumination (examples in Table 3). We use this discretization procedure for our experiments (see Appendix C.1 for details). We use the same algorithm as previous section to extract lexical relations.
Models Most prior work on visual question answering has used pre-trained convolutional networks to encode images, and recurrent networks to encode questions and generate answers. For experiments on CLEVR, we use a simplified model in which both questions and images are mapped to answers by a transformer model, similarly to Ramesh et al. (2021). See Appendix C.2 for details.
Both LEXSYM augmentation and this VQATransformer model operate over sequences of discrete visual codes produced by a vector-quantized variational autoencoder. Once these discrete representations have been produced, we infer lexicons and perform data augmentation directly to these representations, without re-synthesizing images (though such synthesis is possible, as in Table 3, to interpret model behavior).
The COGENT task is very different from the sequence modeling tasks discussed above: inputs contain many tokens, and the training set is orders of magnitude larger. GECA and CSL-Aug, which have a high polynomial dependence on sequence length, could not be applied as they fail to terminate within a reasonable amount of time.
Results In Table 2, a transformer model with LEXSYM achieves state-of-the-art results on the CLEVR-COGENT dataset, reducing errors by roughly 33% relative to the best existing system. LEXSYM also outperforms substitution based-data augmentation (Liu et al., 2021b), particularly on semantically complex utterances involving quantification (App. Table 4). On the IID CLEVR split, LEXSYM’s performance is comparable to humans, and somewhat behind pre-trained models.

heading: 6 Other Related Work
text: Lexicalized neural models Word-level alignments between input and output sequences were an essential feature of statistical phrase- and treebased sequence models (Chiang et al., 2005; Koehn et al., 2003). Neural scoring functions were sometimes integrated into these models (Misra and Artzi, 2016). Neural models with attention (Bahdanau et al., 2015) do not require explicit alignment, though several pieces of past work have shown that incorporating explicit token-level correspondences improves generalization (Akyurek and Andreas, 2021; Prabhu and Kann, 2020; Pham et al., 2018). The semantic correspondence function in Sec. 4 plays the same role as the input–output dictionary in these methods, but LEXSYM as a whole is more general: it is not restricted to modeling sequenceto-sequence problems, and can infer and exploit correspondence relations between component of an example. To the best of our knowledge, this paper is also the first to make use of token-level alignments in joint neural models of text and images.
Compositionality in representation learning While we have focused on compositionality as a property of data distributions or interpretation functions, another line of work in machine learning and language evolution has studied compositionality as an emergent property of learned representations (Andreas, 2019; Resnick et al., 2019; Brighton and Kirby, 2006). In settings where representational compositionality is desirable (e.g. to train communication protocols that can generalize to new states), LEXSYM might provide a tool for promoting it.
Equivariant Sequence Models As mentioned in Sec. 2, our work builds on existing approaches that control generalization with specialized model architectures designed to be equivariant to permutations of a pre-specified lexicon (if f(x1 · · ·xn) = y1 · · · ym then
f(π(x1) · · ·π(xn)) = π(y1) · · ·π(ym) for a permutation π) (Gordon et al., 2020; White and Cotterell, 2022). LEXSYM differs from these approaches in three ways. First, LEXSYM is modelagnostic and compatible with pre-training. Second, LEXSYM is compatible with (and automatically derives transformations for) more complicated relations than input–output correspondences, making it possible to apply to tasks like ALCHEMY where such relations are important. Finally, LEXSYM gracefully handles (possibly noisy) learned lexicons, making it applicable to tasks like COGENT with complex or uninterpretable token mappings.
Data Augmentation Data augmentation approaches are widely used across machine learning application domains featuring known invariances of the data distribution (Japkowicz et al., 2000; Jia and Liang, 2016; Shaw et al., 2021). Substitutionbased schemes that replace words with synonyms, or synchronously replace words and their translations, are widely used for machine translation and general de-biasing (Liu et al., 2021b; Wang et al., 2018; Wei and Zou, 2019).

heading: 7 Limitations and Future Directions
text: While Sec. 3 characterizes the effect of general Lhomomorphisms, LEXSYM specifically produces single-token swaps. In images represented as discrete symbol sequences, if a single symbol simultaneously encodes multiple visual features (e.g. color and texture), these features will remain entangled in synthesized examples. It will not exchange substructures larger than a single token, and thus will not synthesize examples longer than those already present in the training set (Lake et al., 2019). This is because LEXSYM targets compositionality but not recursion, which is also required to model the full range of human-like generalizations in sequence learning problems.
LEXSYM is also sensitive to the nature of the tokenization scheme itself. In morphologically rich languages, for example, LEXSYM may need to be applied not on top of words or segments, but instead canonicalized morphemes produced by learned morphological analyzers (Narasimhan et al., 2015; Bergmanis and Goldwater, 2017; Cotterell and Schütze, 2018) (analogous to the use of learned image patch representations rather than pixels in our VQA experiments).
Finally, LEXSYM does not induce some of the generalizations obtained other methods for improv-
ing compositional generalization, especially those that exploit extra structure (e.g. tree-shaped inputs and outputs) in the semantic parsing domain (e.g. Liu et al., 2021a). It might serve as a platform for future versions of those methods that offer greater generality and formal guarantees.

heading: 8 Conclusion
text: We have presented LEXSYM, a new data augmentation method that improves compositional generalization of neural models in multiple domains. LEXSYM is derived from a characterization of the principle of compositionality as a constraint on the symmetries of data distributions, and a procedure for automatically identifying these symmetries using token-level alignments. Our results highlight the fact that many inductive biases targeted by specialized models in NLP can be alternatively, and often more flexibly, expressed as a hypothesis about the structure of the distribution to be modeled.

heading: Acknowledgements
text: This work was supported by the MachineLearningApplications initiative at MIT CSAIL, the MIT–IBM Watson AI lab, and the National Science Foundation under grant CCF-2217064. Computing resources were provided by a gift from NVIDIA through the NVAIL program and by the Lincoln Laboratory Supercloud.
Ethics Statement
We do not anticipate any ethical issues associated with the techniques decribed in this paper.

heading: A Proof of Theorem 2
text: Proof. The lexicons that we learn only unary type relations and a semantic correspondence relation L = ({rτk}, rϵ). As noted there, we make the following additional assumptions (satisfied by our lexicon learning algorithms):
(i) Types are disjoint, i.e. every symbol belongs to a single type: ∀x ∈ Σ, |τx| = |{rτk | rτk(x) = 1}| = 1.
(ii) Semantic correspondences are one-to-many from text to meaning. This means that no two text symbols can translate into the same meaning symbol: Ei ∩ Ej = 1xi=xj and all rϵ(x /∈ xtext, y) = rϵ(y, x /∈ xmeaning) = 0.
(iii) Semantic correspondence is type preserving: all symbols in a correspondence class have the same type τei∈Ei = {rτEi}.
To show that f is an L-homomorphism, we want to show that rϵ(f(x1), f(x2)) = rϵ(x1, x2) for any x1, x2. The transformation function and all the definitions are symmetric to indices i and j (i − j symmetry), so it is sufficient to show the correspondence relations stay the same for below cases only:
(a) x1 = xi, x2 = xi:
rϵ(f(xi), f(xi)) = rϵ(xj , xj) = 0 = rϵ(xi, xi)
(by ii)
(b) x1 = xi, x2 = xj :
rϵ(f(xi), f(xj)) = rϵ(xj , xi) = 0 = rϵ(xi, xj)
(by ii)
(c) x1 = xi, x2 ∈ Ei:
rϵ(f(xi), f(x2)) = rϵ(xj , x ′ ∈ Ej)
= 1 = rϵ(xi, x2)
(by definition of Ei and Ej)
(d) x1 = xi, x2 ∈ Ej :
rϵ(f(xi), f(x2)) = rϵ(xj , x ′ ∈ Ei)
= 1xi=xj = rϵ(xi, x2)
(by ii)
(e) x1 = xi, x2 /∈ {{xi} ∪ {xj} ∪ Ei, Ej}:
rϵ(f(xi), f(x2)) = rϵ(xj , x2)
= 0 = rϵ(xi, x2)
(f) x1 = xi, x2 /∈ {{xi} ∪ {xj} ∪ Ei, Ej}: same steps as (e)
(g) x1 ∈ Ei, x2 = xi:
rϵ(f(x1), f(xi)) = rϵ(x ′ ∈ Ej , xj)
= 0 = rϵ(x1, xi)
(by ii)
(h) x1 ∈ Ei, x2 = xj : same steps as (g)
(i) x1 ∈ Ei, x2 ∈ {{xi} ∪ {xj} ∪ Ei, Ej}:
rϵ(f(x1), f(x2)) = rϵ(x ′ ∈ Ej , x2)
= 0 = rϵ(x1, x2)
(by ii)
Finally, we require rτ (x) = rτ (f(x)) for any x and τ . Since we assume all items in Ei belong to a type matching xi (likewise for j), and types are disjoint, this follows immediately from the definition of f , which only swaps symbols of the same type.
B Enumerating L-homomorphisms A simple algorithm is given below:
Algorithm 1 L-homomorphism enumeration input: Lexicon L = (Σ, r1, . . . , rn) for f ∈ ΣΣ do
h← 1 for i = 1..n, xa..xb ∈ Σp do
if r(xa, . . . , xb) ̸= r(f(xa), . . . , f(xb)) then h← 0
end if end for if h then
yield f end if
end for
C Implementation Details
C.1 VQVAE Details We use a discrete variational auto-encoder (van den Oord et al., 2017) to encode the images 16 × 16 grids of discrete codes. We used a code-book with n = 32 tokens associated with d = 64 dimensional
learned latent vectors. The original image size (480, 320) is cropped to (440, 300) and resize our images into (128, 128) pixels. The encoder convolutional neural network has three down-sampling layers which output 16 × 16 × d size hidden representations. For encoder and decoder CNN architectures, we follow the implementation provided in a public Pytorch implementation5 by adding one more up-sampling and down-sampling layer to adjust our image size.
We use exponential moving average to update latent vectors as in official implementation6 We train the model on the images of the same training data and did not use any external data.
We use batch size of 512, and learning rate 0.0003 with the Adam optimizer (Kingma and Ba, 2015). We clip the gradients to 5.0. Hyperparameters were selected by sweeping d over {64, 128}, image sizes over {128, 144}, and n over {24, 32, 48} to maximize the the number of aligned tokens in the lexicon. For each experiments in Table 2, we run VQVAE for 4 random seeds and select the codebook that gives the largest IBM model likelihood for training data. Each experiment takes 10 hours in 4 NVIDIA V100 GPUs.
5https://github.com/ritheshkumar95/ pytorch-vqvae
6https://github.com/deepmind/sonnet/blob/v2/ sonnet/src/nets/vqvae.py
C.2 VQA Transformer Details
The Transformer takes tokenized images xI and the question xQ and outputs answers as follows:
cxI = VQVAEenc(xI)
eQ = WQxQ + 1Dpositional(xQ)
exI = WccxI + 2Dpositional(cx)
h = Transformer([eQ exI ])
xA = argmax softmax(Wprojhstart)
(4)
We follow the hyper-paramters provided in (Popel and Bojar, 2018). Transformers have 4 heads, 512- dimensional hidden vectors (same with embedding sizes) and 10 layers. We provide the dimensions in Eq. 4:
xI : 3× 128× 128 cxI : 32× 16× 16 Wc : 512× 32 exI : 512× (16× 16) eQ : 512× |Vtext| WQ : 512× |Vtext|
h : 512× (|Q|+ 16× 16) hstart : 512× 1 Wproj : 512× |Vtext|
(5)
Models are trained using the Adam optimizer with and Noam learning rate scheduler (Vaswani et al., 2017) with lr = 1.0 and 16k warming steps as provided in Popel and Bojar (2018). We use a batch size of 1024 and we train for 200k steps,
which takes 48 hours on 8 NVIDIA V100 GPUs. In Fig. 3, we provide the sketch of overall pipeline.
C.3 Baselines: LSTM Details
We use the implementation provided by (Akyurek and Andreas, 2021), increasing the number of training iterations from 8k to 15k for augmented training runs in COGS, SCAN datasets. For the ALCHEMY dataset, we optimize iteration count over {8k, 15k, 25k, 50k} based on validation accuracy, and found 25k to be optimal. For the CLEVR dataset, we optimize itreation count over {8k, 15k, 25k, 50k} for CLEVR and CLEVRCOGENT dataset based on CLEVR’s validation accuracy.
C.4 Baselines: T5 Details
We use the Huggingface (Wolf et al., 2019) implementation T5-base model. The difference between our T5 baselines results and the results in Qiu et al. (2022) due to their usage of different intermediate representation for the output in order to keep our evaluation consistent with other previous work. We try to optimize (learning rate, learning rate scheduler) and training parameters (iteration count) of Qiu et al. (2022) and (Akyurek and Andreas, 2021), use the best setting for the given dataset.
C.5 Alignment Model Details
In our experiments, we use the best alignment method reported in (Akyurek and Andreas, 2021), which is IBM Model 2 for all datasets except the SCAN dataset that uses their proposed algorithm, to obtain our initial alignments A = {(xi, xj): set of tuples contains aligned tokens. We run alignment algorithms between xtext and xmeaning. For SCAN and COGS, xtext is the actual inputs, xmeaning is the actual outputs. In ALCHEMY, xtext is instructions, xmeaning is beaker states. In VQA experiments, xtext question and answer words, xmeaning VQVAE codes. We disable diagonalization in FastAlign as it includes non-language structured VQVAE codes.

heading: D Lexicons
text: D.1 Lexicon Learning
Extracting semantic correspondences rϵ(xi, xj) Given the initial alignments A in Appendix C.5, we remove every xj that is not aligned to at least 1% of occurrences of xi in the dataset. We then produce a one-to-many lexicon by deleting lexicon
entries (xi, xj) and (x′i, xj) when both exist. With, these alignment creates entries in rϵ(xi, xj) = 1(xi,xj)∈A
Extracting Types rτ (x) Given the partition of the data points (xtext,xmeaning), our type finding algorithm is essentially unsupervised clustering of the text symbols in xtext. The types of matching xmeaning symbols are automatically determined by the correspondence relation, rϵ found above. In all our datasets xtext is English, so the symbols that goes into following clustering algorithm are actual words.
Following Clark and Eyraud (2007) and Andreas (2020), we assign types to individual words based on their environments. For each symbol, x ∈ Σ, that has at least one equivalent symbol in A, we define the context κ(x) = {(α, β) : αxβ ∈ X}: the set of strings (α, β) that appear surrounding x in the training set. (If the two examples in Fig. 1 formed the entire training set, we would have κ(yellow) = κ(green) = {(Q: How many, objects? A: 1)}.). 7 We then represent Σ as a graph with an edge between each xi and xj where κ(xi) ∩ κ(xj) ̸= ∅ (Clark and Eyraud’s syntactic congruence relation) and xi and xj has same part-of-speech tag according to spaCy pipeline with en-core-web-lm language model 8. We assign each connected component of this graph a distinct type. This is only one possible approach to typing; alternatives might use clustering of distributed representations.
D.2 Extracted Lexicons
In this section, we present lexicon entries for symbols that we learned through our typing algorithm.
SCAN We present equivalance relations that we extracted from SCAN training dataset.
7The environment’s window size w = |α| = |β| is a fixed hyper-parameter similar to Andreas (2020). We optimize it over w ∈ {1, 2, 5, 10, 15} in ALCHEMY dataset (used w = 10) by using the validation set, in CLEVR and CLEVRCOGENT based on CLEVR’s validation set (used w = 10). For COGS and SCAN datasets, we resort w = 1 to enable learning of extremely rare items.
8https://github.com/explosion/spacy-models/ releases/tag/en_core_web_lg-3.5.0
Source symbol Type Target Symbol(s)
jump t1 I_JUMP walk t1 I_WALK run t1 I_RUN look t1 I_LOOK left t2 I_LEFT right t2 I_RIGHT
COGS Since the extracted lexicon is large for semantic parsing, we present only some of the equivalance relations that we extracted from COGS training data for reference.
Source symbol Type Target Symbol(s)
baked t1 bake noticed t1 notice helped t1 help dog t2 dog boy t2 boy sailor t2 sailor
COGENT We present equivalance relations that we extracted CLEVR-COGENT training data. The lexicon we found includes all the color symbols. The target symbols given here are learned VQVAE codes. In Appendix E, we show these codes on top of the images to qualitatively verify the alignments.
Source Symbol Type Target Symbols
red t1 9 purple t1 25, 29 cyan t1 28 blue t1 20 green t1 11 yellow t1 23, 18 gray t1 6 brown t1 2

heading: E Samples & Statistics
text: We present examples generated by LEXSYM in Table 3. As we performed augmentation random and online during training, and we do not have a static augmented set to calculate statistics for. Instead, we run a single iteration of our augmentation function over all examples with our augmentation function and obtain following statistics:
Augmentation Statistics COGS CLEVR SCAN ALCHEMY
# Augmented samples 24155 699960 14670 18285 # Novel samples 23301 548277 7304 11786 # Unique novel samples 22617 548277 4851 11786 # Samples in test 121 0 7304 0 # Unique samples in test 109 0 4851 0
Note that, in CLEVR, we consider the novelty based on (question + answer) string since the generated image codes can be novel but the resulting image not. The following differences are significant under a paired t-test:
E.1 Statistical Significance Tests for Table 1
The following differences in Table 1 are significant under a paired t-test:
Alchemy:
• T5+LEXSYM > T5 (p < 0.05)
• LSTM+LEXSYM > LSTM+Substitute, LSTM, LexLSTM (p < .00001)
COGS:
• T5+LEXSYM > T5 (p < .00001)
• LSTM+LEXSYM > LSTM, (p < .00001)

heading: F CLEVR-COGENT Detailed Results
text: COGENT results are presented in Table 4.

heading: G Data
text: For CLEVR-COGENT (Johnson et al., 2017), we use training set for Split-A as our training set, validation set for Split-B as our validation set, and validation set of Split-B as our test set. The CLEVR and ALCHEMY datasets is released under the Creative Commons CC BY 4.0 license. The COGS datasets (Kim and Linzen, 2020; Kim et al., 2022) are released under MIT license. SCAN (Lake and Baroni, 2018) datasets are released under BSD license. The train, validation and test set sizes are given as below.
CLEVR-COGENT VQATransformer (No Pre-Praining)
Baseline 73.3 ±1.0 71.0 ±1.6 85.7 ±0.74 83.5 ±0.1 64.4 ±0.7 81.4 ±1.2 + Substitute (e.g. Liu et al., 2021b) 84.4 ±0.7 76.7 ±1.1 89.5 ±0.3 88.8 ±0.3 85.1 ±1.0 88.0 ±0.6 + LexSym 85.9 ±0.9 80.1 ±0.9 91.1 ±0.5 91.0 ±0.7 85.2 ±1.3 88.9 ±0.7
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: 8 (Limitations)

heading: 3 A2. Did you discuss any potential risks of your work?
text: 9 (Impact Statement)

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Left blank.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Left blank.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: It might serve as a platform for future versions of those methods that offer greater generality and formal guarantees