SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation (2024)

Abstract: Strong inductive biases enable learning from little data and help generalization outside of the training distribution. Popular neural architectures such as Transformers lack strong structural inductive biases for seq2seq NLP tasks on their own. Consequently, they struggle with systematic generalization beyond the training distribution, e.g. with extrapolating to longer inputs, even when pre-trained on large amounts of text. We show how a structural inductive bias can be efficiently injected into a seq2seq model by pre-training it to simulate structural transformations on synthetic data. Specifically, we inject an inductive bias towards Finite State Transducers (FSTs) into a Transformer by pretraining it to simulate FSTs given their descriptions. Our experiments show that our method imparts the desired inductive bias, resulting in improved systematic generalization and better few-shot learning for FST-like tasks. Our analysis shows that fine-tuned models accurately capture the state dynamics of the unseen underlying FSTs, suggesting that the simulation process is internalized by the fine-tuned model.1

heading: 1 Introduction
text: Inductive biases, i.e. the preferences and the abstract knowledge a model brings to the task, enable a model to learn from small amounts of data and generalize systematically outside of the training distribution. While seq2seq models perform very well on in-distribution data, they usually lack structural inductive biases and consequently struggle with systematic generalization. Previous work has shown that this includes generalization to unseen combinations of known sub-strings (Lake and Baroni, 2018; Keysers et al., 2020), extrapolation to longer inputs (Hupkes et al., 2020) and deeper recursion (Kim and Linzen, 2020).
Integrating structural inductive biases into seq2seq models is challenging. One popular ap-
1We release our code at https://github.com/namednil/sip
proach is to develop specialized architectures (Wu and Cotterell, 2019; Zheng and Lapata, 2021; Kim, 2021; Lindemann et al., 2023), which makes it difficult to precisely control and adjust the nature of the inductive bias to changing demands as the architecture would need to be modified and models retrained. Recently, some works instead have tried to inject inductive biases into seq2seq models by pretraining on a well-chosen synthetic task (Krishna et al., 2021; Wu et al., 2021, 2022) or meta-learning on a distribution of synthetic tasks (McCoy et al., 2020; McCoy and Griffiths, 2023) using MAML (Finn et al., 2017). Here, the inductive bias can be controlled by the choice of the synthetic task. However, meta-learning with MAML scales poorly because it requires expensive second-order derivatives and standard pre-training can be less effective (McCoy and Griffiths, 2023).
In this work, we present a computationally inexpensive way of injecting a structural inductive bias into a Transformer. We focus specifically on introducing an inductive bias that is helpful for tasks that traditionally have been approached with Finite State Transducers (FSTs). We choose FSTs
6570
because they are formally well understood, are easy to generate automatically, and are one of the simplest computational devices that are useful in NLP applications. While we focus on FSTs, the methodology is fairly general and our approach also provides a starting point for incorporating more general structural biases, provided by more expressive formalisms such as Pushdown Transducers.
Our approach (SIP, for Simulation-Induced Prior) is simple (see Fig. 1): given a representation of an FST and an input string, a Transformer is pre-trained to predict what the output of the FST is on the given input. We assume that FSTs are not specified for fine-tuning on downstream tasks, so we replace the FST with tunable embeddings and fine-tune the model solely on input/output pairs. Since we fine-tune all parameters, the model can deviate from FST-like behavior if needed.
Contributions. We show that a model pretrained with SIP has an inductive bias that improves systematic generalization and few-shot learning for ‘FST-like’ downstream tasks. SIP not only improves systematic generalization on FST tasks similar to those seen during pre-training but also on ones that are structurally more complex. The same pre-trained model also transfers well to natural data and achieves strong results on few-shot learning of text editing (e.g. Jane Doe → J. Doe) and grapheme-to-phoneme conversion.
Our probing experiments give insights into how the inductive bias is injected: SIP not only leads to the imitation of the input/output behaviour of FSTs, but encourages dynamics to emerge that simulate crucial aspects of FSTs in the hidden representations. Fine-tuning can leverage these dynamics, providing the inductive bias, and learn representations that resemble those of ground truth FSTs.

heading: 2 Related Work
text: Systematic generalization. Systematic generalization refers to the ability of a model to generalize (or extrapolate) beyond its training distribution in a systematic way that aligns with how humans generalize. Systematic generalization is difficult for standard seq2seq models in contexts such as semantic parsing (Finegan-Dollak et al., 2018) and machine translation (Li et al., 2021), in particular to unseen combinations of phrases, longer inputs as well as deeper recursion (Keysers et al., 2020; Kim and Linzen, 2020).
A range of approaches have been developed to tackle this, with many works focusing on special-
ized architectures (Guo et al., 2020; Kim, 2021; Lindemann et al., 2023). Furrer et al. (2020) find that the specialized architectures they consider do not transfer well to tasks beyond the context in which they were designed. This highlights the importance of being able to adjust inductive biases more easily than re-designing the architecture of a model. Large-scale pre-training has also been shown to help with systematic generalization (Furrer et al., 2020). However, challenges remain even for LLMs such as GPT-3 and PALM (Qiu et al., 2022; Dziri et al., 2023). The methodology we present in this work can be used to create additional material for LLM pre-training. Here we focus on smaller models and leave this to future work.
Pre-training with synthetic tasks. Pre-training a model on a synthetic task to introduce specific inductive biases has been explored by several recent works. Krishna et al. (2021) identify useful ‘skills’ for news summarization and develop a pretraining task accordingly. LIME (Wu et al., 2021) targets mathematical reasoning and is pre-trained on string manipulation that resembles formal reasoning. Papadimitriou and Jurafsky (2023) consider pre-training with several synthetic languages to investigate which helps most for language modelling. In contrast to these works, our approach targets simulating a computational device and maintains a closer relation to the pre-training setting because of the tunable prefix.
A challenge for using individually hand-crafted tasks is to cover a sufficient space of phenomena that are relevant to downstream tasks. Instead of training on a single task only, McCoy et al. (2020); McCoy and Griffiths (2023) meta-learn on a distribution of tasks using MAML (Finn et al., 2017). Our approach also uses a distribution of tasks but it scales better than MAML-based methods because MAML requires computing and storing secondorder derivatives. For example, the Transformer we train has a magnitude more parameters than the LSTM of McCoy and Griffiths (2023) and is pretrained on a smaller GPU (A100 vs RTX 2080 TI). In addition, as the complexity of each individual task grows, MAML requires more examples per task. We circumvent this by using a compact and unambiguous description of each task instead.
Simulating execution. The idea of using a neural network to predict the outcome of the execution of a computational device or code has come up in several contexts over the last few years. Early work by Zaremba and Sutskever (2014) investi-
gates it as a challenging benchmark for LSTMbased seq2seq models. Recent works have explored simulating (aspects of) code execution for various down-stream applications, such as program synthesis (Austin et al., 2021), or debugging and code analysis (Bieber et al., 2022) as well as reverse engineering (Pei et al., 2021). Finally, Finlayson et al. (2022) train a Transformer to interpret regular expressions: given a regular expression and a string, the task is to decide if the string is in the regular language. There are three crucial differences between their work and ours: (i) they investigate the empirical capabilities of Transformers while we introduce structural inductive biases for downstream tasks, (ii) they consider binary outputs whereas we consider sequential outputs, and (iii) we perform probing experiments showing strong evidence for FST simulation in the hidden representations.
Emergent World Representations. Our analysis provides evidence that our model trained with SIP internally simulates transitions between FST states even though it was not explicitly supervised to do so. Similar observations have been made for Language Models trained to play Othello (Li et al., 2023) and chess (Karvonen, 2024), where the model was found to acquire a representation of the board state simply from being trained to predict the next move.

heading: 3 Finite State Transducers
text: We briefly review Finite State Transducers (FSTs) which we use in our experiments. FSTs are closely related to Finite State Automata (FSAs). While an FSA describes a set of strings, an FST describes a relation between strings, i.e. a set of pairs (x, y), where x is an input y is an output.
FSTs can be visualized as labelled directed graphs (see Fig. 2), where the nodes are called states and the edges are called transitions. Consider the path q0 0 : 1−−→ q1 0 : 1−−→ q1 1 : 1−−→ q2 in Fig. 2b. This path is called an accepting path because it starts in an initial state (indicated by an arrow ‘from nowhere’ pointing to the state), and it ends in a final state (indicated by double circles). An accepting path shows what an input can be mapped to. In this case, the path shows that the FST transduces the input 001 into the output 111. We can read off which input an accepting path associates an output to by concatenating all the strings along the path occurring before ‘:’. The output can be determined by concatenating the strings after ‘:’. Hence, each transition σ : ρ−−→ can be thought of as
q0
0:ε
q11:12:2
0:0 1:1 2:2
(a) A deterministic FST.
q0
q10:1
q3
0:2
0:1,1:1
q21:1
0:2,1:1
q42:2
(b) A non-deterministic but functional FST.
Figure 2: Examples of functional FSTs. The FST in (a) deletes leading zeros. The FST in (b) replaces any 0 by a 1 if the last input symbol is a 1. Conversely, if the last symbol is a 2, any 0 is replaced by a 2. The output can only be determined after the last input symbol.
‘replacing’ σ by ρ. Inserting and deleting can be achieved by means of the empty string, written as ϵ. For example, Fig. 2a ‘replaces’ leading zeros by an empty string, effectively deleting them.
In general, an input can be paired with arbitrarily many different outputs. We call an FST f functional if every input x is paired with at most one output y, and use the notation f(x) to refer to y. All FSTs we consider here are functional.
In this work, we investigate generalization across different sub-classes of FSTs, namely from the less expressive deterministic FSTs to non-deterministic FSTs. An FST is called deterministic if (i) it has a unique initial state, (ii) for all states q and input symbols σ there is at most one transition q
σ : ρ−−→ q′ and (iii) σ ̸= ϵ. Intuitively, this means that in any state, for an input symbol σ there is at most one possible next state and one possible output, and hence for any input string there is at most one path that is compatible with it. Because of this, we can always infer a prefix of the output by looking only at a prefix of the input string and ignoring the rest. For example, consider the input prefix 001. In the deterministic FST in Fig. 2a, we know that the output has to start with 1 because there is only one path that is compatible with 001. In contrast, in the non-deterministic FST in Fig. 2b, three paths are compatible with 001 that have different outputs. In that case, we can only determine the output once we look at the last symbol of the input. In short, non-deterministic FSTs can take context to the right into account but deterministic FSTs cannot.

heading: 4 Simulation-Induced Prior
text: Our approach largely follows the pre-training and fine-tuning paradigm. We first pre-train on synthetic FST tasks by giving the model a representation of an FST as a prefix and an input string (see Fig. 1). The training objective is to predict the
output of the FST on the input string. Our research hypothesis is that training a model to predict the behaviour of an FST incentivizes the model to acquire reusable dynamics that internally simulate FSTs. When fine-tuning the model using a tunable prefix instead of an encoding of an FST, these dynamics should be easy to leverage and provide a structural inductive bias for FST-like tasks.

heading: 4.1 Pre-training
text: During pre-training, the model is given a representation of an FST and a string in its domain and has to predict the output of that FST on the given input string. The input to the Transformer is a sequence of vectors from Rd, which consist of a prefix that represents the FST f and a suffix comprised of the embeddings of the input string (see Fig. 1):
h1,h2, . . . ,hk︸ ︷︷ ︸ FST encoding ,x1,x2 . . . ,xn︸ ︷︷ ︸ Input to FST
Each hi encodes a transition p σ : ρ−−→ q as a vector:
hi =W [EMBState(p); EMBState(q);
EMBSymbol(σ); EMBSymbol(ρ); EMBFinal(e)]
where [; ] represents vector concatenation, e indicates if q is a final state, and W is linear layer that ensures that h ∈ Rd. All embeddings are simple look-up tables based on the id of the state or symbol. The initial state of the FST is always assigned the id 0, and positional embeddings are used as usual. The model is trained to maximize the log probability of the output y = f(x) of the FST f .

heading: 4.2 Fine-tuning
text: After pre-training, we can apply our model to a downstream task and fine-tune it. We assume we do not have access to an FST for the downstream task, and therefore we replace the FST encoding with a sequence of tunable embeddings. That is, the input to the model is a sequence of vectors:
h′1,h′2, . . . ,h′k︸ ︷︷ ︸ Tunable embeddings ,x1,x2 . . . ,xn︸ ︷︷ ︸ Input
where x1,x2 . . . ,xn are the embeddings of the input tokens, h′i ∈ Rd are the tunable embeddings and k is a hyperparameter. The embeddings h′i are initialized to the average of the encoding of multiple FSTs from the pre-training phase. The most straightforward way to fine-tune is to only modify h′ because we are looking for an FST-like task representation. This is similar to prompt tuning (Lester et al., 2021). However, this does not work
well on tasks outside the pre-training distribution. Hence, we fine-tune the entire model, including the prefix, and use a higher learning rate for the prefix than for the rest of the model (see Appendix E).

heading: 4.3 Constructing Pre-Training Data
text: To create our pre-training data, we sample 40,000 deterministic FSTs. For every FST, we sample 5 input/output pairs with input lengths up to 35. In total, this leads to 200,000 pairs for training along with their FSTs. To describe the sampling procedure in more detail, we use an overall vocabulary V consisting of the printable ASCII tokens and the Unicode block for IPA symbols (used for transcribing speech). Seq2seq tasks in the wild usually do not use the whole space of this vocabulary, so for each task T we first uniformly sample the vocabulary size |VT | between 5 and 25 and then uniformly select a subset VT ⊆ V . Then, we uniformly sample the number of states |QT | between 2 and 4, and the number of final states between 1 and |QT |. For every state q and every symbol σ ∈ VT we introduce at most one outgoing transition to a state q′, chosen uniformly at random. This ensures that the FST is deterministic. We then sample the output for the transition: either a symbol ρ ∈ VT or ϵ. Finally, we minimize the number of states of the FST using OpenFST (Allauzen et al., 2007), and exclude those without cycles, as they express finite relations. See Appendix A.1 for details.
In practical applications of FSTs, in particular for text editing, one often wants to keep certain parts of the input unchanged. This can be achieved with a set of transitions of the form q σ :σ−−→ q′ for all σ ∈ VT . Since it is very unlikely to sample such a set of transitions, we use a special symbol that acts as a shorthand for this, which we also use when encoding the FST for pre-training.

heading: 5 Evaluating SIP’s Inductive Bias
text: To understand the effects of our pre-training procedure on the inductive bias of the model and on the downstream performance, we first explore systematic generalization on synthetic FST tasks. This allows us to precisely control the similarity between the pre-training and the downstream task.

heading: 5.1 Evaluation Methodology
text: To evaluate the degree to which a model has an inductive bias towards FSTs, we now describe two methods for generating training and test data that
reward a model for showing important aspects of FST-like systematic generalization.
Iteration generalization. Cycles are a characteristic feature of FSTs, and iteration generalization tests if a model learns that cycles can be traversed more often than seen during training. More specifically, given an FST, we generate training data which requires visiting any state only a few times (iteration count up to 3). In the test data, the model has to generalize to visiting states more often (iteration count at least 4). This notion is related to length generalization (Lake and Baroni, 2018) but tailored specifically to FSTs.
Unseen combinations of transitions (UC). When an FST processes a string, the set of possible next transitions only depends on the current FST state; it does not matter how the current state was reached. Hence, a model with an inductive bias towards FSTs should also not be overly sensitive to how a state is reached, and correctly handle situations where a specific combination of transitions was unobserved during training. For example, consider the FST in Fig. 2a, which deletes leading zeros from a number. Suppose that a model is trained on examples such as 0012, 2201, 1012 but no training example contains the combination of leading zeros followed by a 2, which corresponds to using the combination of the transitions q0 0 : ϵ−−→ q0 and q0 2 : 2−−→ q1. If the model has an inductive bias towards FSTs, it should generalize to this unseen combination and correctly handle inputs such as 0021. To generate appropriate training and test data for this, we sample a pair of adjacent transitions (such as q0 0 : ϵ−−→ q0 and q0 2 : 2−−→ q1 in Fig. 2a) and ensure that no training example uses both transitions within the same string. In contrast, in the test data, all examples require using the combination of the transitions. To make the generalization setup more challenging, we ensure this for multiple pairs of transitions at the same time. We refer to Appendix A.3 for details on the construction.
UC is related to the method of Keysers et al. (2020) who also withhold combinations of seen elements to assess systematic generalization.

heading: 5.2 Setup and Baselines
text: To make a fair comparison, all models we experiment with in the main paper share the same architecture and are initialized from the same checkpoint before any additional pre-training, namely ByT5small (Xue et al., 2022). ByT5 has 300 million parameters and was pre-trained on the multilingual
C4 corpus. It uses raw bytes as tokens, which enables full Unicode support and is a natural unit to consider for FST-like tasks such as text editing and grapheme-to-phoneme conversion. We report additional results with a T5-Base model in Appendix B.3, where we observe similar trends.
SIP-d4. This is a model using the method we propose in this work. We pre-train on the data generated in Section 4.3 (deterministic FSTs, with up to 4 states) for 20 epochs. This model achieves an average sequence-level accuracy of 98% on predicting the output of an unseen FST from the training distribution. For fine-tuning, we use a prefix of length 50 for all experiments in this paper. As an ablation, we also fine-tune the model without the prefix of tunable embeddings (-prefix).
Naive pre-training. We use the same pretraining data as for SIP-d4 but omit the description of the FST and only train on input/output pairs.
Task embeddings (TE). TE is a simplified version of SIP. Instead of using an encoding of an FST in the prefix, this baseline uses 50 randomly initialized embeddings specific to each FST. The embeddings are learned from examples jointly with the rest of the model. Several works have used a single embedding to encode a domain/task in multitask learning (Tsvetkov et al., 2016; Stymne et al., 2018; Zhang et al., 2022). Using a shorter tunable prefix resulted in considerably worse performance in our setup. TE is fine-tuned analogously to SIP, i.e. with a prefix of tunable embeddings.
Set. Wu et al. (2022) investigate the effectiveness of 18 simple synthetic pre-training tasks and found Set to perform best on average. The task is to deduplicate characters such that every type occurs only once, e.g. the input dabacd becomes dabc. This task can be represented by a deterministic FST, albeit a very large one with 2n states for a vocabulary of size n.

heading: 5.3 Systematic Generalization within the Pre-training Distribution
text: First, we want to establish to what degree the pretraining has conferred any inductive bias on the distribution it was pre-trained on.
Setup. For each generalization setup, we generate 5 unseen FSTs with 4 states each using the same procedure as for the pre-training. We fix the vocabulary size to its maximum value (25) in the pre-training data and only use printable ASCII characters in order to reduce variance across tasks. To evaluate UC, we withhold the combination of up to
20 pairs of transitions and generate 5000 training examples with lengths 3 to 15 and corresponding test data as described in Section 5.1. For iteration generalization, we generate training examples with a maximum iteration count of 3 and test on longer examples of length up to 30 with an iteration count of at least 4. Since the out-of-distribution performance of two checkpoints of the same model can vary significantly, we report averages on the test set of the last 10 epochs.
Results. The results can be found in Table 1. On average, SIP-d4 achieves close to perfect accuracy (with one outlier on UC, skewing the mean). TE also shows a clear improvement over the other baselines but SIP-d4 outperforms TE by a large margin. This suggests that SIP-d4 and TE, to a lesser extent, indeed have acquired a stronger inductive bias for FSTs than the other methods. Using SIP-d4 without the tunable prefix leads to a substantial drop in accuracy, highlighting its importance. We analyze the representations learned by SIP-d4 in the tunable prefix in Appendix D.1.

heading: 5.4 More Complex FSTs
text: Does the inductive bias introduced by SIP extend beyond the pre-training distribution to more complex FST tasks? To investigate this, we use the same sampling methodology but generate FSTs with more states. SIP-d4 was pre-trained on FSTs with up to 4 states, and we evaluate on FST tasks with 5, 7 and 10 states.
We show in Fig. 3 how the individual models deviate from the accuracy of ByT5 as a function of the number of states in the test FST. SIP always performs best by a clear margin regardless of the number of states in the FSTs. As we increase the number of states and move away from the pretraining distribution, SIP improves less over the
baselines. We see a similar pattern for TE but with considerably smaller improvements over ByT5.

heading: 5.5 Non-Deterministic FSTs
text: As shown in the previous section, SIP still works well for more complex FST tasks than seen during pre-training. However, this evaluation focused on the favourable case where both pre-training and evaluation involve the same class of FSTs, namely deterministic FSTs. Deterministic FSTs can only take left context into account (see Section 3), which is a restrictive assumption. Here, we evaluate if the inductive bias conferred by SIP carries over to nondeterministic functional FSTs, i.e. those that can also take context to the right into account.
We automatically generate 5 non-deterministic FSTs with 21 states (see Appendix A.2 for details) and report averages in Table 2. Despite the structural mismatch between pre-training and the downstream tasks, SIP-d4 shows clear improvements over the baselines. Interestingly, TE does not consistently outperform the other baselines, despite its stronger results on deterministic FSTs.
Our pre-training procedure does not hinge on using deterministic FSTs. This raises the question if we can achieve even better performance by adjusting the inductive bias. To investigate this, we further pre-train SIP-d4 on 40,000 non-deterministic FSTs with up to 7 states, which we call SIP-nd7. To control for the additional training data of SIPnd7, we also further pre-train SIP-d4 with the same number of deterministic FSTs with the same characteristics as in Section 4.3 (SIP-d4+). The results in Table 2 show better performance of SIP-nd7, which supports the hypothesis that the inductive bias can be adjusted. SIP-d4+ shows a smaller improvement over SIP-d4. Based on 5 additional FSTs per setup to gain more statistical power, we found that the difference between SIP-nd7 and SIP-
d4+ is statistically significant (p ≈ 0.017, n = 20, paired approx. permutation test).

heading: 6 Transfer to Natural Data
text: In this section, we investigate to what degree the inductive bias from pre-training on synthetic data transfers to tasks with natural data that have been traditionally approached with finite state methods.

heading: 6.1 Low-resource Grapheme-to-Phoneme Conversion
text: Grapheme-to-phoneme conversion is the task of converting a word as a sequence of symbols (for example, letters in the Latin alphabet) into a description of how this word is pronounced as letters in the IPA alphabet. For example, a possible pronunciation of ‘explanation’ is [Ekspl@"neIS@n]. Grapheme-to-phoneme conversion can be part of text-to-speech pipelines and FSTs for this purpose usually are two or three magnitudes larger than the FSTs we constructed for pre-training. Because of this, it enables us to test how far beyond the pretraining distribution SIP remains helpful. We focus on learning from small amounts of data, for which a structural inductive bias towards FSTs should be particularly helpful. We evaluate on 7 low-resource languages from different language families that use their own scripts (Balinese, Coptic, Gothic, Lao, Sylheti, Telugu and Central Atlas Tamazight). We obtained the data from Wikipron (Lee et al., 2020).
As a soft upper bound, we compare with Charsiu (Zhu et al., 2022) which is a ByT5-small model that has been further pre-trained on 7.2 million examples of grapheme-to-phoneme conversion across 100 languages. Although Charsiu was not exposed to the scripts of the languages we chose, it may have seen related languages whose scripts are encoded similarly in Unicode.
We report accuracies in Table 3, and phonemeerror-rates in Appendix B.2; trends are identical. The original ByT5-small model performs worst on average despite being a strong model for grapheme-to-phoneme conversion in general (Xue et al., 2022). On average across the languages, SIPd4 outperforms the other methods that pre-train on synthetic data as well as ByT5. The difference between SIP-d4 and Set is statistically significant (p ≈ 4 × 10−4, paired approx. permutation test). Fine-tuning SIP-d4 without the tunable prefix leads to a drop in performance, except for Gothic. Charsiu performs very well on Telugu, potentially because of its large overlap in lexicon with Sanskrit (Staal, 1963), which is part of its training data.

heading: 6.2 Few-shot text editing
text: Learning simple text editing tasks (Jane Doe → J. Doe) from a handful of examples with a Transformer requires a strong structural inductive bias to overcome competing explanations of the data and hence provides a good benchmark for our approach. While current LLMs may seem like the ideal choice for such tasks, they are prone to hallucinations, e.g. ignoring the input and resorting to frequent entities (see Appendix C for an example).
Text editing has been studied in the context of
program synthesis and we evaluate on 19 such tasks from the SyGuS competition 2017 (Alur et al., 2017). Instead of predicting a program, our model directly operates on input/output examples. We note that 17 of these tasks can be solved by compact FSTs, whereas two cannot. These two tasks are revname (Jane Doe → Doe Jane) and sur-initial (John Doe → Doe, J.), which require tracking information about the first name in the states.
We report results for 5-shot experiments in Table 4. SIP-d4 and TE excel at this, reaching well above 90% accuracy on average whereas the other methods perform worse by a large margin. Charsiu does not perform clearly better than baselines such as Set – even though it obtains excellent results on grapheme-to-phoneme conversion. Interestingly, TE performs better than SIP-d4 on the tasks that can be solved with FSTs, potentially because the initialization of the prefix for TE follows the same distribution as during pre-training, which is not the case for SIP. However, SIP considerably outperforms TE on the two tasks that cannot be compactly represented by FSTs, suggesting that some of the dynamics acquired during pre-training can sometimes be leveraged in other contexts as well. Fine-tuning SIP-d4 without the tunable prefix leads only to a very small drop in accuracy on average.

heading: 7 Analysis: SIP leads to FST simulation
text: We motivated our approach by the hypothesis that SIP’s pre-training encourages the model to simulate FSTs internally, and that this provides the structural inductive bias. In this section, we present evidence that (i) SIP models indeed approximately simulate FSTs in the hidden states, and (ii) that the dynamics responsible for simulation are re-used after finedtuning all parameters on input/output pairs only.
For a model to simulate FSTs in its hidden rep-
resentations, it must be able to track the FST state when processing a string, and it should be possible to extract the FST state with a probe. To test this, we mirror the pre-training setup and provide SIP-d4 with an FST and an input string (Fig. 4, left). For each token, we extract the top-layer activations of the encoder, and learn a linear probe with a softmax layer to predict the ID of the state that the given FST is in before processing that token. Since state IDs are largely arbitrary, the probe has to learn to relate the hidden representations to the FST presented in the input.
The probe achieves 99.3% token-level accuracy on a test set with unseen FSTs, and a wholesequence accuracy of 93.9%. We also evaluate a trivial heuristic that returns a random state that has an appropriate outgoing transition for each token in the input. This heuristic achieves a token-level accuracy of 68.9%, and a whole-sequence accuracy of only 17.8%. A probe trained on ByT5 representations, i.e. before SIP pre-training, performs even worse at 42.9% token-level accuracy and whole-sequence accuracy of only 7.1% (see Appendix D.2). Hence, the model has learned a non-trivial way to simulate transitions between states of the FST encoded in the prefix. This is remarkable because the pre-training procedure for SIP-d4 does not provide supervision for how to process strings.
While this shows that SIP leads to the simulation of state transitions after pre-training, does the model leverage the simulation ability on downstream tasks? Recall that we fine-tune all parameters of the model (Section 4.2), so the model could
employ a very different strategy to fit the data. To investigate this, we set the trained probe aside and freeze it (Fig. 4, right). We then fine-tune SIP-d4 on the iteration generalization tasks with 4 states in the gold FST (cf. Table 1). Finally, we apply the frozen probe to the fine-tuned model to see if the state sequences we extract are similar to those of the ground truth FST. Fine-tuning SIP-d4 could induce the same FST as the ground truth but use a different numbering of the states. To account for this, for each of the five tasks, we find the isomorphism between the predicted state IDs and the ground truth that gives the best match on average.
The results are presented in Fig. 5 as confusion matrices between predicted and gold states. The probe extracts state sequences that resemble the state sequences of the gold FST (up to isomorphism), both on the training data and the out-ofdistribution test data. We also find that deviation from the ground truth state sequence correlates with errors by the fine-tuned model: if the probe extracts correct state sequences, the model achieves an accuracy of 98.6% on the iteration generalization tasks, whereas it drops to 89.8% when the probe extracts state sequences that deviate. The difference is statistically significant (approximate permutation test, p ≈ 5 × 10−5). Overall, this shows that the finetuned model reuses the dynamics for state tracking and learns representations similar to the ground truth FST.

heading: 8 Conclusion
text: We present SIP, a simple, efficient and adjustable method for introducing a structural inductive bias into a seq2seq model. We focus on an inductive bias towards FSTs, one of the simplest computational devices that is useful for NLP applications. We achieve this by pre-training a Transformer to
simulate automatically generated FSTs, i.e. to predict the output of an FST given an input string and a description of the FST. Our experiments show that our method imparts the desired inductive bias, resulting in improved systematic generalization and better few-shot learning for FST-like tasks. In addition, we show with probing experiments that a model trained with SIP simulates transitions between FST states in its hidden representations, and that the dynamics behind this are leveraged during fine-tuning. In future work, we plan to extend this methodology to more expressive formalisms such as Pushdown Transducers which can be used for a wider range of downstream NLP tasks.
Limitations
Our investigation focuses on FSTs with a relatively small number of states. However, the results in Section 5.4 and in the experiments on grapheme-tophoneme conversion show that even pre-training with FSTs with a small number of states has positive impacts for tasks that require larger or more complex FSTs.
The probing experiments show that the model simulates transitions between states similarly to an FST, but we did not perform a mechanistic interpretation of how exactly this is implemented in the weights. One potential mechanism behind the simulation behaviour is the construction of Liu et al. (2022) who show that Transformer decoders can simulate transitions between states of deterministic finite automata for strings of length up to n using O(log(n)) layers.
Acquiring a specific inductive bias by means of learning to simulate a computational device is a general idea that could be applicable beyond FSTs but might be unsuitable in cases where (i) it is difficult to formulate a reasonable computational device to simulate (such as document classification and sentiment analysis beyond keyword spotting), or (ii) the computational device would be very hard or infeasible to simulate (e.g. Turing machines).
Our experiments focus on moderately sized models (300M parameters) with an encoder-decoder architecture, and we did not investigate large decoderonly models. Our methodology can also be applied to decoder-only models, and we do not foresee any reasons why it could be less effective in that setup.
Finally, we only consider the standard Transformer architecture and we leave it to future work to explore the impact of SIP on variants of the Transformer architecture designed for handling
long character sequences (Yu et al., 2023) or in the context of state-space models (Wang et al., 2024).

heading: Acknowledgements
text: We thank Verna Dankers, Victor Prokhorov, and Christine Schäfer for discussions and comments. ML is supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1), the University of Edinburgh, School of Informatics and School of Philosophy, Psychology & Language Sciences, and a grant from Huawei Technologies. IT is supported by the Dutch National Science Foundation (NWO Vici VI.C.212.053).

heading: A Generation of Synthetic Data and Splits
text: A.1 Generating deterministic FSTs
Before describing our procedure for sampling deterministic FSTs, we briefly establish notation. An FST is a tuple ⟨Q,Σ,Γ, I, F,∆⟩, where Q is a finite set of states, Σ is the input alphabet, Γ is the output alphabet, I ⊆ Q is a set of initial states, F ⊆ Q is a set of final states and ∆ ⊆ Q × (Σ ∪ {ϵ}) × (Γ ∪ {ϵ}) × Q are the transitions. We assume Σ = Γ and call it V for vocabulary.
For technical reasons, we exclude the three characters [, ] and \ from the vocabulary as they are interpreted as special characters by OpenFST, which we use for constructing and representing FSTs.
In addition to the shorthand for identity transitions (id), we also have shorthands for converting upper case to lower case and vice-versa (lower-to-upper, upper-to-lower). We describe our procedure to generate a deterministic FST with pseudocode in Algorithm 1. It receives as argument n (the number of states in the FST), f (number of final states), V (the vocabulary of this FST), and probabilities P-ID, P-DROP, P-SHORTHAND. These probabilities control the likelihood of using a shorthand, not drawing an outgoing edge (P-DROP) with a given symbol, and creating a single identity transition (P-ID). We use CHOICE to denote a uniform random choice from a finite set.
We use P-ID = 0.2, P-DROP = 0.4, P-SHORTHAND = 0.15 in our experiments.
A.2 Generating Non-deterministic Functional FSTs
It is not straightforward to directly generate nondeterministic FSTs that are guaranteed to express a function. However, we can directly generate a bimachine, which then can be converted into an FST.
Algorithm 1 Generate a random deterministic FST function GEN-DET-FST(n, f, V, P-ID, P-DROP, P-SHORTHAND) Q = {0, . . . n− 1} ∆ = ∅ I = {0} for q ∈ Q do q′ = CHOICE(Q) with prob P-SHORTHAND s = CHOICE([id, lower-to-upper, upper-to-lower])
∆ := ∆ ∪ {q s : s−−→ q′)} else
for σ ∈ V do with prob P-DROP
no-op ▷ No outgoing edge with σ at q else with prob P-ID ∆ := ∆ ∪ {q σ :σ−−→ q′} else ∆ := ∆ ∪ {q σ : CHOICE(V ∪{ϵ})−−−−−−−−−−−→ q′}
end with prob end for
end with prob end for Eliminate states from Q through which no accepting path can go Choose random subset F of Q with |F | = min(f, |Q|) return minimized FST with states Q, transitions ∆, initial states I and final states F end function
Bimachines (Schützenberger, 1961) represent the functions expressible by FSTs, i.e. for every functional FST there is a bimachine that represents it (and vice-versa). A bimachine consists of two deterministic finite state automata (called left and right) and an output function. Let AL be the left FSA with states QL and transition function δL : QL × Σ → QL), and let AR bet the right FS with states QR and transition function δR : QR × Σ → QR. The output function is ψ : Ql × Σ × Qr → Γ∗. All states of AL and AR are final states. Given an input string x = σ1σ2σ3 . . . σn, a bimachine runs AL from left to right over x, keeping track of the states ql0, q l 1, q l 2, . . . q l n. It also runs A
R over the string x but this time from right to left, again keeping track of the states qr0, q r 1, q r 2, . . . q r n that are visited. Then, the state sequence of the right automaton is reversed and ψ is
Algorithm 2 Generate output function for bimachine
function GEN-OUTPUT-ψ(nL, nR, V, P-ID = 0.2)
for qL ∈ 0, . . . , nL − 1 do for qR ∈ 0, . . . , nR − 1 do
for σ ∈ V do with prob P-ID ψ(qL, σ, qR) := σ
else ψ(qL, σ, qR) := CHOICE(V ∪ {ϵ})
end with prob end for
end for end for return ψ
end function
applied ‘elementwise’ as illustrated in Fig. 7. More formally, the output of the bimachine is ψ(ql0, σ1, q r n−1)ψ(q l 1, σ1, q r n−2) . . . ψ(q l n−1, σ1, q r 0).
Bimachines can be compiled into FSTs with a simple product construction. For a bimachine ⟨AL, AR, ψ⟩, one can construct an equivalent FST as follows:
⟨QL ×QR,Σ,Γ, {sL} ×QR, QL × {sR},∆⟩ where sL and sR are initial states of AL and AR, and ∆ contains all transitions
∆ = {⟨qL, qR⟩ σ : ρ−−→ ⟨q′L, q′R⟩ | δL(qL, σ) = q′L, δR(q′R, σ) = qR, ρ = ψ(qL, σ, q′R)} We refer to Mihov and Schulz (2019) for details and further information about bimachines.
To sample bimachines, we re-use Algorithm 1 with P-SHORTHAND = 0, and ignore the outputs of the transitions, treating them as FSAs. We sample the output function according to Algorithm 2. For the test data creation (Table 2), we use 5 states in the left FSA and 4 states in the right FSA, and set P-DROP = 0.4. For creating the training data for SIP-nd7, we use 2 or 3 states in either left or right automaton and set P-DROP = 0.6 to keep the length of the prefix low to save GPU memory.
A.3 Unseen Combinations of Transitions
We now describe the construction by which we create training and test data for the evaluation of Unseen Combinations of transitions. We first describe how we construct an FST for the training and
test data, respectively, given a choice of transitions whose combination we want to withhold. Then, we briefly describe how those transitions are chosen.
Given an FST f (illustration in Fig. 8, left) and transitions ta and tb (highlighted in red) whose combination we want to withhold, we construct a new FST ftrain as follows: We create two copies fa, fb of the original FST f . In fa, we remove the transition tb; in fb, we remove the transition ta. Then ftrain = fa ∪ fb, which can be constructed by introducing a new initial state with ϵ-transitions into the respective initial states of fa and fb (right side of Fig. 8). This ensures that any accepting path goes through fa or fb but cannot alternate between the two. Hence, ta or tb can be used – but not both in the same string. Note that ftrain still describes a partial function (rather than a relation) because any accepting path in fa and any accepting path in fb is also an accepting path in f . As a result, whenever fa and fb are both defined, they agree on the result fa(x) = fb(x) = f(x). We test exclusively for how a model handles unseen combinations of transitions by generating examples from f for which ftrain is not defined.
To make the generalization setup more challenging, these steps can be applied to multiple pairs of adjacent transitions at the same time, i.e. to withhold ⟨t1a, t1b⟩, . . . , ⟨tka, tkb ⟩: We create the copy fa and remove the transitions t1b , . . . , t k b from fa and analogously remove t1a, . . . , t k a from fb.
Now, we briefly describe how we select which pairs of transitions we want to withhold. We only select adjacent transitions, i.e. transitions where one can be used immediately after the other, excluding self-loops. In addition, some transitions cannot be deleted without cutting off a vital initial or final state, which can lead to ftrain being undefined for any string (and hence no training data). We ensure this never happens by never withhold-
ing the first transition into each state based on a depth-first traversal of the FST.
A.4 Additional dataset information
For all experiments with synthetic data (generated by FSTs), we generate 5000 training examples and 1000 test examples. To reduce variance across tasks, we fix the vocabulary size to its maximum value (25) in the pre-training data and choose the vocabulary only from the printable ASCII characters.
Length distribution. The input strings in the pre-training data we generate for SIP-d4 have a minimum length of 1, an average length of 15.57 and a maximum length of 35. We report the length distributions for the iteration generalization experiments in Section 5 in Table 5.
SyGuS. We took the data from the SyGuS competition github https://github.com/ SyGuS-Org/benchmarks/tree/master/comp/ 2017/PBE_Strings_Track, and extracted the ‘constraints’. For each text editing tasks, there are usually three files, e.g. firstname, firstname-long, firstname-long-repeat. We only consider data from the *-long variant because the non-marked variant (e.g. firstname) is a subset of the *-long variant, and we exclude *-long-repeat as it contains repeated data points. We also exclude some text editing tasks that have insufficient amounts of data for reliable evaluation (bikes) and some tasks where the input is not a single string but a pair of strings if concatenating the strings results in particularly long inputs (univ), or if the concatenation of the
string pair makes the task trivial (name-combine, which would correspond to an identity operation). For a few-shot experiment, we sample 5 training examples and evaluate on the rest.
We note that the original intention in the design of the benchmark data was for program synthesis rather than few-shot learning. The data contains names, and separately it contains phone numbers (but not combined). However, we believe both to be synthetically generated.
Grapheme-to-phoneme. We obtain data from Lee et al. (2020), and conduct experiments mainly on the broad transcription, except for Telugu and Tamazight, where we use the narrow transcription. For each experiment, we randomly sample 100 training examples, and use the rest as test data. The data is available under a permissible license: https://en.wiktionary.org/ wiki/Wiktionary:Copyrights

heading: B Additional Results
text: B.1 Additional Results with More States
In Fig. 3, we show accuracy relative to the accuracy of ByT5. Here, we show the absolute accuracies and edit distances in Table 6.
B.2 Full results for grapheme-to-phoneme conversion
Table 7 shows the full results of our graphemeto-phoneme conversion experiments, including phoneme error rate (PER).
B.3 Additional results with T5-Base
We run a subset of the experiments starting off from a pre-trained T5-Base (Raffel et al., 2020) instead of ByT5. This model is about one-third smaller than ByT5 (around 200 million instead of 300 million parameters). T5-Base uses a different vocabulary than ByT5, so we resize the output layer to the vocabulary size of ByT5 and re-initialize it. For the input embeddings, we re-purpose the first n embeddings in the T5-Base embedding matrix to represent the token ids according to the ByT5 tokenizer. While this is suitable as a starting point for further pre-training, we found that directly finetuning T5-Base with these modifications on downstream tasks led to very poor results and do not include them here. Instead, we train T5-Set (analogous to Set) for a fair point of comparison.
We report a subset of the results from the main paper in for T5-Base in Tables 8 to 10.
We also tried to pre-train a ByT5-style model from scratch (i.e. from random initialization). However, we could not find a setting of hyperparameters that would make the model converge well. We hypothesize that the model already needs to be in a reasonable space to make learning feasible.
B.4 Generalization to longer strings
In the main paper, we report results on iteration generalization where a model is trained on strings such that each state has been visited at most 3 times, and is tested on strings where at least one state is visited at least 4 times. Here, we explore a more extreme version, where there is a large gap between the maximum length seen during training and the minimum length seen during testing. As another point of comparison, we further pre-train SIP-d4 on 40,000 FSTs with strings of length up to 110 (SIP-d4-long).
We report results in Table 11. ByT5 struggles with this generalization setup across the board. SIP-d4 performs remarkably well on lengths 40- 70 which are beyond the lengths seen during its pre-training. However, performance drops starkly when testing on inputs of length 90 to 110. We hypothesize that this is because the relevant positional embeddings were not pre-trained by SIP. In contrast, SIP-d4-long performs well on inputs of length 90 to 110, as it has seen strings of such length during pre-training.

heading: C Hallucination Example
text: We briefly show an example where an LLM ignores a part of the input and resorts to outputting a highfrequency entity. Consider the following in-context examples for a simple text editing task:
Input Output Howard Phillips Lovecraft H.P. Lovecraft John Ronald Reuel Tolkien J.R.R. Tolkien Thomas Stearns Eliot T.S. Eliot
At the time of submission, the current version of ChatGPT frequently outputs “J.K. Rowling” for the name “John Edward Rowling”, hallucinating the K.

heading: D Additional Analysis
text: D.1 Analysis of fine-tuned prefixes
To gain some understanding of how the prefix of tunable embeddings is used by the model and what it contains, we consider the setup of fine-tuning
only the prefix and keeping the rest of the model unchanged. That is, all the task-specific information has to be captured in these embeddings. Specifically, we fine-tune on the 5 FSTs from Section 5.3 for iteration generalization for 20 epochs with a learning rate of 0.5.
We explore two questions:
1. Is the model robust towards different permutations of the fine-tuned prefixes? Intuitively, these permutations correspond to changing the order in which transitions are listed, so ideally the model should not be sensitive to that order.
2. Does the fine-tuned prefix represent the taskspecific information in a similar way to how FSTs were encoded during pre-training?
To address the first question, we randomly permute the tuned prefixes and compute accuracy on the iteration generalization data before and after permuting the tuned prefixes. We use 20 permutations per learned prefix and average results across the 5
FSTs. Overall, we find that this results only in a small drop in accuracy: the median drop in accuracy is only around 1.3 percentage points, and the arithmetic mean of the drop is around 7.1 percentage points. Most permutations do not have a big impact on how the prefix is interpreted but a few permutations do have a stronger negative impact, skewing the arithmetic mean.
To address the second question, we test if the learned prefix for a task t resembles an encoding of an FST that solves t. For each of the 5 FSTs, we generate 10,000 distractors, i.e. FSTs that have the same number of states and use the same vocabulary as the FST solving t. We define the similarity of two prefixes p, q as follows:
sim(p, q) = max π
1
n
∑
i
pTi qπ(i)
||pi||2 · ||qπ(i)||2 where π is a permutation, and pi is the i-th vector in prefix p, and prefixes p and q both have length n. That is, we define the similarity between p and q as the highest possible average cosine similarities between positions in p and q that one can achieve
by assigning a position in p to exactly one position in q and vice-versa.2 Taking the maximum over all permutations is justified by our results to the first question above, which showed that the model is largely invariant to different permutations of the
2Computing the similarity sim(p, q) is relatively expensive because it involves solving the assignment problem (e.g. with the Hungarian algorithm). Instead of solving the assignment problem exactly, we approximate it with the Sinkhorn algorithm (Sinkhorn, 1964). We then take the output of the algorithm (a matrix of ‘soft’ assignments) and for each position in p, we greedily select a matching position in q.
tuned prefix. For every task t, we compute the similarity between the prefix p learned by fine-tuning on input/output pairs and the union of encodings of the distractors and encodings of the gold standard FST for task t. Where necessary, we truncate encodings of FSTs to have the same length as the learned prefix. We present the results in Fig. 9 showing that all learned prefixes are most similar to an encoding of the ground truth FST.
D.2 Probing non-SIP models All probes are trained for one epoch on activations produced by passing 8,000 FSTs with 5 inputs each (i.e. 40,000 instances) through the model.
For the baseline probe, we take the trained SIPd4 model (including matrix W and embeddings from 4.1) and re-initialize the Transformer to ByT5small. The probe achieves only a token-level accuracy of 42.9% and whole-sequence accuracy of 8.1%. We see very similar results for a probe trained on a randomly initialized Transformer in this setup: a token-level accuracy of 42.5% and a whole-sequence accuracy of 7.1%.

heading: E Additional model details & Hyperparameters & Hardware
text: SIP. For completeness, we now describe the order in which we arrange the transitions. While the ordering of the transitions does not matter for expressing FSTs, the Transformer uses positional encodings which might have impacts on the pre-
training (though see Appendix D.1). We assemble the overall prefix by stacking the individual vectors h0, . . . , hn of the transitions p0
σ0 : ρ0−−−−→ q0, . . . , pn
σn : ρn−−−−→ qn. We group the transitions by their originating state (i.e. pi) and go over the states by their id, starting with 0, the initial state.
During pre-training, we might encounter FSTs with different numbers of transitions within the same batch. To handle this, we use padding encodings by reserving a special padding state and padding symbol in the embedding matrices of states and symbols. To initialize the prefix for fine-tuning, we use the average of 32 FST encodings (chosen at random) from pretraining.
For pre-training, we use embeddings of dimensionality 64 for states, embeddings of dimensionality 256 for symbols, and of dimensionality 16 to indicate final/non-final states.
Task embeddings. To enable faster adaption of the task embeddings than the rest of the model to fit a particular task, we use a higher learning rate for the task embeddings (1.0) than for the rest of the model (5 · 10−4) during pre-training. We also use a higher learning rate for the prefix during fine-tuning, analogously to SIP.
Because we have to store 40,000 task embeddings (one for each generated FST), TE requires a lot of memory. To reduce memory consumption, the task embeddings have a dimensionality of 180 and are up-projected to fit into the Transformer, analogously to W in Section 4.1. Nevertheless, the memory consumption of the embeddings is substantial and we store them on a separate GPU. Analogously to SIP-d4, we pre-train for 20 epochs.
Naive. We pre-train for a single epoch only as we found this achieved better results on downstream tasks than training for 20 epochs.
Set. We sample 200,000 examples according to the procedure described by Wu et al. (2022) to match our pre-training dataset size. Again, we found it more helpful for downstream task perfor-
mance to train for a single epoch rather than 20 epochs.
Fine-tuning Hyperparameters. Like pretraining, we finetune with the Adam optimizer. The main hyperparameters involved for both SIP and TE are the learning rates for the main model, and (separately) the learning rate of the tunable prefix. We chose these manually. Generally, we found that using a learning rate of 1.0 was a good choice for the prefix. Lester et al. (2021) report a similarly high learning rate to be useful for prompt tuning. For the rest of the model, we found 3 ·10−4 and 5 · 10−4 to work well for SIP-d4 and TE, respectively. For few-shot experiments, we use a somewhat smaller learning rate for TE for the main model (3 · 10−4). We noticed that T5-SIP-d4 (see Appendix B.3) was more sensitive to the learning rate choice in general than SIP-d4.
Hardware/Computational Budget. We ran our experiments on NVIDIA GeForce RTX 2080 Ti GPUs (11264MiB RAM) with driver version 535.54.03 and cuda version 12.2.
Pre-training SIP-d4 took around 30 hours for 20 epochs. One training run on synthetic data (including evaluation) takes around one hour, and one training run for low-resource grapheme-tophoneme conversion takes between 5 and 10 minutes.

heading: Future Work
text: In future work, we plan to extend this methodology to more expressive formalisms such as Pushdown Transducers which can be used for a wider range of downstream NLP tasks. Limitations
Our investigation focuses on FSTs with a relatively small number of states. However, the results in Section 5. 4 and in the experiments on grapheme-tophoneme conversion show that even pre-training with FSTs with a small number of states has positive impacts for tasks that require larger or more complex FSTs. The probing experiments show that the model simulates transitions between states similarly to an FST, but we did not perform a mechanistic interpretation of how exactly this is implemented in the weights. One potential mechanism behind the simulation behaviour is the construction of Liu et al. (2022) who show that Transformer decoders can simulate transitions between states of deterministic finite automata for strings of length up to n using O(log(n)) layers. Acquiring a specific inductive bias by means of learning to simulate a computational device is a general idea that could be applicable beyond FSTs but might be unsuitable in cases where (i) it is difficult to formulate a reasonable computational device to simulate (such as document classification and sentiment analysis beyond keyword spotting), or (ii) the computational device would be very hard or infeasible to simulate (e. g. Turing machines). Our experiments focus on moderately sized models (300M parameters) with an encoder-decoder architecture, and we did not investigate large decoderonly models. Our methodology can also be applied to decoder-only models, and we do not foresee any reasons why it could be less effective in that setup. Finally, we only consider the standard Transformer architecture and we leave it to future work to explore the impact of SIP on variants of the Transformer architecture designed for handling
long character sequences (Yu et al. , 2023) or in the context of state-space models (Wang et al. , 2024)