Backpack Language Models
Year: 2023

Abstract:
We present Backpacks: a new neural architecture that marries strong modeling performance with an interface for interpretability and control. Backpacks learn multiple non-contextual sense vectors for each word in a vocabulary, and represent a word in a sequence as a contextdependent, non-negative linear combination of sense vectors in this sequence. We find that, after training, sense vectors specialize, each encoding a different aspect of a word. We can interpret a sense vector by inspecting its (non-contextual, linear) projection onto the output space, and intervene on these interpretable hooks to change the model’s behavior in predictable ways. We train a 170M-parameter Backpack language model on OpenWebText, matching the loss of a GPT-2 small (124Mparameter) Transformer. On lexical similarity evaluations, we find that Backpack sense vectors outperform even a 6B-parameter Transformer LM’s word embeddings. Finally, we present simple algorithms that intervene on sense vectors to perform controllable text generation and debiasing. For example, we can edit the sense vocabulary to tend more towards a topic, or localize a source of gender bias to a sense vector and globally suppress that sense.

heading: 1 Introduction
text: Consider the prefix The CEO believes that ___, and the problem of debiasing a neural language model’s distribution over he/she. Intuitively, the bias for he originates in the word CEO, because replacing CEO with nurse flips the observed bias. A successful intervention to debias CEO must reliably apply in all contexts in which the word CEO appears; ideally we would want to make a non-contextual change to the model that has predictable effects in all contexts. In general, in all aspects of interpretability and control, it is desirable to make interventions with a tractable interface (e.g., noncontextual representations) that apply globally.
Such interventions are difficult in Transformer models (Vaswani et al., 2017) because their contextual representations are monolithic functions of their input. Almost any intervention on the model has complex, non-linear effects that depend on context. We would instead like models that enable precise, rich interventions that apply predictably in all contexts, and are still expressive, so they are a viable alternative to Transformers.
We address these challenges with a new neural architecture, the Backpack, for which predictions are log-linear combinations of non-contextual representations. We represent each word in a vocabulary as a set of non-contextual sense vectors that represent distinct learned aspects of the word. For example, sense vectors for the word “science” could encode types of science, connections to technology, notions of science being “settled,” or different aspects of the scientific process (replication or experiment) (Table 1). Sense vectors do not learn classic word sense, but more general aspects of a word’s potential roles in different contexts; in fact, they can be seen as a multi-vector generalization of classic word vectors (Mikolov et al., 2013).1
1Our code, sense vectors, language model weights, and demos are available at https://backpackmodels.science.
9103
To make interventions on sense vectors behave predictably in different contexts, a Backpack represents each word in a sequence as a linear combination of the sense vectors for all words in the sequence. The expressivity of a Backpack comes from the network that computes the weights of the linear combination as a function of the whole sequence; for example, in all our experiments we use a Transformer for this. Since sense vectors are softly selected depending on the context, they can specialize; each sense can learn to be predictively useful in only some contexts. The log-linear contribution of senses to predictions then implies that the interventions on sense vectors we demonstrate in Section 6 apply identically (up to a non-negative scalar weight) regardless of context.
Our experiments demonstrate the expressivity of Backpack language models, and the promise of interventions on sense vectors for interpretability and control. In Section 4 we train Backpack language models on 50B tokens (5 epochs) of OpenWebText; a Backpack with 124M parameters in the contextual network (and 46M parameters for sense vectors) achieves the perplexity of a 124M-parameter Transformer; thus one pays for more interpretability with a larger model size. In Section 5, we show that sense vectors specialize to encode rich notions of word meaning. Quantitatively, on four lexical similarity datasets (e.g., SimLex999), sense vectors of a 170M parameter Backpack outperform word embeddings of the 6B-parameter GPT-J-6B Transformer, and approach the performance of state-ofthe-art specialized methods for this task. Finally, in Section 6 we show that sense vectors offer a control mechanism for Backpack language models. For example, stereotypically gendered profession words (e.g., “CEO” or “nurse”) tend to learn a sense vector associated with this gender bias; by downscaling this sense vector, we greatly reduce disparity in contextual predictions in a limited setting.

heading: 2 The Backpack Architecture
text: In this section, we define the general form of the Backpack architecture. We then show how continuous bag-of-words word2vec (CBOW) (Mikolov et al., 2013) and Self-Attention-Only networks (Elhage et al., 2021; Olsson et al., 2022) are special cases of Backpacks.

heading: 2.1 Backpack General Form
text: A Backpack is a parametric function that maps a sequence of symbols x1:n = (x1, . . . ,xn) to a sequence of vectors o1:n = (o1, . . . ,on), where each symbol xi belongs to a finite vocabulary V and oi ∈ Rd. We call oi the Backpack representation of xi in the context of a sequence x1:n.
Sense vectors. For each x ∈ V , a Backpack constructs k sense vectors
C(x)1, . . . , C(x)k, (1)
where C : V → Rk×d. Sense vectors are a multivector analog to classic non-contextual word representations like word2vec or GloVe: we make this analogy precise in Section 2.2.
Weighted sum. For a sequence x1:n, the representation oi of element xi is a weighted sum of the predictive sense vectors for the words in its context: given contextualization weights α ∈ Rk×n×n,
oi = n∑
j=1
k∑
ℓ=1
αℓijC(xj)ℓ. (2)
The contextualization weights αℓij of a Backpack are themselves defined by a (non-linear) contextualization function of the entire sequence x1:n:
α = A(x1:n), (3)
where A : Vn → Rk×n×n.
The name “Backpack” is inspired by the fact that a backpack is like a bag—but more orderly. Like a bag-of-words, a Backpack representation is a sum of non-contextual senses; but a Backpack is more orderly, because the weights in this sum depend on the ordered sequence.
Backpack Models. A Backpack model is a probabilistic model that defines probabilities over some output space Y as a log-linear function of a Backpack representation o1:n ∈ Rn×d:
p(y|o1:n) = softmax (E(o1:n)) , (4)
where y ∈ Y and E : Rn×d → R|Y| is a linear transformation. Because Backpack models are loglinear in their representations, the sense vectors contribute log-linearly to predictions. This allows us to inspect a sense vector by projecting it onto the vocabulary via E and observe exactly how it will contribute to predictions in any context.
Models parameterized by the prevailing deep neural architectures—including LSTMs (Hochreiter and Schmidhuber, 1997) and Transformers— are not Backpacks because their output representations are (relatively) unconstrained functions of the entire sequence. By contrast, Backpack models may seem limited in expressivity: the representations oi are scalar-weighted sums of non-contextual vectors C(xj)ℓ. Contextual relationships between sequence elements can only be expressed through the weights α = A(x1:n). Nevertheless, our experiments show that an expressive contextualization weight network can represent complex functions by weighted sums of sense vectors, e.g., our 170M parameter Backpack LM uses a 124M-parameter Transformer to compute α, and achieves the loss of a 124M-parameter Transformer LM.
To place Backpacks in some historical context, we now show how two existing architectures can be described as Backpacks.

heading: 2.2 Continuous Bag-of-Words is a Backpack
text: The continuous bag-of-words word2vec model defines a probability distribution over a center word xc ∈ V conditioned on n context words x1:n.2 The model proceeds to (1) construct vector embeddings vx for each x ∈ V , and (2) uniformly average the embeddings of the context words to predict the
2Context in this setting is usually defined as words surrounding the center word.
center word:
vxc =
n∑
i=1
1 n vxi , (5)
p(xc | x1:n) = softmax(Uvxc), (6)
where U ∈ RV×d. We see that vxc is a Backpack representation by setting C(x) = vx ∈ R1×d in Equation (1) using a single sense vector (k = 1) and setting the contextualization weights in Equation (3) to be uniform: αℓij = 1n .
This connection to CBoW foreshadows the emergence of linguistic structures in the predictive sense vectors of Backpack models, just as these structures emerge in CBoW (Mikolov et al., 2013).

heading: 2.3 Single-Layer Self-Attention is a Backpack
text: The Backpack structure—define sense vectors (values), and use the sequence to determine how to sum them (weights)—may remind the reader of a single layer of self-attention. The key-query-value self-attention function is as follows:
oj =
n∑
i=1
k∑
ℓ=1
αℓijOV (ℓ)xj (7)
αℓ = softmax(x⊤K(ℓ)⊤Q(ℓ)x), (8)
where x ∈ Rn×d is (overloaded) to be a noncontextual embedding of the sequence, O ∈ Rd×d/k, and V (ℓ) ∈ Rd/k×d, where k is the number of attention heads. The self-attention function is a Backpack with C(xj)ℓ = OV (ℓ)xj . Self-attentiononly networks are studied in the context of, e.g., mechanistic interpretability (Elhage et al., 2021). A Transformer composes blocks of self-attention and non-linear feed-forward layers that combine information from the whole sequence; unlike a Transformer, the contextualization weights of a Backpack each select a non-contextual sense of a single word.

heading: 3 Language Modeling with Backpacks
text: In this section, we define a neural autoregressive language model parameterized by a Backpack. We use the standard softmax parameterization of the probability over the next token in a sequence, with a weight matrix E ∈ Rd×|V| that maps a representation oj ∈ Rd to logits E⊤oj ∈ R|V|:
p(xj | x1:j−1) = softmax(E⊤oj). (9)
Recall (Section 2.1) that Backpack representations oj are defined by sense vectors C(x) and contextualization weights αj . In Section 3.1 we describe a parameterization of C for the predictive sense vectors in Equation (1), and in Section 3.2 we describe a parameterization of A for the contextualization weight network in Equation (3). When oj is parameterized by a Backpack, we call a model of the form given by Equation (9) a Backpack LM.

heading: 3.1 Parameterizing senses
text: For the sense function C : V → Rk×d, we embed each x ∈ V into Rd and pass these embeddings though a feed-forward network FF : Rd → Rk×d:
C(x) = FF(Ex), (10)
where the embedding/projection matrix E is tied to the output matrix in Equation (9) (Press and Wolf, 2017). Note that we could define all k × |V| sense vectors using a lookup table, but this would be an enormous number of parameters as k grows large. Instead, we embed the words as Ex ∈ Rd, and then blow them up to Rd×k using shared weights. This may explain the related sense roles observed for different word types in Section 5.1.

heading: 3.2 Parameterizing contextualization weights
text: We parameterize A : Vn → Rk×n×n using a standard Transformer, followed by a layer of multiheaded key-query self-attention. That is, we pass an embedded sequence through a Transformer
h1:n = Transformer(Ex1:n) (11)
(with proper autoregressive masking and some position representation) and compute A(x1:n) = α, where
αℓ = softmax(h1:nK(ℓ)⊤Q(ℓ)h⊤1:n), (12)
for each predictive sense ℓ = 1, . . . , k with matrices K(ℓ), Q(ℓ) ∈ Rd×d/k. We can think of the k senses as heads and, for each head, the contextualization weights define a distribution of attention over words.3

heading: 4 Experiments Training Backpack LMs
text: In this section we specify the hyperparameters used to train Backpack and Transformer language models (Section 4.1), data and optimization procedure
3Note that the sense weights are normalized (1) independently for each sense, and (2) to sum to one over the sequence length.
(Section 4.2), evaluations (Section 4.3) and results (Section 4.4). We also show the necessity of learning k > 1 sense vectors to achieve strong language modeling performance (Section 4.5).

heading: 4.1 Models
text: We train three Transformer baseline models, which we label Micro (30M parameters), Mini (70M parameters), and Small (124M parameters; the same size as GPT-2 small). We also train Micro (40M), Mini (100M), and Small (170M) Backpack language models, for which the weighting function (Equation 11) is parameterized using the corresponding Transformer, and almost all extra parameters are in the non-contextual sense vectors.4 Backpacks thus cost extra parameters and compute beyond their underlying contextualization network. Except where stated, we use k = 16 sense vectors in all Backpacks (Section A).
We use a reduced sequence length of 512 for all models, and the 50,257-subword GPT-2 tokenizer. Model hidden dimensionalities, layer counts, and head counts are reported in Table 9.

heading: 4.2 Data & Optimization
text: We train all models on OpenWebText (Gokaslan and Cohen, 2019), a publicly available approximate reconstruction of the English WebText corpus used to train the GPT-2 family of models (Radford et al., 2019). We use a batch size of 524,288 tokens, and train all models for 100,000 gradient steps for a total of 52B tokens; training for longer is known to make marginal difference for small models (Hoffmann et al., 2022). The size of OpenWebText means this is roughly 5 epochs. We use cross-entropy loss and the AdamW optimizer, with a warmup of 5,000 steps and linear decay to zero.

heading: 4.3 Evaluations
text: Before our experiments in interpretability and control, we check the expressivity of Backpacks. We evaluate models on perplexity for a held out set of OpenWebText, perplexity and accuracy for the (OpenAI variant of) LAMBADA evaluation of long-distance dependencies (Radford et al., 2019; Paperno et al., 2016), perplexity on Wikitext (Merity et al., 2017), and BLiMP English linguistic competence accuracy (Warstadt et al., 2020) evaluated using the EleutherAI harness (Gao et al., 2021) (Version 1).
4There are a negligible number of additional parameters in the final key-query Backpack operation (Equation 12)).

heading: 4.4 Discussion
text: Comparing each Backpack LM to a Transformer LM of equivalent specification to the Backpack’s contextualization network, we see that the Backpack performs roughly as well (Table 2). Again, the Backpack has more parameters, a tax for the interface provided by sense vectors. During training, we find that Backpack language models take longer to converge than Transformers. Curiously, while the Small Backpack and Transformer achieve almost identical OWT perplexity, the Backpack language models perform substantially better on LAMBADA and Wikitext, but worse on BLiMP.

heading: 4.5 Effect of varying the number of senses
text: To study the impact of the number of sense vectors on language modeling performance, we train Mini-sized Backpack language models on a reduced schedule of 50,000 gradient steps, for k ∈ {1, 4, 16, 64} sense vectors. The perplexities for k = 1, 4, 16, 64 are 38.6, 29.3, 26.0, and 24.1, demonstrating the necessity of a non-singleton set of sense vectors. Table 8 contains the full results.

heading: 5 Emergent Structure in Sense Vectors
text: Backpack language model sense vectors are not trained using a supervised notion of word sense, but implicitly specialize to encode different shades of a word’s predictive use. In this section, we qualitatively examine sense vectors (Section 5.1) and quantitatively demonstrate their effectiveness in computing lexical similarity and relatedness (Section 5.2). Taken together, this suggests that sense vectors can provide a high-level interface for intervention, which we explore in Section 6.

heading: 5.1 Visualizing Senses
text: Empirically, trained Backpack models associate specific sense vector indices with different roles for
prediction. We interpret these roles by picking a sense ℓ of a word x, and projecting this sense onto the word embeddings: E⊤C(x)ℓ ∈ R|V|. Note that this is exactly (up to a scalar) how this sense contributes to any prediction of the model. We interpret a sense vector’s role by reporting the words with the highest score under this projection.
Table 3 visualizes a few of these senses. For example, sense 12 seems to encode a broad notion of relatedness for almost all words; sense 3 encodes particulars of the bigram distribution given x; sense 14 seems to encode both associated objects for verbs, and noun modifier dependency children for nouns. In Section 5.2 we show that sense 14 encodes a powerful notion of verb similarity.

heading: 5.2 Lexical Relationship Tests
text: Classic lexical-relatedness and similarity tests measure the extent to which a similarity function on pairs of words correlates with human-elicitied notions of similarity. Similarity functions derived from word embeddings are evaluated by Spearman correlation between the predicted and true similarity rank-order. Early non-contextual embeddings like COALS (Rohde et al., 2005), word2vec (Mikolov et al., 2013), and GloVe (Pennington et al., 2014) have recently been outperformed by word embeddings derived by distillation of contextual networks (Bommasani et al., 2020; Gupta and Jaggi, 2021; Chronis and Erk, 2020). We evaluate Backpack LM sense vectors on similarity datasets SimLex999 (Hill et al., 2015), SimVerb3500 (Gerz et al., 2016), and relatedness datasets RG65 (Rubenstein and Goodenough, 1965) and (Agirre et al., 2009).
Senseℓ Cosine. For all ℓ ∈ {1, . . . , k}, we define a similarity function based only on sense ℓ:
Simℓ(x,x′) = cossim(C(x)ℓ, C(x′)ℓ), (13)
where cossim is cosine similarity. Intuitively, we expect that some senses may specialize to learn lexical relatedness or similarity.
Minimum Sense Cosine. Because each sense encodes a different aspect of a word’s meaning, we might expect that highly similar words are similar across all senses. We test for this strong form of similarity using
Simmin(x,x′) = min ℓ Simℓ(x,x′) (14)
Other methods. We evaluate embeddings from the tied softmax/embedding matrices of the much larger GPT-2-1.5B (Radford et al., 2019) and GPTJ-6B (Wang and Komatsuzaki, 2021), classic word embeddings (from Bommasani et al. (2020)) and
state-of-the art specialized methods using either a single vector per word (Gupta, 2021) or many vectors (Chronis and Erk, 2020).
Discussion. Sense 12 (the “synonym” sense) performs well across datasets, matching or outperforming embeddings like GPT-2-1.5B and GPT-J-6B (Except GPT-J-6B on RG-65). Sense 14, the “verb objects” sense, performs best on just verb similarity (VerbSim3500), and the minimum similarity over senses works especially well on noun lexical similarity (SimLex999.) Our methods approach the performance of state-of-the-art methods; despite being trained for a very different task, sense vectors encode substantial lexical information (Table 4).

heading: 6 Sense Vectors for Control
text: In this section, we demonstrate several proof-ofconcept methods that leverage sense vectors for controlling LM behavior.

heading: 6.1 Topic-controlled generation
text: Given a bag-of-words target b ∈ R|V|, e.g., arts, culture, we would like to bias generation towards sequences related to concepts related to these terms. Our algorithm proceeds in three parts. First, we sort sense vectors by log-probability assigned to b, that is, b⊤(E⊤C(x)ℓ).5 Second, based on the scores, we assign a re-weighting factor δ to each sense; senses with the higher scores weighted more. (See Section D for details.) Third, we generate from
5We divide this term by the maximum absolute logprobability of the sense vector, maxx∈V x⊤(E⊤C(x)ℓ).
the Backpack using the re-weighted sense vectors, reducing δ back to 1 as the topic is introduced. The updated backpack equation is
oi =
n∑
j=1
k∑
ℓ=1
αℓijδℓijC(xj)ℓ, (15)
where δijℓ is the re-weighting. Intuitively, the semantic coherence of sense vectors may imply that upweighting senses with affinity to the target bagof-words richly upweights related words and topics. We give details as to how we perform the sense reweighting and the annealing in Section D.
Evaluation. We use the label descriptors of the topic classifier of Antypas et al. (2022), with 17 categories (sports, arts & culture, health,. . . ), as the bag-of-words for control. We evaluate control accuracy as the percent of generations to which the classifier assigns the correct topic label, and overall generation quality and diversity using MAUVE scores (Pillutla et al., 2021).6
Results. We compare to Plug-and-Play Language Models (PPLM; Dathathri et al. (2019)), a considerably slower, gradient-based control method using our Small Transformer model. We generate 500 samples from each model for each topic across a range of strengths of control. We find that sense controlled generation provides at least as strong control as PPLM (Figure 2), though the MAUVE scores of the unmodified Transformer are higher than the Backpack.) Results and examples are provided in the Appendix in Tables 12, 16, 17, 18.
6We concatenate generations across the 17 categories and compute MAUVE against OpenWebText validation examples.

heading: 6.2 Mitigating gender bias
text: Through inspection, we learned that sense vector 10 of many stereotypically gendered profession nouns (nurse, CEO, teacher) coherently express the stereotype through pronouns. Table 13 gives examples of these senses. We attempt to mitigate gender bias in Backpack behavior on these gendered profession nouns by turning down sense 10 (multiplying by a scalar less than 1).
We took an existing set of stereotypically gendered profession nouns from WinoBias (Zhao et al., 2018), and constructed a simplified setting in which a single profession word is in each context, and a third-person nominative pronoun (e.g., he/she/they) is acceptable, e.g., My CEO said that__. The full set of nouns and prompts is in Section D.2. We evaluate models on the average of the bias of probabilities of him vs her as follows:
E x∈prompts
[ max ( p(he | x) p(she | x) , p(she | x) p(he | x) )] .
Baseline. To debias a Transformer with an analogous method, we take inspiration from Bolukbasi et al. (2016). We take Exhe − Exshe as an estimate of a gender bias direction, and project the embedding Exnurse either to the nullspace of this direction or only partially remove it.
Results. A perfectly unbiased model would achieve ratio 1, whereas the unmodified Transformer achieves 7, and with nullspace projection, 6.72 (Table 5). Finding the optimal fraction of the gender bias direction to remove per profession does not improve further. For Backpacks, we find that removing sense 10 from the profession word (setting it to zero) reduces the bias score from 4.34 to 2.88. Learning the optimal removal fraction per profession achieves 2.16, for a total reduction of
65%.7 In Figure 3, we demonstrate the clear effect of ablating sense 10 on the most likely words in one of these contexts.8

heading: 6.3 Knowledge editing
text: Sense vectors show promise for use in knowledge editing (De Cao et al., 2021)—editing a model’s predictions about world knowledge. In particular, many associations with proper nouns can be localized to sense vectors in that noun. In this qualitiative proof-of-concept, we edit the sense vectors of a target word x (e.g., MacBook to remove associations with a word xr (e.g., Apple) and replace those associations with another word xa (e.g., HP). Intuitively, this intervention ensures that whenever the contextualization weights would point to a sense vector in MacBook to predict words associated with Apple, it now predicts words associated with HP.
7Curiously, Backpacks are overall less biased to begin with (in this setting); we don’t have a strong hypothesis as to why.
8It is incidental that sense 10 encodes gender bias as opposed to another sense index; the consistency in index across words may be due to parameter sharing in C.
We project each sense vector of x to the nullspace of Exr, and then add in Exa:
C̃(x)ℓ = C(x)ℓ + C(x)⊤ℓ Exr ∥C(xr)ℓ∥22 ( Exa ϕ − Exr ) ,
where ϕ = ∥Exa∥ 2 2
∥Exr∥22 is a normalization term to ac-
count for the differing norms of Exa and Exr. Intuitively, this projection modifies each sense vector in measure proportional to how much xr was predicted by that sense. So, senses of MacBook that would added mass to Apple now add mass to HP; unrelated senses are not affected. In Table 6, we show samples providing intuition for how MacBook evokes HP instead of Apple, but is otherwise semantically and syntactically maintained.

heading: 7 Related Work
text: Representation learning in NLP. Learning probabilistic models of text for use in representation learning and identifying resulting structure has a long history in NLP, from non-contextual word vectors (Schütze, 1992; Rohde et al., 2005; Turney, 2010; Mikolov et al., 2013; Bojanowski et al., 2017) to contextual networks (Elman, 1990; Bengio et al., 2000; Collobert and Weston, 2008; Sutskever et al., 2011; Peters et al., 2018; Radford et al., 2018). Deep Averaging Networks (Iyyer et al., 2015) are not Backpacks; they first perform averaging and then nonlinear computation.
Interpretability for Control of NLP networks. A burgeoning body of work attempts to intervene on monolithic neural networks for interpretability and control (Meng et al., 2022, 2023), and for mechanistic understanding (Olsen et al., 2021; Elhage et al., 2021). Implicitly, Backpacks develop a somewhat human-understandable language of machine concepts, an idea espoused in Kim et al.
(2018); Koh et al. (2020). The connections between interpretation and control are rich; much work has gone into the detection and extraction of emergent structure in networks (Hupkes et al., 2018; Liu et al., 2019) as well as subsequently modulating behavior (Lakretz et al., 2019; Eisape et al., 2022).
Generalized Additive Models. Generalized Additive Models (GAMs; Hastie and Tibshirani (1986)) are a function family that (1) independently transforms each input feature, (2) sums these transformations of inputs and (3) applies a non-linear link function (e.g., softmax):
f(x1:n) = Φ (r1(xi) + · · ·+ rn(xn)) (16)
Treating each word-position pair as a feature, Backpacks are not GAMs because they include a weighting α that depends on all features. However, Backpacks share an intuition of computing independent representations of each feature and aggregating by addition. Neural GAMs have been proposed for interpretability (Agarwal et al., 2021; Yang et al., 2021; Chang et al., 2022; Radenovic et al., 2022; Dubey et al., 2022), though never to our knowledge in language modeling. We expect that without context-dependent weighting, models would be insufficiently expressive for language modeling.

heading: 8 Discussion
text: In this section, we address a few natural questions about the expressivity and interpretability of Backpacks, highlighting the limits of our knowledge.
How do Backpacks compare to architecture X? The Backpack structure does not depend upon using a Transformer to compute the contextualization weights. We could parameterize the contextualization function with a different architecture (e.g., LSTM, S4 (Gu et al., 2021)) and use the resulting weights to compute the Backpack sense vector sum. This architecture, e.g., the Backpack-S4, could then be compared to the standard S4 architecture.
Are Backpacks as expressive as Transformers? We don’t know. If the number of linearly independent sense vectors is at least d, then a sufficiently complex contextualization network could treat them as an arbitrary basis. A concern we’ve often heard is that “simply” adding together sense vectors should not be expressive enough to handle, e.g., negation. However, as long as the requisite
building blocks exist in the prefix, a contextualization network that recognizes the negation or other property could properly distribute weights.
Are Backpacks inherently interpretable? No, but we believe no architecture is. Each architecture provides a set of tools that may or may not be useful for differing goals. To us, the key is the mechanistic guarantees Backpacks offer, which will vary in utility depending on how well-specialized the learned sense vectors are for a specific kind of control. Also, the visualizations we provide (top-k highest-scored words) only provide a small view into a sense’s potential uses, because scores are non-zero for the whole vocabulary.
Are Backpacks as compute-efficient as Transformers? At a glance, no. Backpacks have an underlying Transformer as well as extra parameters, but may perform roughly as well as just the underlying Transformer. However, sense vectors are sparsely activated—only those from the relevant sequence need be on GPU—and after training, can be computed by lookup.
Why do sense vectors specialize? Ablations in Table 8 show that they should at least learn to be linearly independent, since linear dependence is equivalent to having having fewer sense vectors, which causes higher perplexity. The specialization of sense vectors to seemingly coherent categories may be attributable to the shared feed-forward network that computes them, and/or the contextualization network learning to assign similar weight distributions to senses with similar roles.
Are sense vectors like “word senses?” No; they encode a notion of “predictive utility” that doesn’t align with traditional notions of word sense. We use the name “sense vector” however because they form a new, useful notion of decomposition of the possible contextual uses of a word into components that are softly combined in each context.

heading: 9 Conclusion
text: Non-contextual word2vec embeddings initiated modern deep learning research in NLP, and have fascinating geometric structure. Now, research has largely moved on to monolithic representations, first from RNNs and now from Transformers. Our work suggests that we can have both rich lexical structure and interventions, and strong contextual performance, in a single model.

heading: 10 Acknowledgements
text: The authors would like to thank Amita Kamath, Steven Cao, Xiang Lisa Li, Ian Covert, and the Stanford NLP Group community for useful discussions. Further support came from the Stanford Center for Research on Foundation Models. Christopher Manning is a CIFAR Fellow. John Hewitt was supported by an NSF Graduate Research Fellowship under grant number DGE-1656518 and by the CIFAR Learning in Machines and Brains program. We gratefully acknowledge the support of a PECASE Award to Percy Liang.

heading: 11 Limitations
text: There is a fundamental uncertainty in whether Backpack language models will continue to scale with parameters and data and be viable alternatives to Transformers at larger model scales. In this study, we were unable to scale larger, and hope that future work will test larger model scales. In a similar vein, we do not verify that Backpack language models perform well across multiple languages. We also do not consider, e.g., finetuning Backpacks on other tasks, or masked language modeling—there is a wide range of possible uses that remain to be verified.
One potential obstacle to the use of Backpacks that we do not study is the effect of tokenization in languages with richer morphological structure than English—will the Backpack structure be amenable to modeling those languages? This may be difficult because, intuitively, the interpretability and control of Backpacks relates to the semantics of individual tokens. Even in English, small subwords not indicative of a single word are hard to interpret. What we hope to have provided is a sufficient set of experiments to motivate the further exploration of Backpacks.

heading: 12 Ethics
text: This paper describes and releases an open-domain language model trained on a largely unfiltered subsection of the (mostly English portions of the) textual internet, and describes methods for interpreting and controlling said model. Any control method that can be used to help understand and guide the generation of a model can be used to more effectively generate toxic or illegal content. Despite this, we do expect that, overall, the benefit of deeper insight into Backpack language models is a step
in the right direction. In particular, explanations based on the structure of Backpacks may be able to provide insights into the mechanisms behind model behaviors, increasing transparency.
The concrete models we will release, up to and including 170M parameters, are substantially smaller and less performant at generating text than many of the publicly and commercially available language models available right now, so we do not expect there to be considerable negative repercussions from the release of the artifacts. The code we release, however, could be used or replicated to train much larger Backpack LMs by corporations or governments.

heading: A Language Model Training Details
text: We use the FlashAttention codebase (Dao et al., 2022) which in turn relies on the Huggingface codebase (Wolf et al., 2020) and NumPy (Harris et al., 2020). We perform no preprocessing of OpenWebText. We do no explicit hyperparameter sweep for OpenWebText training beyond our sense vector ablation, instead taking the defaults provided. We train our models on 4 A100 (40GB) GPUs. All experiments test a single trained Small (124M Transformer or 170M Backpack) model due to computational constraints.
A.1 The feed-forward sense network.
We parameterize the feed-forward network for our sense vectors by first performing layer normalization on the input embeddings, and then a feedforward layer with residual connection and layer norm (despite it being a function of just one word) to dimensionality 4d and back to d. Then a subsequent feed-forward network to hidden dimensionality 4d and then up to k ∗ d. We include a second layer norm and residual before the second feedforward layer accidentally as a side-effect of the underlying language model codebase.
For our experiments ablating k in Section 4.5, the second feed-forward component maps to d and then kd, not 4d → kd.

heading: B Extra evaluations
text: B.1 Timing Benchmarking
To benchmark the speed of each model, we used a single A100 GPU, running the forward pass of each model with a sequence length of 512 and a batch size of 32. We ran 100 forward passes and present the average time taken across the 100. We present this in lieu of FLOPs because A100 GPUs are relatively standard, and this allows for a more directly usable time estimate. Results are in Table 7. We find that Backpacks take roughly 1.4x as long to run as their underlying Transformers.

heading: C Lexical Similarity Details
text: To handle words in the lexical similarity datasets that don’t appear as single words in the tokenizer, we use one of two methods. We either average all subwords, or take the first subword. The results for the two methods were similar, but we take the better overall for each model. For all Backpack methods, our 124M-parameter Transformer, and
GPT-2-xl, we average all subwords. For GPT-J (which uses the same tokenizer), we take the first subword.

heading: D Sense Vector Control Details
text: D.1 Topic control details
The full results are in Table 12. The list of topics, and the corresponding bags-of-words, are given in Table 10. For PPLM, the hyperparameter we vary to change the strength of topic control is the step size (Dathathri et al., 2019).
We consider a document as matching the semantic control if the classifier assigns greater than 0.5 probability to the attempted class. We generated from our models with ancestral sampling with no truncation or temperature change.
Topic control. Let b ∈ R|V| be the many-hot vector defined by the bag of words input to the control problem. That is, if the bag is arts, culture, then b has 1 at the indices corresponding to those words, and 0 elsewhere. To determine the initial weights δ for each sense vector, we first sort all |V| ∗ k sense vectors by decreasing normalized dot product with the bag of words vector:
s(C(x)) = b⊤E⊤C(x)
max(E⊤C(x)) (17)
We then take the 0.95, 0.80, and 0.60 quantiles of these scores to determine how to weight the vectors. Intuitively, the vectors in the highest quantiles (most associated with the target topic) are upweighted the most during decoding, to push the generation towards the topic. The three quantiles partition the set of scores into 4, which are given separate δ values; the exact 4 depend on the strength of control (i.e., different points in Figure 2.) The exact δ upweighting for each point are given in Table 11.
Topic annealing. From the the beginning value of δ given above, we anneal back to 1 as follows. For each sense C(xj)ℓ, we compute the total sum of non-negative log-probability assigned by the sense to the set of words generated so far, intuitively to compute whether the words already generated express the meaning intended by the sense:
aC(xj)ℓ =
n∑
i=1
max ( x⊤i E ⊤C(xj)ℓ), 0 ) . (18)
We then re-weight by a term dependent on the sequence index to upweight terms near to the most recently generated text:
bC(xj)ℓ = σ ( −aC(xj)ℓf + 6 ) ∗ (1 + j) /100
(19)
where j is the index of the word of the sense vector in the generated text, and f is a scaling constant set to 7.5 divided by the maximum δ in the experiment (the maximum of each row in Table 11.)
Finally, we compute the annealed δ as a soft combination, weighted by bC(xj)ℓ , of the maximum delta and the default of 1:
δℓij = bC(xj)ℓδℓij + (1− a) ∗ 1. (20)
D.2 Gender bias mitigation details For the third-person singular verb they, we found that our sense intervention on sense 10 slightly increases the probability of they relative to he or she.
The full set of nouns and prompts we use is as follows. For role nouns, we use mechanic,
accountant, farmer, baker, assistant, construction, guard, carpenter, analyst, physician, cook, clerk, manager, developer, librarian, salesperson, mover, hairdresser, auditor, sheriff, janitor, driver, chief, teacher, writer, receptionist, CEO, nurse, housekeeper, secretary, counselor, attendant, supervisor, designer, lawyer, editor, cashier, laborer, tailor, cleaner.
For evaluation, we use the prompts in Table 14. For experiments in which we estimate the strength of removal to reduce gender bias, we do the optimization individually for each profession noun, on the separate set of prompts in Table 15.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
9
3 A2. Did you discuss any potential risks of your work? 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
3 A4. Have you used AI writing assistants when working on this paper? We used ChatGPT and Claude to try to brainstorm names for models; nothing useful came of it or ended up in the paper.
B 3 Did you use or create scientific artifacts? Section 5,6,7
3 B1. Did you cite the creators of artifacts you used? Section 5,6,7, Appendix
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.2
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.1
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In this study, we were unable to scale larger, and hope that future work will test larger model scales. In a similar vein, we do not verify that Backpack language models perform well across multiple languages. We also do not consider, e. g. , finetuning Backpacks on other tasks, or masked language modeling—there is a wide range of possible uses that remain to be verified. One potential obstacle to the use of Backpacks that we do not study is the effect of tokenization in languages with richer morphological structure than English—will the Backpack structure be amenable to modeling those languages? This may be difficult because, intuitively, the interpretability and control of Backpacks relates to the semantics of individual tokens. Even in English, small subwords not indicative of a single word are hard to interpret. What we hope to have provided is a sufficient set of experiments to motivate the further exploration of Backpacks