A Measure-Theoretic Characterization of Tight Language Models
Year: 2023

Abstract:
Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings. In most cases, the estimated distribution sums to 1 over all finite strings. However, in some pathological cases, probability mass can “leak” onto the set of infinite sequences. In order to characterize the notion of leakage more precisely, this paper offers a measure-theoretic treatment of language modeling. We prove that many popular language model families are in fact tight, meaning that they will not leak in this sense. We also generalize characterizations of tightness proposed in previous works.

heading: 1 Introduction
text: Language modeling is a core task in natural language processing. As canonically defined, language modeling involves estimating a distribution over the set of strings over a given alphabet. If the alphabet is the set of words in a language,1 then a language model can be thought of as a distribution over the language’s sentences. Since Shannon (1948), language modeling has been used to estimate statistical properties of language and has become essential for computational linguistics research (Hale, 2001; Meister et al., 2021). Further, it is also central to a wide range of natural language processing applications, whether as a source model in a noisy channel architecture (Weaver, 1955; Jelinek, 1976), as a way of learning better representations of language (Peters et al., 2018), or, more recently, for prompted generation (Brown et al., 2020), where the distribution defined by a language model is employed in tasks like question-answering (Petroni et al., 2019), style transfer (Reif et al., 2022), and even sequence tagging (Liu et al., 2022).
More formally, a language model is typically defined to be a distribution over the countably infinite
1Or perhaps alphabetic symbols or subwords; see, e.g., Bostrom and Durrett (2020).
set Σ∗ of all (finite) strings (Booth and Thompson, 1973).2 However, it has been shown that some classes of autoregressive language models have parameter settings in which the generative process terminates with probability < 1. Welleck et al. (2020) discuss this issue for recurrent neural network language models. Models whose generative process may fail to terminate are called non-tight (Chi, 1999, who discussed non-tight PCFGs). If an autoregressive language model is non-tight, it may generate infinite sequences and MCMC algorithms over such models will not mix to the correct distribution.
It is here that a subtlety arises: the set of infinite sequences is uncountably infinite. Properly treating a distribution over this sample space requires a modicum of measure theory.3 To clarify the situation, we review the measure-theoretic treatment of distributions over infinite sequences. We then make use of a termination symbol EOS to define a random variable whose value can be a string, i.e., an element of Σ∗, or an infinite sequence. In a tight language model, this random variable has probability 1 of being a string and hence finite.
Beyond offering a measure-theoretic formalization, our paper also demonstrates how tightness relates to the Borel–Cantelli lemmata, simplifying a recent result by Meister et al. (2022). To conclude our paper, we analyze several language modeling architectures and give conditions on their tightness. We demonstrate that n-gram language models— and more generally, language models defined by stochastic finite-state automata—can be non-tight, and we give a simple necessary and sufficient condition for tightness in terms of the inverse of the automaton’s transition matrix. This builds on a known
2Recall that Σ∗ def= ⋃
n Σ n where for n ≥ 0, Σn is the set
of strings of length n. The ∗ is the Kleene closure operator. 3Indeed, our treatment resolves an imprecision present in the literature. For instance, Welleck et al. (2020) discusses the probability of infinite sequences despite using the canonical definition of a language model as a distribution over Σ∗.
9744
result due to Lehmann (1977). We also discuss when neural language models become non-tight. We prove that Transformer-based language models (Vaswani et al., 2017; Brown et al., 2020) are always tight and that recurrent neural language models are always tight when they employ a bounded activation function. However, we also exhibit a recurrent neural network (RNN) language model with a ReLU activation (Nair and Hinton, 2010) that is non-tight in a simpler construction than the one offered by Chen et al. (2018). As a byproduct, we also generalize and strengthen the results given by Welleck et al. (2020), who give a sufficient condition for tightness of recurrent neural language models in terms of the norm of the hidden state.

heading: 2 Motivating Examples
text: Let Σ be an alphabet, i.e., a finite set of symbols, and let EOS /∈ Σ be a distinguished end-ofsequence symbol. Let Σ def= Σ∪{EOS}. A string of length n ≥ 0 is a finite sequence x = x1x2 . . . xn where each xt ∈ Σ. By convention, we say that xn+1 = EOS, although xn+1 is not an element of the sequence x. For any integer 1 ≤ t ≤ n+ 1, we write x<t for the prefix x1x2 · · ·xt−1.
We now begin to distinguish between “language models” and “sequence models.” As is traditional in the NLP literature, we henceforth define a language model to be a probability distribution over the countable set Σ∗ of all strings (see Def. 3.4). It is popular to specify such a distribution in terms of its conditional probabilities p̄(xt | x<t). Definition 2.1. An autoregressive sequence model (ASM) is a conditional probability distribution p̄(xt | x<t) where xt ∈ Σ and x<t ∈ Σ∗.
If p̄ is an ASM, then we define a non-negative function p over Σ∗ by p(x) def= ∏n+1 t=1 p̄(xt |
x<t) = p̄(EOS | x) ∏n t=1 p̄(xt | x<t), where n denotes the length of x. But is p a language model? Not always, since as we will see below, it is possible for p(Σ∗) def=∑ x∈Σ∗ p(x) < 1. Of course this “bad” case never happens if the ASM’s conditional probabilities are derived from a known LM, in which case p simply recovers that LM.4 In this case clearly p(Σ∗) = 1.
4That is, suppose the ASM’s conditional probabilities match the conditional probabilities of some known language model p0: that is, p0(Xt = xt | X1 . . . Xt−1 = x<t) = p̄(xt | x<t) whenever the former conditional probability is well-defined under the language model p0, i.e., whenever xt ∈ Σ and x<t ∈ Σ∗ with p0(X1 . . . Xt−1 = x<t) > 0.
It follows that if p(Σ∗) < 1, then the ASM’s conditional probabilities do not match the conditional probabilities of any language model p0.
We now exhibit such a “bad” ASM. Although the conditional probability distributions p̄(· | x<t) each sum to 1 over Σ, they fail to combine into a model p that sums to 1 over Σ∗ (i.e., a language model).
Example 2.2 (non-tight bigram model). Consider the bigram model defined in Fig. 1a over the alphabet Σ = {a, b}. Under this model, any finite string that contains the symbol b will have probability 0, since p̄(EOS | b) = p̄(a | b) = 0. This implies p(Σ∗) = ∑∞ n=0 p(a n) = ∑∞ n=0(0.7) n ·0.1 =
0.1 1−0.7 = 1 3 < 1.
Example 2.3 (tight bigram model). In contrast, in Fig. 1b, obtained from Ex. 2.2 by changing the arcs from the b state, p(Σ∗) = 1. See App. B for details of this calculation.
Ex. 2.2 above confirms that the autoregressive formulation does not necessarily yield p that is a valid distribution over Σ∗.
But if p is not a language model, what is it? It is intuitive to suspect that, in a model with p(Σ∗) < 1, the remainder of the probability mass “leaks” to infinite sequences, i.e., the generative process may continue forever with probability > 0. We will make this intuition formal in §3. By analogy with Chi and Geman (1998) and Cohen and Johnson (2013), we refer to such models as non-tight.5
The non-tightness of Ex. 2.2 is related to the fact that the probability of EOS is 0 at some states, in contrast to Ex. 2.3. However, requiring p̄(EOS | x<t) > 0 for all prefixes x<t is neither necessary nor sufficient to ensure tightness. It is not necessary because one can, for example, construct an ASM in which p̄(EOS | x<t) = 0.1 when t is even but = 0 otherwise. Such a model generates only odd-length strings but is tight. It is not sufficient because of the following example, in which p̄(EOS | x<t) is always positive but decays so rapidly toward 0 that the generative process might continue forever.
Example 2.4 (non-tight RNN). Consider an RNN over a small alphabet Σ = {a, EOS} with the fol-
Then by the chain rule of probability, p(x) = p0(x) for each x ∈ Σ∗. Thus p = p0, so p is a language model.
5In Chi and Geman (1998) and Cohen and Johnson (2013), a PCFG is non-tight if its generative process may not terminate, and consequently the total probability of all finite trees is less than 1.
lowing hidden state recurrence:
h0 = 0, ht = ReLU(ht−1 + 1). (1)
In this case, the hidden state admits a closed-form expression ht = t ∈ R. Setting the (1-dimensional) embedding of the alphabet to be va = 1 and vEOS = 0, we arrive at
p̄(EOS | x<t) = softmax(vaht, vEOSht)EOS = e
0·t e1·t+e0·t = 1 et+1 > 0. (2)
The EOS probability is always strictly positive, but Thm. 4.7 shows that this sequence model is nontight. Numerically, p(Σ∗) ≈ 0.702 < 1.
On the other hand, an ASM may be tight after all if the probability of EOS decays more slowly—even when it still approaches 0.
Example 2.5 (tight RNN). Consider again an RNN over the alphabet Σ = {a, EOS} with the following recurrence using softplus activation:6
h1 = 0, ht = log(exp(ht−1) + 1). (3)
Starting from h1 = 0 = log 1, a simple induction argument shows that
ht = log(exp log(t− 1) + 1) = log t. (4)
Again, setting va = 1 and vEOS = 0, we arrive at
p̄(EOS | x<t) = softmax(vaht, vEOSht)EOS (5) = e
0·log t e1·log t+e0·log t = 1 t+1 > 0.
This decays slowly to 0: limt→∞ p̄(EOS | x<t) = 0, but since ∑∞ t=1 p̄(EOS | x<t) = ∞, Prop. 4.3 below implies that this ASM is tight. Finally, we illustrate the peril of not treating distributions over uncountable sets carefully.
6We use softplus instead of ReLU to simplify arithmetics.
Example 2.6 (infinite coin toss). Consider the infinite independent fair coin toss model, where we aim to place a distribution over {H, T}∞, the uncountable set of infinite sequences of {H, T}. Intuitively, such a distribution corresponds to an ASM in which for all x<t, p̄(H | x<t) = p̄(T | x<t) = 12 and p̄(EOS | x<t) = 0. Clearly, each individual infinite sequence over {H, T} should be assigned probability (12)
∞ = 0. Without a formal foundation, one may arrive at the following paradox:
1 = p ({H, T}∞) = p (⋃ ω∈{H,T}∞{ω} )
(6)
= ∑
ω∈{H,T}∞ p({ω}) =
∑
ω∈{H,T}∞ 0
? = 0.
Together, these examples suggest that one must take care to characterize tightness. And, to the authors’ surprise, it does not appear as if such a careful characterization yet exists in the NLP literature.

heading: 3 The Language Model Measure
text: In this section, we rigorously characterize the kind of distribution induced by an ASM. As mentioned earlier, an ASM can lose probability mass to the set of infinite sequences, Σ∞. However, Σ∞, unlike Σ∗, is uncountable, and it is due to this fact that we need to work explicitly with the measure-theoretic formulation of probability.

heading: 3.1 Measure-Theoretic Background
text: The goal of measure-theoretic probability is to assign probability to subsets of an outcome space Ω. For instance, in Ex. 2.6, Ω = {H, T}∞. However, in the course of the study of measure theory, it has become clear that for many common Ω, it is impossible to assign probabilities in a way that satisfies a set of reasonable desiderata.7 Consequently, the
7Measure theory texts commonly discuss such desiderata and the dilemmas that comes with them. See, e.g., Chapter 7 in Tao (2016), Chapter 3 in Royden (1988) or Chapter 3 in Billingsley (1995). We also give an example in Thm. 3.5.
standard approach to probability theory resorts to only assigning probability to certain “nice” subsets of Ω, which are referred to as events or measurable subsets, as in the theory of integration or functional analysis. The set of measurable subsets is commonly denoted as F (Def. 3.1), and a probability measure P : F → [0, 1] is the function that assigns a probability to each measurable subset. As it turns out, the following simple and reasonable requirements imposed on F and P are enough to rigorously discuss probability (Kolmogorov, 1933).
Definition 3.1. Let P(Ω) be the powerset of Ω. Then F ⊆ P(Ω) is called a σ-algebra (or σ-field) over Ω if the following conditions hold:
1) Ω ∈ F , 2) if E ∈ F , then its complement Ec ∈ F , 3) if E1, E2, . . . is a finite or infinite sequence of
sets in F , then⋃nEn ∈ F . If F is a σ-algebra over Ω, we call the tuple (Ω,F) a measurable space.
A measurable space guarantees that operations on countably many sets are always valid, and hence permits the following definition.
Definition 3.2. A probability measure P over a measurable space (Ω,F) is a function P : F → [0, 1] such that
1) P(Ω) = 1, 2) if E1, E2, . . . is a finite or infinite sequence
of disjoint sets in F , then P(⋃nEn) =∑ n P(En).
In this case we call (Ω,F ,P) a probability space. Note that it assigns measure only to the sets in F; other sets are said to be non-measurable.

heading: 3.2 Sequence Models
text: As we saw in §2, sampling successive symbols from a non-tight ASM has probability > 0 of continuing forever. Hence, we hope to regard the ASM as defining a probability space over Ω = Σ∗ ∪Σ∞, where Σ∞ denotes the set of infinite strings8 over Σ. Note that this set Ω is uncountable whenever |Σ| ≥ 2. We will first need to turn it into a measurable space by defining an appropriate σ-algebra.
This type of distribution is more general than a language model, which takes Ω to be the set Σ∗ of finite strings. To distinguish the two, we refer to a distribution over Σ∗ ∪ Σ∞ as a sequence model.
8We will use the phrase “infinite string” in this paper when it is natural to do so, e.g., in the context of Σ∗∪Σ∞. However, this is nonstandard terminology: in computer science, string generally refers to a finite object.
Definition 3.3. A sequence model is a probability measure P over the set Σ∗ ∪ Σ∞.
Intuitively (we will make this precise later), the event Σ∞ ⊂ Σ∗ ∪ Σ∞ in Def. 3.3 represents nontermination of the generating process, i.e., it attempts to generate an infinitely long sequence. If this never happens, we have a language model. Definition 3.4. A language model is a probability measure P over just Σ∗. Equivalently, it is a sequence model P such that P (Σ∞) = 0.
Our goal in the rest of this section is to rigorously construct a sequence model P that encodes the conditional probabilities of a given ASM. Since the ASM specifies conditional distributions over the augmented alphabet Σ, we first use it to construct a probability measure P over a measurable space (Σ ∞ , σ(C)). We then derive our sequence model P from P as the probability measure of a random variableX in a measurable space (Σ∗∪Σ∞, σ(C)). The σ-algebras σ(C) and σ(C) will be built below.

heading: 3.3 Pre-Measure
text: As mentioned in §3.1, it is often impossible to measure the probability of every single subset of Ω. For example, in the infinite coin toss model in Ex. 2.6, we might begin by reasonably assigning probability 0 to each individual sequence ω ∈ {H, T}∞. Unfortunately, it is then impossible to assign probability to every subset of {H, T}∞; we must restrict our measurable space to a strict subset of P(Ω), where P() is the powerset operator. Theorem 3.5. Assuming the Axiom of Choice and the Continuum Hypothesis, there exists no probability measure P over ({H, T}∞,P({H, T}∞)) such that P({ω}) = 0 for each ω ∈ {H, T}∞. Proof. This is a direct consequence of Ulam (1930). See App. C.1.1 for a discussion.
We will address this with well-known methods. A versatile theorem of Carathéodory provides a natural way to construct a probability space for sequences, in which prefix probabilities are welldefined. We first review two needed definitions. Definition 3.6. A ⊆ P(Ω) is called an algebra (or field) over Ω if
1) Ω ∈ A, 2) if E ∈ A, then Ec ∈ A, 3) if E1, E2 ∈ A, then E1 ∪ E2 ∈ A.
Definition 3.7. Let A be an algebra over some set Ω. A probability pre-measure over (Ω,A) is a function P0 : A → [0, 1] such that
1) P0(Ω) = 1, 2) if E1, E2, . . . is a (countable) sequence of dis-
joint sets inA whose (countable) union is also in A, then P0(∪∞n=1En) = ∑∞ n=1 P0(En). Note that the only difference between a σalgebra (Def. 3.1) and an algebra is that condition 3 is weakened from countable to finite, and the only difference between a probability measure (Def. 3.2) and a pre-measure is that the latter is defined with respect to an algebra instead of a σ-algebra.
The idea behind Carathéodory’s Extension Theorem is that there is often a simple construction of an algebra A over Ω such that there is a natural way to define a probability pre-measure. One can then extend this probability pre-measure to a probability measure that is both minimal and unique in a precise sense. For example, the standard Lebesgue measure over the the real line can be constructed in this way. For our case of infinite sequences, we will first construct an algebra over Ω = Σ∞ for some alphabet Σ. Then, assuming we are given an ASM p̄ over Σ, we can associate the algebra with a pre-measure that is consistent with p̄. We will make use of the following definition to construct the algebra:
Definition 3.8. Given any set H ⊆ Σk, define its cylinder set (of rank k) to be
C(H) def = { xω : x ∈ H,ω ∈ Σ∞ } (7)
In essence, a cylinder set of rank k is the set of infinite strings that share their k-prefix with some string x ∈ H ⊆ Σk. For a length-k string x = x1 · · ·xk, the rank-k cylinder set C(x) def= C({x}) is the set of all infinite strings prefixed by x.9 We denote the collection of all rank-k cylinder sets by Ck def= { C(H) : H ∈ P(Σk) } and define C def= ⋃∞ k=1 Ck to be the collection of all cylinder sets over Ω.10
Lemma 3.9. C ⊂ P(Ω) is an algebra over Ω = Σ ∞.
Proof. See App. C.1.2.
We are now ready to define the pre-measure P0 for the cylinder algebra C. Given an ASM p̄ and any set C(H) ∈ C, let
P0(C(H)) def = ∑ x∈H p̄(x) (8)
9This type of cylinder set, i.e., one that is generated by a singleton, is also called a thin cylinder.
10Observe that C1 ⊂ C2 ⊂ C3 ⊂ · · · .
where, denoting the length of x by k,
p̄(x) def = ∏k t=1 p̄(xt | x<t). (9)
We confirm in Prop. C.2 that P0 is well-defined even though the cylinder set C(H) may also arise as C(H ′) where H ′ 6= H .11
Lemma 3.10. P0 is a pre-measure over C.
Proof. See App. C.1.2.

heading: 3.4 Extension of Pre-Measure
text: We have now gathered all the ingredients needed to state Carathéodory’s theorem.
Theorem 3.11 (Carathéodory’s Extension Theorem). Given an algebra A over some set Ω and a probability pre-measure P0 : A → [0, 1], there exists a probability space (Ω,F ,P) such thatA ⊂ F and P|A = P0. Furthermore, the σ-algebra F depends only on A and is minimal and unique—thus we may denote it by σ(A)—and the probability measure P is unique.
Proof Sketch. See App. C.2.1.
Applying Carathéodory’s extension theorem to our cylinder algebra C and pre-measure P0, we see that there exists a probability space (Σ∞, σ(C),P) over Σ∞ that agrees with the ASM p̄’s probabilities. It is a fair question to ask what kinds of sets are non-measurable under this construction; we discuss this in App. C.2.2.

heading: 3.5 A String-Valued Random Variable
text: Having constructed the probability space (Σ ∞ , σ(C),P), we now demonstrate how to use it to induce a probability space over Σ∗ ∪ Σ∞ as required by Def. 3.3. We will achieve this through the use of a random variable.
Definition 3.12 (random variable). A mapping X : Ω → S between two measurable spaces (Ω,F) and (A,G) is an (A,G)-valued random variable, or a measurable mapping, if, for all B ∈ G,
X−1(B) def= {ω ∈ Ω : X(ω) ∈ B} ∈ F . (10)
To construct a random variable that takes values in Σ∗∪Σ∞, Def. 3.12 requires us to first construct a σ-algebra over Σ∗∪Σ∞. We will do so in a similar
11For example, in the infinite coin toss model, C(H) = C({HH, HT}).
fashion as we constructed (Σ∞, C). Given H ⊆ Σk, define a rank-k cylinder set in Σ∗ ∪ Σ∞ to be
C(H) def = {xω : x ∈ H,ω ∈ Σ∗ ∪ Σ∞}. (11)
Let Ck be the set of all rank-k cylinder sets. Define C def= ∪∞k=1Ck. Then, σ (C) is a σ-algebra by the same reasoning as in Lemma 3.9 and Thm. 3.11. We can now define the random variable X by12
X(ω) = { ω<k if ωk is the first EOS in ω ω otherwise (if EOS /∈ ω) (12)
where ω ∈ Σ∞. We claim that X is well-defined: Proposition 3.13. The function X : (Σ ∞ , σ(C)) → (Σ∗ ∪ Σ∞, σ(C)) defined in Eq. (12) is a measurable mapping.
Proof. See App. C.3.
Any measurable function induces a probability measure on the output space, called the pushforward measure (cf. §2.4 in Tao, 2011), given by
P (X ∈ E) def= P(X−1(E)). (13)
One can check that P , defined using P, is indeed a probability measure on (Σ∗ ∪ Σ∞, σ(C)) and hence (Σ∗ ∪ Σ∞, σ(C), P ) is a probability space. We have therefore shown that, given any ASM, we can construct an associated sequence model as defined in Def. 3.3.
Under the formulation of a probability space together with a random variable, useful probability quantities arise naturally and intuitively. In particular, when x ∈ Σ∗ is a finite string, we have
P (X = x) def = P (X ∈ {x}) = p(x) (14)
with the definition of p from §2. Additionally, as we will show in the next section, the probability of the set of infinite strings P (X ∈ Σ∞) is the probability of generating an infinite string.13
Deriving EOS As an aside, the preceding section allows us to motivate the EOS token in ASM as a construct that emerges naturally. Specifically, for any x ∈ Σ∗, rearranging Eq. (14):
p̄(EOS | x) = P (X=x)p̄(x) = P (X=x) P (X∈C(x)) (15a)
12In this definition, the position k ≤ ∞ of the first EOS—a stopping time—is itself a random variable.
13An important detail left out in this discussion is that both the singleton set {x} and Σ∞ need to be measurable in (Σ∗∪ Σ∞, σ(C)) for the above to make sense. This is addressed by Prop. C.7 and Prop. C.8.
= P (X = x | X ∈ C(x)) (15b)
where we have used p̄(x) = P(C(x)) = P(X−1(C(x))) = P (X ∈ C(x)). This means that the EOS probability in an ASM emerges as the conditional probability that, given that we must generate a string with a prefix x ∈ Σ∗, the string is exactly x.

heading: 4 Characterizing Tightness
text: Beyond the measure-theoretic formalization, a goal of this paper is to provide an exact characterization of tightness in ASMs. The results presented in this section generalize Lemma 3.2 in Welleck et al. (2020). First, we consider the event
Ak def = {ω ∈ Σ∞ : ωk = EOS} (16)
in the probability space (Σ∞, σ(C),P). Intuitively, Ak is the event that an EOS symbol appears at position k in the string. Note that under this definition the Ak are not disjoint. For example, the string ω = ab EOS c EOS dd · · · lives in the intersection of A3 and A5 since EOS appears at both position 3 and position 5. Using Eq. (16), we can express the event consisting of all finite strings as ⋃∞ k=1Ak. It follows that we can express the event of an infinite string as ( ⋃∞ k=1Ak) c = ⋂∞ k=1A c k. Thus, using the random variable X , we can express the probability of generating an infinite string as
P (X ∈ Σ∞) = P(X−1(Σ∞)) (17a) = P ( ⋂∞ k=1A c k) . (17b)
Hence, we can now formalize the notion of tightness, which we have introduced in §2 and Def. 3.4.
Definition 4.1. A sequence model P is said to be tight if P (X ∈ Σ∞) = 0, in which case it is also a language model (cf. Prop. C.9). Otherwise, we say that it is non-tight.
Note that the definition of Ak only uses a string’s k-prefix, and hence is a cylinder set of rank k. Recalling that the cylinder sets are measurable and so are the sets countably generated by them, we see that both the event consisting of all finite strings and the event consisting of all infinite strings are measurable. Thus, P (∪∞k=1Ak) and P (∩∞k=1Ack) are well defined.

heading: 4.1 A Lower Bound Result
text: We have characterized tightness in terms of the probability of a specific event P (∩∞k=1Ack), a quantity we now seek to determine.
Lemma 4.2. If ∑∞ n=2 P ( An | ∩n−1m=1Acm ) = ∞, then P (∩∞m=1Acm) = 0. Proof. See App. D.
Using Lemma 4.2, we can derive the following useful sufficient condition for a sequence model derived from an ASM to be tight. It applies when the probability of EOS does not decay too rapidly with the length of the prefix.
Proposition 4.3. If p̄(EOS | x) ≥ f(t) for all t ≥ 1,x ∈ Σt−1, and ∑∞t=1 f(t) = ∞, then P(∩∞k=1Ack) = 0. In other words, P is tight. Proof. See App. D.2.
This test implies tightness for all of the tight examples in §2, but not for the non-tight ones. Note that the lower-bounding function f depends only on the length of the prefix, not its content. f does not have to be monotonic—in the case of the even/odd example from §2, it is not.

heading: 4.2 The Borel–Cantelli Lemmata
text: It turns out that Prop. 4.3 admits a converse statement in which we can prove a similar property of p̄ by assuming that the model is tight. To prove this result, we will use a fundamental inequality from probability theory—the Borel–Cantelli lemmata. The Borel–Cantelli lemmata are useful for our purposes because they relate the probability measure of sets of the form ⋂∞ n=0An or ⋃∞ n=0An
to a series ∑∞
n=0 pn. We will only state the lemmata here without supplying their proofs;14 however, we point out that Lemma 4.2 can be viewed as a parallel statement to the Borel–Cantelli lemmata and one can prove the lemmata using a very similar proof (cf. proof of Thm 2.3.7 in Durrett, 2019).
Concretely, given a sequence of events {An}∞n=1 in some probability space, the Borel–Cantelli lemmata are statements about the event
{An i.o.} def= ⋂∞ m=1 ⋃∞ n=mAn (18)
where i.o. stands for “infinitely often.” Intuitively, {An i.o.} is the set of outcomes that appear in infinitely many sets in the collection {An}∞n=1 (hence the name). We will not use Borel–Cantelli directly, but they offer a probabilistic proof of a key result (Cor. 4.6) which will in turn lead to the desired statement about tightness. We formally state the first and second Borel–Cantelli lemmata below.
14See §2.3 in Durrett (2019) or §4 in Billingsley (1995) instead.
Lemma 4.4 (Borel–Cantelli I). If∑∞ n=1 P(An) <∞, then P(An i.o.) = 0. Lemma 4.5 (Borel–Cantelli II). If∑∞ n=1 P(An) =∞, then P(An i.o.) = 1, provided that {An} is a sequence of independent events. Using the Borel–Cantelli lemmata, we can prove the following useful fact. Corollary 4.6. Given a sequence {pn} where pn ∈ [0, 1). Then, ∏∞ n=1(1− pn) = 0⇐⇒ ∑∞ n=1 pn =∞. (19)
Proof. See App. D.3.
We now turn to proving a more general version of Prop. 4.3, which would imply its converse. First, we define the following quantity
p̃EOS(t) def = P(At | Ac1 ∩ · · · ∩Act−1) (20)
which can be viewed as the EOS probability at step t, given that EOS was not generated at any earlier step. In Eq. (48a) in App. D.2, we show that, when p̃EOS(t) is defined, it has the same value as
p̃EOS(t) = ∑ ω∈Σt−1 p̄(ω)p̄(EOS | ω)∑ ω∈Σt−1 p̄(ω) . (21)
We can now completely characterize the tightness of an ASM with the following theorem. Theorem 4.7 (Proposition 2.4 in Meister et al., 2022). An ASM is tight if and only if p̃EOS(t) = 1 for some t or ∑∞ t=1 p̃EOS(t) =∞.
Proof. See App. D.4. The proof uses Cor. 4.6, which accounts for the form of the condition.
We remark that Thm. 4.7 is a generalization of Prop. 4.3 since if p̃EOS(t) is lower-bounded by f(t) whose series diverges, its own series would also diverge. However, since p̃EOS(t) involves the computation of a partition function in its denominator, it can be intractable to calculate (Lin et al., 2021). Hence, Prop. 4.3 will be our main tool for determining tightness.
Finally, we note that Thm. 4.7 generalizes claims in previous work. For example, Welleck et al. (2020) require f(t) = c > 0 for some constant c to determine tightness. Hence, their bound is not helpful in determining the tightness in either Ex. 2.4 or Ex. 2.5, because the EOS probability can be arbitrarily small in both cases. Applying Prop. 4.3, we see that (1) the ASM in Ex. 2.4 is non-tight, because the series ∑∞ t=1 1 et+1 is convergent, and (2) the ASM in Ex. 2.5 is tight, since the series ∑∞ t=1 1 t+1 is divergent.

heading: 5 Analysis of Common Language Models
text: We now put into practice the foundations built up in the previous sections and discuss the tightness of several classes of ASMs.

heading: 5.1 Stochastic Finite-State Language Models
text: Language modeling based on n-grams has been historically influential in NLP (Jurafsky and Martin, 2009, Ch. 4). However, as Fig. 1 illustrates, n-gram language models are specific cases of the more general stochastic finite-state language models (Vidal et al., 2005). Tightness is more naturally characterized in this more general setting, as it turns out. We begin with a linear-algebraic definition of stochastic finite-state language models—or, more precisely, sequence models, since in this paper we do not consider the non-tight ones to be language models.
Definition 5.1. A Q-state stochastic finitestate sequence model (SFSSM) is a quadruple( Σ, s, {P(a)}a∈Σ, t ) , where Σ is an alphabet of symbols, P(a) ∈ RQ×Q≥0 is a symbol-specific transition matrix for a ∈ Σ,15 s ∈ RQ≥0 is a vector of initial state probabilities, and t ∈ RQ≥0 is a vector of termination probabilities, i.e., probabilities of generating EOS in each state.16 We further require that ∑Q q=1 sq = 1 and that tq + ∑Q q′=1 Pqq′ = 1 for all 1 ≤ q ≤ Q, where P def= ∑a∈Σ P(a) is the transition sum matrix.
Given an SFSSM ( Σ, s, {P(a)}a∈Σ, t )
, the probability of a string x ∈ Σ∗ is defined by
p̄(x1 · · ·xn) = s> (∏n t=1 P (xt) ) t. (22)
Definition 5.2. A state q of an SFSSM (1 ≤ q ≤ Q) is accessible if there is a positive-probability path to q from some state r with sr > 0; it is coaccessible if there is a positive-probability path from q to some state r with tr > 0. It is useful if it is both accessible and co-accessible, i.e., q appears on some positive-probability accepting path.
Def. 5.2 allows a simple characterization of tight SFSSMs, namely Thm. 5.3, and a straightforward proof of Cor. 5.4.17
15For simplicity, we have disallowed ε-transitions. 16We use Q to denote the number of states as Q is the traditional notation for the set of states in a finite-state automaton. 17Cor. 5.4 is a special case of Chi and Geman (1998), who showed that MLE estimates of PCFGs are tight.
Theorem 5.3. An SFSSM is tight iff all accessible states are also co-accessible.
Proof. See App. E.1.1.
Corollary 5.4. Maximum likelihood estimates of n-gram models based on some corpus are tight.
Proof. See App. E.1.1.
In fact, we can express the termination probability of an SFSSM in simple linear algebra terms.
Definition 5.5. Trimming an SFSSM means removing its non-useful (useless) states to obtain a substochastic finite-state sequence model.18 This does not affect the string probabilities (22). Removing the non-useful states means removing their rows and columns from P as well as their rows from s and t, yielding possibly smaller P′, s′ and t′.
Theorem 5.6. Let P′ be the transition sum matrix of a trimmed substochastic FSSM. Then I−P′ is invertible and P (X ∈ Σ∗) = s′>(I−P′)−1t′ ≤ 1. Proof. See App. E.1.2.
The well-known matrix inversion formula used above finds the total weight of all accepting paths in any weighted graph (Tarjan, 1981).19 The formula can be seen as a special case of Lehmann’s (1977) algebraic path algorithm.

heading: 5.2 Transformer Language Models
text: We now prove that all Transformer language models are tight. Key to our proof of the tightness of various neural architectures, including the Transformer, is the following basic fact in topology.
Theorem 5.7. Let X be a compact topological space and Y be any topological space. If f : X → Y is continuous, then f(X) ⊆ Y is also compact. Proof. See App. E.2.
To address the variable-length nature of modern deep NLP models, we will mathematically abstract them as a function on vector tuples,20 f : ( Rd )+ → ( Rd )+, that is length-preserving in the sense that f ( Rt×d ) ⊆ ( Rt×d ) for all t > 0.
18We use the term substochastic rather than stochastic here because the trimmed model satisfies t′q + ∑Q′ q′=1 P ′ qq′ ≤ 1, but might no longer achieve equality as required by Def. 5.1. 19This is assuming the total weight is finite (which we guarantee by substochasticity) and the matrix is invertible (which we guarantee by trimming)
20Here ( Rd )+
is the set of nonempty tuples of vectors in Rd. This is formally the disjoint union (coproduct)
∐ t∈Z>0 R t×d.
Intuitively, this definition is saying that f is a function that maps a nonempty vector tuple {vi}ti=1 to another vector tuple {hi}ti=1 of the same length,
f(v1, . . . ,vt) = (h1, . . . ,ht) ∈ Rt×d, (23)
where vi ∈ Rd is commonly the embedding of the input symbol xi. In particular, we can take the function f : ( Rd )+ → ( Rd )+ to be the function defined by a stack of Transformer layers. This setup will help us state the following. Lemma 5.8. Let f : ( Rd )+ → ( Rd )+ be the function defined by a finite number of Transformer layers (e.g., n layers) with any continuous activation function. Given a compact set K ⊂ Rd. Then, there exists a compact set K ′ ⊂ Rd such that for every t ∈ Z>0,
f ( Kt ) ⊆ ( K ′ )t . (24)
Proof. See App. E.2.
Recall that a Transformer language model—or more precisely, a Transformer ASM—defines the conditional probabilities using the softmax transformation
p̄(xt+1 | x≤t) = exp(u>xt+1ht)∑ y∈Σ exp(u > y ht)
(25)
where ux ∈ Rd is the output symbol embedding of x ∈ Σ and ht is defined from the input embeddings of x≤t via Eq. (23). Using Lemma 5.8, together with the finiteness of the vocabulary Σ and the continuity of the softmax transformation (25), readily yields our main result on Transformers.
Theorem 5.9. The autoregressive sequence model defined by any (fixed-depth) Transformer is tight.
Proof. See App. E.2.

heading: 5.3 Recurrent Neural Language Models
text: Recall that the hidden state of an RNN is typically defined by the recurrence
ht = σ (Wvt +Uht−1 + b) (26)
where vt ∈ Rd is the embedding of the input symbol xt, as above, and σ(·) is some activation function (Elman, 1990). The conditional probabilities are usually defined in the same way as Eq. (25). Using Thm. 5.7 and the same strategy of proof as in Thm. 5.9, one can also easily prove the tightness of any RNN ASM with bounded activations (e.g.,
tanh or sigmoid). However, as we saw in Ex. 2.4, an unbounded activation function (e.g., ReLU) can indeed lead to non-tightness by making the probability of EOS decay too fast. The condition derived in Thm. 4.7 precisely determines how fast such decay can be without losing the tightness of the language model. Below, we generalize this result as well as Lemma 3.2 of Welleck et al. (2020), and show that if the norm of the activations eventually grows sub-logarithmically, the RNN is still tight.
Proposition 5.10. Given an RNN ASM over Σ. Again let the output symbol vector be ux ∈ Rd for x ∈ Σ, and set k def= supx∈Σ ‖ux − uEOS‖2. Additionally, for each t > 0, let ‖ĥt‖2 be the maximum attainable hidden state norm for any context x ∈ Σt. Such a sequence model is tight if k‖ĥt‖2 ≤ log t for all sufficiently large t. Proof. See App. E.3.
This result is weaker than Thm. 5.9 because in an RNN, unlike a Transformer, the depth of the computation graph grows with the sequence length.

heading: 6 Conclusion
text: This paper presents a measure-theoretic treatment of language modeling and its tightness. Practical implications of our results include determining when sampling from an autoregressive sequence model is guaranteed to terminate and whether MCMC algorithms over such models will mix to the correct distribution.
To this end, we first defined various components of language modeling in measure-theoretic terminology. This in turn allows us to understand the portion of probability mass allocated to infinite-length strings. Importantly, this presentation formalizes a definition of sequence modeling under which the probability of producing an infinite-length sequence is non-zero; while today’s models are often capable of producing such strings, previously there was no rigorous treatment of this case.
Indeed, such a definition is useful when considering a number of neural architectures (e.g., a simple RNN as in Elman, 1990) and language generation systems (e.g., the distribution induced by nucleus sampling; Holtzman et al., 2020). In particular, we showed that perhaps the most commonly-used NLP architecture, the Transformer language model, is indeed a language model—a tight distribution over finite strings—a property that had been called into question by previous work.

heading: Limitations
text: Our discussion in this paper leaves out the consideration of computability of measures over languages. Specifically, we note that there exist works on computable measure theory developed in the context of theoretical computer science (de Leeuw et al., 1956) and probabilistic programming languages (Roy, 2011). Additional machinery needs to be developed in order for a proper treatment and we leave this for future work.
Another notable limitation is that we exclusively focused on the autoregressive production of language. Importantly, our formalism might not be compatible with other models of language production such as those induced by a PCFG.
Finally, our proofs of Thm. 5.9 and Prop. 5.10 exploit the strictly positive property of the softmax function. Importantly, they do not apply to models with sparse distributions (Martins and Astudillo, 2016; Peters et al., 2019; Martins, 2021).

heading: Ethics
text: There are no ethical implications of this paper to the best knowledge of the authors.

heading: Acknowledgments
text: We thank Holden Lee for helpful discussion and suggestions, Anej Svete for help with the graphics and Chu-Cheng Lin and Xinyan Velocity Yu for helpful comments. LD is partially supported by the Johns Hopkins Mathematical Institute for Data Science (MINDS) Fellowship. Finally, we thank the students of the LLM course at ETH Zürich (263- 5354-00L) for carefully reading this paper as part of their lecture notes, and in particular, Valentin Bieri for making a valuable remark.

heading: A Related Work
text: The issue of tightness has been studied extensively in the context of probabilistic context-free grammars (PCFG; Chi and Geman, 1998; Chi, 1999; Cohen and Johnson, 2013), although Chi (1999) refers to non-tight models as improper. Specifically, Chi (1999) gave algorithms for determining the tightness of a PCFG by formalizing a PCFG as a branching process. Chi (1999) further proved that any maximumlikelihood estimator yields a tight PCFG. Several previous works study the ability of language models to place probability mass on infinite-length strings (Booth and Thompson, 1973; Nederhof and Satta, 2006; Chen et al., 2018; Welleck et al., 2020), where they refer to the non-tight language models as inconsistent. In some cases, this behavior can be attributed to the discrepancy between the language model itself and the distribution induced by a (possibly stochastic) decoding algorithm: the decoder may have a lower probability of generating the EOS token. For example, on the tight bigram model of Ex. 2.3, a greedy decoder will always generate a and never EOS. Yet in other examples, it is the model itself that leaks probability mass to infinite-length strings, i.e., it may be non-tight, which is the problem we focus on in this work, providing a characterization of tightness. Notably, the conditions we propose are more general than those of Welleck et al. (2020).
Several other works consider the limitations of common neural network architectures for modeling distributions over finite sequences (strings), albeit focusing specifically on other attributes, such as their computational complexity for problems like equivalence or undecidability (Chen et al., 2018; Lin et al., 2021; Lin and McCarthy, 2022; Lin, 2022). In contrast, this work provides a formal treatment of language models by enlarging the sample space to Σ∗ ∪ Σ∞, although to ensure tightness, Σ∞ must receive probability 0. Such definitions are not uncommon in probability theory. For example, while the Wiener process (i.e., the standard Brownian motion) is a distribution over all functions, the definition ensures that the set of discontinuous functions is assigned probability 0 (Durrett, 2019, Ch. 7).
Meister et al. (2022) similarly address the notion of a language model as a distribution over infinite sequences by casting such models as stochastic processes. They use this framing in order to motivate decoding, without providing comprehensive measure-theoretic foundations of such distributions.

heading: B Details for Motivating Ex. 2.3
text: Here, we lay out the steps to calculate P(Σ∗) from Fig. 1b:
P(Σ∗) = ∞∑
n=0
( P(an+1) + ∞∑
m=0
P(an+1bm+1) ) (27a)
=
∞∑
n=0
( 1 · (0.7)n · 0.1 + ∞∑
m=0
1 · (0.7)n · 0.2 · (0.9)m · 0.1 )
(27b)
= ∞∑
n=0
(0.7)n · ( 0.1 + 0.2 · ( ∞∑
m=0
(0.9)m ) · 0.1 ) (27c)
=
∞∑
n=0
(0.7)n · (
0.1 + 0.2 · 1 1− 0.9 · 0.1
) (27d)
= ∞∑
n=0
(0.7)n · 0.3 = 0.3 1− 0.7 = 1 (27e)

heading: C Measure Theory Details

heading: C.1 Proofs and Details in §3.3

heading: C.1.1 Details of Thm. 3.5
text: Theorem 3.5. Assuming the Axiom of Choice and the Continuum Hypothesis, there exists no probability measure P over ({H, T}∞,P({H, T}∞)) such that P({ω}) = 0 for each ω ∈ {H, T}∞.
This theorem is an impossibility of measure theorem. Generally speaking, the existence of a non-measurable set implies some form of impossibility of measure. The most famous example of non-measurable sets are Vitali sets, which exist given the Axiom of Choice. Vitali’s 1905 construction is typically described in introductory texts on measure theory (Royden, 1988; Billingsley, 1995; Axler, 2020). The existence of Vitali sets shows that it is impossible to define a probability measure that satisfies translational invariance on the measurable space ( [0, 1),P([0, 1)) ) . Thus, to achieve translational invariance, Lebesgue measure uses a σ-algebra smaller than P([0, 1)), in which the Vitali sets are among the non-measurable sets. However, the translational invariance desideratum is not relevant to our space of discrete sequences. A theorem by Ulam (1930) reveals a deeper reason that some sets must be non-measurable. We shall state the theorem below as given in Oxtoby (1980) and omit its proof. We refer interested readers to Chapter 5 in Oxtoby (1980), which contains an accessible proof and an excellent discussion of the theorem including its generalizations and historical context.
Theorem C.1 (Ulam, 1930). Assuming the Axiom of Choice, a finite measure µ defined for all subsets of a set X of cardinality ℵ1 vanishes identically [that is, equals zero for all subsets] if it is equal to zero for every one-element subset.
In the statement above, ℵ1 denotes the cardinality of the first uncountable ordinal. We can see that Thm. 3.5 is a straightforward consequence of Thm. C.1.
Proof of Thm. 3.5. Recall that card({H, T}∞) = 2ℵ0 . Assuming the Continuum Hypothesis, 2ℵ0 = ℵ1, and hence by Thm. C.1, such a measure is uniformly 0, and hence cannot be a probability measure.

heading: C.1.2 Other Proofs in §3.3
text: Lemma 3.9. C ⊂ P(Ω) is an algebra over Ω = Σ∞.
Proof. First, Ω ∈ C since it is a cylinder set of rank 0 or indeed of any rank k: Ω = C(Σk) ∈ Ck ⊂ C. Second, C is closed under complements: given a cylinder set of rank k, that is, C(H) where H ⊆ Σk, its complement ( C(H) )c = C ( Σ k \H ) is also a cylinder set of rank k. Finally, C is closed under union: the union of cylinder sets of ranks k1 ≤ k2 is a cylinder set of rank k2, since both can be regarded as cylinder sets of rank k2. Hence, C is an algebra over Ω.
Proposition C.2. P0 as defined in Eq. (8) is a well-defined function.
Proof. Suppose a cylinder set arises in two ways, C(H1) = C(H2), where H1 ⊆ Σk1 and H2 ⊆ Σk2 . We must show
∑ x∈H1 p̄(x) = ∑ x′∈H2 p̄(x
′). Without loss of generality, assume that k1 ≤ k2. The definition of C(H2) (Def. 3.8) implies that H2 consists of all length-k2 prefixes of strings in C(H2). But C(H2) = C(H1), so the definition of C(H1) (Def. 3.8) implies that its length-k2 prefixes are exactly the strings of the form xy where x ∈ H1,y ∈ Σk2−k1 . Hence we can write H2 in terms of H1 as H2 = {xy : x ∈ H1,y ∈ Σk2−k1}. Thus
∑
x′∈H2 p̄(x′) =
∑
x∈H1
∑
y∈Σk2−k1 p̄(xy) =
∑
x∈H1 p̄(x) (28)
where the last equality is true because p̄ is defined by the locally normalized product (9).
Lemma 3.10. P0 is a pre-measure over C. For the proof of Lemma 3.10, we will mostly follow the proof of Thm 2.3 in Billingsley (1995), with the exception of invoking the Tychonoff theorem directly. This proof depends on the following lemma, which is Example 2.10 in Billingsley (1995). We repeat the statement and proof here for the reader’s convenience.
Lemma C.3. Let P0 be a finitely additive probability pre-measure over C such that, given a decreasing sequence of sets A1 ⊃ A2 ⊃ · · · in C where ⋂∞ n=1An = ∅, limn→∞ P0(An) = 0. Then, P0 is also countably additive over C.
Proof. Let {An} be a sequence of disjoint sets in C such that A = ⋃ nAn ∈ C. Then, defining
Bn = ⋃ m>nAm, we see that B1 ⊃ B2 ⊃ · · · and ⋂ nBn = ∅. Notice that
A = A1 ∪B1 = A1 ∪A2 ∪B2 = · · · = A1 ∪ · · · ∪An ∪Bn (29)
for any n and hence by finite additivity of P0
P0(A) = P0(A1) + · · ·+ P0(An) + P0(Bn) (30)
or equivalently
P0(A1) + · · ·+ P0(An) = P0(A)− P0(Bn). (31)
Since, Bn ↓ ∅ implies that P0(Bn) ↓ 0 by assumption, taking the limits on both sides of Eq. (31) yields ∑
n
P0(An) = lim n→∞
∑ i≤n P0(Ai) = P0(A)− lim n→∞ P0(Bn) = P0(A) (32)
which shows countable additivity.
We also recall the Tychonoff theorem.21
Theorem C.4 (Tychonoff). Let {Xα}α∈J be an indexed family of compact topologies. Then, their product topology ∏ α∈J Xα is also compact.
We can now give the proof for Lemma 3.10.
Proof of Lemma 3.10. We first show that P0 is finitely additive over C. Let C(H1) and C(H2) be two disjoint cylinder sets. By Prop. C.2, we can assume they are of the same rank without loss of generality. Then,
C(H1) ∪ C(H2) = ⋃
x∈H1 {xω : ω ∈ Σ∞} ∪
⋃
x∈H2 {xω : ω ∈ Σ∞} (33a)
= ⋃
x∈H1∪H2 {xω : ω ∈ Σ∞} (H1 and H2 equal rank and disjoint) (33b) = C(H1 ∪H2) (33c)
which leads to
P0(C(H1) ∪ C(H2)) = P0(C(H1 ∪H2)) = ∑
x∈H1∪H2 p̄(x) = P0(C(H1)) + P0(C(H2)). (34a)
Hence, P0 is finitely additive. Now, equip Σ with the discrete topology. Since Σ is finite, it is compact under the discrete topology and so is Σ∞ by Thm. C.4. Then, by properties of the product topology over discrete finite spaces, all cylinder sets in Σ∞ are compact. To apply Lemma C.3, let C1 ⊃ C2 ⊃ · · · be a decreasing sequence of cylinder sets with empty intersection. Suppose to the contrary that P0 ( ⋂ nCn) > 0. This would imply that all Cn are nonempty (any of these being empty would result in a measure 0). However, by Cantor’s intersection theorem22, ⋂ nCn is nonempty, contradicting the assumption. Hence, P0 ( ⋂ nCn) = 0, and by Lemma C.3, P0 is countably additive. 21See §37 in Munkres (2000) for a detailed and well-written treatise. 22Cantor’s intersection theorem states that a decreasing sequence of nonempty compact sets have a nonempty intersection. A version of this result in introductory real analysis is the Nested Interval Theorem.

heading: C.2 Details in §3.4

heading: C.2.1 Carathéodory’s Extension Theorem
text: Theorem 3.11 (Carathéodory’s Extension Theorem). Given an algebra A over some set Ω and a probability pre-measure P0 : A → [0, 1], there exists a probability space (Ω,F ,P) such that A ⊂ F and P|A = P0. Furthermore, the σ-algebra F depends only on A and is minimal and unique—thus we may denote it by σ(A)—and the probability measure P is unique.
Proof Sketch. First, construct an outer measure by approximation with countable coverings. Then, show that the collection of sets that is measurable with respect to this outer measure is a σ-algebra F that containsA. Finally, restricting the outer measure to this σ-algebra, one is then left with a probability space. To show minimality, one can show that F is contained in any σ-algebra that contains A. Uniqueness is given by applying Dynkin’s π-λ theorem (Theorem 3.2 in Billingsley, 1995).
Great care must be taken in each step involved in the outline above. To address these is well beyond the scope of this paper and we refer reader to the many excellent texts with a proof of this theorem, such as Chapter 12 in Royden (1988) and Chapter 11 in Billingsley (1995).

heading: C.2.2 The Space of Non-measurable Sets
text: Non-measurable sets are, in general, difficult to find. Even when we can exhibit such sets, they tend to be very abstract and counter-intuitive. Vitali’s and Bernstein’s sets are two prominent examples for the Lebesgue measure. Blackwell and Diaconis (1996) offers a construction of a non-measurable set in the cylinder σ-algebra.23
As another approach to understand this better, we can consider how our collection σ(C) of all measurable sets, i.e., our σ-algebra, is constructed from our algebra C of cylinder sets (as opposed to simply knowing from Carathéodory’s Extension Theorem that it exists). Concretely, as in §1.6 in Folland (1999), we can intuitively consider the following process to build from the collection of cylinder sets C, which is a countable collection, all the way up to its generated σ-algebra, whose cardinality is unknown just yet:
• Let C0 = C, • Let C1 be the set that includes all countable unions of sets in C0 or the complements of such, • Repeat this process to build Cn for every n ∈ N.
One might then take the union ⋃ n∈N Cn of this increasing sequence of collections of sets, and ask if it is the same as σ(C). In general, the answer is no (as one might expect if one is familiar with the Borel Hierarchy). However, we can obtain σ(C) if we perform this construction for every countable ordinal α. Abbreviating the operation in the second step above as δ, i.e., C1 = δ(C0), and letting ω1 be the collection of all countable ordinals,24 we can define
Cα = { δ(Cβ) if α = β + 1 for some β ∈ ω1,⋃ β∈ω1:β<α Cβ otherwise.
(35)
This will give us the desired set as follows: Proposition C.5 (Proposition 1.23, Folland, 1999). σ(C) = ⋃α∈ω1 Cα. Next, we recall the following basic fact from cardinality theory.
Proposition C.6 (Proposition 0.14, Folland, 1999). If card(A) ≤ 2ℵ0 and card(Xα) ≤ 2ℵ0 for all α ∈ A, then card (⋃ α∈AXα ) ≤ 2ℵ0 .
Noting that card(ω1) ≤ 2ℵ0 and card(C) = ℵ0, we can conclude that card(σ(C)) ≤ 2ℵ0 from Prop. C.5 and Prop. C.6. In other words, the cardinality of σ(C) is at most that of the continuum, and since card ( P(Σ∞) ) = 22
ℵ0 (= i2), σ(C) is, in terms of cardinality, an almost negligible subset of P(Σ∞)! That is, most subsets in Σ∞ are non-measurable—though explicit examples have rarely been constructed
23The following assumes basic familiarity with the theory of ordinal numbers. Readers without such background may skip to the last paragraph for conclusion.
24ω1 is, in fact, the same as the first uncountable ordinal. Its existence (and hence the existence of the collection of all countable ordinals) can be guaranteed by exhibiting a well-ordered uncountable set using the Axiom of Choice.
(Blackwell and Diaconis, 1996). App. C.3 below establishes that common subsets of Σ∞ that we work with are measurable.

heading: C.3 Proofs in §3.5
text: Proposition 3.13. The function X : (Σ∞, σ(C))→ (Σ∗ ∪Σ∞, σ(C)) defined in Eq. (12) is a measurable mapping.
Proof. To show that X is measurable, it suffices to show the measurability of preimages of a generating set25 of the σ-algebra σ(C) on Σ∗ ∪ Σ∞. Such a generating set is formed by the thin cylinders C(x) def= C({x}) for x ∈ Σ∗. (Recall that cylinders in Σ∗ ∪ Σ∞ are defined by Eq. (11).) Given x ∈ Σ∗:
X−1(C(x)) =X−1({xω : ω ∈ Σ∗ ∪ Σ∞}) (36a) =X−1({xω : ω ∈ Σ∗}) ∪X−1({xω : ω ∈ Σ∞}) (36b)
=
( ⋃
ω∈Σ∗ C(xω EOS)
) ∪ ( C(x) ∩ ∞⋂
k=1
Ack
) (36c)
Note that the set Ak above, defined by Eq. (16), is a cylinder of Σ ∞, representing the event of terminating by step k. Then, from the derivation above, we can see thatX−1(C(x)) is formed by countable operations over measurable sets (cylinders) of Σ∞, and is hence measurable. So X is a measurable function.
Proposition C.7. In measure space (Σ∗ ∪ Σ∞, σ(C)), {x} is measurable for all x ∈ Σ∗. Proof. We will show that {x} = C(x) \⋃a∈ΣC(xa) and hence is measurable. By definition in Eq. (11), for any x ∈ Σ∗,
C(x) = {xω : ω ∈ Σ∗ ∪ Σ∞} (37a) = {xω : ω ∈ Σ∗} ∪ {xω : ω ∈ Σ∞} (37b)
where
{xω : ω ∈ Σ∗} = {x} ∪ ⋃
a∈Σ {xaω : ω ∈ Σ∗} (38a)
and
{xω : ω ∈ Σ∞} = ⋃
a∈Σ {xaω : ω ∈ Σ∞}. (39)
So
C(x) = {x} ∪ ⋃
a∈Σ
( {xaω : ω ∈ Σ∗} ∪ {xaω : ω ∈ Σ∞} ) (40a)
= {x} ∪ ⋃
a∈Σ C(xa) (40b)
where the union is disjoint. This implies {x} = C(x) \⋃a∈ΣC(xa) as desired. Proposition C.8. In the measure space (Σ∗ ∪ Σ∞, σ(C)), Σ∞ is measurable. Proof. First, Σ∗ ∪ Σ∞ is the entire outcome space, which is measurable by the definition of σ-algebra. Notice that
Σ∞ = (Σ∗ ∪ Σ∞) \ ⋃
x∈Σ∗ {x}. (41)
Since each {x} in the above is measurable by Prop. C.7 and Σ∗ is a countable set, Σ∞ is then measurable.
25A set G is said to be a generating set of a σ-algebra F if F is the smallest σ-algebra that contains G.
The measurability of Σ∞ in (Σ∗ ∪ Σ∞, σ(C)) (Prop. C.8) was assumed by our definition of tightness (Def. 4.1). As we have also established that each {x} is measurable (Prop. C.7), we can give an alternative characterization. Proposition C.9. A sequence model (Σ∗ ∪ Σ∞, σ(C), P ) is tight if and only if∑x∈Σ∗ P ({x}) = 1. Proof. We defined a sequence model to be tight if and only if P (Σ∞) = 0 (Def. 4.1). By Propositions C.7 and C.8, we can write
1 = P (Σ∗ ∪ Σ∞) = P (Σ∞) + P (Σ∗) (finite additivity) (42a) = P (Σ∞) + ∑
x∈Σ∗ P ({x}). (countable additivity) (42b)
Hence, a sequence model is tight if and only if ∑
x∈Σ∗ P ({x}) = 1.

heading: D Proofs on Characterizing Tightness (§4)

heading: D.1 Proof of Lemma 4.2
text: The result below is stated without proof as Exercise 4.3.5 in Durrett (2019). Lemma 4.2. If ∑∞ n=2 P ( An | ∩n−1m=1Acm ) =∞, then P (∩∞m=1Acm) = 0.
Proof. First, recall an elementary inequality that for x > 0,
x− 1 ≥ log x ⇔ 1− x ≤ log 1 x . (43)
Note that P(∩nm=1Acm) > 0 for any n, for otherwise the conditional probabilities would be undefined. Let pn def = P(∩nm=1Acm). Then we have that pn > 0 for all n, and
∞ = ∞∑
n=2
P(An | ∩n−1m=1Acm) (44a)
=
∞∑
n=2
1− P(Acn | ∩n−1m=1Acm) (44b)
= lim N→∞
N∑
n=2
1− P(Acn | ∩n−1m=1Acm) (44c)
≤ lim N→∞
N∑
n=2
log 1/P(Acn | ∩n−1m=1Acm) (by Eq. (43)) (44d)
= lim N→∞
N∑
n=2
log P(∩n−1m=1Acm) P(∩nm=1Acm)
(44e)
= lim N→∞
N∑
n=2
log pn−1 pn
(44f)
= lim N→∞
N∑
n=2
(log pn−1 − log pn) (44g)
= lim N→∞
(log p1 − log pN ) (44h)
= log p1 − lim N→∞ log pN (44i)
which implies that
lim N→∞
log pN = −∞ (45a)
⇔ lim N→∞ pN = 0 (45b) ⇔ lim N→∞ P(∩Nm=1Acm) = 0 (45c) ⇔ P(∩∞m=1Acm) = 0. (by continuity of measure) (45d)

heading: D.2 Proof of Prop. 4.3
text: Proposition 4.3. If p̄(EOS | x) ≥ f(t) for all t ≥ 1,x ∈ Σt−1, and∑∞t=1 f(t) =∞, then P(∩∞k=1Ack) = 0. In other words, P is tight.
Proof. In the proof, we rename the index t to n to match the usual presentation of the Borel-Cantelli lemmata. We are given that p̄(EOS | x) ≥ f(n) for all x ∈ Σn−1. To apply Lemma 4.2, we observe that
An ∩ (Ac1 ∩ · · · ∩Acn−1) ={ω ∈ Σ ∞ : ωn = EOS} ∩ ( n−1⋂
i=1
{ω ∈ Σ∞ : ωi 6= EOS} )
(46a)
={ω ∈ Σ∞ : ω = EOS,∀ i < n,ω 6= EOS} (46b) ={ω ∈ Σ∞ : ω’s first EOS is at position n} (46c)
and similarly
Ac1 ∩ · · · ∩Acn−1 = {ω ∈ Σ ∞ : there is no EOS in ω’s first n− 1 positions} (47)
Setting G def= {ω EOS : ω ∈ Σn−1} ⊂ Σn, we get
P(An | Ac1 ∩ · · · ∩Acn−1) = P(An ∩ (Ac1 ∩ · · · ∩Acn−1))
P(Ac1 ∩ · · · ∩Acn−1) (48a)
= P(C(G))
P(C(Σn−1)) (definition of G) (48b)
= ∑ ω∈Σn−1 p̄(EOS | ω)p̄(ω)∑
ω∈Σn−1 p̄(ω) (by Eq. (8)) (48c)
≥ ∑
ω∈Σn−1 f(n)p̄(ω)∑ ω∈Σn−1 p̄(ω)
(definition of f(n)) (48d)
= f(n) ∑ ω∈Σn−1 p̄(ω)∑ ω∈Σn−1 p̄(ω)
(48e)
= f(n). (48f)
Since ∑∞ n=1 f(n) = ∞ and hence ∑∞
n=2 f(n) = ∞, the above inequality shows that the condition of Lemma 4.2 holds. Hence by Lemma 4.2, the event of a string never terminating, i.e., ∩∞k=1Ack, has probability measure P(∩∞k=1Ack) = 0.
In summary, if the EOS probability of a language model is lower-bounded at ever steps by the terms of a divergent series, then the event that this language model terminates has probability 1.

heading: D.3 Proof of Cor. 4.6
text: To show Cor. 4.6, we first show the following simple consequence of Borel–Cantelli. Corollary D.1. If P(An i.o.) = 1, then ∑∞
n=1 P(An) =∞. Proof. Suppose to the contrary that ∑∞ n=1 P(An) < ∞, then, by Borel–Cantelli I (Lemma 4.4),
P(An i.o.) = 0, which contradicts the assumption. Hence, ∑∞ n=1 P(An) =∞.
Cor. 4.6 below is also stated without proof as Exercise 4.3.4 in Durrett (2019).
Corollary 4.6. Given a sequence {pn} where pn ∈ [0, 1). Then, ∏∞ n=1(1− pn) = 0⇐⇒ ∑∞ n=1 pn =∞. (19)
Proof. We can use a product measure to construct a sequence of independent events {An}∞n=1 such that P(An) = pn. (The product measure ensures independence.) Then, by definition in Eq. (18),
{An i.o.}c = ∞⋃
m=1
⋂
n≥m Acn (49)
So,
1− P(An i.o.) = P
 ⋃
m
⋂
n≥m Acn
  (50a)
= lim m→∞
P
  ⋂
n≥m Acn
  (50b)
= lim m→∞
∏
n≥m P(Acn) (An are independent by construction) (50c)
= lim m→∞
∏
n≥m (1− pn) (50d)
(⇒): Assume∏∞n=1(1− pn) = 0. Then, for any m,
0 = ∏
n≥1 (1− pn) =
  ∏
1≤n<m (1− pn)
 
︸ ︷︷ ︸ >0
 ∏
n≥m (1− pn)
  (51)
So it must the case that, for any m, ∏ n≥m(1− pn) = 0. Therefore,
1− P(An i.o.) = lim m→∞
∏
n≥m (1− pn) = 0 (52)
which implies P(An i.o.) = 1. Cor. D.1 implies that ∑∞
n=1 pn =∞. (⇐): Assume∑∞n=1 pn =∞. Then by Borel–Cantelli II (Lemma 4.5), P(An i.o.) = 1 which implies
0 = 1− P(An i.o.) = lim m→∞
∏
n≥m (1− pn) (53)
Observe that {∏ n≥m(1− pn) } m is a non-decreasing sequence in m; to see this, note that as m grows larger we multiply strictly fewer values (1 − pn) ∈ (0, 1]. However, since we know the sequence is non-negative and tends to 0, it follows that for any m, we have
∏
n≥m (1− pn) = 0. (54)
It follows that, for any m, we have
∞∏
n=1
(1− pn) = ∏
n<m
(1− pn) ∏
n≥m (1− pn) ︸ ︷︷ ︸ =0
= ∏
n<m
(1− pn) · 0 = 0. (55)

heading: D.4 Proof of Thm. 4.7
text: Theorem 4.7 (Proposition 2.4 in Meister et al., 2022). An ASM is tight if and only if p̃EOS(t) = 1 for some t or ∑∞ t=1 p̃EOS(t) =∞.
Proof. Recall the definition of p̃EOS, as previously defined in Eq. (20), is
p̃EOS(t) def = P(At | Ac1 ∩ · · · ∩Act−1). (56)
Case 1. Suppose that p̃EOS(t) < 1 for all t. Consider the termination probability again:
P
( ∞⋂
t=1
Act
) = lim
T→∞ P
( T⋂
t=1
Act
) (57a)
= lim T→∞
T∏
t=1
P(Act | Ac1 ∩ · · · ∩Act−1) (57b)
= lim T→∞
T∏
t=1
(1− p̃EOS(t)) (57c)
=
∞∏
t=1
(1− p̃EOS(t)). (57d)
In the above, we have assumed that P(Ac1 ∩ · · · ∩ Act) > 0 for all t, which is true by assumption that p̃EOS(t) < 1.. Hence, by Cor. 4.6, Eq. (57d) is 0 if and only if ∑ t p̃EOS(t) =∞. Case 2. If p̃EOS(t) = 1 is true for some t = t0, then P(Ac1 ∩ · · · ∩Act0) = 0 and hence P ( ⋂∞ t=1A c t) = 0 and such a language model is guaranteed to terminate at t0.

heading: E Proofs for Analyses of Common Language Models (§5)

heading: E.1 Proofs for FSSMs (§5.1)

heading: E.1.1 Proofs for Stochastic FSSMs
text: Theorem 5.3. An SFSSM is tight iff all accessible states are also co-accessible.
Proof. We refer to a state q as initial if sq > 0 and as final if tq > 0. (These are sometimes called source and sink states.) We prove each direction of the theorem in turn:
(⇒): Assume the SFSSM is tight. Let q be an accessible state. Since the SFSSM has at least one positive-probability path from an initial state, there is a positive probability of reaching q during generation. If there were no positive-probability path from q to a final state, then the SFSSM would never terminate on the occasions when it reached q, contradicting the assumption of tightness. Hence q must be co-accessible.
(⇐): Assume that all accessible states are co-accessible. We construct a Markov chain whose states are the SFSSM’s accessible states QA ⊆ {1, . . . , Q} together with an EOS state. In this Markov chain, the initial probability of q is given by sq when q ∈ QA and by 0 when q = EOS; the transition probability from q to q′ is given by Pqq′ when q, q′ ∈ QA, by tq when q ∈ QA and q′ = EOS, by 1 when q = q′ = EOS, and by 0 otherwise. The probability that the Markov chain is in state q ∈ QA after t steps equals the probability that the SFSSM is in state q after t steps (note that the SFSSM never reaches any state q /∈ QA). The probability that it is in state EOS after t steps equals the probability that the SFSSM has terminated after ≤ t steps.
Clearly EOS is an absorbing state of the Markov chain, meaning that once the Markov chain reaches this state, it never leaves. A fundamental result on finite-state Markov chains (Grinstead and Snell, 1997, Theorem 11.3) is that if every state can reach an absorbing state, then with probability 1, the chain reaches an absorbing state (“is absorbed”) in finite time. Every state can in fact reach EOS, by coaccessibility of QA. This further implies that EOS is the only absorbing state (as an absorbing state cannot reach any
other state). So by the result cited above, the Markov chain reaches EOS with probability 1 in finite time. Consequently, the SFSSM terminates after finitely many steps with probability 1; that is, the SFSSM is tight.
Corollary 5.4. Maximum likelihood estimates of n-gram models based on some corpus are tight.
Proof. The SFSSM for an n-gram model has states that correspond to (n − 1)-grams and transitions that correspond to characters (unigrams), as illustrated by Fig. 1. When the SFSSM’s probabilities are estimated with MLE, the accessible states are (n − 1)-grams that have appeared in some string in the corpus. Such states must also be co-accessible so that they can generate the rest of that string. Hence, by Thm. 5.3, this SFSSM is tight.

heading: E.1.2 Proofs for Substochastic FSSMs
text: To prove Thm. 5.6, we will make use of the following useful lemma. Lemma E.1. Let P′ be the transition sum matrix of a trimmed substochastic FSSM. Then ρ(P′) < 1 where ρ(·) denotes the spectral radius. Proof. To begin with, we wish to apply the following result, which connects the row sums of a matrix to its spectral radius. Below, Mn denotes the set of n× n matrices, and |||A|||∞ = max1≤i≤n ∑n j=1 |Aij | denotes the operator∞-norm. Proposition E.2 (§6.2.P8, Horn and Johnson, 2012). For any A ∈Mn, ρ(A) ≤ |||A|||∞. Additionally, if A is irreducible and not all absolute row sums of A are equal, then ρ(A) < |||A|||∞.
However, the transition sum matrix P of a substochastic FSSM may be reducible, whereas the irreducibility condition in Prop. E.2 cannot be dropped. Hence, we need to “decompose” P′ in a way that recovers irreducibility. We use the Frobenius normal form (also known as irreducible normal form) to achieve this.
Proposition E.3 (§8.3.P8, Horn and Johnson, 2012). Let A ∈ Mn be non-negative. Then, either A is irreducible or there exists a permutation matrix Π such that
Π>AΠ =   A1 ∗
. . . 0 Ak
  (58)
is block upper triangular, and each diagonal block is irreducible (possibly a 1 × 1 zero matrix). This is called a Frobenius normal form (or irreducible normal form) of A. Additionally, λ(A) = λ(A1) ∪ · · · ∪ λ(Ak) where λ(·) denotes the set of eigenvalues of a matrix.
Notice that the permutation in the Frobenius normal form merely renumbers the states of the trimmed FSSM. We may check that as a result, the termination probability given in Thm. 5.6 is unchanged:26
(Π>s′)>(Π>P′Π)k(Π>t′) = (s′>Π)(Π>P′kΠ)(Π>t′) = s′>P′kt′ (59)
Hence, with an appropriate renumbering, we may assume without loss of generality that P is already given in Frobenius normal form
P′ =   P′1 ∗
. . . 0 P′k
  (60)
where each P′i is irreducible. Since the transition sum matrix P′ of a trimmed substochastic FSSM is a substochastic matrix, each P′i is also substochastic. In fact, each P′i is strictly substochastic, meaning that there is at least one row that 26The equalities here use the fact that the inverse of a permutation matrix Π is its transpose: Π Π> = I .
sums to less than 1. To see this, suppose to the contrary that there is a stochastic P′i. Since the FSSM is trimmed, every state is both accessible and co-accessible. Being accessible implies that there is a positive probability of reaching every state in P′i. However, the stochasticity of P ′ i forces the corresponding t ′ entries to be 0. Hence, none of these states can transition to EOS, meaning that they’re not co-accessible, contradicting the assumption. Hence, every P′i is strictly substochastic. Then, either all row sums of P′i are less than 1 (in which case |||P′i|||∞ < 1) or some row sums are 1 and some are less than 1 (in which case |||P′i|||∞ = 1 and P′ has unequal absolute row sums). In either case, Prop. E.2 implies that ρ(P′i) < 1, for all 1 ≤ i ≤ k.
Finally, the last sentence of Prop. E.3 entails that ρ(P′) = max{ρ(P′1), . . . , ρ(P′k)}. Since each ρ(P′i) < 1, we have ρ(P ′) < 1. Theorem 5.6. Let P′ be the transition sum matrix of a trimmed substochastic FSSM. Then I − P′ is invertible and P (X ∈ Σ∗) = s′>(I−P′)−1t′ ≤ 1. Proof. By Lemma E.1, ρ(P′) < 1, in which case I−P′ is invertible and the Neumann series∑∞k=0 P′k = I + P′ + P′2 + · · · converges to (I−P′)−1 (Horn and Johnson, 2012, §5.6). Thus
P (Σ∗) = ∞∑
k=0
P (Σk) =
∞∑
k=0
s′>P′kt′ = s′> ( ∞∑
k=0
P′k ) t′ = s′>(I−P′)−1t′. (61)

heading: E.2 Proofs for Transformer Result (§5.2)
text: Again, the following theorem is well-known:
Theorem 5.7. Let X be a compact topological space and Y be any topological space. If f : X → Y is continuous, then f(X) ⊆ Y is also compact. Proof. Let {Uα}α∈A be any open cover of f(X). By continuity, f−1(Uα) ⊂ X is open for any α ∈ A, and hence {f−1(Uα)}α∈A is also an open cover ofX . By the compactness ofX , there is a finite sub-cover {f−1(Uαi)}ni=1, in which case {Uαi}ni=1 forms a finite sub-cover for f(X). Lemma 5.8. Let f : ( Rd )+ → ( Rd )+ be the function defined by a finite number of Transformer layers (e.g., n layers) with any continuous activation function. Given a compact set K ⊂ Rd. Then, there exists a compact set K ′ ⊂ Rd such that for every t ∈ Z>0,
f ( Kt ) ⊆ ( K ′ )t . (24)
Note. We make use of the following notations in the proof below: 4t−1 = {y ∈ Rt : y ≥ 0,1>y = 1} denotes the (t − 1)-dimensional simplex; Br(z) = {v ∈ Rn : dist(z,v) < r} denotes the open ball centered at z with radius r; A denotes the closure of set A.
Proof. Let K0 = K. In an autoregressive transformer, each of the n layers consists of two blocks: a self-attention block and a feedforward block. We will use induction on the 2n blocks to build up compact sets K1,K2, . . . ,K2n that contain the output vectors of these respective blocks, and then take K ′ = K2n.
The self-attention block is a function on (Rd)+ → (Rd)+. So, let t ∈ Z>0 be arbitrary and consider any sequence of input vectors (v1, . . . ,vt) such that for all i, vi ∈ K0. Denote the output vectors of the attention block with (v′1, . . . ,v ′ t). By definition of attention, each output vector v ′ j = ∑t i=1 α (j) i vi where α(j) ∈ 4t−1 are the attention weight vectors obtained through the softmax function. Compact sets in Rd are bounded (by the Heine–Borel theorem), and hence there exists M > 0 such that K0 ⊆ BM (0). Noting that the norm function ‖ · ‖ on Rd is convex, we have the following
‖v′j‖ = ∥∥∥∥∥ t∑
i=1
α (j) i vi ∥∥∥∥∥ (62a)
≤ t∑
i=1
α (j) i ‖vj‖ (∗)
≤ t∑
i=1
α (j) i M = M (62b)
where (∗) results from Jensen’s inequality. Eq. (62b) shows that each of the output vectors v′j lies in BM (0) which is compact. Hence, setting K1 = BM (0), we have shown that, for any t ∈ Z>0, the attention block maps Kt0 into K t 1.
Note that we cannot use Thm. 5.7 here because the attention block defines a different function on Rt×d → Rt×d for each t, and Thm. 5.7 only implies that there exists a separate length-dependent output compact set Kt ⊂ Rt×d for each t, which is different from this lemma’s statement.
The feedforward function is a continuous function on Rd → Rd, and therefore, by Thm. 5.7, maps its input compact set K1 to an output compact set, which we call K2.
Finally, residual connections and layer norms are also continuous functions acting on each of the input vectors, and hence by the same reasoning would also preserve compactness.
Now we can use induction and show that there exist compact sets K3,K4, . . . ,K2n−1,K2n where K2n contains the output set of the final layer. Set K ′ = K2n and we have proven the statement.
Theorem 5.9. The autoregressive sequence model defined by any (fixed-depth) Transformer is tight.
Proof. Given the Transformer, there exists a fixed compact set K that will contain all inputs vi ∈ Rd to the first layer. This is true because each vi is the sum of a word embedding, which falls in a finite set since Σ is finite, and a position embedding, which lies in the compact set [−1, 1]d. Hence, by Lemma 5.8, there exists a fixed compact set K ′ that contains all output embedding vectors (regardless of how long the sequence is).
The final output probability is given by a multiplication with the word embedding matrix followed by the softmax function as in Eq. (25). This process amounts to composing two continuous functions. In particular, we can extract the EOS probability as a continuous R-valued function gEOS : K ′ → (0, 1) (neither 0 or 1 is in the range of the softmax function). By continuity of gEOS and Thm. 5.7, K ′′ def= gEOS(K ′) ⊆ (0, 1) is compact. Since K ′′ is compact, and hence closed, inf K ′′ ∈ K ′′. Thus inf K ′′ ∈ (0, 1) and in particular inf K ′′ > 0. Therefore, taking = inf K ′′, we have shown that the EOS probability of a Transformer is bounded below by some > 0 (regardless of the length of the sequence).
Hence, by Prop. 4.3, any Transformer ASM is tight and thus defines a language model.

heading: E.3 Proofs for RNN Result (§5.3)
text: Proposition 5.10. Given an RNN ASM over Σ. Again let the output symbol vector be ux ∈ Rd for x ∈ Σ, and set k def= supx∈Σ ‖ux − uEOS‖2. Additionally, for each t > 0, let ‖ĥt‖2 be the maximum attainable hidden state norm for any context x ∈ Σt. Such a sequence model is tight if k‖ĥt‖2 ≤ log t for all sufficiently large t.
Proof. Let Xt(ω) be the random variable that is equal to the tth token in an outcome ω ∈ Ω. Also let hx be the hidden representation of the RNN after processing some finite list of tokens x ∈ Σ∗. Further, let ux ∈ Rd be the output embedding of x ∈ Σ, Then for any t ∈ N and any x ∈ Σt, we have:
P(Xt+1 = EOS | X≤t = x) = expu>EOShx∑ y∈Σ expu > y hx
(63a)
= 1∑
y∈Σ expu > y hx / expu > EOShx
(63b)
= 1
1 + ∑ y∈Σ exp(uy − uEOS)>hx (63c)
≥ 1 1 + ∑ y∈Σ exp (‖uy − uEOS‖2‖hx‖2) (Cauchy–Schwarz) (63d) ≥ 1 1 + ∑ y∈Σ exp(k‖hx‖2) (63e) = 1
1 + |Σ| exp(k ‖hx‖2) (63f)
Now define ‖ĥt‖2 def= supx∈Σt ‖hx‖2. We then have that ∀t ∈ N and ∀x ∈ Σt:
P(Xt+1 = EOS | X≤t = x) ≥ 1
1 + |Σ| exp(k‖ĥt‖2) (64)
Now, by Prop. 4.3, we have that if ∑∞
t=0 1 1+|Σ| exp(k ‖ĥt‖2) diverges, then the language model is tight.
We will show that this condition holds if ∃N ∈ N such that ∀t ≥ N , k‖ĥt‖2 ≤ log t. First, note that limt→∞ 1t 1+|Σ|t 1 = limt→∞ 1 t + |Σ| = |Σ| ∈ (0,∞). Hence, by the limit comparison
test, since ∑∞
t=1 1 t diverges, this means ∑∞ t=1 1 1+|Σ|t must also diverge.
Now, suppose there exists N such that that k ‖ĥt‖2 ≤ log t for all t ≥ N . This implies that for t ≥ N we have 1
1+|Σ| exp(k‖ĥt‖2) ≥ 11+|Σ|t , which combined with the above and the comparison test, implies that∑∞
t=N 1 1+|Σ| exp(k‖ĥt‖2) diverges. This in turn means that
∑∞ t=0
1
1+|Σ| exp(k‖ĥt‖2) diverges.
Hence, if k ‖ĥt‖2 ≤ log t for all sufficiently large t (that is, for all t ≥ N ), then the RNN ASM is tight and thus defines a language model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
7 A2. Did you discuss any potential risks of your work? Our paper provides theoretical analysis of language models. We do not expect potential risks as a result of this work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Additional machinery needs to be developed in order for a proper treatment and we leave this for future work. Another notable limitation is that we exclusively focused on the autoregressive production of language. Importantly, our formalism might not be compatible with other models of language production such as those induced by a PCFG. Finally, our proofs of Thm. 5. 9 and Prop. 5. 10 exploit the strictly positive property of the softmax function. Importantly, they do not apply to models with sparse distributions (Martins and Astudillo, 2016; Peters et al. , 2019; Martins, 2021)