SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuchen Li
henl4@cs.cmu.edu
Bingbin Liu
bingbinl@cs.cmu.edu
Transformer interpretability aims to understand the algorithm implemented by a learned Transformer by examining various aspects of the model, such as the weight matrices or the attention patterns. In this work, through a combination of theoretical results and carefully controlled experiments on synthetic data, we take a critical view of methods that exclusively focus on individual parts of the model, rather than consider the network as a whole. We consider a simple synthetic setup of learning a (bounded) Dyck language. Theoretically, we show that the set of models that (exactly or approximately) solve this task satisfy a structural characterization derived from ideas in formal languages (the pumping lemma). We use this characterization to show that the set of optima is qualitatively rich; in particular, the attention pattern of a single layer can be "nearly randomized", while preserving the functionality of the network. We also show via extensive experiments that these constructions are not merely a theoretical artifact: even with severe constraints to the architecture of the model, vastly different solutions can be reached via standard training. Thus, interpretability claims based on inspecting individual heads or weight matrices in the Transformer can be misleading.
Transformer-based models power many leading approaches to natural language processing. With their growing deployment in various applications, it is increasingly essential to understand the inner working of these models. Towards addressing this, there have been great advancement in the field of interpretability presenting various types of evidence (Clark et al., 2019; Vig & Belinkov, 2019; Wegreffe & Pinter, 2019; Nanda et al., 2023; Wang et al., 2023), some of which, however, can be misleading despite being highly intuitive (Jain & Wallace, 2019; Serrano & Smith, 2019; Rogers et al., 2020; Grimsley et al., 2020; Brunner et al., 2020; Meister et al., 2021).
In this work, we aim to understand the theoretical limitation of certain interpretability methods by characterizing the set of viable solutions. We focus on myopic interpretability methods, i.e. methods based on examining individual components only. We adopt a particular toy setup in which Transformers are trained to generate Dyck grammars, a classic type of formal language grammar consisting of balanced parentheses of multiple types. Dyck is a useful sandbox, as it captures properties like long-range dependency and hierarchical tree-like structure that commonly appear in natural and programming language syntax, and has been an object of interest in many theoretical studies (Hahn, 2020; Yao et al., 2021; Liu et al., 2022b, 2023). Dyck is canonically parsed using
(a) With Position Embedding
(b) With Position Embedding
(c) Without Position Embedding
Figure 1: Second-layer attention patterns of two-layer Transformers on Dyck: typical attention patterns do not exactly match the intuitively interpretable stack-like pattern prescribed in Ebrahimi et al. (2020); Yao et al. (2021). The blue boxes indicate the locations of the last unmatched open brackets, as they would appear in a stack-like pattern. All models reach  $\geq 97\%$  accuracy (defined in Section 4.1). In the heatmap, darker color indicates larger value.
(d) Without Position Embedding
a stack-like data structure. Such stack-like patterns (Figure 1) have been observed in the attention heads (Ebrahimi et al., 2020), which was later bolstered by mathematical analysis in Yao et al. (2021).
From a representational perspective and via explicit constructions of Transformer weights, recent work (Liu et al., 2023; Li et al., 2023) show that Transformers are sufficiently expressive to admit very different solutions that perform equally well on the training distribution. Thus, the following questions naturally arise:
(Q1) Do Transformer solutions found empirically match the theoretical constructions given in these representational results (Figure 1)? In particular, are interpretable stack-like pattern in Ebrahimi et al. (2020) the norm or the exception in practice?
(Q2) More broadly, can we understand in a principled manner the fundamental obstructions to reliably "reverse engineering" the algorithm implemented by a Transformer by looking at individual attention patterns?
(Q3) Among models that perform (near-)optimally on the training distribution, even if we cannot fully reverse engineer the algorithm implemented by the learned solutions, can we identify properties that characterize performance beyond the training distribution?
Our contributions. We first prove several theoretical results to provide evidence for why individual components (e.g. attention patterns or weights) of a Transformer should not be expected to be interpretable. In particular, we prove:
- A perfect balance condition (Theorem 1) on the attention pattern that is sufficient and necessary for 2-layer Transformers with a minimal first layer (Assumption 1) to predict optimally on Dyck of any length. We then show that this condition permits abundant non-stack-like attention patterns that do not necessarily reflect any structure of the task, including uniform attentions (Corollary 1).
- An approximate balance condition (Theorem 3), the near-optimal counterpart of the condition above, for predicting on bounded-length Dyck. Likewise, non-stack-like attention patterns exist.
- Indistinguishability from a single component (Theorem 2), proved via a Lottery Ticket Hypothesis style argument that any Transformer can be approximated by pruning a larger random Transformer, implying that interpretations based exclusively on local components may be unreliable.
We further accompany these theoretical findings with an extensive set of empirical investigations.
Is standard training biased towards interpretable solutions? While both stack-like and non-stack like patterns can process Dyck theoretically, the inductive biases of the architecture or the optimization process may prefer one solution over the other in practice. In Section 4.1, based on a wide range of Dyck distributions and model architecture ablations, we find that Transformers that generalize near-perfectly in-distribution (and reasonably well out-of-distribution) do not typically produce stack-like attention patterns, showing that the results reported in prior work (Ebrahimi et al., 2020) should not be expected from standard training.
Do non-interpretable solutions perform well in practice? Our theory predicts that balanced (or even uniform) attentions suffice for good in- and out-of-distribution generalization. In Section 4.2, we
empirically verify that with standard training, the extent to which attentions are balanced is positively correlated with generalization performance. Moreover, we can guide Transformers to learn more balanced attention by regularizing for the balance condition, leading to better length generalization.
There has been a flourishing line of work on interpretability in natural language processing. Multiple "probing" tasks have been designed to extract syntactic or semantic information from the learned representations (Raganato & Tiedemann, 2018; Liu et al., 2019; Hewitt & Manning, 2019; Clark et al., 2019). However, the effectiveness of probing often intricately depend on the architecture choices and task design, and sometimes may even result in misleading conclusions (Jain & Wallace, 2019; Serrano & Smith, 2019; Rogers et al., 2020; Brunner et al., 2020; Prasanna et al., 2020; Meister et al., 2021). While these challenges do not completely invalidate existing approaches (Wiegreffe & Pinter, 2019), it does highlight the need for more rigorous understanding of interpretability.
Towards this, we choose to focus on the synthetic setup of Dyck whose solution space is easier to characterize than natural languages, allowing us to identify a set of feasible solutions. While similar representational results have been studied in prior work (Yao et al., 2021; Liu et al., 2023; Zhao et al., 2023), our work emphasizes that theoretical constructions do not resemble the solutions found in practice. Moreover, the multiplicity of valid constructions suggest that understanding Transformer solutions require analyzing the optimization process, which a number of prior work has made progress on (Jelassi et al., 2022; Li et al., 2023; Deng et al., 2023).
Finally, it is worth noting that the challenges highlighted in our work do not contradict the line of prior work that aim to improve mechanistic interpretability into a trained model or the training process (Elhage et al., 2021; Olsson et al., 2022; Nanda et al., 2023; Chughtai et al., 2023; Li et al., 2023), which aim to develop circuit-level understanding of a particular model or the training process.
We defer discussion on additional related work to Appendix A.
Dyck languages A Dyck language (Schützenberger, 1963) is generated by a context-free grammar, where the valid strings consist of balanced brackets of different types (for example, “ $[()]$ ” is valid but “ $(())]$ ” is not).  $\mathrm{Dyck}_k$  denote the Dyck language defined on  $k$  types of brackets. The alphabet of  $\mathrm{Dyck}_k$  is denoted as  $[2k] \equiv \{1, 2, \dots, 2k\}$ , where for each type  $t \in [k]$ , tokens  $2t - 1$  and  $2t$  are a pair of corresponding open and closed brackets. Dyck languages can be recognized by a push-down automaton. For a string  $w$  and  $i \leq j \in \mathbb{Z}_{+}$ , we use  $w_{i:j}$  to denote the substring of  $w$  between position  $i$  and position  $j$  (both ends included). For a valid prefix  $w_{1:i}$ , the grammar depth of  $w_{1:i}$  is defined as the depth of the stack after processing  $w_{1:i}$ :
$$
\operatorname {d e p t h} \left(w _ {1: i}\right) = \# \text {O p e n B r a c k e t s i n} w _ {1: i} - \# \text {C l o s e d B r a c k e t s i n} w _ {1: i}.
$$
We overload  $\mathrm{depth}(w_{1:i})$  to also denote the grammar depth of the bracket at position  $i$ . For example, in each pair of matching brackets, the closing bracket is one depth smaller than the open bracket. We will use  $\tau_{i,d}$  to denote a token of type  $i \in [2k]$  placed at grammar depth  $d \in \mathbb{N}$ .
We consider bounded-depth Dyck languages following Yao et al. (2021). Specifically,  $\mathrm{Dyck}_{k,D}$  is a subset of  $\mathrm{Dyck}_k$  such that the depth of any prefix of a word is bounded by  $D$ ,
$$
\mathrm {D y c k} _ {k, D} := \left\{w _ {1: n} \in \mathrm {D y c k} _ {k} \mid \max  _ {i \in [ n ]} \operatorname {d e p t h} \left(w _ {1: i}\right) \leq D \right\}. \tag {1}
$$
While a bounded grammar depth might seem restrictive, it suffices to capture many practical settings. For example, the level of recursion occurring in natural languages is typically bounded by a small constant (Karlsson, 2007; Jin et al., 2018). We further define the length- $N$  prefix set of  $\mathrm{Dyck}_{k,D}$  as
$$
\mathrm {D y c k} _ {k, D, N} = \left\{w _ {1: N} \mid \exists n \geq N, w _ {N + 1: n} \in [ 2 k ] ^ {n - N}, s. t. w _ {1: n} \in \mathrm {D y c k} _ {k, D} \right\}. \tag {2}
$$
Our theoretical setup uses the following data distribution  $\mathcal{D}_{q,k,D,N}$ :
Definition 1 (Dyck distribution). The distribution  $\mathcal{D}_{q,k,D,N}$ , specified by  $q\in (0,1)$ , is defined over  $\mathrm{Dyck}_{k,D,N}$  such that  $\forall w_{1:N}\in \mathrm{Dyck}_{k,D,N}$ ,
$$
\mathbb {P} \left(w _ {1: N}\right) \propto (q / k) ^ {\# \{i | w _ {i} \text {i s o p e n , d e p t h} \left(w _ {1: i}\right) > 1 \}} \cdot (1 - q) ^ {\# \{i | w _ {i} \text {i s c l o s e d , d e p t h} \left(w _ {1: i}\right) <   D - 1 \}}. \tag {3}
$$
That is,  $q \in (0,1)$  denote the probability of seeing an open bracket at the next position, except for two corner cases: 1) the next bracket has to be open if the current grammar depth is 0 (1 after seeing the open bracket); 2) the next bracket has to be closed if the current grammar depth is  $D$ .
Training Objectives. Given a model  $f_{\theta}$  parameterized by  $\theta$ , we train with a next-token prediction language modeling objective on a given  $\mathcal{D}_{q,k,D,N}$ . Precisely, given a loss function  $l(\cdot ,\cdot)\to \mathbb{R}$ ,  $f_{\theta}$  is trained to minimize the loss function  $\min_{\theta}\mathcal{L}(\theta ;\mathcal{D}_{q,k,D,N})$  with
$$
\mathcal {L} (\theta ; \mathcal {D} _ {q, k, D, N}) = \mathbb {E} _ {w _ {1: N} \sim \mathcal {D} _ {q, k, D, N}} [ \frac {1}{N} \sum_ {i = 1} ^ {N} l \left(f _ {\theta} \left(w _ {1: i - 1}\right), z \left(w _ {i}\right)\right) ] \tag {4}
$$
in which  $z(w_{i}) \in \{0,1\}^{2k}$  denotes the one-hot embedding of token  $w_{i}$ . We will omit the distribution  $\mathcal{D}_{q,k,D,N}$  when it is clear from the context. We will also consider a  $\ell_2$ -regularized version  $\mathcal{L}^{\mathrm{reg}}(\theta) = \mathcal{L}(\theta) + \lambda \frac{\|\theta\|_2^2}{2}$  with parameter  $\lambda > 0$ .
For our theory, we will consider the mean squared error as the loss function:
$$
l := l _ {s q} (x, z _ {i}) = \| x - z _ {i} \| _ {2} ^ {2}. \tag {5}
$$
In our experiments, we apply the cross entropy loss following common practice.
Transformer Architecture. We consider a general formulation of Transformer in this work: the  $l$ -th layer is parameterized by  $\theta^{(l)} \coloneqq \{W_{Q}^{(l)}, W_{K}^{(l)}, W_{V}^{(l)}, \mathrm{param}(\mathbf{g}^{(l)})\} \in \Theta$ , where  $W_{K}^{(l)}, W_{Q}^{(l)} \in \mathbb{R}^{m_{a} \times m}$ , and  $W_{V}^{(l)} \in \mathbb{R}^{m \times m}$  are the key, query, and value matrices of the attention module;  $\mathrm{param}(\mathbf{g}^{(l)})$  are parameters of a feed-forward network  $\mathbf{g}^{(l)}$ , consisting of fully connected layers, (optionally) LayerNorms and residual links. Given  $X \in \mathbb{R}^{m \times N}$ , the matrix of  $m$ -dimensional features on a length- $N$  sequence, the  $l$ -th layer of a Transformer computes the function
$$
f _ {l} (X; \theta^ {(l)}) = \mathrm {g} ^ {(l)} \left(\mathrm {L N} \left(W _ {V} ^ {(l)} X \underbrace {\sigma \left(\mathcal {C} + \left(W _ {K} ^ {(l)} X\right) ^ {\top} \left(W _ {Q} ^ {(l)} X\right)\right)} _ {\text {a t t e n t i o n p a t t e r n}}\right) + X\right), \tag {6}
$$
where  $\sigma$  is the column-wise softmax operation defined as  $\sigma(A)_{i,j} = \frac{\exp(A_{i,j})}{\sum_{k=1}^{N} \exp(A_{k,j})}$ ,  $\mathcal{C}$  is the causal mask matrix defined as  $\mathcal{C}_{i,j} = -\inf \cdot \mathbb{1}[i > j]$  where inf denotes infinity. We call  $\sigma\left(\mathcal{C} + (W_K^{(l)}X)^\top(W_Q^{(l)}X)\right)$  the Attention Pattern of the Transformer layer  $l$ . LN represents columnwise LayerNorm operation, whose  $j_{th}$  output column is defined as
$$
\mathrm {L N} _ {C _ {L N}} (A) _ {:, j} = \frac {\mathcal {P} _ {\perp} A _ {: , j}}{\max  \left\{\| \mathcal {P} _ {\perp} A _ {: , j} \| _ {2} , C _ {L N} \right\}}, \mathcal {P} _ {\perp} = \mathcal {I} _ {m} - \frac {1}{m} \mathbf {1 1} ^ {\top}. \tag {7}
$$
Here  $\mathcal{P}_{\perp}$  denotes the projection orthogonal to the  $11^{\top}$  subspace  $^2$  and  $C_{LN}$  is called the normalizing constant for LayerNorm.
We will further define the attention output at the  $l$ -th layer as
$$
a _ {l} (X; \theta^ {(l)}) = W _ {V} ^ {(l)} X \sigma \left(\mathcal {C} + \left(W _ {K} ^ {(l)} X\right) ^ {\top} \left(W _ {Q} ^ {(l)} X\right)\right). \tag {8}
$$
When  $C_{LN} = 0$ , we will also consider the unnormalized attention output as
$$
\tilde {a} _ {l} (X; \theta^ {(l)}) = W _ {V} ^ {(l)} X \tilde {\sigma} \left(\mathcal {C} + \left(W _ {K} ^ {(l)} X\right) ^ {\top} \left(W _ {Q} ^ {(l)} X\right)\right). \tag {9}
$$
where  $\tilde{\sigma}(A)_{i,j} = \exp(A_{i,j})$  and it holds by definition that  $\mathrm{LN}_0(\tilde{a}_l(X; \theta^{(l)})) = \mathrm{LN}_0(a_l(X; \theta^{(l)}))$ .
An  $L$ -layer Transformer  $\mathcal{T}_L$  consists of a composition of  $L$  of the above layers, along with a word embedding matrix  $W_{E} \in \mathbb{R}^{m \times 2k}$  and a linear decoding head with weight  $W_{\mathrm{Head}} \in \mathbb{R}^{2k \times w}$ . When
<sup>1</sup>The challenge of applying our theory to cross-entropy loss is that for some prefixes, their grammatical immediate continuations strictly exclude certain tokens in the vocabulary (e.g. " $\)" cannot immediately follow "\(\{\}$ ", so the optimal cross-entropy loss can only be attained if some parameters are set to infinity. However, when label smoothing is added, the optima is finite again, and analysis similar to ours could apply.
2this is just a compact way to write the standard mean subtraction operation
inputting a sequence of tokens into Transformer, we will append a starting token  $t_{S}$  that is distinct from any token in the language at the beginning of the sequence. Let  $\mathcal{Z} \in \mathbb{R}^{2k \times (N + 1)}$  denote the one-hot embedding of a length- $N$  sequence, then  $\mathcal{T}_L$  computes for  $\mathcal{Z}$  as
$$
\mathcal {T} (\mathcal {Z}) = W _ {\text {H e a d}} \left[ f _ {L} \left(\dots \left(f _ {1} \left(W _ {E} \mathcal {Z}\right)\right)\right) \right] _ {1: 2 k, (N + 1)}. \tag {10}
$$
3 Theoretical Analyses
Many prior works have looked for intuitive interpretations of Transformer solutions by studying the attention patterns of particular heads or some individual components of a Transformer (Clark et al., 2019; Vig & Belinkov, 2019; Dar et al., 2022). However, we show in this section why this methodology can be insufficient even for the simple setting of Dyck. Namely, for Transformers that generalize well on Dyck (both in-distribution and out-of-distribution), neither attention patterns nor individual local components are guaranteed to encode structures specific for parsing Dyck. We further argue that the converse is also insufficient: when a Transformer does produce interpretable attention patterns, there could be limitations of such interpretation as well, as discussed in Appendix B. Together, our results provide theoretical evidence that careful analyses (beyond heuristics) are required when interpreting the components of a learned Transformer.
3.1 Interpretability Requires Inspecting More Than Attention Patterns
This section focuses on Transformers with 2 layers, which are sufficient for processing Dyck (Yao et al., 2021). We will show that even under this simplified setting, attention patterns alone are not sufficient for interpretation. In fact, we will further restrict the set of 2-layer Transformers by requiring the first-layer outputs to only depend on information necessary for processing Dyck:
Assumption 1 (Minimal First Layer). We consider 2-layer Transformers with a minimal first layer  $f_{1}$ . That is, let  $Z \in \mathbb{R}^{2k \times (N + 1)}$  denote the one-hot embeddings of any input sequence  $t_{S}, t_{1}, \ldots, t_{N} \in [2k]$ , then the  $(j + 1)_{th}$  column of the output  $f_{1}(W^{E}Z)$  only depends on the type and depth of  $t_{j}, \forall j \in [N]$ .
Assumption 1 requires the first layer output to depend only on the bracket type and depth, disregarding any other information such as positions; one such example is given by Yao et al. (2021). The construction of a minimal first layer can vary, hence we directly parameterize its output instead:
Definition 2 (Minimal first layer embeddings). Given a minimal first layer,  $e(\tau_{t,d}) \in \mathbb{R}^m$  denotes its output embedding of  $\tau_{t,d}$  for  $t \in [2k]$ ,  $d \in [D]$ .  $e(t_S) \in \mathbb{R}^m$  is the embedding of the starting token.
It is important to note that while the minimal first layer is a strong condition, it does not weaken our results: We will show that the function class allows for a rich set of solutions, none of which are necessarily interpretable. Relaxing to more complex classes will only expand the solution set, and hence our conclusion will remain valid. See Appendix C.2 for more technical details.
3.1.1 Perfect Balance Condition: Ideal Generalization of Unbounded Length
Some prior works have tried to understand the model by inspecting the attention patterns (Ebrahimi et al., 2020; Clark et al., 2019; Vig & Belinkov, 2019). However, we will show that the attention patterns alone are too flexible to be helpful, even for the restricted class of a 2-layer Transformer with a minimal first layer (Assumption 1) and even on a language as simple as Dyck. In particular, the Transformer only needs to satisfy what we call the balanced condition:
Definition 3 (Balance condition). A 2-layer Transformer (Equation (10)) with a minimal first layer (Assumption 1 and Definition 2) is said to satisfy the balance condition, if for any  $i$ ,  $j_1, j_2 \in [k]$  and  $d', d_1, d_2 \in [D]$ ,
$$
\left(\boldsymbol {e} \left(\tau_ {2 i - 1, d ^ {\prime}}\right) - \boldsymbol {e} \left(\tau_ {2 i, d ^ {\prime} - 1}\right)\right) ^ {\top} \left(W _ {K} ^ {(2)}\right) ^ {\top} W _ {Q} ^ {(2)} \left(\boldsymbol {e} \left(\tau_ {2 j _ {1}, d _ {1}}\right) - \boldsymbol {e} \left(\tau_ {2 j _ {2}, d _ {2}}\right)\right) = 0. \tag {11}
$$
The following result shows that under minor conditions the balance condition is both necessary and sufficient:
Theorem 1 (Perfect Balance). Consider a two-layer Transformer  $\mathcal{T}$  (Equation (10)) with a minimal first layer (Assumption 1) and  $C_{LN} = 0$  (Equation (7)). Let  $\mathcal{O}$  denote the optimal prediction scenario, that is, when the first layer embeddings  $\{e(\tau_{i,d})\}_{d\in [D],i\in [2k]}$  (Definition 2) and second layer parameters  $\theta^{(2)}$  satisfy
$$
\theta := \left\{\boldsymbol {e} (\tau_ {i, d}) \right\} _ {d \in [ D ], i \in [ 2 k ]}, \theta^ {(2)} \} = \arg \min _ {\tilde {\theta}} \mathcal {L} (\tilde {\theta}; \mathcal {D} _ {q, k, D, N}), \forall N,
$$
where the objective  $\mathcal{L}$  is defined in Equation (4). Then,
- Equation (11) a necessary condition of  $\mathcal{O}$ , if  $W_V^{(2)}$  satisfies  $\mathcal{P}_{\perp}W_V^{(2)}e(\tau_{t,d}) \neq 0, \forall t \in [k], d \in [D]$ .
- Equation (11) is a sufficient condition of  $\mathcal{O}$ , if the set of  $2k + 1$  encodings  $\{\pmb{e}(\tau_{2i - 1,d}),\pmb{e}(\tau_{2i,d})\}_{i\in [k]}\cup \{\pmb{e}(t_S)\}$  are linearly independent for any  $d\in [D]$ , and the projection function  $\mathrm{g}^{(2)}$  is a 6-layer MLP<sup>3</sup> with  $O(k^2 D^2)$  width.
Remark: Recall from Equation (7) that  $\mathcal{P}_{\perp}$  projects to the subspace orthogonal to  $\mathbf{11}^{\top}$ . The assumption in the necessary condition can be intuitively understood as requiring all tokens to have nonzero contributions to the prediction after the LayerNorm.
Recall that  $e(\tau_{2i-1,d'})$ ,  $e(\tau_{2i,d'-1})$  denote the first-layer outputs for a matching pair of brackets. Intuitively, Equation (11) says that since matching brackets should not affect future predictions, their embeddings should balance out each other. The balance condition Equation (11) is "perfect" in the sense that the theory assumes the model can minimize the loss for any length  $N$ ; we will see an approximate version later in Theorem 3.
Proof of the necessity of the balance condition. The key idea is reminiscent of the pumping lemma for regular languages. For any prefix  $p$  ending with a closed bracket  $\tau_{2j,d}$  for  $d \geq 1$  and containing brackets of all depths in  $[D]$ , let  $p_\beta$  be the prefix obtained by inserting  $\beta$  pairs of  $\{\tau_{2i-1,d'}, \tau_{2i,d'-1}\}$  for arbitrary  $i \in [k]$  and  $d' \in [D]$ . Denote the projection of the unnormalized attention output by
$$
u \left(\tau_ {t _ {1}, d _ {1}}, \tau_ {t _ {2}, d _ {2}}\right) := \mathcal {P} _ {\perp} \exp \left(e \left(\tau_ {t _ {1}, d _ {1}}\right) ^ {\top} \left(W _ {K} ^ {(2)}\right) ^ {\top} W _ {Q} ^ {(2)} e \left(\tau_ {t _ {2}, d _ {2}}\right)\right) W _ {V} ^ {(2)} e \left(\tau_ {t _ {1}, d _ {1}}\right). \tag {12}
$$
We ignored the normalization in softmax above, since the attention output will be normalized directly by LayerNorm according to Equation (6).
By Equation (10), there exists a vector  $v \in \mathbb{R}^m$  such that for any  $\beta \in \mathbb{N}$ , the next-token logits given by Transformer  $\mathcal{T}$  are