SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinyi Wang $^{1}$ , Wanrong Zhu $^{1}$ , Michael Saxon $^{1}$ , Mark Steyvers $^{2}$ , William Yang Wang $^{1}$
$^{1}$ Department of Computer Science, University of California, Santa Barbara
$^{2}$ Department of Cognitive Sciences, University of California, Irvine
{xinyi_wang, wanrongzhu, saxon}@ucsb.edu,
msteyver@uci.edu, william@cs.ucsb.edu
In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. However, existing literature has highlighted the sensitivity of this capability to the selection of few-shot demonstrations. Current understandings of the underlying mechanisms by which this capability arises from regular language model pretraining objectives remain disconnected from the real-world LLMs. This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models. On this premise, we propose an algorithm to select optimal demonstrations from a set of annotated data with a small LM, and then directly generalize the selected demonstrations to larger LMs. We demonstrate significant improvement over baselines, averaged over eight GPT models on eight real-world text classification datasets. We also demonstrate the real-world usefulness of our algorithm on GSM8K, a math word problem dataset. Our empirical findings support our hypothesis that LLMs implicitly infer a latent variable containing task information.
Transformer-based [41] pre-trained large language models (LLMs) have demonstrated significant advancements in a variety of natural language processing (NLP) tasks. As the size of these LLMs increases, they gain "in-context learning" capabilities, whereby the models achieve state-of-the-art (SOTA) or near-SOTA performance by conditioning on a small number of demonstration examples at inference time, without any need for updating model parameters [4]. Below is an example input sequence for semantic analysis with in-context learning:
Great movie. Positive.\n The worst movie ever. Negative.\n Can't wait to see the second movie!
The first two lines are two demonstrations, and the third line is a test input. We expect an LLM to output the correct label Positive as a continuation.
In-context learning has been demonstrated to be an effective technique for a wide range of NLP tasks. However, it is sensitive to the choice, format, and even the order of the demonstrations used [29, 20]. This makes achieving optimal performance with in-context learning a significant challenge, requiring real human effort to adjust the format and selection of demonstration examples. Heuristic solutions, such as selecting demonstrations based on the similarity between the demonstrations and test input
$^{1}$ Code: https://github.com/WANGXinyiLinda/concept-based-demonstration-selection.
[19, 37] have been proposed, but a comprehensive understanding of why certain demonstrations are effective while others are not remains elusive. Additionally, the mechanisms by which LLMs acquire in-context learning capabilities through training on natural text under the standard language model pre-training objective are not fully understood. Recent works on understanding in-context learning provide valuable insights and theoretical results [5, 1, 42, 14, 12], but are limited in scope, focusing on synthetic experiments to validate their hypotheses, while it remains unclear if these results generalize to LLMs pre-trained on real-world natural language data. Xie et al. [50] introduced a prominent result providing a latent topic (concept) variable interpretation for in-context learning. They showed that the in-context learning predictor approaches the Bayes optimal predictor when the number of demonstrations approaches infinity, under the assumption that both the pre-training data distribution and task-specific data distribution are Hidden Markov Models (HMM). However, the assumption that the data generation process is Hidden Markovian makes extrapolation of the result to natural language questionable, and restricts empirical verification to synthetic data with toy models.
We are inspired by this prior work and introduce a more general and natural explanation built on realistic assumptions, which gives rise to a practical demonstration selection algorithm. Our explanation is inspired by the generation process of a topic model, i.e. a simple latent variable model:
$$
P (\boldsymbol {w} _ {1: T}) = \int_ {\Theta} P (\boldsymbol {w} _ {1: T} | \boldsymbol {\theta}) P (\boldsymbol {\theta}) d \boldsymbol {\theta}
$$
Where  $\theta \in \Theta$  represents a potentially high dimensional topic/concept variable,  $\Theta$  is the space of the topic/concept variable, and  $w_{1:T}$  refers to the token sequence of a piece of text. Note that the topic model here refers to the modern neural topic models [23, 22]. On the other hand, generative LLMs model text data according to the general probabilistic decomposition:
$$
P \left(\boldsymbol {w} _ {1: T}\right) = \prod_ {i = 1} ^ {T} P \left(\boldsymbol {w} _ {i} \mid \boldsymbol {w} _ {i - 1}, \dots , \boldsymbol {w} _ {1}\right)
$$
While in practice, LLMs generate new tokens based on all previous tokens, we investigate whether a simplified assumption similar to that of topic models can be made for LLMs:
$$
P _ {M} \left(\boldsymbol {w} _ {t + 1: T} \mid \boldsymbol {w} _ {1: t}\right) = \int_ {\Theta} P _ {M} \left(\boldsymbol {w} _ {t + 1: T} \mid \boldsymbol {\theta}\right) P _ {M} (\boldsymbol {\theta} \mid \boldsymbol {w} _ {1: t}) d \boldsymbol {\theta}
$$
In this scenario, the generated tokens are assumed to be conditionally independent of previous tokens, given the latent topic (concept) variable that acts like an approximate sufficient statistic for the posterior information related to the prompt  $\boldsymbol{w}_{1:t}$ . For in-context learning, this concept variable includes format and task information. By conditioning on an appropriate latent concept variable, LLMs would generate the desired continuation with  $P(\boldsymbol{w}_{t+1:T}|\boldsymbol{\theta})$ . As LLMs do not explicitly learn a latent variable distribution like LDA-style topic models [3], we can instead utilize this formulation under an Empirical Bayesian formulation inspired by Lester et al. [17] to only approximate the optimal latent variable value for a desired task, using a small LLM (with less than 1B parameters), which is computationally efficient.
We empirically validate our explanation by selecting examples  $(\pmb{w}_{1:t}$  in the equations) that are most likely to infer the optimal latent variable value (those with the highest posterior probability  $P(\pmb{\theta}|\pmb{w}_{t+1:T})$ ). We then directly use them as demonstrations for in-context learning with other larger LLMs (up to 175B parameters) and observed a significant performance improvement. The generalization of demonstrations between LLMs is likely a result of similar pre-training data distributions.
While our work is inspired by that of Xie et al. [50], our approach differs significantly in both theoretical analysis and experimental settings. Our main contributions are as follows:
- We assume a general data generation process specified by a three-variable causal graph, without constraints on the distribution function or the number of demonstrations.
- We prove under these realistic assumptions that the in-context learning predictor can reach the Bayes optimal predictor with a finite number of demonstrations chosen using the latent concept variable.
- We introduce an efficient, practical demonstration selection algorithm based on our theoretical results, which can select demonstrations using a small LLM and then directly generalize the demonstrations to other LLMs. The effectiveness of our algorithm is empirically validated using real-world LLMs on both text classification tasks and math word problems.
Our goal is to close the gap between theoretical understandings and real-world LLMs. To the best of our knowledge, our proposed latent variable explanation of in-context learning is the first Bayesian explanation that yields an effective algorithm in real-world scenarios.
2 Theoretical Analysis
In in-context learning, the prompt  $w_{1:t}$  is composed of several demonstrations and a test input. The generated tokens  $w_{t+1:T}$  represent the model's prediction for the test input.
2.1 Notations and Problem Setting
Suppose the objective of our task is to predict a discrete target variable  $Y \in \mathcal{Y}$ , given a token sequence  $X \in \mathcal{X}$ , where  $\mathcal{X}$  is the space of all possible token sequences.  $\theta \in \Theta$  is a potentially high dimensional latent variable, where  $\Theta$  is the high dimensional space of the variable. Unlike the traditional topic model,  $\theta$  is not assumed to be discrete, but continuously distributed over  $\Theta$ . To define the data generation process, we posit the existence of an underlying causal relation between  $X, Y$ , and  $\theta$ . We examine two potential directions of this causal relation, namely  $X \to Y \gets \theta$  and  $Y \to X \gets \theta$ , which can be represented mathematically as the following structural equations:
$$
Y = f (X, \boldsymbol {\theta}, \epsilon) \quad X = g (Y, \boldsymbol {\theta}, \epsilon)
$$
Here  $\epsilon \in \mathcal{E}$  is an independent noise variable,  $f: \mathcal{X} \times \Theta \times \mathcal{E} \to \mathcal{Y}$  and  $g: \mathcal{Y} \times \Theta \times \mathcal{E} \to \mathcal{X}$  are two deterministic functions. Furthermore, we denote the joint data distribution by  $X, Y, \theta \sim P$ , and assume that  $Y$  is sampled from a uniform distribution over  $\mathcal{Y}$ . The distinction between these two directions is crucial, as it allows us to utilize the direction in which the child variable ( $Y$  or  $X$ ) is independent of the other variables, given its parents.
We hypothesize that the causal direction depends on the nature of the task. For instance, in the task of predicting the sentiment  $(Y)$  of a movie review  $(X)$ , it is reasonable to assume that the opinion about the movie is formed before writing the review, thus making  $Y$  the cause of  $X$ , along with the task concept of "writing a passage to express one's opinion about the movie"  $(\theta)$ . Conversely, for the task of classifying whether a product review  $(X)$  is helpful to other customers  $(Y)$ , it is the quality of the review  $(X)$  that cause other customers to upvote it  $(Y)$ , along with the task concept of "rating the helpfulness of this review"  $(\theta)$ . In the rest of the paper, we will focus on the  $X \to Y \leftarrow \theta$  direction and leave a detailed discussion of the other direction in the Appendix.
Suppose we are interested in a task (e.g. semantic analysis) denoted by  $d \in \mathcal{T}$ , where  $\mathcal{T}$  is the space of all possible tasks. We assume there is an injective function between  $\mathcal{T}$  and  $\Theta$ . i.e. for each task  $d$ , there is a concept variable  $\theta^d$ , such that each data  $(X^d, Y^d)$  sampled from task  $d$  is generated by:
$$
Y ^ {d} = f (X ^ {d}, \theta^ {d}, \epsilon)
$$
To perform in-context learning with an LLM (generically denoted by model label  $M$ ), we condition on a fixed set of  $k$  demonstration examples  $(X_1^d, Y_1^d)$ ,  $(X_2^d, Y_2^d)$ , ...,  $(X_k^d, Y_k^d)$  sampled from task  $d$ .
Following previous works [24, 26], as we are not using any instruction fine-tuned models, we do not include a task description in the prompt, with the aim of focusing on the examination of the demonstrations. To naturally project  $\mathcal{V}$  into the token space  $\mathcal{X}$ , we define injective mappings  $\tau^d:\mathcal{V}\to \mathcal{X}$ , which are typically defined by human understanding of the task  $d$ . e.g. for sentiment analysis,  $\tau^d$  map positive class to the token "positive" and negative class to the token "negative". Additionally, a delimiter token  $\boldsymbol{w}^{d}$  is defined, typically an empty space or a new line token, to separate the demonstrations when concatenated. We denote the LLM output probability of  $X,Y$ , and  $\theta$ , with the aforementioned preprocessing applied, by  $P_M^d$ :
$$
P _ {M} (\tau^ {d} (Y) | X _ {1} ^ {d}, \tau^ {d} (Y _ {1} ^ {d}), \pmb {w} ^ {d}, \dots , X _ {k} ^ {d}, \tau^ {d} (Y _ {k} ^ {d}), \pmb {w} ^ {d}, X) = P _ {M} ^ {d} (Y | X _ {1} ^ {d}, Y _ {1} ^ {d}, \dots , X _ {k} ^ {d}, Y _ {k} ^ {d}, X)
$$
2.2 Problem Analysis and Theoretical Results
Suppose a set of observed data sampled from task  $d$ , denoted as  $\mathcal{D}^d$ , is available, allowing for the selection of the  $k$  most suitable demonstrations from it. For any incoming test example  $X$ , we have:
$$
P _ {M} ^ {d} \left(Y \mid X _ {1} ^ {d}, Y _ {1} ^ {d}, \dots , X _ {k} ^ {d}, Y _ {k} ^ {d}, X\right) = \int_ {\Theta} P _ {M} ^ {d} (Y \mid \boldsymbol {\theta}, X) P _ {M} ^ {d} (\boldsymbol {\theta} \mid X _ {1} ^ {d}, Y _ {1} ^ {d}, \dots , X _ {k} ^ {d}, Y _ {k} ^ {d}, X) d \boldsymbol {\theta} \tag {1}
$$
Figure 1: An overview of our proposed two-phased algorithm. Demonstration selection and latent concept learning share the same LLM as demonstration selection needs to reuse the learned concept tokens, while at the in-context learning time, any other generative LLMs can be used. Here we only illustrate the  $X \to Y \gets \theta$  direction. The  $Y \to X \gets \theta$  direction can be illustrated similarly by exchanging  $X$  and  $Y$  in the above figure.
Here, we assume the sampling of the test example is independent of the sampling of the demonstrations, so  $Y$  is independent of the demonstrations given  $\theta$  and  $X$ . We also assume that the pre-trained data distribution  $P_M^d$  is a suitable approximation of the assumed data distribution  $P$ :
Assumption 2.1. Assume that  $P_M(X) = P(X)$ , and  $P_M^d (Y|\pmb {\theta},X)\propto P(Y|\pmb {\theta},X)$  for  $X\to Y\gets \pmb{\theta}$ .
Note that the assumption that a large language model captures the true distribution of language is fairly common in the literature studying LLMs [50, 34, 47]. With this assumption, we establish:
Proposition 2.2. If task  $d$  follows the  $X \to Y \leftarrow \theta$  direction, then  $\arg \max_{y \in \mathcal{Y}} P_M^d(Y = y|\theta^d, X)$  is the Bayes optimal classifier.
In this case, only when  $P_M^d(\pmb{\theta}|X_1^d,Y_1^d,\dots,X_k^d,Y_k^d,X)$  completely concentrate on  $\theta^d$ , can the in-context learning classifier become the Bayes optimal classifier [11]:
Theorem 2.3. If task  $d$  follows the  $X \to Y \leftarrow \theta$  direction, then the in-context learning classifier
$$
\operatorname *{arg  max}_{y\in \mathcal{Y}}P_{M}^{d}(Y = y|X_{1}^{d},Y_{1}^{d},\ldots ,X_{k}^{d},Y_{k}^{d},X)
$$
always has a higher or equal probability of misclassification to the Bayes optimal classifier  $\arg \max_{y\in \mathcal{Y}}P_M^d (Y = y|\theta^d,X)$ . Equality only holds when
$$
\forall x \in \mathcal {X}, P _ {M} ^ {d} \left(\theta^ {d} \mid X _ {1} ^ {d}, Y _ {1} ^ {d},..., X _ {k} ^ {d}, Y _ {k} ^ {d}, X = x\right) = 1.
$$
A similar argument can be made for the  $Y \to X \leftarrow \theta$  direction. 2 Here, Equation (1) would become:
$$
P _ {M} ^ {d} \left(X \mid Y _ {1} ^ {d}, X _ {1} ^ {d}, \dots , Y _ {k} ^ {d}, X _ {k} ^ {d}, Y\right) = \int_ {\Theta} P _ {M} ^ {d} \left(X \mid \boldsymbol {\theta}, Y\right) P _ {M} ^ {d} \left(\boldsymbol {\theta} \mid Y _ {1} ^ {d}, X _ {1} ^ {d}, \dots , Y _ {k} ^ {d}, X _ {k} ^ {d}, Y\right) d \boldsymbol {\theta} \tag {2}
$$
Note that the left-hand side of Equation (1) and Equation (2) are similar to the direct and channel method introduced by Min et al. [24]. However, our analysis differs from theirs in that we do not treat  $(Y\rightarrow X\leftarrow \theta)$  as the universally superior channel direction for modeling in-context learning, rather arguing that depending on the end task, the causal direction  $(X\rightarrow Y\leftarrow \theta)$  is sometimes better. This view is supported by our empirical results in Appendix B.
Here we demonstrate how the proposed theory can be practically applied to select optimal demonstration examples. Since latent variable  $\theta$  encodes both the task and format information, the whole distribution over  $\Theta$  is too complex to model. Unlike traditional topic models, we will only focus on estimating an optimal value  $\theta^d$  corresponding to task  $d$ .
First, we perform latent concept learning, wherein the task latent  $\theta^d$  is learned as a set of new token embeddings using prompt tuning over the full demonstration candidate set. With this optimal task latent, we then perform demonstration selection, where a smaller set of demonstrations is chosen to maximize the likelihood of postfixing the latent concept tokens. We only need to use a small LLM to do the above steps to obtain an optimal set of demonstrations that can be directly transferred to other LLMs. Figure 1 is an overall illustration of our proposed method.
2The detailed argument of the  $Y\rightarrow X\gets \theta$  direction can be found in Appendix A.2.
Algorithm 1 Latent concept learning
Input: Dataset  $\mathcal{D} = \{(x_i, y_i, d_i)\}_i$  associated with a set of tasks  $S$ , LLM  $M$ , number of concept tokens per task  $c$ , learning rate  $\alpha$ , and number of training steps  $N$ .
Output: LLM  $M^{\prime}$  with fine-tuned concept tokens.
Add  $c|S|$  new tokens to the vocabulary. i.e. The concept tokens  $\hat{\theta}^d$  for each task in  $S$ . Randomly initialize their embeddings  $E_{new}$ . Freeze all parameters in  $M$  except  $E_{new}$ ;
Sample a random batch  $B$  in  $\mathcal{D}$  and initialize gradient  $g\gets 0$
for each data point  $(x,y,d)$  in  $B$  do
$$
g = g + \frac {\partial \ell (X , Y ; \hat {\theta} ^ {d})}{\partial E _ {n e w}};
$$
$$
E _ {n e w} = E _ {n e w} - \alpha g;
$$
3.1 Latent Concept Learning
We want to first find the optimal value of the latent concept variable  $\theta^d$  corresponding to a task  $d\in \mathcal{T}$ . As  $\arg \max_{y\in \mathcal{Y}}P_M^d (Y = y|\theta^d,X)$  is the Bayes optimal classifier according to Proposition 2.2,  $\theta^d$  should be able to minimize  $-\mathbb{E}_{X,Y,d}[\log P_M^d (Y|\theta^d,X)]$  for the  $X\to Y\gets \pmb{\theta}$  direction. In practice, we try to align  $\theta^d$  to the token embedding space by adding new tokens to the vocabulary. After this alignment, we hope to be able to use the learned new tokens of  $\theta^d$  as regular tokens.
More specifically, building upon the methodology proposed by Lester et al. [17], for each specific task  $d$ ,  $c$  new concept tokens (denoted as  $\hat{\theta}^d$ ) are added to the original vocabulary of LLM  $M$  to represent the corresponding task concept  $\theta^d$ . Subsequently, the embedding of these new tokens  $E_{new}(\hat{\theta}^d)$  is fine-tuned while freezing the remaining parameters of LLM  $M$ . The variable  $c$  is treated as a hyperparameter. In practice, in order to condition on  $\theta^d$ , the corresponding  $c$  concept tokens are appended to the input  $X$  (or  $Y$ ) as shown in the example provided below, where  $c = 2$ :
<sentiment_token_1><sentiment_token_2> Can't wait to see the second movie!
By giving the above input tokens, we ask the LLM to predict the correct label Positive for us. Note that <sentiment_token_1> here is just a label assigned to the newly added concept token. It can be anything as long as it does not overlap with the original vocabulary of LLM.
The fine-tuning objective would then be minimizing  $\mathcal{L}(\hat{\theta}^d) = \mathbb{E}_{X,Y}[\ell (X,Y;\hat{\theta}^d)]$ , where
$$
\ell (X, Y; \hat {\theta} ^ {d}) = \left\{ \begin{array}{l l} - \log P _ {M} ^ {d} (Y | \hat {\theta} ^ {d}, X) & \text {i f} X \to Y \leftarrow \boldsymbol {\theta} \\ - \log P _ {M} ^ {d} (X | \hat {\theta} ^ {d}, Y) & \text {i f} Y \to X \leftarrow \boldsymbol {\theta}. \end{array} \right.
$$
Theoretically, if we can minimize the above loss function, a Bayes optimal classifier can be obtained, and the concept tokens would be a reasonable delegate of the real latent concept variable:
Proposition 3.1. When  $\mathcal{L}(\hat{\theta}^d)$  is minimized,  $P_M^d (Y|\hat{\theta}^d,X) = P(Y|\theta^d,X)$  for  $X\to Y\gets \pmb{\theta}$ . If the LLM  $M$  is invertible, then  $\hat{\theta}^d = \theta^d$ .<sup>3</sup>
We denote the LLM  $M$  with fine-tuned concept tokens by  $M'$ . Since we add the concept tokens into the regular token vocabulary, the raw LLM output probability  $P_{M'}(\hat{\theta}^d | \boldsymbol{w}_{1:t})$  ( $\boldsymbol{w}_{1:t}$  denote a given prompt) would be in the token sequence space  $\mathcal{X}$  instead of the concept space  $\Theta$ . Since learning all possible  $\theta^d \in \Theta$  is infeasible, we propose to approximate the concept space  $\Theta$  by sampling a diverse subset of tasks  $S \subseteq \mathcal{T}$ . Then the estimated conditional probability of  $\theta^d$  would be:
$$
\hat {P} _ {M ^ {\prime}} ^ {d} (\hat {\theta} ^ {d} | \boldsymbol {w} _ {1: t}) = \frac {P _ {M ^ {\prime}} ^ {d} (\hat {\theta} ^ {d} | \boldsymbol {w} _ {1 : t})}{\sum_ {t \in \mathcal {S}} P _ {M ^ {\prime}} ^ {t} (\hat {\theta} ^ {t} | \boldsymbol {w} _ {1 : t})}
$$
To obtain the concept tokens for all tasks in  $\mathcal{S}$ , we fine-tune all tasks together with the loss  $\sum_{d\in \mathcal{S}}\mathcal{L}(\theta^d)$ . We summarize the proposed algorithm in Algorithm 1.
3 More discussion can be found in Appendix A.3.
Algorithm 2 Demonstration selection
Input: dataset  $\mathcal{D}^d$  for a task  $d$ . LLM with fine-tuned concept tokens  $M'$ . The number of demonstrations  $k$ .
Output: A set of selected demonstrations.
for each  $(X^d,Y^d)$  in  $\mathcal{D}^d$  do
Compute  $\hat{P}_M^d (\hat{\theta}^d |X^d,Y^d)$
Select top  $k$  examples with the largest  $\hat{P}_M^d (\hat{\theta}^d |X^d,Y^d)$  , denoted as  $(X_1^d,Y_1^d),\dots,(X_k^d,Y_k^d)$
Note that the embedding matrix of a generative LLM is shared on both the input and output sides. So while we only see the concept tokens on the input side at the training time, they can be viewed as regular word tokens that can be generated on the output side.
3.2 Demonstration Selection
According to Theorem 2.3, for a task  $d$ , to make the in-context learning classifier closer to the Bayes optimal classifier, we need to select demonstrations  $(X_1^d,Y_1^d),\ldots,(X_k^d,Y_k^d)$  that maximize  $P_M^d(\theta^d|X_1^d,Y_1^d,\ldots,X_k^d,Y_k^d,X)$  for all  $X\in \mathcal{X}$ . Then our goal then becomes selecting demonstrations that can best infer the task concept for all test inputs on average:
$$
\operatorname * {a r g   m a x} _ {X _ {1} ^ {d}, Y _ {1} ^ {d}, \dots , X _ {k} ^ {d}, Y _ {k} ^ {d}} \mathbb {E} _ {X} [ P _ {M} ^ {d} (\theta^ {d} | X _ {1} ^ {d}, Y _ {1} ^ {d}, \dots , X _ {k} ^ {d}, Y _ {k} ^ {d}, X) ]
$$
As test examples are sampled independent of the demonstrations, and  $P_{M}(X) = P(X)$  according to Assumption 2.1, we have
$$
\mathbb {E} _ {X} \left[ P _ {M} ^ {d} \left(\theta^ {d} \mid X _ {1} ^ {d}, Y _ {1} ^ {d}, \dots , X _ {k} ^ {d}, Y _ {k} ^ {d}, X\right) \right] = P _ {M} ^ {d} \left(\theta^ {d} \mid X _ {1} ^ {d}, Y _ {1} ^ {d}, \dots , X _ {k} ^ {d}, Y _ {k} ^ {d}\right)
$$
If we assume each demonstration is also sampled independently, we have:
$$
P _ {M} ^ {d} (\theta^ {d} | X _ {1} ^ {d}, Y _ {1} ^ {d}, \dots , X _ {k} ^ {d}, Y _ {k} ^ {d}) = \frac {\prod_ {i = 1} ^ {k} P _ {M} ^ {d} (\theta^ {d} | X _ {i} ^ {d} , Y _ {i} ^ {d})}{P _ {M} ^ {d} (\theta^ {d}) ^ {k - 1}}
$$