Venue: NAACL 2025

# Token-based Decision Criteria Are Suboptimal in In-context Learning

Hakaze Cho\({}^{1,\diamondsuit}\) &Yoshihiro Sakai\({}^{1}\) &Mariko Kato\({}^{1}\) \\ Kenshiro Tanaka\({}^{1}\) &Akira Ishii\({}^{1}\) &Naoya Inoue\({}^{1,2}\) \\ \({}^{1}\)Japan Advanced Institute of Science and Technology  \({}^{2}\)RIKEN *Primary Contributor, Correspondence to: yfzhao@jaist.ac.jp &

###### Abstract

In-Context Learning (ICL) typically utilizes classification criteria from output probabilities of manually selected label tokens. However, we argue that such token-based classification criteria lead to suboptimal decision boundaries, despite delicate calibrations through translation and constrained rotation applied. To address this problem, we propose Hidden Calibration, which renounces token probabilities and uses the nearest centroid classifier on the LM's last hidden states. In detail, we assign the label of the nearest centroid previously estimated from a calibration set to the test sample as the predicted label. Our experiments on 6 models and 10 classification datasets indicate that Hidden Calibration consistently outperforms current token-based baselines by about 20%\(\sim\)50%, achieving a strong state-of-the-art in ICL. Our further analysis demonstrates that Hidden Calibration finds better classification criteria with less inter-class overlap, and LMs provide linearly separable intra-class clusters with the help of demonstrations, which supports Hidden Calibration and gives new insights into the principle of ICL. Our official code implementation can be found at [https://github.com/hc495/hidden_Calibration](https://github.com/hc495/hidden_Calibration).

## 1 Introduction

**In-context Learning (ICL) Dong et al. (2022) is a few-shot learning paradigm without model parameter updates on Language Models. In detail, as shown in Fig. 1-(A, B), given a prompt consisting of demonstrations and a query, LMs conduct causal language modeling operation from the prompt to assign probabilities to the label token candidates designed by hand, and ICL chooses the one with the highest probability as the prediction.**

One well-known issue of ICL is that the predicted probabilities are biased (_under-calibrated_), leading to a decrease in prediction performance Fei et al. (2023); Han et al. (2022); Zhao et al. (2021); Zhou et al. (2023). To address this issue, previous work _calibrates_ the predicted label token probabilities by performing affine transformations with estimated parameters to adjust these probabilities for more precise predictions.

These previous works and also the vanilla ICL are based on a potential assumption: the affine manifolds spanned by the decoding vectors in the LM head (_un-embedding vectors_) of _manually selected_ label tokens are good subspaces of the hidden space to distinguish hidden vectors (i.e., the last hidden states to be fed to the LM head) onto various label appropriately, so that the label token probabilities decoded from these subspaces are accurate classification logits. However, although using the label un-embedding with task-related semantics (e.g. "positive" and "negative") seems intuitive, it should be noted that we have no reason to believe that these label un-embeddings have any explicit

Figure 1: In an ICL diagram, **A. The prompt of ICL consists of a concatenation of demonstrations and a query. LMs encode the prompt into the last hidden state \(h\), then B. (previous works) use the un-embedding vectors of the label tokens (\(E_{+}^{U}\), \(E_{-}^{U}\)) to decode the \(h\) to prediction \(\hat{y}\), then calibrations are used to adjust the predicted logits. C. Our work uses the calibration dataset to calculate centroids (\(\bar{h}_{+}\), \(\bar{h}_{-}\)) to decode the \(h\).**

guarantee for decoding the last hidden states into accurate classification logits (token-based decision criterion), even if various delicate calibrations are used to move these boundaries _inside the subspaces_ (see SS3.1). Also, some works have pointed out that randomly changing label spaces doesn't critically influence ICL performance Min et al. (2022c); Wei et al. (2023), which means the selected label subspaces are trivial and arbitrary, making a suspicion of: using manually selected label un-embeddings to decode the last hidden states, i.e., **utilizing manually selected label probabilities as classification criteria may not be good ICL practices**.

Previous work has shown that using the output probabilities of the _full vocabulary_ increases ICL performance Xu et al. (2022); Abbas et al. (2024). This is a good start to avoid the manually selected classification criteria, but there is still doubt that output probability distributions are not informative enough for classification (see SS3.2). Therefore, we utilize the last hidden states instead, which are informative precursors of the token probabilities.

Concretely, we propose Hidden Calibration, training centroid classifiers on the last hidden states of ICL prompts. As shown in Fig. 3, **during the training**, we build standard ICL prompts similarly to Fig. 1-A from a supervised calibration set and input them into the LM to get the last hidden states of the last tokens of ICL prompts. Then, we calculate the centroids of the last hidden states w.r.t. the queries' label to get a centroid for each label, as an anchor for inference. **During the inference**, we input the test prompt, find the nearest centroid to the last hidden states of the test prompt, and assign the corresponding label of the centroid as the prediction.

Empirically, Hidden Calibration improves the ICL performance by approximately more than 20% on 10 text classification datasets and 6 modern LMs (SS4.1), with an equal computational cost with previous calibration methods. To the best of the author's knowledge, Hidden Calibration consistently outperforms the calibration baselines, achieving a strong state-of-the-art in ICL. Additional experiments indicate that Hidden Calibration effectively alleviates the stress of prompt engineering, performing robust accuracy under different prompt designs.

Moreover, our subsequent analysis indicates that Hidden Calibration does find better logits-mapping subspaces that effectively separate data points. In detail, we find that the distribution of classification logits calculated from Hidden Calibration have less inter-class overlapping than from label probabilities, while such overlapping is proportional to the lower bound of the classification error. This suggests Hidden Calibration finds subspaces with essentially better classification performance.

Furthermore, we investigate the principle of Hidden Calibration, that is, the reason why a simple centroid-based linear decision boundary can divide the ICL hidden state properly. We find that LMs provide linearly separable clusters in the hidden states w.r.t. query labels, while more demonstrations can promote such a process.

Our contributions can be summarized as:
* We analyze the previous calibration practices on ICL, and find their consistent limitations: using predicted probabilities of manually selected label tokens for classification criteria, which is often under-guaranteed.
* We propose Hidden Calibration to address the problem before, eliminating the unreliable decoding on the hand-selected label, and using a centroid classifier on the LM's last hidden states. Our experiments suggest that Hidden Calibration reaches a strong state-of-the-art.
* Our further analysis indicates that Hidden Calibration does find better classification criteria with less inter-class overlap, and LMs provide linearly separable intra-class clusters with the help of demonstrations, which supports Hidden Calibration to classify samples accurately.

## 2 Background

This section reviews previous work on ICL and denotes their mathematical descriptions as an introduction to the main motivation of this work.

### In-context Learning

PromptingGiven a few-shot natural language classification dataset (_demonstration set_) \({\cal D}=\left\{\big{(}x^{(i)},y^{(i)}\big{)}\in{\cal X}\times{\cal Y}\right\} _{i=1}^{n}\), where \(x^{(i)}\) and \(y^{(i)}\) are the input sequence and label token of \(i\)-th data point, and \({\cal X},{\cal Y}\) is the input and label space, respectively, we sample a set of \(k\) samples \({\cal D}^{de}=\left\{\big{(}x^{(c_{i})},y^{(c_{i})}\big{)}\right\}_{i=1}^{k}\) from \({\cal D}\) with an index set \(\{c_{i}\}_{i=1}^{k}\) for a given query \(x^{q}\) as the demonstrations. Then, we use a template \(T\) to concatenate them in a natural language form into a prompt token sequence: \(s=T\left({\cal D}^{de},x^{q}\right)\), as shown in Fig. 1-A.

EncodingA decoder-structured LM receives the prompt token sequence \(s\) and encodes it into the 

_last_ (from the last Transformer layer) hidden state matrix as with a length of token and embedding dimension of . We denote the hidden state of the last token as .

**Constrained Decoding.** In a typical ICL setup, one chooses the un-embedding vectors of the label candidates in the output head1 to decode is the prediction. That is, for each label , the similarity (usually the dot-product similarity) between and the un-embedding vector is calculated as the output logits , as shown in Fig. 1-B for a binary classification example. Then, the label with the highest logits is chosen as the prediction , that is: .

Footnote 1: We omit the bias term in the output head (if any) for the sake of simplicity, which can be overridden by a fixed-to-one dimension, or covered by the calibration described below.

### Token-probability Calibration for ICL

However, Zhao et al. (2021) find that simply using the original logits for classification can not lead to a good ICL practice, since these logits have considerable prior bias and often tend towards specific labels even if the query is blank or meaningless (Zhao et al., 2021; Fei et al., 2023). Some calibrations have been proposed to mitigate such bias in a linear form: first, the logits are transformed into probabilities as , then affine-transformed as calibrated classification criteria , where is the calibration terms estimated from training examples from a calibration set, and is the Hadamard multiplication. Various estimations for and are used: some practices use examples with pseudo queries terms (Fei et al., 2023; Zhao et al., 2021), while other practices use Gaussian estimation on real prompts (Han et al., 2022) or the mean value of during the inference (Zhou et al., 2023).

However, as to be discussed in SS3.1 current calibrations are affine transformations on label token probability, without modifying the , causing only limited improvement to ICL performance.

## 3 Methodology

Based on the above background, in this section, we demonstrate the limitations of the above calibrations, and then propose Hidden Calibration to address such limitations fundamentally.

### Token Probabilities Are Not Good Classification Criteria

To better understand the limitations of the label token probability-based ICL, we show a prototypical visualization of the hidden states of ICL prompts (aforementioned ). Specifically, we input 2,048 ICL prompts (with built from of SemEval 2014-Task 4 Restaurant (Pontiki et al., 2014) into OPT-2.7B (Zhang et al., 2022) and plot the on a 2D-Principal Component plane in Fig. 2 (detailed in Appendix A.4).

As a simple 2-way case, focusing on the data points labeled "positive" and "negative", we plot the difference between the un-embedding vectors of these two label tokens2. Then, the coordinates of the projected hidden states in this direction are the difference of predicted logits between these two labels, serving as a _token-based classification criteria_, i.e., when the coordinate is positive, a "positive" label will be assigned, and vise versa. Therefore, in this visualized scenario, the orthogonal line at the zero point is the original decision boundary, points below this boundary are classified as "positive", and

Figure 2: Token probability-based decision boundaries (original & batch calibrated) are suboptimal comparing to centroid-based boundary. Points and contour lines are ICL’s last hidden states and kernel densities mapped by Principal Component Analysis. Oblique coordinate axis is the direction of the un-embedding difference of label tokens , where the kernel densities of mapped data points are plotted. The rotating calibration by (e.g. Contextual Calibration, Domain Calibration) has a limited feasible mapping direction3.

 

-wise versa. The batch calibrated boundary (Zhou et al., 2023) is always parallel to the original one, and the other calibrations (Contextual Calibration Zhao et al., 2021), Domain Calibration Fei et al. (2023)) produce rotated mapping directions \(\big{(}A_{+}E^{U}_{+}-A_{-}E^{U}_{-}\big{)}\), by positive-definite term \(A\) and thus rotated decision boundaries, with limited direction3 between \(E^{U}_{+}\) and \(-E^{U}_{-}\).

Footnote 3: In current practices, the \(A\) are calculated from reciprocals of probabilities, which are positive-definite (Note that the calibration is trivial when \(A\) is not positively definite: the label with negative \(A\) components will never be assigned), and usually do not have significant relative values.

Intuitively, as shown in Fig. 2, the token-based decision boundaries cannot effectively classify these data points, which is due to the inherent direction of the token un-embedding vectors, regardless of limited affine transformation by calibration. A straightforward better linear boundary is the equidistant points between both classes' centroids, we try to find it as follows.

### Hidden Calibration

Motivated by the visualization, we propose Hidden Calibration, using the centroid similarity as the classification logits. In practice, we use a 2-step paradigm as shown in Fig. 3: first, as training, we calculate the centroid of the last hidden states of data points within each class on some (of amount \(m\)) prompt-label training examples. Then, in the inference, we select the closest centroid of the test prompt's hidden state as the prediction.

In detail, **(1) Training:** Given a _calibration set_ with \(m\) supervised prompt-label pair \(\left\{\big{(}s^{(i)},y^{(i)}\big{)}\right\}_{i=1}^{m}\), where the \(s^{(i)}\)s (_training examples_) are standard ICL prompts with \(k\) demonstrations, and \(y^{(i)}\)s are the ground-truth labels of corresponding \(s^{(i)}\)s' query, we input each training example \(s^{(i)}\) to LM, and extract the last hidden state \(h^{(i)}\). Repeating on the whole training example set, we can get a supervised hidden state set \(\mathcal{H}=\left\{\big{(}h^{(i)},y^{(i)}\big{)}\right\}_{i=1}^{m}\). Then, we calculate the centroids of label \(l\) as: \(h_{l}=\mathbb{E}_{\big{(}h^{(i)},y^{(i)}\big{)}\in\mathcal{H},y^{(i)}=l}\left[ h^{(i)}\right]\).

Then, we utilize the calculated centroids in **(2) Inference:** Given a test ICL prompt, we input it into the LM and get the last hidden state \(h\), then calculate the similarity between \(h\) and every centroid \(\bar{h}_{l}\) as the centroid-based logits \(\alpha_{l}\). In practice, the additive inverse of Euclidean distance is used as the similarity (that is, \(\alpha_{l}=-\left\|h-\bar{h}_{l}\right\|_{2}^{\frac{1}{2}}\)), while Appendix C.1 shows that Hidden Calibration acts equally on cosine similarity. We assign the label with the highest logits as the prediction.

**"Why hidden states?"** Notice that another intuitive solution to the problem in SS3.1 is utilizing the logits or probabilities of the _whole vocabulary_, as shown in previous works Xu et al. (2022); Abbas et al. (2024). However, since the input, hidden states, and logits form a Markov chain, no input-relevant information gain is propagated to the full-vocabulary logits. Moreover, the dimensionality of the full-vocabulary logits is typically significantly larger than the hidden states, therefore we choose the hidden states, a dense and informative precursor of token probabilities, as the classification feature.

**"Why centroid classifier?"** Moreover, more complex classifiers, such as a KNN classifier, or a multi-layer perceptron, can be used on the last hidden states instead of a centroid classifier. However, we choose the centroid classifier as the simplest implementation to avoid attribution confusion, that is, if even a rudimentary classifier on hidden states still outperforms, it is powerful enough to demonstrate that our hypothesis is robust. Also, a centroid classifier has a minimal computation cost to fit the scenario of low-resource (SS4.3).

## 4 Experiments & Main Results

In this section, we empirically prove the effectiveness of Hidden Calibration by classification performance on 6 models and 10 datasets. Hidden Calibration outperforms all the baselines and reaches a strong state-of-the-art of ICL with high efficiency in calculation, data, and prompt engineering.

### Experimental Settings

**Models.** We use 6 models: OPT-2.7B Zhang et al. (2022), Llama 2 Touvron et al. (2023) (6.9B, 13B, 34B), Llama 3 AI@Meta (2024) (8B) and GPT2XL Radford et al. (2019) (1.6B). Models larger than 10B are quantized.

**Baselines.** We use 6 baselines from the previous

Figure 3: The diagram of Hidden Calibration. **Step 1:** Calculating the hidden state centroid of each label. **Step 2:** Find the label of the nearest centroid of the text sample to be the prediction.

 

works, with 4 label token-based methods: **Vanilla ICL**(None) [20], **Contextual Calibration**(Con.C) [17], **Batch Calibration**(Bat.C) [35], and **Domain Calibration**(Dom.C) [68]; 2 whole vocabulary probabilities-based methods KNN [29] and **Centroid Calibration**(Cent.C), which we propose as a fair comparison with the same processing on the whole output vocabulary probability vectors instead of the hidden states. Details can be found in Appendix A.2.

**Datasets.** We use 10 commonly used classification datasets with some of the overlength data points excluded. See Appendix A.1 for details.

**Other details.** All the model checkpoints and datasets are loaded from HuggingFace. Macro F1 is used as the classification metric. We use a simple template to generate the prompt, see Appendix A.3. We set \(m=16|{\mathcal{Y}}|\) training examples (16 examples per class), and for fairness, every baseline is given equal training examples for calibration. All the experiments are repeated 5 times.

### Main Results: Hidden Calibration is A New State-of-the-art of ICL

The tested classification performance of Hidden Calibration and baselines is shown in Fig. 4, where Hidden Calibration (Hidd.C) consistently outperforms all the label token-based or vocabulary-based baselines. Comparing to the vanilla ICL (None), Hidden Calibration produces an improvement up to around 100%. In general, compared to the strongest baseline, Hidden Calibration improves the performance by approximately 20%. Detailed numeric and Accuracy results are in Appendix B.1.

Especially, compared to the Cent.C baseline proposed by us for a controlled trial, which conducts the same calculation but uses the whole output token probabilities instead of the hidden states, Hidden Calibration outperforms, which confirms our idea that token probability distribution is a less informative classification feature mentioned in SS3.2.

### Efficiency: Low Complexity towards Time, Space, Data, and Prompting

**Time and Space Complexity.** Intuitively, Hidden Calibration has little additional computational cost compared to the calibration baselines, since they require almost equivalent feedforward calculations, making it competitively efficient as listed in Table 1. Here, we are most concerned about the inference time cost, and Hidden Calibration is the fastest among all the non-label-based methods since the product \(|{\mathcal{Y}}|d\) is usually not very large.

**Training Data Complexity.** Hidden Calibration requires additional annotated data compared to label

Figure 4: The classification performance (Macro F1(%)) of 6 models averaged on 10 datasets. Hidden Calibration (Hidd.C) is a new state-of-the-art of ICL, where demonstrations consistently improve the performance.

 

token-based calibration methods of the same scale in an acceptable range. In detail, in Con.C and Dom.C of \(k\) demonstrations, \(k\) supervised data is needed with a synthetic query to build a training example, while Hidden Calibration needs a real query **for each label**, requiring one more supervised data. However, for classification tasks, preparing an example for each label can be easily done whether in an industry or laboratory scenario, furthermore, these data and trained centroids can be reused (Appendix C.2) to further reduce the requirement of annotated data.

Training Sample EfficiencyRegarding the efficiency of training examples, we repeat the experiments with various \(m\) on OPT-2.7B (see Appendix A.7 for details), from 1 to 128 calibration examples **per class**. Also, for the vanilla ICL, we give equivalent demonstrations for a fair comparison. The results are shown in Fig. 6, which indicate that Hidden Calibration stably benefits from the size of the calibration set, while even one sample per class can still make it outperform. Meanwhile, vanilla ICL and label token-based methods can not benefit from more available data, making Hidden Calibration a better practice no matter how much supervised data can be accessed: data can be used to estimate the centroid to improve the classification in a linear cost, rather than increase the demonstrations in a quadratic cost with less benefit.

Prompting ComplexityWe find Hidden Calibration reduces the pressure on prompt engineering for Hidden Calibration and baselines on Llama 2-6.9B and Rotten_Tomatoes (\(k=4\)) in three aspects: **(1) Prompt template**[53]. We select 7 different prompt templates (shown in Appendix A.3) and test ICL performance on them, shown in Fig. 5 (left). **(2) Label distribution in demonstrations.** We construct prompts with various numbers of "positive" demonstrations presented, and test ICL performance shown in Fig. 5 (middle), **(3) Demonstration order**[10]. We enumerate the full arrangements from a fixed demonstration set of \(k=4\), and test the ICL performance using each demonstration arrangement, shown in Fig. 5 (right). All the results show that: compared to baselines, Hidden Calibration keeps narrow and high-performance distribution against all the three variables, i.e., Hidden Calibration stably works for various contexts, providing higher efficiency on prompt designing.

Figure 5: Sensitivities on **(left)** prompt template, **(middle)** demonstration label distribution, and **(right)** demonstration order on Llama 2-6.9B and Rotten_Tomatoes. Legend is consistent with Fig. 4, omitted.

Figure 6: Classification performance against the number of training examples (\(m\)) of calibrations. As a comparison, we plotted the results of vanilla ICL using equivalent demonstrations, due to the quadratic overhead against the context length, we can test up to \(k=4|{\cal Y}|\).

 



## 5 Analysis

This section attempts to enhance our understanding of Hidden Calibration through comprehensive observations: **(1)** Similar to Fig. 2, we measure the inter-class overlapping area on data points projected into classification criteria, to find whether Hidden Calibration maps data into logits with lower inter-class overlap, i.e., better separability. **(2)** We further investigate why simple linear boundaries can effectively classify ICL hidden states, as observed in Hidden Calibration. We find that LMs provide a primary linear clustering in hidden states responding to query classes, and such clustering is enhanced by more demonstrations.

### Effectiveness: Hidden Calibration Finds Criteria with Lower Overlap

In Fig. 2, we projected the data points into the difference of the label logits (_vanilla classification criteria_) on the oblique coordinate axis, then a significant _overlap_ between the projected data point cloud in two classes can be observed, making it difficult to find suitable classification boundaries vertical to the projection direction. Therefore, such overlap can be used to evaluate classification criteria, so, in this section, we quantify the intuitive observation as the area of overlap serving as a metric for classification criteria.

In detail, we first decompose the multi-way classification dataset into all possible binary classification combinations w.r.t. the ground-truth labels (for example, in a binary combination with labels "positive" and "negative", only "positive" and "negative"-labeled data is obtained). Then, for each combination, we build standard ICL prompts with queries labeled with a specific one of the selected binary combination. Input these prompts into the LM and map the last hidden state of the prompts onto the normal vector of the decision boundary formed by the calibration method, we get the mapped coordinate, as what is shown in the oblique coordinate axis of Fig. 2. To get a continuous distribution of the distance, we run kernel density estimations on the calculated coordinate, then repeat this processing on the other label in the binary combination, and get two density estimations for both labels in a binary combination, as shown in Fig. 7 (curves). Then, we calculate the overlap area of these two kernel density curves. We repeat such processing for each binary combination as shown in Fig. 7 (heat maps), and the final Averaged Overlap is the macro average of overlap area among all possible binary combinations (operation details are in Appendix A.5).

The overlap area of the two distribution curves is double to the _lower bound_ of the classifier's error rate among these two labels (Appendix A.5.3), so Averaged Overlap is an intuitive metric of the classification criteria: the larger the overlap, the more difficult it is for the classifier, even (further) calibrated or ideal, to classify data points correctly,

Figure 8: ICL hidden states clustering dynamics visualized on OPT-2.7B and SemEval 2014-Task 4 Laptopsp [6]. The densities of data points appear in clusters responding to their query labels originally when no demonstrations are given, and gradually converge to the centroid w.r.t. the demonstrations number (\(k\)).

Figure 7: Diagrammatic sketch of the overlap calculation with GPT2-XL on SemEval 2014-Task 4 Restaurant, \(k=4\). **Curves:** The kernel density of probability difference of \(l_{1}=\)“positive” and \(l_{2}=\) “negative”. **Heatmaps:** The overlap of 2-combinations (we plot the combination with the same label with overlap 1, but omit them in averaging).



resulting in a potential decrease in accuracy.

We measure the Averaged Overlap of 4 un-quantized models on 5 datasets (see Appendix A.5.2 for experimental details). The result on GPT2-XL is shown in Fig. 9 (see Appendix B.2 for other models), where the Averaged Overlaps from token-based methods are consistently high, causing that better classification performance cannot be achieved on such methods, which confirms our hypothesis in SS3.1. Meanwhile, the overlaps from Hidden Calibration is much less than from token-based methods, meaning that Hidden Calibration produces better classification criteria with better _possible_ classification performance than the token-based methods, even if delicate calibrations transfer or rotate these classification boundaries.

### Principle: The Inner Linear-separability

In the practice of Hidden Calibration, simple linear boundaries are used to classify ICL examples, raising curious on the linear separability of hidden states. In this section, We find that LMs primarily produce linearly separable hidden state clusters corresponding to the ground-truth label, and the demonstrations facilitate this process.

As an intuitive visualization, we plot curves the same as the Fig. 7 but with various numbers of demonstrations \(k\) to visualize the _clustering dynamics_ of hidden states in Fig. 8, where we find that: **(1)** the data points have a little linear separability when \(k=0\), and **(2)** such linear separability is being enhanced among the increment of \(k\), performing increasing intra-class converging tendency.

We further characterize this process. First, we calculate the Averaged Overlap similar to SS5.1 against \(k\) in Fig. 10. We find that the token-based overlaps remain high and stable w.r.t. \(k\), which indicates that the token-based methods can not benefit much from the demonstrations. However, the overlaps from Hidden Calibration significantly decrease with the increase of \(k\), indicating that Hidden Calibration benefits from the demonstrations as expected, aligning with our observations in SS4.2.

More generally, we visualize the distribution of the last hidden states from similar inputs of Fig. 8 on the **second** principal components of hidden states to get an essential observation in Fig. 11, where as \(k\) increases, the hidden state shows more clear intra-class clustering, enabling separability through a linear boundary.

More directly, on the last hidden states, we measure the intra-class standard error and the inter-class averaged centroid distance against \(k\) (see Appendix A.6 for details), both are a first-order moment for a joint measurement of intra-class clustering and inter-class clustering. The results are shown in Fig. 12, where the two curves are both diminishing, showing an obvious intra- and inter-class clustering trend w.r.t. \(k\). However, the inter-class clustering has weaker and less persistent decreasing trends, presenting only in early demonstrations, or even ascending, which indicates that demonstration enhances intra-class clustering stronger than the inter-class clustering, which is beneficial to linear classification. Moreover, a model with more parameters shows a stronger difference between these clustering.

## 6 Related Works

Given the topic of enhancing in-context learning, we classify the literature into 3 categories.

**Model parameter update-based method:** Although it is pointed out that the ICL objective is implicitly included in pre-training data Han et al. (2023), explicitly fixing the gap between the ICL objective and causal language modeling objective can still be beneficial. Such methods are usually based on supervised fine-tuning Min et al. ( 

2022b; Gu et al., 2023; Wei et al., 2021, 2023; Iyer et al., 2022; Wang et al., 2022), and also self-supervised training Chen et al. (2022) and non-gradient method Zhao et al. (2024). Such methods usually require huge amounts of computation and data overhead to update billions of LM parameters.

In contrast, lightweight solutions focus on **classification criteria-based method (calibration)**. Such methods focus on re-calculating output label probabilities, keeping the main feed-forward calculation processes and model parameters un-modified. The original motivation for these works is to eliminate prior bias and unfaithful confidence in ICL, by calibrating the output label probabilities Holtzman et al. (2021); Shi et al. (2022); Fei et al. (2023); Zhao et al. (2021); Han et al. (2022); Zhou et al. (2023); Jiang et al. (2023). While, as described in the main text, some practices without the usage of label-specific probabilities have also been proposed Xu et al. (2022); Abbas et al. (2024); Min et al. (2022a). Also, a careful **design of input prompts** can help improve the ICL performance. **(1) Demonstration selection.** Gonen et al. (2023) finds that selecting the demonstrations with lower perplexity improves the ICL performance, similarly, Kim et al. (2022) generate the demonstrations from pre-trained LMs, etc. **(2) Demonstration ordering.** It is found that the ordering of demonstrations can significantly influence the performance Lu et al. (2022); Liu et al. (2024); Xu et al. (2024), as also shown in our experiments in Fig. 5. Specifically, Lu et al. (2022) detect the optimal demonstration ordering by some synthetic detecting sequences, while Liu et al. (2024) orders the demonstrations from easy to hard, following a curriculum learning form.

## 7 Discussion

**Conclusion.** In this paper, we analyze the current token-based ICL decision boundaries and point out a limitation of using token probability for ICL prediction. To address such a drawback, We propose Hidden Calibration by decoding the classification logits from centroid classifiers on LM's last hidden states. Our experiments show that Hidden Calibration is a new state-of-the-art of ICL, with high efficiency on time & space, data, and prompt engineering. Then, we confirm that Hidden Calibration indeed creates better classification logits by reducing the inter-class overlap. Moreover, we discover the hidden state convergence promoted by demonstrations, as an explanation of the principle of the performance improvement by a single linear classification boundary in Hidden Calibration. We hope this work can inspire exploration of the ICL by investigating the hidden state instead of token probabilities, and update the community's understanding of ICL calibration.

**Comparison to Previous Works.****(1) Comparison to Probe Methods.** One concern is that our work can be regarded as a degraded linear probe Abbas et al. (2024) of the hidden states. However, we believe our work has more advantages: In terms of application, we use fewer samples and require no gradient-based training, which makes our method more user-friendly, efficient, elegant, and interpretable. Moreover, compared to fitting a universal approximation Hornik et al. (1989), our method and settings fully utilize the hidden state convergence on decoder LMs (described in SS5.2), making it a true ICL practice. **(2) Comparison to Supervised Fine-tuning.** Some practices Gu et al. (2023); Min et al. (2022b) build training objectives to fine-tune models for better ICL performance. These efforts are efficient but costly, while our work avoids such an enormous overhead, making it more usable and elegant. **(3) Comparison to Other Calibrations.** Our method can be seen as a disruptive innovation for methods based on token probability (even the ones based on the whole vocabulary). Experimental comparisons of these methods have been given throughout this paper.

Figure 12: The averaged intra-class standard error of data points and the inter-class averaged centroid distance against \(k\). **Solid curves**: means on 5 datasets; **Dashed and pale curves**: Individual results for each dataset.

 



## 8 Limitations

Due to computability limitations, we are not able to compare the performance of Hidden Calibration with the baseline based on supervised fine-tuning. However, we believe that Hidden Calibration is not within the same methodology as the fine-tuning method, due to the significant difference in computational cost. So such a lack of comparison will not seriously hurt the soundness of this paper.

We argue that human intuition in the label token choice is not reliable. However, we have not eliminated such human intuition completely from the ICL loop: when we build prompts, we still choose the label token. How to automatically select the optimal label token in the prompt will be an important issue, leaving as future research directions for improving the performance of ICL further.

Other label probability calibrations (e.g. Batch Calibration) can be combined with Hidden Calibration for further performance improvements, since the \(0\)-point is not necessarily an exact classification boundary, as shown in Fig. 8. Also, more complex prompts can be used. However, due to space constraints, we have not attempted this incremental approach, remaining it for an empirically possible practice.

Observation in SS5.2 needs more theoretical and experimental analysis. As we can see, some models (GPT2-XL) do not benefit from demonstrations even through the lens of hidden state clustering or Hidden Calibration, which needs to be explained. An explanation of "why such clustering occurs or not", and "how to enhance the intra-class clustering by fine-tuning or prompt engineering" will be considerably beneficial for understanding ICL.

## Acknowledgments

This work is supported by the Nakajima Foundation.

## References

* Abbas et al. [2024] Momin Abbas, Yi Zhou, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Salonidis, and Tianyi Chen. 2024. Enhancing in-context learning via linear probe calibration. _arXiv preprint arXiv:2401.12406_.
* Al@Meta [2024] AL@Meta. 2024. Llama 3 model card.
* Basile et al. [2019] Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 54-63, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Chen et al. [2022] Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zornitska Kozareva. 2022. Improving in-context few-shot learning via self-supervised training. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3558-3573.
* de Gibert et al. [2018] Ona de Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montee Cuadros. 2018. Hate Speech Dataset from a White Supremacy Forum. In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_, pages 11-20, Brussels, Belgium. Association for Computational Linguistics.
* Dong et al. [2022] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhijong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. _arXiv preprint arXiv:2301.00234_.
* Fei et al. [2023] Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselt. 2023. Mitigating label biases for in-context learning. In _The 61st Annual Meeting Of The Association For Computational Linguistics_.
* Gonen et al. [2023] Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 10136-10148.
* Gu et al. [2023] Yukian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Pre-training to learn in context. In _The 61st Annual Meeting Of The Association For Computational Linguistics_.
* Han et al. [2023] Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. 2023. Understanding in-context learning via supportive pretraining data. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12660-12673.
* Han et al. [2022] Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. 2022. Prototypical calibration for few-shot learning of language models. In _The Eleventh

International Conference on Learning Representations_.
* Hendel et al. (2023) Roee Hendel, Mor Geva, and Amir Globerson. 2023. In-context learning creates task vectors. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
* Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn't always right. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7038-7051.
* Hornik et al. (1989) Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators. _Neural networks_, 2(5):359-366.
* Hovy et al. (2001) Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. 2001. Toward semantics-based answer pinpointing. In _Proceedings of the First International Conference on Human Language Technology Research_.
* Ilharco et al. (2022) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. In _The Eleventh International Conference on Learning Representations_.
* Iyer et al. (2022) Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihalyov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. Opt-impl: Scaling language model instruction meta learning through the lens of generalization. _arXiv preprint arXiv:2212.12017_.
* Jiang et al. (2023) Zhongtao Jiang, Yuanze Zhang, Cao Liu, Jun Zhao, and Kang Liu. 2023. Generative calibration for context learning. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 2312-2333.
* Kim et al. (2022) Hyuhng Joon Kim, Hyungsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2022. Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. _arXiv preprint arXiv:2206.08082_.
* Li and Roth (2002) Xin Li and Dan Roth. 2002. Learning question classifiers. In _COLING 2002: The 19th International Conference on Computational Linguistics_.
* Liu et al. (2024) Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, and Wei Lu. 2024. Let's learn step by step: Enhancing in-context learning ability with curriculum learning. _arXiv preprint arXiv:2402.10738_.
* Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8086-8098.
* Malo et al. (2014) P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. _Journal of the Association for Information Science and Technology_, 65.
* Min et al. (2022a) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a. Noisy channel language model prompting for few-shot text classification. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5316-5330.
* Min et al. (2022b) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022b. Metacl: Learning to learn in context. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2791-2809.
* Min et al. (2022c) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022c. Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11048-11064.
* Mohammad et al. (2018) Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In _Proceedings of the 12th international workshop on semantic evaluation_, pages 1-17.
* Pang and Lee (2005) Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In _Proceedings of the ACL_.
* Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_, pages 27-35, Dublin, Ireland. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Rosenthal et al. (2017) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In _Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017)_, pages 502-518.
* Sheng and Uthus (2020) Emily Sheng and David Uthus. 2020. Investigating societal biases in a poetry composition system.
* Shi et al. (2022) Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. 2022. Nearest neighbor zero-shot inference. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3254-3265 

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Voronov et al. (2024) Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. Mind your format: Towards consistent evaluation of in-context learning improvements. _arXiv preprint arXiv:2401.06766_.
* Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR.
* Wang et al. (2022)Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In _2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022_.
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. In _International Conference on Learning Representations_.
* Wei et al. (2023) Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, et al. 2023. Symbol tuning improves in-context learning in language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 968-979.
* Xu et al. (2022) Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. 2022. \(k\) nn prompting: Beyond-context learning with calibration-free nearest neighbor inference. In _The Eleventh International Conference on Learning Representations_.
* Xu et al. (2024) Zhichao Xu, Daniel Cohen, Bei Wang, and Vivek Srikumar. 2024. In-context example ordering guided by label distributions. In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 2623-2640.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihalyov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Kouura, Anjalil Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models.
* Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _NIPS_.
* Zhao et al. (2024) Yufeng Zhao, Yoshihiro Sakai, and Naoya Inoue. 2024. Noisyicl: A little noise in model parameters calibrates in-context learning. _arXiv preprint arXiv:2402.05515_.
* Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _International conference on machine learning_, pages 12697-12706. PMLR.
* Zhou et al. (2023) Han Zhou, Xingchen Wan, Lev Proleev, Diana Miccu, Jilin Chen, Katherine Heller, and Subhrajjit Roy. 2023. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. _arXiv preprint arXiv:2309.17249_.

## Appendix A Experimental Details

### Datasets

In this paper, 10 datasets are used as shown in Table 2. Some datasets do not provide valid splitting, so we randomly split all of them into calibration sets and test sets: For each dataset, we first shuffle it with the random seed 42. Then, we choose the 512 data at the tail as the testing data, and the 512 data at the head (all the datasets have more than 1024 examples) as the calibration data. Each data point in a test set is used once for each experiment trial to build a prompt example and test for performance.

AGNews and GLUE-RTE have over-length examples. So, in the main experiments, we filter out those examples: for Llama 2-6.9B, when \(k=8\), we filter out all the examples with a string length greater than 512 in AGNews and 128 in GLUE-RTE. Also, for Llama 3-8B, when \(k=8\), we filter out all the examples with a string length greater than 128 in GLUE-RTE and omit the experiments on AGNsews. In the experiments in SSSS.2, for all the models, we filter out all the examples with a string length greater than 256 for all the \(k\).

### Baselines

6 baselines (1 vanilla and 5 improved) are used in this paper. Here we introduce the 5 improved baseline.

**Contextual Calibration (Con.C).** Proposed by Zhao et al. (2021), Con.C uses empty queries with normal demonstrations as calibration samples to estimate the calibration term \(A\). In detail, Con.C inputs \(m\) samples with empty queries into the model and gets the averaged normalized label probabilities \(\vec{p}^{f}\) among \(m\) samples. We take the reciprocal of the probabilities as calibration term \(A=\vec{p}^{f-1}\), while the \(B=\mathbf{0}\) 

Batch Calibration (Bat.C)Proposed by Zhou et al. (2023), Bat.C is an inference-time calibration, using the negative averaged normalized label probabilities \(-\bar{p}\) of \(m\) samples in inference time as the calibration term \(B=-\bar{p}\), while the \(A=\mathbf{1}\), where \(\mathbf{1}\) is the all-one vector.

Domain Calibration (Dom.C)Proposed by Fei et al. (2023), Dom.C acts similarly to the Con.C, with the difference that it uses a random sequence sampled on the random tokens from the calibration dataset as queries instead of empty ones. We fix the sampled length to 32.

KNN Prompt (KNN)Proposed by Xu et al. (2022), KNN uses the whole output vocabulary probability distribution as the classification feature, instead of the label tokens. In detail, first, features of calibration examples are calculated as \(k\)-NN anchors. Then, during the inference, a \(k\)-NN classifier is used to classify the feature from the test samples. We use \(m\) examples to calculate the anchors for k-NN, and the nearest neighbor number is set to 3.

Central Calibration (Cent.C)This is the control method proposed by us with a calculation process completely consistent with the Hidden Calibration, except that the usage of the hidden state is replaced by the whole output vocabulary probability distribution consistent with KNN. This method compares with Hidden Calibration to prove that the output probability distribution is not a good classification feature for ICL in a controlled setting.

Notice that: these label-probability-based methods (Con.C, Bat.C, Dom.C) use \(A\) or \(B\)_along_, which may be another major drawback of these calibration methods: According to Fig. 2, if a calibration rotates the mapping direction suitably, and transfer the \(0\)-point properly, a decision boundary close to the Hidden Calibration can be found. This also leads to a new research direction for calibration: the simultaneous usage of translation and rotation methods.

### Prompts

In this paper, we use a minimum prompt template shown in Table 3. The separator between demonstrations is "\(\backslash\)n".

To facilitate the replication of label probability-based methods, we limit all the labels in the label space to one token by synonymous conversion. Note that Hidden Calibration does not need to meet such a one-token requirement.

Especially, in SS4.3, we use 6 more prompt templates to test the stability of each ICL method against the prompt templates. We list these extra templates in Table 4.

### Details of Visualization in SS3.1

Principle Component Analysis (PCA)Given a hidden state set \(\mathcal{H}=\left\{h^{(i)}\right\}_{i=1}^{n}\), we span all the hidden state vector into a matrix \(H\in\mathbb{R}^{n\times d}\). The covariance matrix is \(\operatorname*{cov}\left(H\right)=\frac{1}{n}\left(H-\bar{H}\right)^{T}\left( H-\bar{H}\right)\), where the \(\bar{H}\) is the matrix spanned by the element-wise average vectors \(\bar{h}\) of hidden state set \(\mathcal{H}\). We conduct Eigenvalue Decomposition on \(\operatorname*{cov}\left(H\right)\) and adjust the dimensions to arrange the eigenvalues \(\Lambda\) in a descending order along the row:

\[\operatorname*{cov}\left(H\right)=P\Lambda P^{T}, \tag{1}\]

where the \(P\in\mathbb{R}^{d\times d}\) is an orthogonal matrix. Taking the top-\(\tilde{d}\) lines of \(P\) and span them into \(\tilde{P}\in\mathbb{R}^{d\times\tilde{d}}\), we get the principle component mapping:

\[\mathrm{PCA}_{\mathcal{H}}(h)=\left(h-\bar{h}\right)\tilde{P}=h\tilde{P}- \bar{h}\tilde{P}. \tag{2}\]

Notice that \(\tilde{P}\tilde{P}^{T}=I\), where \(I\) is the identity matrix.

Dot-product after PCASuppose we have dot-product with vector4\(h\) and \(E\) in the original space

\begin{table}
\begin{tabular}{l l} \hline \hline
**Dataset** & **Abbr.** \\ \hline AGNews Zhang et al. (2015) & AGNews SentEval 2014-Task 4 Restaurant Pontiski et al. (2014) & SemE.R \\ SentEval 2014-Task 4 Lastouts Pontiski et al. (2014) & SemEL \\ Penent Sentiment Sheng and Uluus (2020) & Poems \\ GLUE-RTF Wang et al. (2019) & RTE \\ tweet\_eval\_emotion Mohammad et al. (2018) & TEE \\ tweet\_eval\_late Basile et al. (2019) & TEH \\ tweet\_eval\_seminten Rosenthal et al. (2017) & TES \\ financial\_phrasebank Gall agreey Malo et al. (2014) & FP \\ rotten\_tomtoes Pang and Lee (2005) & Rot.T \\ \hline \hline \end{tabular}
\end{table}
Table 2: Datasets and Abbreviations used in this paper.

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Dataset** & **Prompt Template** & **Verbalizer** \\ \hline AGNews Input: \(\propto\)Label: \(<\)\textgreater{} & world; spotset, business, science- \\ SenE.R Input: \(<\)\textgreater{} Aspect: \(\leftrightarrow\)Lab: \(<\)\textgreater{} & positive, neutral, negative \\ SenE.L Input: \(<\)\textgreater{} Aspect: \(\leftrightarrow\)Lab: \(<\)\textgreater{} & positive, neutral, negative \\ PersonS & Import: \(\leftrightarrow\)Lab: \(<\)\textgreater{} & positive, neutral, negative, mix \\ RTI & Imti: \(<\)\textless{}? Text \(\leftrightarrow\) Lab: \(<\)\textgreater{} & include, neutral \\ TIE & Import: \(<\)\textless{}¿ & anger, joy, positive, sad \\ TERI & Import: \(<\)\textless{}¿ & normal, late \\ TTS & Imti: \(<\)\textless{}¿ & positive, neutral, negative \\ PP & Import: \(<\)\textless{}¿ & positive, neutral, negative \\ Root.T & Import: \(<\)\textless{}¿ & positive, negative \\ \hline \hline \end{tabular}
\end{table}
Table 3: Prompt templates used in this paper.

 

\({\mathbb{R}^{d}}\), producing the dot-product similarity classification criterion \(\alpha\):

\[\alpha=h\left(E^{T}-\mathbf{0}^{T}\right). \tag{3}\]

When we conduct a same PCA on both \(h\) and \(E^{T}\) to get dot-product similarity in a dimensionality-reduced space similar to Fig. 2:

\[\tilde{\alpha}=\mathrm{PCA_{\mathcal{H}}}\left(h\right)\underbrace{\left( \mathrm{PCA_{\mathcal{H}}}\left(E\right)^{T}-\mathrm{PCA_{\mathcal{H}}} \left(\mathbf{0}\right)^{T}\right)}_{\text{Mapping direction selected after PCA}} \tag{4}\]

\[=\left(h\tilde{P}-\bar{h}\hat{P}\right)\left(E\tilde{P}\right)^{T} \tag{5}\]

\[=h\tilde{P}\tilde{P}^{T}E^{T}-\bar{h}\tilde{P}\tilde{P}^{T}E^{T} \tag{6}\]

\[=\alpha-\bar{h}E^{T}. \tag{7}\]

Notice that we use the mapping direction \(\left(\mathrm{PCA_{\mathcal{H}}}\left(E\right)^{T}-\mathrm{PCA_{\mathcal{H}}} \left(\mathbf{0}\right)^{T}\right)\) after the PCA, instead of \(\left(\mathrm{PCA_{\mathcal{H}}}\left(E\right)^{T}-\mathbf{0}^{T}\right)\), and this is the reason why the oblique axis in Fig. 2 does not necessarily pass through the coordinate origin. In such a scenario, the dot productions after PCA only differ by a fixed constant bias \(-\bar{h}E^{T}\) from the ones before PCA. This is the reason why the normal line of oblique axis on the \(0\)-point doesn't pass the coordinate origin of the 2D-plane in Fig. 2.

**Decision Boundary after PCA.** Notice that the decision boundary of two classes \(l_{1}\) and \(l_{2}\) in an non-rotated ICL scenario is:

\[\mathcal{B}=\left\{h|hE_{l_{1}}^{T}-hE_{l_{2}}^{T}=C\right\}. \tag{8}\]

Where the \(C\) is the calibration term without rotation. Notice that it is a hyperplane in \({\mathbb{R}^{d}}\) with normal vector \(\left(E_{l_{1}}-E_{l_{2}}\right)^{T}\). Also, the normal plane which pass the \(0\)-point of direction \(\left(E_{l_{1}}-E_{l_{2}}\right)^{T}\) in \({\mathbb{R}^{\widehat{d}}}\) after PCA is:

\[\begin{split}&\tilde{\mathcal{B}}=\left\{\mathrm{PCA_{ \mathcal{H}}}(h)|\mathrm{PCA_{\mathcal{H}}}(h)\right.\\ &\left.\left(\mathrm{PCA_{\mathcal{H}}}(E_{l_{1}}-E_{l_{2}})- \mathrm{PCA_{\mathcal{H}}}(\mathbf{0})\right)^{T}=0\right\}.\end{split} \tag{9}\]

By the aforementioned transformation, we have:

\[\tilde{\mathcal{B}}=\left\{\mathrm{PCA_{\mathcal{H}}}(h)|hE_{l_{1}}^{T}-hE_{ l_{2}}^{T}=\bar{h}\left(E_{l_{1}}^{T}-E_{l_{2}}^{T}\right)\right\}. \tag{10}\]

That is, the dimensionality-reduced decision boundary \(\tilde{\mathcal{B}}\) is perpendicular to the mapped direction \(\left(\mathrm{PCA_{\mathcal{H}}}\left(E_{l_{1}}-E_{l_{2}}\right)-\mathrm{PCA_ {\mathcal{H}}}(\mathbf{0})\right)\), and biased only by a constant \(\left(\bar{h}\left(E_{l_{1}}^{T}-E_{l_{2}}^{T}\right)-C\right)\) on the classification criteria comparing to the original space. Specifically, in the two-dimensional case, it is a straight line that may not necessarily pass through the coordinate origin, as shown in Fig. 2.

### Details of Experiment in SS5.1

#### a.5.1 Calculation Details of Averaged Overlap

First, we divide the \(|\mathcal{Y}|\)-way classification task into \({\mathbb{C}}(|\mathcal{Y}|,2)\) 2-way classification task5, to allow us to use a scalar to characterize the classification criteria for each 2-combination (similar to what we do to the "positive" and "negative" examples in Fig. 2). Then, for each chosen 2-combination, w.l.o.g, given labels denoted as \(l_{1}\) and \(l_{2}\), we build prompt-label sets6 as:

Footnote 5: The \({\mathbb{C}}(m,n)\) is the \(n\)-combination number from \(m\) elements.

Footnote 6: Notice that the \(T\) is the prompting function.

\[\mathcal{S}_{l_{j}}=\left\{T\left(\mathcal{D}^{de,(i)},x^{(c_{i})}\right) \Big{|}y^{(c_{i})}=l_{j}\right\}_{l_{j}\in\left\{l_{1},l_{2}\right\}}^{n_{l_{ j}}}, \tag{11}\]

where \(c_{i}\) is the sampled query index. That is, we sample queries annotated with these two labels and build prompt sets, then collect the prompts with the same query label \(l_{j}\) into \(\mathcal{S}_{l_{j}}\), with a size \(n_{l_{j}}\).

Then, for each prompt \(s^{(i)}=T\left(\mathcal{D}^{de,(i)},x^{(i)}\right)\in\mathcal{S}_{l_{j}}\), we run decoders (vanilla, Con.C, Dom.C and Hidden Calibration) with probability normlization \(f_{l_{1}}(\cdot)\) and \(f_{l_{2}}(\cdot)\) to get the classification probabilities of assigning label \(l_{1}\) and \(l_{2}\) as \(\alpha_{1}^{(i)}=f_{l_{1}}\left(s^{(i)}\right)\) and \(\alpha_{2}^{(i)}=f_{l_{2}}\left(s^{(i)}\right)\). We calculate the difference between \(\alpha_{1}^{(i)}\) and \(\alpha_{2}^{(i)}\) and collect them into a set:

\[\mathcal{A}_{l_{j}}=\left\{\alpha_{1}^{(i)}-\alpha_{2}^{(i)}\Big{|}s^{(i)}\in \mathcal{S}_{l_{j}}\right\}_{i=1}^{n_{l_{j}}}. \tag{12}\]

Now, for the 2-combination of labels \((l_{1},l_{2})\), we get \(\mathcal{A}_{l_{1}}\) and \(\mathcal{A}_{l_{2}}\), whose elements are the probabilities difference between assigning \(l_{1}\) and assigning \(l_{2}\) to example \(s^{(i)}\). The difference between \(\mathcal{A}_{l_{1}}\) and \(\mathcal{A}_{l_{2}}\) is: the elements in \(\mathcal{A}_{l_{1}}\) are from \(s^{(i)}\)s with queries labeled by ground-truth \(l_{1}\), and vice versa. We obtain continuous probability density functions

\begin{table}
\begin{tabular}{r p{142.3pt}} \hline \hline \# & Prompt Template \\ \hline Original & Input: <\infty>, Label: <\varphi> \\
1 & Sentence: <\(\times\infty\), Label: <\varphi> \\
2 & sentence: <\(\times\infty\), Label: <\(\gamma>\) \\
3 & sentence: \(\setminus n\) <\(\times\infty\), Label: <\(\gamma>\) \\
4 & Input: <\(\times\infty\), Sentiment: <\(\gamma>\) \\
5 & Input: <\(\times\infty\), sentiment: <\(\gamma>\) \\
6 & x: <\(\times\infty\), y: <\(\gamma>\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: The 7 prompt templates used in the experiment (on Rotten_Tomatoes) of Fig. 5 (left).

 

of \({\cal A}_{l_{1}}\) and \({\cal A}_{l_{2}}\) as \(p_{l_{1}}(\cdot)\) and \(p_{l_{2}}(\cdot)\) by kernel density estimation, as the curves in Fig. 7.

Then, we calculate the overlap area of these curves:

\[S_{l_{1},l_{2}}=\int_{-1}^{1}\min\left[p_{l_{1}}(x),p_{l_{2}}(x)\right]\mathrm{d }x. \tag{13}\]

For each combination7 in the \(\mathbb{C}(|\mathcal{Y}|,2)\) 2-combinations, we repeat to calculate the \(S_{..,*}\), and average them as the **Averaged Overlap**\(\bar{S}\).

Footnote 7: Notice that on \(S_{..,*}\), the labels are rotational symmetry.

\[\bar{S}=\frac{1}{\mathbb{C}(|\mathcal{Y}|,2)}\sum_{i=1}^{|\mathcal{ Y}|}\sum_{j=i+1}^{|\mathcal{Y}|}S_{l_{i},l_{j}}. \tag{14}\]

#### a.5.2 Experimental Details in SS5.1

We conduct experiments resulting Fig. 9 on 3 models with SemEval 2014-Task 4 Restaurant, SemEval 2014-Task 4 Laptops, AGNews, Poem Sentiment, and financial_phrasebank, given the demonstration number \(k=4\) and calibration example numbers \(m=16\). We use the whole 512 examples on the test split for each dataset and repeat 5 times.

#### a.5.3 Proof: _the Overlap Area is Double to the Error's Lower Bound_

Suppose a label combination \(l_{1}\) and \(l_{2}\), w.l.o.g., we have a ground truth probability density function \(p_{l_{1}}(x)\) and \(p_{l_{2}}(x)\) on a criterion \(x\in\mathbb{X}\), same as the curves in Fig. 7. Given a specific value of criterion \(x\), the upper-bound classification performance is determined by majority vote, which is the most accurate method on such a point, resulting in a density of error classification:

\[e(x)_{l_{1},l_{2}}\geqslant\min\left[p_{l_{1}}(x),p_{l_{2}}(x)\right]. \tag{15}\]

So, the integral error rate:

\[{\cal E}_{l_{1},l_{2}} \geqslant\frac{\int_{x\in\mathbb{X}}\min\left[p_{l_{1}}(x),p_{l_{ 2}}(x)\right]\mathrm{d}x}{\int_{x\in\mathbb{X}}p_{l_{1}}(x)dx+\int_{x\in \mathbb{X}}p_{l_{2}}(x)\mathrm{d}x} \tag{16}\] \[=\frac{1}{2}\int_{x\in\mathbb{X}}\min\left[p_{l_{1}}(x),p_{l_{2}} (x)\right]\mathrm{d}x\] (17) \[=\frac{1}{2}S_{l_{1},l_{2}}. \tag{18}\]

### Details of Experiment in SS5.2

#### a.6.1 Calculation of the Distance and Standard Error

**Averaged Centroid Distance.** Given a \(|\mathcal{Y}|\)-way classification task, for each label \(l\) we build its corresponding prompt set \({\cal S}_{l}=\big{\{}s^{(c_{i})}|y^{(c_{i})}=l\big{\}}_{i=1}^{n_{l}}\), where \(s^{(c_{i})}\) is the prompt with query labeled by \(l\), and \(c_{i}\) is the sampled query index. We encode it into a hidden state set \({\cal H}_{l}=\big{\{}h^{(i)}\big{\}}_{i=1}^{n_{l}}\), and calculate its centroid \(\bar{h}_{l}\), as what we do in Hidden

Figure 13: The classification performance (Accuracy(%)) of 3 models averaged on 10 datasets.

 

Calibration:

\[{\bar{h}_{l}=\frac{1}{n_{l}}\sum_{h^{(i)}\in\mathcal{H}_{l}}h^{(i)}.} \tag{19}\]

For every 2-combination of labels \(l\) and \(l^{\prime}\), we calculate the distance of their centroid, and the average among all the 2-combination is used as the Averaged Centroid Distance:

\[{\rm ACD}=\frac{1}{{\mathbb{C}}(|\mathcal{Y}|\,,2)}\sum_{i=1}^{|\mathcal{Y}|} \sum_{j=i+1}^{|\mathcal{Y}|}\left\lVert{\tilde{h}_{i}-{\tilde{h}_{j}}}\right\rVert _{2}. \tag{20}\]

**Averaged Intra-class Standard Error.** Given the hidden state set \(\mathcal{H}_{l}=\big{\{}h^{(i)}\Big{\}}_{i=1}^{n_{l}}\) w.r.t. the label \(l\), we span all the hidden state vectors into a matrix \(H_{l}\in\mathbb{R}^{n_{l}\times d}\). The covariance matrix is \(\big{(}H_{l}-{\bar{H}_{l}}\big{)}^{T}\begin{pmatrix}H_{l}-{\bar{H}_{l}}\end{pmatrix}\), where the \({\bar{H}_{l}}\) is the matrix spanned by the element-wise average vectors of hidden state set \(\mathcal{H}_{l}\). Notice that the \({\rm ACD}\) is a first-order moment, for a proper comparison, we use the average on the diagonal elements of the element-wise square root of the covariance matrix as the intra-class standard error metric for label \(l\). We average all the standard errors from all the classes as the Averaged Intra-class Standard Error:

\[{\rm AIS}=\frac{1}{|\mathcal{Y}|d}\sum_{i=1}^{|\mathcal{Y}|}{\rm tr}\left[ \sqrt{\big{(}H_{i}-{\bar{H}_{i}}\big{)}^{T}\begin{pmatrix}H_{i}-{\bar{H}_{i}} \end{pmatrix}}\right]. \tag{21}\]

#### a.6.2 Experimental Details in SS5.2

We conduct experiments resulting Fig. 12 on 4 models with SemEval 2014-Task 4 Restaurant, SemEval 2014-Task 4 Laptops, AGNews, Poem Sentiment, and financial_phrasebank, given the calibration example numbers \(m=16\). We use the whole 512 examples on the test split for each dataset and repeat 5 times.

### Experimental Details for Fig. 6

We conduct experiments resulting Fig. 6 on OPT-2.7B with 4 datasets: SemEval 2014-Task 4 Restaurant, SemEval 2014-Task 4 Laptops, AGNews, and Poem Sentiment, given the demonstration numbers \(k=4\) and repeat 5 times.

Figure 14: The augmented results on 2 models of Fig. 9.

\begin{table}
\begin{tabular}{c|c c c c} \hline  & **0\(\rightarrow\)1** & **4\(\rightarrow\)1** & **(I\(\rightarrow\)1)** & **1\(\rightarrow\)4** & **(4\(\rightarrow\)4)** \\ \hline
**SemE.R** & **+9.46** & **+22.50** & **(+26.14)** & **+17.95** & **(+38.75)** \\  & +13.58 & **+14.45** & **+5.16** & **+7.51** & **+22.78** \\
**SemE.L** & **+26.80** & **+17.18** & **(+26.65)** & **+10.79** & **(+37.33)** \\  & +3.20 & **+5.61** & **+22.72** & **+14.96** & **+24.77** \\
**AGNews** & **+42.38** & **+40.20** & **+41.02** & **+43.12** & **(+46.66)** \\  & +2.24 & **+1.24** & **+2.40** & **+2.02** & **+3.77** \\
**PoemS** & **+0.16** & **+2.12** & **(+21.49)** & **+8.79** & **(+12.96)** \\  & +1.87 & **+2.68** & **+2.52** & **+2.81** & **+1.75** \\
**Fina.P** & **+1.35** & **+21.40** & **(+16.70)** & **+10.00** & **(+37.30)** \\  & +1.88 & **+2.90** & **+3.80** & **+13.68** & **+2.30** \\ \hline \end{tabular}
\end{table}
Table 6: Transferability of centroid among various \(k\) on the same dataset. \(k_{1}\to k_{2}\) is to use centroids estimated by \(k_{1}\) demonstrations for inference on test examples with \(k_{2}\) demonstrations. Other annotations are the same as Table. 5 

Detailed Results

### Details of Main Results

Numerical details of Fig. 4 are shown in Table 8, 9, 10, 11, 12 and 13. Accuracy results are shown in Fig. 13.

### Details of Averaged Overlaps Results

The augmented results on the other 3 models (we skip this experiment on the quantitated model) of Fig. 9 are shown in Fig. 14.

## Appendix C Additional Discussion

### The Similarity Measures Used in Hidden Calibration

In SS3.2, we use the Euclidean distance as the similarity measure, while this is not the only option. Intuitively, we can choose other similarity measures as alternatives. Moreover, since we get inspired by observation with dot-production similarity, we should check the performance on such a measure instead of the Euclidean distance. This section uses cosine similarity as an example to illustrate that there is no significant performance difference between these measures. We use cosine similarity to repeat the results in SS4.2 on Llama 2-6.9B and GPT2-XL.

The results are shown in Fig. 15, where the performance based on these two measures is close, without statistical difference. This indicates that the hidden space has good properties of both metric and vector space, and Hidden Calibration acts equally on these measures.

### Transferability of the Centorid

We have proven that it is not advisable to use the _common_ token probability criteria, while, since the centroid criteria are proven to be better than token probability, we are curious: can the centroid calculated in one task be transferred to other tasks with the same label space? Among the datasets sharing the same label space "positive", "neutral", and "negative", we calculate centroids by one dataset and evaluate Hidden Calibration with it on another dataset, on OPT-2.7B, with \(k=4\), \(m=16|{\cal Y}|\). The results are shown in Table 5, where only limited transferability is demonstrated in different domains of the same task (SemE.R and SemE.L), whose behavior is similar to _task vector_[12, 2], while other combination of datasets can not demonstrate considerable transferability. This further exacerbates our doubts about the token-based method: We find that the hidden state distributions have significant differences among various datasets, even if they share a common label space, then utilizing fixed token unembedding vectors to decode these classification criteria is highly unreliable.

Moreover, we repeat this experiment on various \(k\), instead of various datasets, as shown in Table 6. The transferabilities among \(k\) are better than on datasets, but still worse than the un-transferred scenario. Notice that \(4\to 1\) results are much better than \(0\to 1\), which support our results in SS5.2: hidden states with higher \(k\) are further converged.

### A Demonstration towards ICL Principles

Our findings may lead to an explanation of the principle of ICL and traditional calibrations. LMs generate distributed representations into separate clusters in the last hidden state. At this point, by dot-product, any non-collinear _arbitrary or plausible_ mapping directions should be able to capture and classify these clusters to some extent. Note: The absolute distance in such a direction is not faithful (since the centroids of these hidden states and the coordinate origins in these mapping directions are not necessarily aligned), which leads to the generation of so-called bias, and calibrating these biases can improve the performance to a certain extent. However, in such a paradigm, high-dimensional features are discarded, resulting in overlapping originally linearly separable features in high-dimensional space, leading to a loss of classification accuracy, even if the calibration aligns the coordinate origin.

Figure 15: The classification performance (Macro F1(%)) of Hidden Calibration with difference similarity measure.

 



### Applicability on More-way Classification and Biased Dataset

To further verify the applicability of Hidden Calibration on harder tasks, we test the performance of Hidden Calibration and baseline methods on TREC [12] for more-way (\(|{\cal Y}|=6\)) classification and Hate_Speech18 [40] for a biased dataset (label frequency distribution: \([0.87,0.11,0.007,0.015]\)), shown in Table 7. For results on TREC, Hidden Calibration produces a significant improvement compared to all the baselines on all the models. While, Hidden Calibration outperforms on Hate_Speech18 in most cases, and sometimes Hidden Calibration is weaker than baseline methods, but consistently produces competitive results. However, we believe that this slightly weaker result cannot be fully attributed to biased datasets, given that there are biased datasets among the 10 standard datasets where Hidden Calibration perform SotA results (e.g. SemEval 2014-Task 4 Restaurant, refer SSB.1).

## Appendix D Statements

### Author Contributions Statement

The vast majority of contributions of this paper are attributed to Hakaze Cho (also called Yufeng Zhao), who provided ideas, designed and conducted experiments, collected and described data, wrote papers, and revised them.

Y.S., M.K., K.T., and A.I. participated in our discussion and provided some comments, with sequentially decreasing participation and contribution. They also helped write a non-peer-viewed seminar version of this paper in Japanese.

Naoya Inoue is their supervisor, he provides an excellent laboratory environment, funding, necessary guidance, and paper revision.

### License for Artifacts

**Models.** GPT2-XL and OPT-2.7B is under the MIT license, Llama family is under its specific license.

**Datasets.** We list the open-source license for the datasets used in this paper as follows:

* CC-by-4.0: Poem Sentiment, SemEval 2014-Task 4 Restaurant, SemEval 2014-Task 4 Laptops, tweet_eval_emotion, tweet_eval_hate, tweet_eval_hate
* CC-by-SA-3.0: financial_phrasebank, GLUE-RTE
* Unknown: AGNews, rotten_tomatoes

**Consistency of Usage.** Models and data are used with their original usage.

### AI Agent Usage

AI Agents are only used for writing improving and grammar checking in this paper.

\begin{table}
\begin{tabular}{l c c c c c c} \hline  & **GPT2-XL-1.6B** & **OPT-2.7B** & **Llama 2-6.5B** & **Llama 3-SB** & **Llama 2-13B** & **Llama 2-34B** \\ \hline  & \multicolumn{6}{c}{TREC} \\
**None** & 13.02 & 15.85 & 23.12 & 16.79 & 23.22 & 21.13 \\
**Con.C** & 14.10 & 8.12 & 23.72 & 18.34 & 23.56 & 22.35 \\
**Bat.C** & 14.44 & 17.31 & 23.26 & 20.28 & 22.88 & 22.15 \\
**Dom.C** & 14.10 & 8.58 & 23.14 & 19.10 & 23.42 & 23.23 \\
**KNN** & **27.53** & 33.80 & 49.25 & 43.15 & 54.76 & **53.66** \\
**Cent.C** & 23.46 & **33.80** & **49.18** & **46.12** & **56.79** & 53.03 \\
**Hidd.C** & **55.91** & **61.14** & **64.90** & **71.59** & **75.64** & **68.39** \\ \hline  & \multicolumn{6}{c}{Hate\_Speech18} \\
**None** & **23.96** & 23.95 & **23.96** & **23.96** & **23.96** & 23.92 \\
**Conf.C** & 21.57 & 25.47 & 14.68 & **23.74** & 21.49 & 21.25 \\
**Bat.C** & 20.59 & 21.25 & 20.54 & 18.25 & 20.66 & 23.68 \\
**Dom.C** & 21.23 & 23.68 & 20.73 & 23.67 & **24.36** & 23.28 \\
**KNN** & 17.30 & 16.74 & 16.50 & 16.26 & 16.23 & **27.05** \\
**Cent.C** & 16.32 & **25.47** & 15.42 & 5.46 & 9.15 & 22.23 \\
**Hidd.C** & **24.47** & **27.03** & **23.44** & 20.67 & 20.06 & **29.94** \\ \hline \end{tabular}
\end{table}
Table 7: Performance of all 6 models on TREC for a more-way classification, and on Hate_Speech18 for a biased dataset. \(k=4\), top-2 results are in **bold**.



* [14] M. C. Bender, J. R. Friedman, and A. S. Gheorghi, _Theoretical Population of the Universe: An Introduction to the Theory of Evolution_, volume 207 of _Lecture Notes in Physics_, vol. 389 of _Lecture Notes in Physics_, vol. 563 of _Lecture Notes in Physics_, vol. 564 of _Lecture Notes in Physics_, vol. 565 of _Lecture Notes in Physics_, vol. 566 of _Lecture Notes in Physics_, vol. 567 of _Lecture Notes in Physics_, vol. 568 of _Lecture Notes in Physics_, vol. 569 of _Lecture Notes in Physics_, vol. 569 of _Lecture Notes in Physics_, vol. 570 of _Lecture Notes in Physics_, vol. 571 of _Lecture Notes in Physics_, vol. 572 of _Lecture Notes in Physics_, vol. 573 of _Lecture Notes in Physics_, vol. 574 of _Lecture Notes in Physics_, vol. 575 of _Lecture Notes in Physics_, vol. 576 of _Lecture Notes in Physics_, vol. 577 of _Lecture Notes in Physics_, vol. 578 of _Lecture Notes in Physics_, vol. 579 of _Lecture Notes in Physics_, vol.

 

* [14] J. M. Cannon, R. A. Thorne, and D. S. Bostrom, Phys. Rev. Lett. **68**, 2359 (1997)
* [15] L. P. Landau, _Quantum Mechanics: The Classical Theory of Fields_, Oxford University Press, New York, 1960.

[MISSING_PAGE_POST]

 

* [14] M. C. Bender, J. R. Friedman, and A. S. G. Wilson, Phys. Rev. D **58**, 036902(R) (1997)

[MISSING_PAGE_POST]

 

* [14] J. M. Cannon, R. A. Shannon and S. W. H. Freeman, _The Theory of Computation_, vol. 28, no. 3, pp. 57-60, Springer, New York, 1974.

[MISSING_PAGE_POST]

 

* [14] J. M. Cannon, R. A. B. Wilson, and D. W. Anderson, _The Theory of Quantum Mechanics: An Introduction to the Theory of Quantum Mechanics_, Cambridge University Press, Cambridge, MA, USA, 2003.
* [15] S. G. Edwards, _Quantum Mechanics: The Theory of Quantum Mechanics_, Oxford University Press, New York, NY, 1987.
* [16] L. H. Landau, _Methods in Theoretical Physics_, Clarendon-Yvette, New York, 1965.
* [17] P. K. Townsend, _Introduction to quantum mechanics_, Wiley & Sons, Inc., 1975.
* [18] T. E. Shannon, _A Course on Mathematical Statistics_, John Wiley & Sons, Inc., 1975.
* [19] N. F. Smith, _Statistical Mechanics_, Addison Wesley, Reading, MA, 1987.
* [20] V. I. Arnold, _On the theory of quantum mechanics_, Academic Press, Boston, MA, 1987.

[MISSING_PAGE_POST]

 

* [14] M. C. Bender, J. R. Friedman, and A. S. Gheorghi, "Theoretical Population of the Universe: The Role of Life in the Early Universe," _Astrophys. J._**639**(2), 507-583, doi:10.1088/0004-655X/639/639/507

[MISSING_PAGE_POST]

10