Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning
Authors: Yiming Wang
Pei Zhang
Baosong Yang
Derek F. Wong
Zhuosheng Zhang
Rui Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Out-of-Distribution Detection
Mathematical Reasoning
Generative Language Models
Primary area: natural_language_processing
TL;DR: We reveal the core bottleneck of OOD detection under mathematical reasoning scenarios, and propose a novel trajectory-based OOD detection algorithm for mathematical reasoning on generative language models.

Abstract: Real-world data deviating from the independent and identically distributed (\textit{i.i.d.}) assumption of in-distribution training data poses security threats to deep networks, thus advancing out-of-distribution (OOD) detection algorithms. Detection methods in generative language models (GLMs) mainly focus on uncertainty estimation and embedding distance measurement, with the latter proven to be most effective in traditional linguistic tasks like summarization and translation. However, another complex generative scenario mathematical reasoning poses significant challenges to embedding-based methods due to its high-density feature of output spaces, but this feature causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces. Hence, we propose a trajectory-based method TV score, which uses trajectory volatility for OOD detection in mathematical reasoning. Experiments show that our method outperforms all traditional algorithms on GLMs under mathematical reasoning scenarios and can be extended to more applications with high-density features in output spaces, such as multiple-choice questions.

Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning
Yiming Wang"
Pei Zhang
Baosong Yang 8. Derek F. Wong"
Zhuosheng Zhang" Rui Wang D
"Shanghai Jiao Tong University
Tongyi Lab "NLPACT Lab, University of Macau
Corresponding Author
Email:
fyiming wang, vangrui12]asjtt edu. en
yangbaosong-yybaallbbba www.yyybsaaabbbbbeeeeaaaaaa www.llaaaaaina..aa.a. com
Abstract
Real-world data deviating from the independent and identically distributed (ii.d.) assumption of in-distribution training data poses security threats to deep networks, thus advancing out-of-distribution (OOD) detection algorithms. Detection methods in generative language models (GLMs) mainly focus uncertainty estimation and embedding distance mcasurement, with the latter proven to most effective traditional linguistic tasks like summarization and translation. However, another complex generative scenario mathematical reasoning poses significant challenges embedding based methods due its high-density feature of output spaces, but this feature causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces. Hence, propose trajectory- -based method TV Score, which uses trajectory volatility for OOD detection in mathematical reasoning. Experiments show that method outperforms all traditional algorithms GLMs under mathematical reasoning scenarios and can be extended to more applications with high-density features output spaces, such as multiple-choice questions. -ata------tt-----innnnn
Introduction
The rapid development of generative language models (GLMs)
50 has empowered them fit diverse and challenging datasets, showing strong generalization over in-distribution (ID) test data satisfying the independent and identically distributed (i.i.d) assumption. However, unconstrained inputs real-world settings frequently trigger distributional drifts, called out-of- distribution (OOD) data. In such scenarios, model performance often deteriorates unexpectedly, yielding harmful outcomes. Thus, OOD detection
38 is critical safeguarding model security.
A highly scalable detection method must not rely on specific OOD data distributions, simulating scenarios where OOD data is unavailable is more promising. Most existing research mainly focuses on vision and text classification tasks 12 detection on GLMs remain relatively niche, despite the more severe risks associated with OOD perils in GLMss due to the potential for error propagation in autoregressive generated sequences [3 Existing methods on GLMs only focus on traditional text generation scenarios like summarization and translation, and these methods did not step outside the research framework of uncertainty estimation and embedding distance measure
26 48
In contrast, studies addressing OOD
33
Among these, 43 has demonstrated that embedding-
based methods are currently the only optimal solution for text generation scenarios, they determine whether a new sample ID or OOD by calculating the Mahalanobis Distance 22 between the new sample embedding and ID embedding distribution in the static input output space.
Recently, mathematical reasoning has emerged as challenging generative task and crucial benchmark for evaluating model abilities. However, it presents unique phenomena in both input and output spaces that render embedding- based methods inapplicable, as shown in Figure |
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Output
Input
Output
Coumetry
WINT News
Sirmor Theory
TED Tall
seeertt hattenboooeeyy "nleowened 1100 2,103 4200 weelt "W
Outurt dear U.I.A. aue Bot o 4.0 8.4-0
heal Phepodore an morr pooluct maor fectrn
of dittons
Gooeerta and treany
(a) Mathematical Reasoning
(b) Text Generation (Translation)
Figure I: Embedding projection and cases of input and output spaces under mathematical reasoning and text generation scenarios. We select MATH [6] dataset for mathematical reasoning and OPUS 49 for text generation, each with four diverse domains. Different colors represent different domains, with lighter and darker shades indicating input and output. We use SimCSE [9] for sentence embeddings and UMAP 34
for dimensionality reduction. Appendix B shows detailed settings and examples.
Input space of mathematical reasoning exhibits vague clustering features across various domains, contrast the more defined clusters observed in text generation. This indicates that embedding may struggle to capture the complexity of mathematical questions.
Output space of mathematical reasoning exhibits high-density characteristics with significant overlap between different domains. we call this phenomenon Pattern Collapse. Since the output is mathematically symbolic 153 37 it compresses the search space, increasing the likelihood of overlap between questions from disparate domains (As cases of Figure More importantly, the sequence tokenization used in GLMs allows for substantial token sharing among mathematically distinct expressions, as these mathematical tokens are primarily drawn from digits 0-9 and finite special symbols such decimal points and square roots, rather than diverse linguistic elements. These scalar outputs lack distinctive features associated with specific domain distributions.
More discussion of the two phenomena, especially the "patlern collapse", will be presented in Section Given the limitations of traditional methods, we aim explore innovative OOD detection solutions in mathematical reasoning scenarios. We transform our focus from static embedding space to the dynamic embedding trajectory, this motivation stems from a theoretical insight, as shown Figure [ "pattern collapse" causes the convergence of the trajectory endpoints of different samples, leading to significant trajectory differences across samples. In Section 2.1 we model and prove this hypothesis to elucidate the intuition of using trajectories as measure. Subsequently. in Section 2.2, we perform empirical experiments to investigate the underlying causes of trajectory differences between ID and OOD samples. Our findings reveal a phenomenon we term Early Stabilization, wherein GLMs achieve primary reasoning in later stages for ID samples, a pattern not observed in OOD samples. This observation provides direct evidence for the rationale behind using trajectory as measure.
Based these analyses, in Section we propose the Trajectory Volatility detection algorithm (TV Score) for mathematical reasoning. Then in Section $ and S we conduct extensive OOD detection experiments with diverse datasets and GLMs validate the effectiveness of our method. We also tackle the challenging scenario of OOD quality estimation, which raises higher precision demands on the OOD scores. Results indicate that our method surpasses all traditional algorithms for GLMs under the mathematical reasoning scenario. Additionally, we demonstrate the extension of our method to broader range of tasks with high-densic features output spaces, such as multiple-chote questions.
Problem Statement: OOD Detection on GLMs. We start by formalizing GLMs. Let X' and y be the input and output spaces with Px and Py be the marginal distributions for respective space, and Px. is the joint data distribution defined over X' y/ GLMs are trained given input sequence x = 11 ~ Px of length / to autoregressively generate the next token in the corresponding output sequence y = yn U1 ~ Py of length n over the likelihood model po (y(y)r)) = I[ Po (y: y y=s, a)), where each z; and y are taken from vocabulary V and 0 is sampled from the parameter space θ

---

Mathematical Reasoning
Text Generation Trandtationl
0:O1 Embeddinng eyer
Output Space
Output Space
Su5) comples in the daszser
tualt)
Input Space
Input Space
0.65), su
||A (06))A(s) 1,
[-.00+ 1,
200
Figure The "pattern collapse" phenomenon only exists mathematical reasoning scenarios, where two samples initially distant in distance will converge approximately at the endpoint after undergoing embedding shifts, and does not occur in text generation scenarios. This produces greater likelihood of trajectory variation under different samples in mathematical reasoning.
Assume that Px.y denote a distribution sufficiently different from Px.y, the goal of OOD detection in GLMs is to find score function fax y. 0) for each sample and threshold which may rely on the features of X', V/ and O to achieve a high discrimination accuracy goal:
max P P(x.y)~Pxx [f y. 0) < <e| (..)) [f ÿ. 0) c).
(I)
2 Dynamic Embedding Trajectory Distinguishes ID and OOD Samples in Mathematical Reasoning
We begin by defining the embedding trajectory and its volatility features. Given sample we put its input to the language model with L layers, and the output sequence consists of T tokens. For the i-th token, we denote its output embedding at layer as hξ, with each embedding d-dimensional vector. Following the definitions in
we define the average embedding y Σ hi as the sentence embedding at layer 1. Then, the embedding trajectory is formed as a progressive chain of these embeddings: y/o - y/1 + > y1 + - YL-1 + yL- To measure the change magnitudes of the embedding trajectory in the latent space, we define two types of trajectory volatilities:
43 51.
Dimensson--ndependdee volatility Vs(s) R4; For sample first obtain the embedding difference vector 1t yt-1| between each adjacent-layer pair / - and l. V, is the average of all differences across layers, it captures the local trajectory changes across individual dimensions:
L
1 Vi(s) = T. Σ
E(wv wi-1.-- ................................ I w--l))
yı -
yh-
Dimension-joint volatility (s(s)= E For sample s, we first obtain the L2-norm of the embedding difference |ll1 91-1l12 between each adjacent-layee pair | - 1 and 1. V, is the average of all such differences across layers, it captures the global changes in the trajectory:
1 Vs(s) = Σ yy 91-1 1
Σ
(ut
v 1)
(3)
In this section, will clarify our motivation: Why Dynamic Embedding Trajectory As The Measure? This stems from the phenomenon of "pattern collapse" in the output space, where the endpoints of different sample trajectories converge to nigh-density region. This constraint potentially increases the likelihood of trajectory differences across samples. We will model and prove this theoretical intuition in Section 2.1 through the Dimension ciimenssion-nddeenddenn volatility. After gaining insight into trajectory differences, we specifically explore the differences between ID) and OOD sample trajectories. We will empirically investigate this in Section 2.2 through the Dimension-join volatility.
2.1 Theoretical Intuition: Trajectory Differences with Higher Likelihood
Figure has illustrated the "pattern collapse" phenomenon in the output space in mathematical reasoning scenarios. We abstract this phenomenon in Figure ? which compares the trajectory

---

100
120
2400 veku
3.40
5.00
420
20
25
25 35
25
Figure 3: Trajectory volatility curve comparisons between one ID data and ten OOD data from diverse mathematical domains. Each trajectory represents the average of all samples from the corresponding datasets, with color shading being the sample standard deviation. Llama2-7B is used for the backbone.
trend between different samples in the mathematical reasoning and text generation scenarios: In mathematical reasoning. when initial points of two trajectories are separated by any distance in the input space, they typically converge to significantly closer distance in the output space after undergoing an embedding shift. However, in text generation, outputs from different samples may not exhibit this same convergence. This finding inspires the following theoretical intuition: Hard constraints on trajectory endpoints in mathematical reasoning allow for a higher probability of trajectory differences under different samples, expressed by the key hypothesis:
Hypothesis In scenarios charucterized by pattern collapse the output space, the probability of trajectory volatility differences across samples increases.
We now formalize this hypothesis. Assume that the output space is Gaussian distribution (c,, E²), so the output embedding yı. ((c, E²). For two different samples and 8j, their dimension- independent volatilities are V1 (s;) and (s;),, respectively. We want to show that when pattern collapse exists, the probability of (s:) Vr(ss) & 0 will increase. According to the pattern collapse features under different scenarios, we can constraint the yr. as follows:
For mathematical reasoning with pattern collapse, Σ - +0, so we approximate that y = c;
For text generation without pattern collapse, Σ = diag (1, 62, ... ÓD) + O, so yr # C.
With such formalized constraints, we model our Hypothesis as main theorem:
Theorem 2.1 (Main Theorem) We assume that {y) are all independent variables sampling from vector space R.d. For different samples 8i and 83, their embedding sets are (y:)) and { 1 respectively. The likelihood of trajectory volatility differences between 8; and 8, under mathematical reasoning scenarios is higher than that under text generation scenarios, which means:
E Had (1usle2, ~ (((2))
(V: (s3) * +V L. 18,1 = c)
E ( (lusl:/ (()))
(Vr(s:) Vy(s;) + 0l j., [5) ~ N(c,E))
Due to space limits, we present the Complete Proof in Appendix C This demonstrates that when pattern collapse occurs, the probability of trajectory volatility differences across samples increases.
2.2 Empirical Analysis: Early Stabilization of ID samples
The theoretical analysis in Section 2.1
provides an intuition for using embedding trajectory as the measure for distinguishing between different samples. However, the specific trajectory differences between ID and OOD samples remain unclear. Thus, we investigate this empirically in this section.
We select eleven mathematical reasoning datasets: one for ID data and ten for OOD data. The MultiArith dataset serves as the ID data, while the OOD datasets include GSM8K, SVAMP, AddSub, SingleEq. SingleOp, and MATH. The latter encompasses five tasks across various mathematical domains: Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus. This selection includes varying task types and levels of difficulty. We first train a base Llama2-7B (32 layers) using the ID training set, then inference on the ID test set and all OOD test sets. Details regarding the datasets, model, and implementation can be found in Section

---

We measure the dimension-join volatility for all samples from each ID and OOD test set. Specifically, we compute each adjacent-layer change magnitude |/y yt-1 1 and connect all values as a change curve, with higher value means the higher volatility. Figure 3 shows ten curve comparisons, with each sub-figure including the IDD data and one OOD data, We find that the change magnitude slightly until the 20th layer. After 20 layers, for ID data, the change magnitude is again suppressed after few layers of inference, while for OOD data, the magnitude is maintained at a relatively high level.
We term this phenomenon "Early Stabilization": For ID data, GLMs largely complete their reasoning in the mid-to-late stages, and simple adjustments are sufficient after that. However, for OOD data, GLMs can still not complete accurate reasoning at a later stage. They thus can only randomly switch to a specific output pattern, i.e., the scalar mathematical expression pattern. This provides strong evidence that using embedding trajectory for OOD detection may be effective.
TV Score: Trajectory Volatility Score for OOD Detection
In Section 2.2 we have identified a significant difference in embedding trajectory volatilities between ID and OOD samples in mathematical reasoning. We now aim to leverage this phenomenon to develop a lightweight OOD detection solution tailored for mathematical reasoning scenarios.
Inspired by static embedding methods that use the ID sample embedding cluster as the reference to measure the Mahalanobis Distance (MaDis) 22 we similarly aim to use the ID sample trajectory cluster as the reference to measure the difference between them and a new sample trajectory. While measuring the difference between an embedding and an embedding cluster only requires a simple MaDis calculation, quantifying the difference between a trajectory and a trajectory cluster less intuitive. Thus, we seek a transition idea starting from the static space Gaussian assumption.
First, we obey the assumption under static embeddings to Gaussian distribution Gi (M(, Et). Next, for
/ B 43 to fit all ID embeddings at each layer
sample with yı be its embedding at layer
1, we map it to its MaDis [(()) RRd + R with G/ as follows:
f(y)) = (y/1 p4:) (E:)
-1 (yn 14)
())
(4)
Finally, we treat the MaDis differences [ f ) between adjacent layer-pairs / - 1 and as the adjacent-layer volatility of the new sample, and average all adjacent-layer volatilities as the final trajectory volatility score (TV Score S) of this sample:
- Σ 1=1
s -
f(y)) f(x-
(TV Score)
The anticipated trend is that when the two adjacent embeddings after MaDis mapping exhibit a greater difference, the embedding change between the two layers more volatile compared ID data. This enables us to identify new samples with trajectory volatility features that significantly deviate from those of the ID samples, thus increasing the likelihood that they are OOD samples.
Furthermore, as a trajectory, outliers in the trajectory may significantly impact feature extraction 21 28 To mitigate this, we explore higher-order differential smoothing techniques to enhance trajectory smoothness. We first define the k-order embedding Viyi and Gaussian distribution v*g = V Σi)
for all S I
k based on the backward difference:
V* y Σ 1-1) + Clgy4,
V'g =
E
Chmppi
Σ C
(5)
k
where C₁
k! i) Similarly, we mapping vh yt to its MaDis
f*y()) : Rd + E with V*G::
V*((())
4 y
CL
Σ C
V4 Σ (-1)+tic (6)
1=0
Following the definition of TV Score, we define the trajectory volatility score after differential smoothing (TV Score w/ DiSmo V*5) as follows:
v I
-
1 Σ f(y)) v f(n--)))
(TV Score w/ Dismo)
1=1
The Algorithmic Process and Computational Complexity are detailed in Appendix D
5

---

4 Experiments
4.1 Setup
Dataset Selection. For the ID dataset, the MultiArith 44 which consists of Math Word Problems on arithmetic reasoning. For the OOD datasets, we intuitively introduce two types of detection scenarios following 43 (i) Far-shift OOD scenario, select the MATH []] dataset with five domains of algebra, geometry, counting and probability, number theory, and precalculus; (ii) Near-shift OOD scenario, we select five independent datasets: GSM8K 6, SVAMP 39, AddSub
13. SingleEq [8], and SingleOp
[8]. We consider the ID data negative(-) and the OOD data
positive(+). Refer Appendix E. for basic information and OOD features of these datasets.
Data Split and Sampling. Given the limited data size of MultiArith, totaling only 600 samples and lacking a standard division, we allocate 360 samples for training and 240 for testing. However, with such a small test set, randommess in evaluation becomes a concern. To mitigate this, we conduct test sampling and set the sampling size 1000. Specifically, we denote ID dataset Din and OOD dataset as D For each sampling. the collection {Din, P where Dont D Dout and [Pin] Da this guarantees positive and negative sample balance. We report both the mean and standard variance of the results to enhance the reliability of evaluations. Refer to Appendix [..2 for the dataset split.
Implementation. To measure the application value of our method used in outting-edge GLMs, we use Llama2-7B 7B [50 and GPT2-XL (1.5B) as our backbones for ID dataset training. Refer to Appendix E.3 for training details. However, there exists uncertainty about the data used in the pre-training phase, especially for Llama2 because its data is closed-source. Some research [57 59
have confirmed the absence of data leakage in Llama2 for the MATH and GSM8K datasets, we still conduct pre-experiments to examine the rationality of the OOD data selection rigorously. Our criterion is the claim that dataset can be categorized as OOD if it exceeds the capabilities of the base model, as proposed in prior studies 47 27
Results of the pre-experiments are shown in Appendix E.4 and they can confirm that these datasets can be considered OOD data for the two GLMs.
Baseline. We compare our method with five training-free baselines where OOD training data are unavailable. We refer to the latest survey 220 to select them for the scarcity of OOD detection methods on GLMs: (1) Maximum Softmax Probability (Prob.) 12: (2) Monte-Carlo Dropout M] (3)
Seguence Perplexity 3); (4) Input Embedding
43 (5) Output Embedding 43 Refer to Appendix
E.5 for details. Additionally, we set the smoothing order & ranges from to 5 for TV Score w/ DiSmo
and report the highest among them when with smoothing.
Evaluation. We divide the OOD detection evaluation into two scenarios: (1) Offline Detection, which classifies whether samples from a given list belong to OOD. For each collection (Di, Dout report the AUROC [35] and FPR95 metrics. The former represents the area under the ROC curve and the latter represents the value of FPR at 95% TPR. (2) Online Detection, which utilizes the offline detection results to calculate an optimal classification threshold for direct determination of whether new samples belong to OOD. We introduce the metrics in the corresponding result sub-sections.
4.2 Main Results
Offiine Detection. Table I presents the results of the offline detection scenarios.
Performance Analysis: In the far-shift OOD setting, our average performance surpasses 98 (Llama2-7B) and 96 (GPT2-XL) under the AUROC metric, surpassing the optimal baseline by 10+ points. Moreover, our performance stands at impressive 5.21 (Ilama2-7B) and 9.89 (GPT2-XL) under the FPR95 metric, representing a remarkable 80 % reduction compared to the optimal bascline, far surpassing all baseline methods. In the near-shift OOD setting, the robustness of our method is even more impressive, All of the baseline methods show significant performance degradation, especially in Llama2-7B, with the AUROC metric decreasing below 60 and the FPR95 elevating above 80. However, our method maintains excellent performances, with AUROC scores surpassing 90 and FPR95 below 30. This indicates that for more fine- ine-grained OOD detection scenarios, our method demonstrates greater adaptability.
Model Analysis: Comparing performances of Llama2-7B and GPT2-XL, we find two phenomena: (i) Results on GPT2-XI. are more stable, performance differences between GPT2-XI. on far- and drir- settings are not significant, while Llama2-7B shows significant performance

---

Table I: AUROC and FPR95 results of the Offline Detection scenario. Underline and bold denote SOTA among all baselines and all methods, respectively. We report the average results under each setting in the main text, results of each dataset are shown in Table and 12 (Appendix
Llama2-7B
GPT2-XL.
Metric
Far-shift OOD
Nearshift OOD
Far-shift OOD
Near-shift OOD
Method
AUROC
FPR95
AUROC
FPR95
AUROCT
FPR95
AUROC1 t
FPR95
Max Softmax Prob. P
78. 38
81.44+136
60.14
88.91+241
70.54+142
78.29+2
67. 17.12+120
76,27=224
Monte-Card Dropout
68.631mm
17.04
52.331am
91.9211 s0
66.18
84.69
63
78.08
Perplexity
85
53.0611
59.35
86.09.1
80.82.101
64.5317
13
72.39
Iaput Embedding
75,89+102
67.87=3x0
60.33+1.2
84.65+252
86.26-0s4
49.33+2210
83.22+aaa
52,90+336
Output Embodding
13
74,86+13p
75.21
14.50=100
86.46+155
77.95
65,64++342
79:28
64,70=272
TV Score (Ours)
98.762811
5.21
92.642009
28.39
93,47
24, 10
94,86
13,82
w/ DiSmo (Ours)
75
12.82
26.99
88.01
96.54
9.89
94. 19
13.66:00
A (bold underline)
+13.12
47.85
+32.31
-56.26
+10.28
-39.44
+1 1.66
-39.24
Table 2: Accuracy and Robustness results of the Online Detection scenario. We mainly compare our method with embedding- based methods, and bold denotes the best among these methods.
Far-shift OOD Setting
Near-shift OOD Setting
Accuracy!
Robustness
Accuracy
Robustness.
Dataset
Dataset
I-Fmb. / /O-Emh. / TV (ours)
I-Fmb. / O-Fmm.. TV (ours)
Algebra
76.43 / 45542 / 93.88
5.27 6.94 / 0.97
GSMSK
81.49/ 93.39
10.08 / 3.36 2.05
Geometry
74.32 / 54.79 / 94.47
2.44 2,43 1.65
SVAMP
68.66 / 63.33 94,88
5,26 3,54 2.13
Cnt.&Prob
50.31 / 27.55 93.74
9,99 2,34 / 2,36
AddSub
79.16 78.09 77.11
3,21 / 6.98 / 2277
Num.Theory
85.80 54.38 / 92.08
3.31 11.45 2.34
SingleEq
/9.83/ 93.15
11.57 3.14 /3.17
Precalculus
80.33 88.50 19.28
6.13 1.38 0.67
SingleOp
69.38 62.20 95.75
4.00 2.37 2244
Average
73.44/ 54.13 94.69
5,43 / 4.91
Average
71.70 70.30 90.26
6.82 3.88//8// 2.51
degradation (mainly for baselines) on near-shift setting; (ii) the DiSmo technique is more effective on GPT2-XL, which suggests that there are more anomalous learning tendencies in latent spaces of small models, and the smoothing helps minimize these anomalies.
In addition, we conduct significant tests (Details are shown in Table methods almost pass all significance tests, while the embedding based methods have the lowest pass rate among baselines, suggesting that their results are more susceptible to sampling error. We also find that the performance of differential smoothing fluctuates greatly in different settings. Therefore, we conduct the ablation of the smoothing order k. Results and analyses are shown in Appendix F.I
We find that our
Online Detection. In this part, utilize the TV score for online OOD discrimination. For each collection (Pi, ). we obtain a detector and computer the optimal cut-off r; of Youden Index, which is at the point in the AUROC curve where TPR - FPR is maximum. Then for all OOD samples E Dout Dour. we donate as the sampling size and computer the discrimination accuracy:
EBED... D T(TV-Score(s) 2 T
Accuracy
Daut
Dout
In addition, the discrimination accuracy should vary less under different data collections, reflecting the discriminator's 's robustness. Therefore, we denote the Robustness metric as sampling variance.
Table 2 presents the results in Llama2-7B. Compared to the embedding- based methods, our TV score obtains about an average of 20-point accuracy improvement in both far-shift OOD and near-shift OOD settings, and on some datasets, such as Cnt.&Prob our TV score achieves more than 40 points of improvement. These all imply that TV Score can perform online discrimination of OOD samples more accurately. In addition, our TV score also possesses stronger robustness, which means that real scenarios, we can find the optimal threshold more consistently in the face of different accessible ID and OOD data, reducing the potential riskiness due uncontrollable data acquisition.
7

---

5 Generalizability Exploration
5.1 Beyond Detection: OOD Quality Estimation
In this part, we utilize the TV score for generative quality estimation (QE). For text generation, the QE performance is usually measured by calculating the correlation coefficient between automatic scores and human ratings. However, QE in mathematical reasoning scenarios is not well-defined problem. For one mathematical question, its answer is either right or wrong, the intermediate state does not exist. For example, when the correct answer is 12.5, it is difficult to judge which is better between generated answers of 1.25 and 13.6. The human approach may be to judge by comparing the difference value or similarity like Rouge 24 and BertScore 660 between the generated answer and the correct answer, which is unfair to the machine because there is a lot of randomness in the intermediary process of computation, and the solution pattern of machines is case-based [4] it is not suitable to judge the machine- machine-generated results with customized mathematical rules.
Therefore, we use the binary direct matching to compare the model-generated answers with the correct answers. Considering the open-ended output of the GLMs, we give a loose matching condition, i.e., long as the correct answer is included in the generated answer by the model, the generated answer recognized as correct and the matching score is 1, otherwise the matching score is O. We compute the Kendall rank correlation coefficient +5 and Spearman rank correlation coefficient
36 between each OOD score and the matching score.
Table 3: OOD Quality Estimation: Kendall's T and Spearman correlation between various OOD scores and benchmark quality metric binary matching. Each column shows the correlation when ID and OOD samples are merged. Underline denotes the SOTA among all baselines, and bold denotes the SOTA among methods. We report the average results under each setting in the main text, results of each dataset are shown in Table 13 and 14 (Appendix
Model
Llama2-7B
N
GPT2-XL
Metric
Far-biff OOD
Near-shift OOD
Far-shift OOD
Near-shift OOD
Method
Kendall 1
Spearman
Kendall
Spearman
Kendall 1
Spearman
Kendall
Spearman
Max Softmax Prob.
0.0241a001
0.038
0.038__omm
0.026_ems
0,066
0.044
i.057
0.057
Perplexity
0.050eams
0.045
0.074-0mm
0.050=smx
0.036=0m4
0.038+300
0.035
0.058=0a0
Input Embedding
0.078
0.10+
0.036-ooms
0.115
0.059=amm
0.098+3me
0.012easss
0.06
Output Embedding
0.058
0.025
0.038_noms
0.012gam
0.050gomm
0.016ga00
0.036g
0.029
TV Score (Ours)
0.161:am2
0.147
0.159=aa1
0.158
0.138=00m
0.1 0.123
0.131
0.1 146420ss
w/ DiSmo (Ours)
jaats
0.1 13 Loos
0.1 0.134_emm
0.139 euee
0,141 2404
0.123
0.154 jasis
(hold- underline)
+0.083
40,050
+0.085
+0.043
+0.073
+0.043
+0.074
+0.086
Table presents the results. For Llama2-7B, when compared with Kendall correlation, the correlation improvement TV scores over SOTA baselines reaches up to 100% under both far-shift and near-shift OOD settings. Compared with Spearman correlation, TV scores demonstrate a correlation enhancement over SOTA baselines by up to 100% under far-shift OOD setting and 30% under near-shift OOD setting. GPT2-XL also demonstrates excellent performance. These findings indicate that TV scores not only facilitate the binary discrimination of ID and OOD samples but also substantially reflect the quality and precision of generated mathematical reasoning.
5.2 Beyond Mathematical Reasoning
Apart from mathematical reasoning, method also has a wider range of potential applications that can be extended to any task where the output space exhibits the pattern collapse property. An example would be multiple-choice questions, which is a popular evaluation tool in the era of large language models and also display the pattern collapse property due to the limited output space being confined to the "ABCD" four options. To verify the generalizability of our method, we conduct experiments using the multiple-choice dataset MMLU [10,, and our method also outperforms all traditional algorithms in this setting. Results and analyses are shown in Appendix F.4
Although direct matching is the most accurate solution, it suffers from two issues: (i) Generated answers may include much noisy content, increasing the matching difficulty; (ii) Performances on GLMs of mathematical reasoning is poor, which unbalances the positive and negative samples and increases the randomness.
8

---

Rethink Inapplicability of Static Embedding Methods
In Figure 1 we find that the static embedding space for both input and output fails to capture distinct features across different domains. This renders traditional embedding- baasdded-- methods unsuitable for mathematical reasoning tasks. Experimental results in Section | also verify this disadvantage. In this section, we conduct a detailed analysis of these phenomena, and reveal the root cause of the poor performance of static embedding methods in mathematical reasoning scenarios.
6.1 Input Space: Representation Dilemma on Mathematical Expressions
The input space in mathematical reasoning exhibits vague clustering features compared to text generation scenarios shown in Figure I
We speculate that semantic representations, embeddings cannot accurately measure mathematical expressions in the mathematical sense.
We begin by considering a counter-intuitive toy example. We construct benchmark expression "A: 2+3+4=?" and three diverse expressions: "X: 8-0+9=?", "Y: x"2*y"3** "exxy""*x44==" "=x""" and "Z: From the ID for A, whereas and 7. are OOD for A since the difficulty and knowledge used to solve Y and Z are largely different from A compared to X. Table 4 presents the cosine similarity between each of them and A. Notably, X exhibits the lowest similarity to A, primarily due to having fewer token sharing with A. This is inconsistent with human perception of mathematics.
Table 4: A toy example about cosine similarities between different mathematical expressions.
Muthematical Expression
Cosine with Benchmark
Benchmark: 2+3+4-?
mathematical sense,
8-0+9=7
2"2 "=+
0.83
0.78
Table 5: AUROC score matrix produced after alternating the MATH dataset's ' five domains as ID and OOD data measured by (a) Input Embedding Mahalanobis Distance and (b) Output Embedding (w/ Co Mahalanobis Distance. Darker colors represent better performances.
OOD Domain
Input Embedding MaDis
Output Embedding (w/ CoT) MaDis
algehra
geometry
ent.&prob
num.theory
precakeluus
algebra
goometry
ent.&prob
num.theory
precalculus
algebra
58.79
71.37
81.24
ID Domain
geometry
85.68
86.55
72.12
cot.&prob
49.95
38.02
46.19
theoey
63.02
78.76
preculculus
79.84
To demonstrate this phenomenon, we use one of five domains from the MATH dataset as the ID and the remaining four domains as OOD. We rotate this setting across five iterations. Under each ID-OOD pair, we allocate half of the ID samples to compute the embedding distribution, while the other half and all OOD samples constitute the test samples. The Mahalanobis distance is then calculated between embeddings and the ID distributions for each sample input, yielding the OOD score. We employ SimCSE g to generate sentence embeddings. Table 5(a) presents the AUROC score matrix, in nearly half the settings, the AUROC value hovers around 50, i.e., completely random. This suggests that embeddings can not distinguish between domains in mathematical sense.
6.2 Output Space: High-density "Pattern Collapse"
The output space in mathematical reasoning exhibits unique high-density characteristic, which call "pattern collapse". As shown in Figure [ the distribution of output embeddings across various domains in the mathematical reasoning scenarios is significantly concentrated and indistinguishable.
Two reasons cause this phenomenon: (1) Expression level: Output space of mathematical reasoning is scalar 37 resulting in compressed search space with higher probability of overlap in two widely divergent questions. For example, "1 + 3 =" and "f.3 ædr =" are both "4"; (2) Token level: We usually categorize different mathematical reasoning domains from a mathematical sense. However, GL.MS model real numbers or mathematical expressions not a mathematical sense, but based on the average embedding of all discrete token embeddings after sequence tokenization. Through

---

tokenization, two expressions that are very different in the mathematical sense (e.g., Two numbers that are far apart on the real number line) may share many tokens because regular mathematical expressions are only taken from 0-9 number tokens and a limited number of special symbols, such as decimal points, slashes, and root signs. Thus, the collapse occurs the token level during the autoregressive prediction of each token.
We conduct the following statistical analysis to demonstrate the universality of "pattem collapse" across various mathematical reasoning tasks: We categorize the mathematical tasks into various types across different domains and difficulties. In each task, we count the token number N and the token type number Ny the dataset, then compute the token duplication rate D and the vocab coverage C' as follows: Llama-2 tokenizator [S0], whose token type number in vocab Ny 32000. The computation metric is:
Table Statistics about output tokens on mathematical reasoning (seven different task types) and text generation (translation and summarization).
Task Type
NT
C
Mathematical Reasoning
Arithmetic(easy)
16136
14
99.9%
0.04%
Arithmetic(hard)
5663
16
99.7%
0.05%
Algebra
5234
107
98.0%
0.33%
use
Gcometry
2615
75
97.1%
0.23%
Cnt.&Prob.
2524
43
98.3%
0.13%
Num. Theory
2395
71
97.1% %
0.22%
Precalculus
3388
84
97.5%
0.26%
NT N
NT N
Average
5422
58
9%%
0.18%
D
C
(8)
Text Generation
We also test translation and summariza- tion tasks by taking samples with the same token size as the mathematical reasoning dataset for a clear comparison. Table 66 presents the statistics data, we find that: (i) The average token duplication rate was 98.9% on all math tasks, and even a staggering 99.9% on some easy arithmetic tasks; In contrast, the token duplication rate on the text generation task is only about 60%, with about 2000 token types, and still increasing the total number of tokens increases. These data and comparisons demonstrate that pattern collapse occurs on mathematical reasoning and not on text generation. (ii) Token repetition rate exceeded 97% on all seven math tasks of different difficulties and types. From these conclusions, we can demonstrate that the "pattern collapse" occurs on generally all types of mathematical reasoning tasks.
2500
1065
57.4%
3.32%
Translation
5000
1832
63.3%
5.10%
10000
2980
70.2%
9.31%
Average
5833
1959
66.4%
6.12%
2500
1265
49.4%
4.01%
Summarization
5000
1970
60.6%
6.16%
10000
3192
68.0%
9.98%
Average
5833
2142
63.2%
6.69%
6.3 Can Chain-of-Though Address "pattern collapse"?
straightforward approach to addressing "pattern collapse" in the output space is to leverage chain- of-thought (CoT) techniques [54]
61 expand the output space size. Likewise, adopt the solution steps associated with each sample in the MATH dataset as the output and employ SimCSE to derive embedding representations. The experimental setup aligns with Section 4.1 and the results are shown in Table 5tb). We note similar phenomenon as in Table S i.e., the detection accuracy under different ID-OOD pairs varies greatly, and thus the detection randomness is more pronounced. This suggests that despite that CoT expands the output space size, the output answer still essentially related to the difficulty and digit of mathematical reasoning, and the semantic embedding representation cannot reflect these features accurately.
7 Conclusion
We propose the TV Score, a lightweight OOD detection method for mathematical reasoning, which distinguishes between ID and OOD samples by the embedding trajectory volatility in the latent space. We identify bottlenecks in OOD detection for mathematical reasoning and prove them empirically and theoretically. Experiments show that our method substantially outperforms all traditional algorithms, and can be extended to more application scenarios beyond mathematical reasoning.
10

---

Limitations
Our limitation mainly lies in the relatively small sizes of datasets used our experiments. Due to the difficulty of collecting and labeling mathematical reasoning data, dataset sizes in this field are generally small, mostly in the hundreds or thousands, making it difficult to obtain millions of data for training and reasoning in translation summarization tasks. To address this, we adopt test sampling to reduce the randomness under small-scale testing and mitigate the data imbalance.
Ethics Statement
The data and models used in this work are sourced from the official version of the original paper, and we strictly adhere to the provided usage protocol. Regarding the data, no modifications have been made the original dataset. Regarding the models, supervised fine-tuning and OOD data inference are involved. To mitigate the risk of uncontrollable outputs, all generated outputs in the experiments have been reviewed to ensure their safety. Furthermore, as our focus is solely on mathematical reasoning and does not involve sensitive content, we would not cause any potential societal impact.
Acknowledgment
This work supported by the Alibaba Innovative Research Program. Yiming and Rui are partially supported by the General Program of the National Natural Science Foundation of China (62176153), the Shanghai Municipal Science and Technology Major Project (2021SHZDZX0100), and SMP- Fund, the Science and Technology Development Fund.
Zhipu.AI Large Model Cross Cross-Disciplinaat Zhuosheng partially supported by the National Natural Science Foundation of China (62406188) This work is also partially supported by the Science and Technology Development Fund, Macau SAR (Grant Nos. FDCCT/600202//FF, FDCT/0070/2022/MMM)) and the Multi Multi-year year Research Grant from the University of Macau (Grant No. MYRGG-GRG2222-00006-99T----
References
[1] Vahdat Abdelzad, Krzysztof Czamecki, Rick Salay, Taylor Denounden, Sachin Vernekar, and Buu Phan. Detecting distr--disttibution inputs in deep neural networks using early-layer output. arXiv preprint 190iv:1910.10300 2019.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[3] Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10687-10701, 2021.
[4] Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in- domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations, 2019.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.16688,. 2021.
[7] Taylor Denouden, Rick Salay, Krzysztof Czamecki,, Vahdar Abdelzad, Buu Phan, and Sachin Vernekar. Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance. arXiv preprint arXiv:1812.02765 2018.

---

[8] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[9] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 6894-6910, 2021.
[10] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2020.
[11] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
[12] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of distribbtiin examples in neural networks. International Conference on Learning Representations, 2016.
[13] Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman, Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 523-533, 2014.
[14] Yi Hu, Xiaojuan Tang, Haotong Yang, and Muhan Zhang. base- or rule-based: How do transformers do the math? arXiv preprint 177.,9990..,7799, 2024.
[15] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677- 689, 2021.
[16] Wenyu Jiang, Yuxin Ge, Hao Cheng, Mingeai Chen, Shuai Feng, and Chongjun Wang. Read: Aggregating reconstruction error into distr--dist detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14910-14918, 2023.
[17] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
[18] Rik Concel-Kedzioree Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1552-1157, 2016.
[19] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[20] Hao Lang, Yinhe Zheng, Yixuan Li, SUN Jian, Fei Huang, and Yongbin Li. A survey out-of-distributio detection in nlp. Transactions on Machine Learning Research, 2023.
[21] Rikard Laxhammar and Göran Falkman. Online learning and sequential anomaly detection in trajectories. IEEE transactions on pattern analysis and machine intelligence, 3666):1158-1175 2013.
[22] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distriutioon samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
[23] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
[24] Chin-Yev Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
12

---

[25] Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural symbol-y graph semantic parsing via uncertainty. Findings of the Association for Computational Linguisties: ACI. 2022, 2022.
[26] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based oitttiiiiiiiiiiiiitiiiinnnn detection. Advances in neural information processing systems, 33:21464-214755 2020.
[27] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz. Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models' alignment. In Socially Responsible Language Modelling Research, 2023.
[28] Yiding Liu, Kaiqi Zhao, Gao Cong, and Zhifeng Bao. Online anomalous trajectory detection with deep generative sequence modeling. In 2020 IEEE 36th International Conference Data Engineering (ICDE), pages 949-960, IEEE, 2020.
[29] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[30] Christos Louizos and Max Welling, Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pages 2218-2227. PMLR, 2017.
[31] Denis Lukovnikov, Sina Daubener, and Asja Fischer. Detecting compositionally out-of- distribution examples in semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 591-598, 2021.
[32] Andrey Malinin, Neil Band, Yarin Gal, Mark Gales, Alexander Ganshin, German Chesnokov, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[33] Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, 2020.
[34] Leland Melnnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
[35] Charles E Metz. Basic principles of roc analysis. In Seminars in nuclear medicine, volume pages 283-298. Elsevier, 1978.
[36] Leann Myers and Maria J Sirois. Spearman correlation coefficients, differences between. Encyclopedia of statistical sciences, 12, 2004.
[37] Inderjeet Nair and Lu Wang. Midgard: Self-consistency using minimum description length for structured commonsense reasoning, 2024.
[38] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan. and Jasper Snoek. Can you trust your model's 's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
[39] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2080-2094, 2021.
[40] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[41] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
13

---

[42] Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan, A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprini arXiv: 106.0902,, 2021.
[43] Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter Liu. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2023.
[44] Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1743-1752, 2015.
[45] Pranab Kumar Sen. Estimates of the regression coefficient based on kendall's tau. Journal of the American statistical association, 63(324):1379-4
1968.
[46] Rupesh Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. Advances in neural information processing systems, 28, 2015.
[47] Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.0556/ 2024.
[48] Yiyou Sun, Yifei Ming. Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20844 PMLR, 2022.
[49] Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, pages 2214-2218, 2012.
[50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092880 2023.
[51] Yiming Wang, Pei Zhang, Baosong Yang, Derek F Wong, and Rui Wang. Latent space chain-of- embedding enables output-free Ilm self-evaluation. arXiv preprint arXiv:2410. 13640, 2024.
[52] Yiming Wang, Zhuosheng Zhang, and Rui Wang. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of- thought method. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8640-8665, 8665, 2023.
[53] Yiming Wang, Zhuosheng Zhang, Pei Zhang, Baosong Yang, and Rui Wang. Meta Meta-rcasoning: Semantics-symbol deconstruction for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 622-643, 2024.
[54] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24877, 2022.
[55] Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout:
Efficient pseudo- independent weight perturbations on mini-batches. In International Conference on Learning Representations, 2018
[56] Tim Z Xiao, Aidan Gomez, and Yarin Gal. Wat zei jei detecting out-of-distribution translations with variational transformers. 2020.
[57] Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404. /8824, 2024.
[58] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribttion detection: A survey. arXiv preprint arXiv:2110.11334 2021.
[59] Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wa, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.008: 2024.
14

---

[60] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. 2019.
[61] Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang. Xinbei Ma. Zhiwei He, Yiming Wang. Mark Gerstein, Rui Wang, Gongshen Liu, et al. Igniting language intelligence: The hitchhiker's guide from of-thuht reasoning to language agents. arXiv preprint arXiv:231.11799,, 2023.
[62] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv22210.03493, 2022.
15

---

Appendix
A Related Work
Pata-unavailable OOD Detection: Generic Methods. In scenarios where OOD data is unavailable. Detection methods are categorized into three main types: (1) Output- based methods assess confidence using predicted probabilities. (2) Ensemble- based methods assess the uncertainty of collection of supporting models, classical techniques are Monte-Carlo dropout Ao g based on Bayesian inference and deep ensemble [99 3T: (3) Feature-bass based methods assess the Mahalanobis Distance between OOD samples and the distribution of the ID data feature space, usually considering the input and output spaces [22
occasionally extending to specific hidden layer spaces []. Besides, there are some fragmented methods, such as gradient methods [3 [5] and autoencoder reconstruction methods 7 Still, these methods suffer from optimization and computational complexity with serious performance bottlenecks 58 and thus are not mainstream detection methods.
OOD Detection GLMs.
Relatively few studies have explored OOD detection GLMs, mainly speech recognition 3 machine translation [6 32
in semantic parsing 3 25
summarization
43 and they do not jump out of the frameworks of uncertainty estimation, ensemble- methods, and embedding-basee based methods. To our knowledge, we are the first to study OOD detection mathematical reasoning, and we have found the failure of traditional algorithms in this scenario. Mathematical reasoning is an important and difficult research topic in the era of LLMs, and this research is valuable for the scenario expansion of OOD detection algorithms on language models.
B Motivation Visualization Details of Figure
B.1 Projection Setting
We use UMAP for projection visualization, which is a nonlinear dimensionality reduction technique for mapping high-dimensional data into low-dimensional spaces. It uses optimization of the local and global structure of the original data to produce high-quality mapping that can preserve the original data's is local and global characteristics.
The UMAP algorithm consists of two steps: (i) Calculate the local similarity between each data point and its immediate neighbors to construct local similarity map. (ii) Map the high-dimensional data to diwddieenionall space by optimizing an objective function that maintains the local and global structure. The hyperparameter "n no_neighbors" in the first step is key, we set it as 10 in our paper.
B.2 More Examples
We select the MATH O dataset for mathematical reasoning and the OPUS 40 (translation). In the MATH, we choose four domains: algebra, geometry, number theory, and precalculus; In the OPUS, we also choose four domains: Bible, news, TED, and health. We present examples of inputs and outputs under different domains for mathematical reasoning and translation scenarios in Table T Table respectively. They correspond to the case projection of Figure] Obviously, under mathematical reasoning, the outputs under different domains may appear exactly the same, while it is impossible under translation.
for text generation
C Theoretical Analysis of Section 2.1
In this section, we theoretically analyze Why pattern collapse in the output space leads greater likelihood of volatility differences in trajectories under different samples, which corresponds to Hypothesis in the main paper.
C.1 Problem Setup
Latent Space Embedding. For given sample 8;, [gn] E (1 < S L.) denote the hidden embedding of the 1-th layer. We define the embedding coordinate in the latent space as {(l, (x.b))
16

---

Table Examples of inputs/outputs from different domains in the mathematical reasoning scenario.
Input
Output
Domain I: Algebra
Suppose that f is function and f-1 is the inverse of f. If f(1) (2) 6. and
f(3) then what f ( (6))?
Domain 2: Geometry
Two chords, AB and CD, meet inside a circle at AP CP 7, then what BP
Domain 3: Number Theory
The product the proper positive integer factors of n can be written as n (az+bb// where is the number of positive divisors has, is positive integer, and the greatest common factor of the three integers a. b, and 1. What is b c?
Domain 4: Precalculus
Simplity
1 tan x
cot x
Table Examples of inputs/outputs from different domains in the translation scenario.
Input
Output
Domain 1: Bible
The earth yielded grass, herbs yielding seed after their kind, and bearing fruit, with its seed in it, after their kind; and God saw that it was good.
+ # AHE
*; # &ALE R t (
Domain 2: News
Supporters say the tunnels would benefit the environment and offer Californians more secure water supply.
Domain 3: TED
(Applause) June Cohen: Frank, that was beautiful, so touching.
1
Frank R A
Domain 4: Health
The White House Coronavirus Task Force was established on January 29.
I
(White House Coronavirus
Task Force) 1 v 21 ALL
ECOVID- 19
Embedding Interpolation. We assume the F₂(x) (Fa Fiz(x), F(((=) - with independent components fits the L coordinates, representing a continuous learning trajectory for st, so 1/sl: (.)... (.()) is taken from the functional space X, (F(()) are all independent variables. We constrain each component function of this (x) to satisfy the m-order (m 2 3) derivability property. Under this setting, the definition of dimension-indepeedenn trajectory volatility (Eq. equates to
V(s,) (s:(s:),
(9)
L Σ (1aa()=Fa(111 (a(()) Fiz(1 1)], (Fa(1) - 1)
Modeling. We observe the "paltern collapse" phenomenon in the output space in Figure We abstract this phenomenon in Figure g which compares the trajectory trend between different samples in the mathematical reasoning and text generation scenarios.
17

---

We specify that 1s12 - Fi(L) ~ Nic, )² where
c c= ,1, cal I = diag(6): 02, 6,))) 64)
(10)
According to the pattern collapse property under different tasks, we can constraint the endpoint embedding LL) the output space:
For mathematical reasoning with pattern collapse, - O, so we approximate that (L For text generation without pattern collapse, Σ 0, F.(L) #c. c.
=
With such constraints, we model the main theorem:
Theorem C.1 (Main Theorem) For different samples and 8j. the likelihood of variations in
trajectory volatility under mathematical reasoning scenarios higher than that under text generation scenarios, which means:
(RAO) (P(0)): ~ U((R) (V(s:) - V(s,) + (I), = cc} > E(RU)) (F,()). ~u(())) (V(s:) V(s,) + O.F,),, FLL) N(c, E²}},
where ) denotes the uniform distribution defined in d-dimensional real number space.
C.2 Prelinimary
the formal proofs. We begin with some propositions and lemmas that will be useful in the main theorem.
Next,
Proposition C.I (Lagrange Remainder Term) In the Taylor Expansion expression
(x) + df
1 d (x a)² +..++
d f (x a)" + Rn+1(x),
dr (x - a) 21
dx²
nl dx
the remainder (1) has the following property:
d=+1f
M
[Bu+1(x)] = (n +++11 1)!
(a - a)n+1
1(x - a))
dettI
(n 1)!
where e [a, x| or c E [r. a). and sup
amt1 ag
: E [a, x] or g E (x, ar >0
Proof. We consider the case of a < a with a > x identically. We use the Fundamental Theorem of
Calculus (FTC) for the most basic expansion of f(z):
f(x) f(a) f" df dr
(11)
dr
Continue to use the FTC to expand the derivatives in integrals:
/" df dæi [(a) +
/ df
d dxa
f(x) f(a)
dæi
dri
da
day
dʃ x a)
+ 10
21 d2
f(aa)
dr dæ,
dr
Ja
dx?
(12)
dkj
12
11
Fn d f
(z - a)* +
don+I drat1 dz, dæ
k=0 k! da
Therefore, the generalized remainder is known
d I
Rn+1 (x)
dx drn ... dr
d+
(13)
d=+1f
(x - 1)" dt.
m! d
18

---

We let m1+1 minae[a,a) d and Ma+1 max4e[a,a] f 80
ma+1 [ tt" dt
( jutif
dtn+l (x- dt S Mn+1 / (o (e- t)" dt
fa dutif (I t)
(14)
11+1
S
dem
S Mn+1.
x+1
According to the Lagrange's Mean Value Theorem, there must exist a number e E [a, x] with
d f fa duti f deatt
(x t)
(15)
den+I
(r-a)=+1 n+1
this gives
d=tif dp++I -x-
(x - a)= d
de
+
(16)
d=+if
f
(1))
(x m
(n+ + 1)!
den+I
and the equation the left side of the proposition is proved completely. The inequality the right-hand side is clearly established by the Lagrange's Mean Value Theorem.
Lemma C.1 (Error Bound for the Midpoint Rule) Suppose that f(x) is a m-th (m 2 2) order differentiable function on the interval (-00, +00), and K = sup
8²1
x ||, [a E R, then
dx
1. ((x) dx (b
a a
K 5
a), -b-a)s
(b a)3
24
Proof. [a, bl:
We do the first order Taylor Expansion for (xx at the midpoint
atb 2 of the interval
a (a+b)
+ b) 5/ f(z) -f
(a+t)) (aa+t) ) dr
f(x) dr af)
2
df
+b)
a+b
x
2
+ Ri (2)
dx
(17)
/
0
Ri (x) dr
Rí (2)) dx
Following Theorem
C.1
we do the inequality scaling below:
1/"0 (z(x) dx
<
σ a a+b>
dz = 6
K
a
3
a
b
37
K
(b
a)²
2!
2
2
2
24
(18)
Lemma C.2 (Difffrenniall-nneegr Error Order Estimation) Suppose that f(x) is m-th
(m
> d's dæl
3) order differentiable function on the interval (-o,, +00) and K;
: E (-00,+000) < +00 (i 1,2, ",, .,), then on the closed interval [a, we
sup
{
have
b b-) - (a (a b- (a+b
(i (i-1)) 1)) L
df dx
dr 5 R(L) ol
A
f
19

---

Proof. For each sub-interval (a+ (i- 1 we perform a Taylor Expansion at the midpoint
- (i 2)) obtain the following expression:
f(a- + --11)= f(a + b a (i- 1))
6-
L
b b- a (i - こ
af
b a L
b (b- a
27
a
L
2
+
d (a b-ط (i ( 2
+ O
2L
b b-a a (i - 2 +
df
a
(b-o) a
27
+
+ 0
L
(a+ T. (i 2)
L
2L
(19)
df
- a
0 P
d
d(a+ + b- (i 1)) L
L
df
Σ
a 21 - ())
a
df
+ + 2
d ( + ( 2))
Following Lemma
C.I we do the inequality scaling for the first-order derivative summation:
b a
df
df
K3
b - a
dx
(i- d (a + ' (1-11) 11)
dx
24
L
(20)
df
Ky(b- a)3
df
Ky(b - -a)4
dr
dz +
dx
24L
dz
2412
Ka(b a)1
+ 0 + 1=)=R(() -R(L)~ ~ +
A
(21)
24L
C.3 Main Theorem Proof
Finally, we formally prove our main theorem C.I in this sub-section, thereby verifying the Hypotheses
1 in the main paper. Since the d components of V(x)) - V(s;) are independent of each other, we
only consider the d-th dimensional component and the rest of the components Va(s;) Va(s;) can be proved in the same way.
Proof. According to Lemma
we can approximate Va(ss) Va(s;) as
1.
Va(s:) Va(s,)
FIa(1) Fea(1
-111 Σ
Ffa(1)
Fja(l 1)1
1=1
dF
L dx (/
da
|F jd
da + O 7
(22)
dx
dFu
dF dx
da
dx
dx
Now we want remove the absolute value of the integrated function. Due to the continuity of the first-order derivative, there must be several zero points that are not extreme points, dividing the domain of function definition into several open intervals with alternating constant positive and negative function values. We first define such a set of zeros on the domain of definition D = [1, L]:
dFid
X,
x x E Dj
= 0 }
= fii, 1g. ..p).
dx
(23)
dFfs
X3
x
x
E
D
dx
20

---

For all zero points is E X or Je E Zj. they must satisfy the following properties to ensure that they are not extreme points:
dFa
dFis
3e 2>0, <6 d(4 Ar) d(i, + Ax) <0, 0,
(24)
and je is the same as dg. Therefore, in the sub-interval (1, (11, 22), (2p, L), (ra()) is alternately constant positive and constant negative, and the same (f)(
We assume that the first interval (1, i)) and (1,J1) are constant positive intervals (the same can be proven for constant negative intervals). In this setting,
continue simplify 22
dFid
dFj
dæ
dæ
dx
dx
dFdd
dFid
dFid
dx +
dr
....
dæ
dx
dr
i dx
dl
{² dl
dx +
dx ......4
da
dr
dx
dx
51
(25)
Fla(x)) + (-1) -Fi(()) +... (-1)²r Fia(())
[Ba(()))) + (-1)) Pia(z)) +...+ (-1)ax P
P
F-Fu(11 + Fza(1)]+ 2 ((11)a Fal(s)) - -(-1)) Fja(in)
k=1
k=1
+ +[(-1)0-- Faall) (-11---1 (Fj(LL
Since Fia(x)) and FT((#) are taken from the functional space, (Fa(1), (Fa(())) and (Fja(1), (Fa((i)) can be seen as independent variables. We let
c= [(-1)+
Fi((L) - (-1) FFa(L)),
(26)
then rewrite Eq 25 the matrix form:
J
dF dr
sd
dæ
dx Aa + c,
(27)
dx
where the coefficient matrix, and æ is the unknown variable vector:
A = 1,,,,,2, 2, 2,, 2, 2, -2, (.. (-1)² 2 RRIx(ttt
p dimensions
dimensions
(28)
x: (1), (1)(1,, Fia(ii). Fialip), Fza(i), Fallig) TT E R ((++++))x1
We let ip = jn (mod 2), and return to the condition in the mathematical expectation both sides of the inequality in the main theorem C.T for the following categorical discussion:
Fi(() = FL)(L) = cas then c = O, so (a - Va(s,) = - = Ax 0; If FFj(LL,FFa(AA) (cc, 0 then * 0, Va(s;) Va(s,) 0 Ax=
We denote N(A) the solution space of {a|Ar = 0} and P(A) the solution space of {z|Ar -c). For x E P(A), x Zp or r Zp + Fn where Ep the special solution and V, the solution in zero space, i.e., the size of the solution space (AA) is larger than N(A). In this case, when the variable æ is sampled from the real number space, its probability of being in N(4) is smaller, namely its probability of not being ((A) is larger. This exactly equivalent the form of the proof of Main Theorem the proof is complete.
(A). Since rank(A) (A) = < + 2, the special solution Ip exists, so
so
21

---

C.4 Extended Conclusion
During the proof of the main theorem, we can unlock a hidden conclusion due the embedding interpolation: GLMy with larger number of hidden layers may achieve more sluble detection performance, i.e., discrepancies embedding volatility of different samples will be more obvious.
In Lemma C.2 we have proved the upper bound of inteerentiil-integrre error order estimation is the
equivalent infinitesimals of 1/ Actually, when L - +00, we have
s a 1 (1 -
b-a (a+ b
(
d dr
lim
dx,
(29)
1=1
which is clear according to the definition of Riemann Sum. This means that when the value of L increases, the differential-integue approximation error will be reduced, the conclusion of the main theorem will be more accurate. In our experiments, we use GPT2-XL (48 layers) and Llama2-78 (32 layers) as training backbones and find that the average performance GPT2-XL. higher and more stable than that of Llama2-7B, while the number of layers of GPT2-XL is 1.5 times that of Llama2-7B. This further validates the correctness our extended conclusions.
D Algorithm: TV Score Computation
The pseudo-code of our TV Score computation pipeline is shown Algorithm
Algorithm Trajectory Volatility (TV) Score Computation
Input:
L: The number of hidden layers
N: The size of ID dataset
k: the smoothing differential order of TV score
{ythiisl the average hidden embedding of the OOD sample in each layer
[golk)ssssr.sssss the average hidden embedding of all N ID samples in each layer
1: for 1 [ to I. do
2: Fitting ID samples ([il)sssssssssss to Gaussian distribution 9 NN E,)
3: f(y)) < (31 -wi (Ex) (y1 - m))
4: end for
5: for i + to k do
6: for
to I do
7: A N(4, (1) 2(9) t (-1)+ Eimo C(E (()) (9 (9) (z
8: end for
(v) (M)
9:
10: end for
for i to do
12: TV, < average (()) 1gs1.
13: end for
Output: (TVi)sssk
As for the computational complexity. once we have all yı. our TV score requires only scalar addition and multiplication. During the ID distribution fitting phase, both operations are O (Ldn), where is the dataset size. During the score computation phase, both operations are ((Lk)). In practice, the computation time is measured in milliseconds.
E Experimental Setting Details
F.1 Basic Information Dataset
For the ID dataset, we use the Muith which consists of Math Word Problems on arithmetic reasoning. For the OOD datasets, we intuitively introduce two types of detection scenarios following
4
(i) Far-shift OOD setting. we select the MATH M| the OOD data, which spans across
22

---

distinct mathematical domains encompassing algebra, geometry, counting and probability, number theory, and precalculus. It contains college difficulty level math problems, whereas MultiArith has only elementary school difficulty, and thus can be considered as sourced from far-different distributions; (ii) Near-shift OOD setting, we select five arithmetic reasoning datasets as the OOD data: GSM8K ( SVAMP 39, AddSub [3), SingleFq [88 and SingleOp [8] they all consist of Math Word Problems like the MultiArith but require different reasoning hops and knowledge points for solving the problems, and thus can be considered as sourced from near-different distributions. In addition, we present the data sizes and examples of all ID and OOD datasets, as shown in Table
E.2 ID Dataset Split
For the ID dataset MultiArith, we find that every 100 consecutive samples show a similar quadratic operation pattern (e.g. samples with id 0-100 are a mixture of addition and subtraction, and id 100-200 are mixture of addition and multiplication). Therefore, we divide it into 6 subsets (6* 100) in order. each subset, we take the first 60 training samples and the last 40 test samples.
E.3 Training Implementation
We train Llama2-7B 50 and GPT2-XI. S models on the training split of MultiArith. Llama2-7B is trained with A optimizer 29 for 10K steps and 8 batch sizes in 4-card RTX 3090 (2 per card). The learning rate is set to le-5, the warmup step to 10, and the maximum gradient normalization to 0.3. GPT2-XL is trained for 3K steps and 128 batch sizes in single RTX 3090, and other configurations are the same as Llama2-7B.
E.4 OOD Dataset Selection Rationality
In this part, examine the rationality of the OOD data selection, ensuring that the OOD data distribution significantly differs from the pre-trained data distribution and has not been fully learned during the pre-training phase. Some research 57 59 have confirmed the absence of data leakage in Llama2 for MATH and GSM8K datasets, we still conduct experiments and analyses to ensure this.
For GPT2-XL, We can determine this from the time dimension. GPT-2 was released in 2019, but the MATH and GSM8k datasets were released in 2021, so they are unlikely to appear the pre-training data. However, for Llama2-7B, due to the closed-source data, we cannot confirm which data the model used in the pre-training phase, so we cannot fully determine whether the selected dataset was OOD for the model from a data perspective.
Therefore, we argue that a dataset can be considered OOD when it is beyond the capability of the base model as claimed by prior work 47 [7 We test all ten datasets we select as the OOD data in the pre-trained Llama2-7B and GPT2-XL model, and Table shows the results. We find that GPT2-XL cannot handle any of the ten mathematical reasoning tasks, and Llama2-7B 7B performs with very low accuracy. Therefore, from a capability standpoint, we can ensure that these datasets are OOD for these two GLMs.
E.5 Baseline
Let represent the input sequence and y the output sequence, with lengths denoted n, and ny respectively. In addition, we assume that p(: 0) represents the GLM parameterized by 0 that has been trained in ID dataset D, outputting sequence of softmax probabilities. We compare some training-free baseline methods as below:
Maximum Softmax Probability
p(u Wi- 0).
(30)
Ml
Monte- Carlo Dropout
1 ply) ())(0)
(31)
where (()) p(0 D).
23

---

Table 9: ID and OOD datasets used in this paper.
In-Distribution Dataset
MultiArith (Data Size: 600)
Q: Kaleb was collecting cans for recycling. On Saturday he filled 5 hags up and Sunday he filled 5 more bags. If each bag had cans in it, how many cans did he pick up total?
A: 40.0
Far Shift cut-of-Distriburt Dataset
Near Shift Out-of-Distribute Dataset
MATH--lgebra (Data Size: 1187)
GSM8K (Data Size: 1318)
How many numbers are the domain the function
Judy teaches 5 dance classes, every day, on the weekdays and classes on Saturday. If each class has 15 students and she charges $15.00 per student, how much money does she make in 1 week?
f(x)
64
2²
64
2 64
A:
A: 7425
MATH-Geemetry (Data Size: 479)
SVAMP (Data Size: 1000)
Suppose we are given seven points that are equally spaced around circle. If P, Q, and are chosen any three of these points, then how many different possible values are there for mLPQI."
A mailman has to give 38 pieces of junk mail each the 78 blocks. there are 19 houses on a block. How many pieces of junk mail should he give each house?
5:
A: 2.0
MATH-Counting and Probability (Data Size: 474)
AddSub (Data Size: 395)
Amy's grandmother gave her identical chocolate chip cookies and 4 identical sugar cookies. how many different orders can Amy eat the cookies such that either she cats chocolate chip cookie first, she cats chocolate chip cookic last, or both?
While taking inventory her pastry shop. Kelly realizes that had 0.4
baking powder yesterday, but the supply is now down 0.3 box. How much more baking powder did Kelly have yesterday?
2:
A: 0.1
MATH-Number Theory (Data Size: 540)
SingleEq (Data Size: 508)
Notice that
Fred had dimes his bank. His sister borrowed his dimes. How many dimes Fred have now?
31 37 = 1147.
Find some integer with 0 2293 such that
31n 3 (mod 2293).
222
4.0
Mreal-Prealluuuus (Data Size: 546)
SingleOp (Data Size: 562)
11
Q: Pamela starts with 30 bottle caps. Jean takes 26 away. How many bottle caps does Pamela end with?
Q: a
and
2
1 V5
Find if the vectors b and
5
3(b c) 8(c x + k(a x
4.0
are orthogonal.
A:
24

---

Table 10: Accuracies of all datasets we select the OOD data in pre-trained GLMs.
Far-shift OOD Setting
Near-shift OOD Setting
Llama2-7B
GPT2-XL
Llama2-7B
GPT2-XL
Dataset
Dataset
Accuracy of pre-trained model
Accuracy of pre-trained model
Algebra
6/ 1187
0/ 1187
GSM8K
0/ 1318
0/ 1318
Geometry
2 / 479
0/479
SVAMP
8/ 1000
0/ 1000
Cnt.&Prob
4/ 474
0/4474
13/ SingleFq
AddSub
0/ 1000
Num. Theory
0/ 540
0/5
5/
0/508
Precalculus
1/ 546
5/
SingleOp
7/ 508
5/508
Sequence Perplexity
"
II p( 2;; 00:00
(32)
Input Embedding
(x ( p2 - pbe) where M4r and Σg represent the mean and variance of the Gaussian distribution associated with the first-layer hidden state.
(33)
Output Embedding
43
(y/ 4 Σ (y My) where My and Eg represent the mean and variance of the Gaussian distribution associated with the final-layer hidden state.
(34)
F Supplementary Experimental Results
F.
Hyperparameter Ablation: Smoothing Order
Preck
GSMIK SVAMP
AddSab Singleii
SingleUp
Goomaty
Num Tharry
80
AUROC
60
60
40
01
100
100
80
80
17R85
40
20
20
Smoothing Order k
Figure 4: Smoothing order analysis: ranges from 0 5 (k 0 corresponds to the original TV Score). The upper part is for the OOD detection scenario and the lower part is for the OOD quality estimation scenario: the left part is for the far-shift OOD datasets and the right part is for the shar- OOD datasets.
In the main experiments, we found that differential smoothing is not as effective as the basic TV score, with excellent results occurring only few datasets. Figure visualizes the results for the
25

---

OOD detection scenario with no smoothing (k = 0) and smoothing order k 1-5. We find that there significant effect of differential smoothing in two cases: (1) very good performance without smoothing, e.g., the precalculus dataset, where the AUROC results are almost close to 100%; and (2) significantly poor performance without smoothing, e.g. the AddSub dataset, where the FPR95 results on all other near-shift OOD datasets are 20 below, the results on this dataset are more than 80, when differential smoothing helps to eliminate some of the noise features and helps better detection.
In addition, for the case of k > O, the peak performance basically occurs in the case of k = 1 or 2, which indicates that when is too large, the phenomenon of smoothio tends to occur. Too much useful feature information is lost, leading to decrease in detection accuracy.
F.2
OOD Detection
Results of each dataset are shown in Table 11 (Llama2-7B) and Table 12 (GPT2-XL).
Table 11: AUROC and FPR95 results in Llama2-7B of the Offline Detection scenario (p-value 0.05 are grayed out). Underline and bold denote SOTA among all baselines and all methods.
Far shift O0D Setting
Dounet
Aleztea
Gormatry
Counting and Pribabiiity
Nurbhr Thrrey
Prisicelux
Anwrgge
Meshod
AUROUT
1
Mia Sclima Purk
/00::://880.970: 880.97
82 22 82 yaat
63.67erar 96. 19
76 a0 85 M5,20a
00 nil htt.s//ww/222.///// /.2.27
78.66iss as 1 4444
Cizlo Dropout
72.12-1aal 95,43
55.64 19715 Lae
67.53 $9.09.54
73 61 222
68.63 $7.04
Peppexity
6e.17 52, $2.78
21.42a 28
gi 90gatt
94 05.0e/ 19108-a117
83.64e1 a 06
Eapu Embollnng
61.51 62, 54
75.41 69.66 zan
62.53 88.35
B4.42 sas
S4.80-us
75 59 a9 (G3.7) sa
75,89 aa 67.57 sus
Ougun Enbeoding
L8.55 19/9519//449220 13392
78.72.13. (22)
61.43 38
74.86.124 75.21123 1as
TV uarz (Oun)
98.7 4.67 gaat
Mulhgase 3.P0
V7. Wi.M0aaxe 8.8
98.48geax 19,37
90 78esee// / 1,47aaos
Di5mo
04 31.55
94/08-eia/
a3.08 50.07.
30,2444
I.
93.25
A (bold enterlino)
+14.20 -18.06
+10 101336--8 49000
-20780 1-00-00 as
+86.00 D1 2-47:44
+5 777 917999
-13.12
Near-shitt OOD Setting
CSMBK
SVAMP
AddSaah
Singiefly
SingleOp
Atrnage
Meshod
AUROCT
EPR95 1
Mas Solime Purk T
Stitlleree 194.0711 os
S6. Shet va /M0 Ofigai
63.3 o /88 S6gna:
66 66.888gt07/ 16. Suga
51 17 so lat
60) 40l su /88 Mlann
Minic Curlo Dropout
48/87-3 99
44.90 992.2
$7.34 59
$4.00 99007.sss
56 /9122
$2.33 191.92
Perpleut [
52 52 /955566 1a
39.12 uaa// 39 14.2444 pas
62 B8 su 20
6 / 18138 uss
50 39 L...
/83 13
59.35 1 $6.09 19
Inper Eimbedding
B
45 95508511
60/ ar7
66 2aann T6 tHaass
67
77.77k21
600
Oupua Eibbecling
Fagira 4lgr an
6M00g 17gn
Migger 86 Aigeer
4% Sigars/ 18 Kigr
50gg su 66gg w
TV soace (Ous)
9428 was/ 14.22
94.51 /aza 12.89 Lat
85, 14114 $2.63 223
93.97 iase/ 17.39 use
94,00 uzs/ 14,61:
92.64 aus 228 cas
w/
21 290
34.24 20000
56.00 M3.7feuss
50.99 aini
A (oold underine)
+41 -47.800
+33.50/-74.0
-2076//3307
+26.83 113377-0 63.09
+26 / 40.17
-32.31 0.2.11--50 26
Table 12: AUROC and FPR95 results in GPT2-XL of the Offline Detection scenario (p-value > 0.05 are grayed out). Underline and bold denote SOTA among all baselines and all methods.
Far-shift OOD Setting
Datase
Alachra
Geoceetry
Cousting and Probukiitt
Namber Theory
Precalcutas
Arerge
Menhoa
AUROOC 1/
FPRES
Mix Softnas Prot. 1+
72. Basuez ass
64.0 577 47742
68 45 Lus $2.35 uist
60.37 suu 82.09%
78.67 poar ol 23.33
70.54 29229 Lac
Mone-Curio Drigoor
x
62 anaa 10 1i
66 56.6321 N2 49a
59 / 92 34g ds
63 88 so M6 M..71 on
70 Ma T3 M0garr
66 18 and 8i ti9
Porpesity
17axw/
12
40g
Inper Embediing
56 / do 12
$3, 20 55 200
39 Staue 10 uns
st ddsas// 552 12.89144
52 N6 s 34 52aaas
86 20 as Bhaa 9hii wi 16 diggne
Farbodding L
1% Vige 66 Afgrro
77 M8al / 68 Sigg
(77gg fil
25 56 21gan
$3, / 54 Diggino
scoce (Duss
98.35 ioun/ 6:24 1424 93 2uu 23 24
97.27 /ass/// 9.23
85.52 uus Sn
aas/ 19.861 Las
93, iou 68 uan
93.47 iass 24.10 V9 19.62.000 / 1175..i
wi Disio (Ours)
94.391 39133 12. t7
45.83 29.6
99.37 2,43.422
96.54 Le
dould
121-23355 55
+14.07
7-50.
/3067
-32.557
+10.38 7 7.30
Near-dhft Setting
Datanel
GSMIK
SVAMP
AddSub
SizpleEy
Sapl=Op
Arvraa
Mashol
Al AUROCT
FPRES
Mis Softuas Puob. H
14.06
12.45 a37
65.50 73.62 4.4
70,00 uou 27.56
70.96 cas/ 57.35 use
65.47 uua 50.37 axs
67.12 76.27 aas
Mooee Caro Drageor Ki
63 (3:4222 211
13:13 28
67.97 67.97 al 71 Alazas
71 21129 / 23:24
$2.59 /92 78
63.54210/ 78 1230
Perplesity
12
74463-a12
saa 87, 87.76
$7.14 43 33.09
30 7400
73/74 72.3722330 42
Inper Enbedding
3
17.3 45 105
$4 14.L /0275 ssa
50 54 /ue 46 d2la
$1.44inm/ 49 pu Las-
11.91 uoa 57 65
85 22 uaa// /2 MO
limbedding
Uf (0 A4
06, 72g
801002 / 70
2912 si/ D0a3m
TV scues (Oun)
95.26 1.78
95,99-0xe oua ve33
56 am 52.23
97.35 13.01smm
29.12 -.o7
94.96 sazi a2
wi Disno (Qus)
97.62 ioai 3359 lous
92. 19:3sw/ 13.07) Lok
$3.35 a 4177
97.83 .aw 19.4
99.94 isac 0883 exc
94.19 11 13.66
dhold undeding)
+10.66 /466.6
+5.01/44
+10,80 1-38.15
+18.00/555
+1.66 /-39
26

---

F33
Beyond Detection: OOD Quality Estimation
Results of each dataset are shown in Table 13 (Llama2-7B) and Table 14 (GPT2-XL).
Table 13: OOD Quality Estimation (Llama2-7B): Kendall's 7 and Spearman correlation between various OOD scores and benchmark quality metric binary matching. Each column shows the correlation when ID and OOD samples are merged. Underline denotes the SOTA among all baselines, and bold denotes the SOTA among our methods.
Ib Far-shit 00D
ID Naar-bilt O0D
Mirtoo
Algelra
Geonety
CatAPoo
Nan Theocy
Precalaatas
Aserae
GSMIK
SVAMP
MiSSbb
SingleBq
SingkOpp
Aarrgge
Kendall Rank Cuerakaioon Conflicnnt
Softax
0,05-axo
0.24
0.044.om
.0043..01
0.04
0.007.am:
0076-ams
0.57eece
-0075wee
Perplssity B
0071ge0
0.112
0005
0.000
0.073ase
0062.ax
0.0klace
0.07
0.0Maaan
Iepur Finbeliingg PD
0.139
0054gv
0.11ga
D0M
00Mare
Ouepur Embbakiin
11
0.77kgow
0.064g00x
-0001.003
0.022oon
0 0.028
0.182-aas
0.136-mm
0.891
0.174
0.342
0.161-4a2
0,309.a07
0,155.am:
0.159
Dinno (Dan)
6,885
0.59
0.123
0
0.148
0.11
0.178
0.113
0.121
0.77
0.005
0.113a
(bol walediag)
+0/021
00003
10,079
10.009
10.122
10.003
-0.073
-0.142
+0.089
10,055
10.022
10.05
Spearman Rank Corriation Coofliciont
Mui Softua Prob
[
0,027-asa
0.057avn
0.33
0:057
0000
0.038
4.07
0.005
0003-wxr
0.83aov
0.02ace
0.025seea
Pepplesity
6009
0018
2051
0.066
Iepus Firtedding
0.166use
0/081goer
0.130yon
0.100ces
Ouapei Enbealing
0005
0.03
20122
0.025
40040psen
0.101gee
-0M27gran
TV some (Ous)
0.122-ax:
0.169
0.102
0.12
0.316-xm
0. 0.1674403
0.124
0,188
0:127-a4r
0.165
0.188
0.158aume
6.19
0.87
0.367
0.131
galeling)
+0.033
+0.103
10.127
+0.89
-0.050
-0.008
+0.013
10.055
10.045
Table 14: OOD Quality Estimation (GPT2-XL.: Kendall's and Spearman correlation between various OOD scores and benchmark quality metric binary matching. Each column shows the correlation when ID and OOD samples are merged. Underline denotes the SOTA among all baselines, and bold denotes the SOTA among our methods.
Datanel
ID Far-shft 0OD
ID Near-shfft OOD
Alpytes
Geomenry
Cn.&Phob
Nam: Throry
Preclisils
GSMBK
SNAMP
AddSish
StngieFa
Singleli
Confiriien
Sofuras Pob..
0,07
0.32=ow
0,000
0.09
0:08
0,066occ
0032-xo:
0.88-1xx
0:03
0.007 0001
Poppisity [
-0000g0cn
0.006
go00
0006
Enbeedinng B
0102
0,t9uce
0010L.
Do0Muses
0,000
Ourput Fibbaddigg
00892000
0004ggen
TV somr (Chin)
0
0.14
8.1R amm
0.167 smn
0.13uace
1.152
Disno (Ouss)
0.134
0.132
0.132
0.161 ccm
0.15
0.139umum
0.002
0.099
D:126anm
0. 0,393
1123
/bold undering)
+0.045
+0.065
+0,009
-0,080
+0.014
+0.005
+0,022
10.129
+0.112
-0.004
Spcarmen Bunk Corelation Coctiiscat
00012
00M
0.0%
0.04
2007
0.872
Peplloit +
0,860mm
0,004.ess
0003
0.000
0,038amn
9/023
0.005
0.01
lapei Enneedline L
0.41
0.135
0.09
a.22
0,008ooc
0.06
0,006-xx 00084
0.00 0.200aas
Ourput Eibbadingg
4000
0004
0.04
00000
1000
TV some (Dus)
0.665
0.100
0.341 aaz
0.005
0.115
0.123
01:12
0.164-am:
0.025-aax
0.168
0.161
Di8mi (Dun)
0.733
0.266
(L072,awe
a.145
0.100
0.141
8.134:aar
0.127
0.180 pan
0. 0.155
(bold underine)
10,022
104073
10.054
10.045
10.009
10043
10.001
10070
10.091
10.000
10.071
10.06
F,4 Beyond Mathematical Reasoning
Our method has a wider range of application scenarios beyond mathematical reasoning. To verify generalizability, we choose the multiple-choice quizzing task, which has the same "pattern collapse" property as mathematical reasoning, since the output space is limited to the "ABCD" four options.
We select the MMLU dataset 10 and choose eight domains among it: high school mathematics, high school physics, high school chemistry, high school biology, high school geography, high school government and politics, high school psychology, high school statistics. We test eight rounds, each using one of the domains as the ID dataset and the remaining seven domains as the OOD datasets.
We use Ilama2-7B as the training backbone, each model is trained for 3K steps and batch sizes in 4-card NVIDIA Tesla V V100 (2 per card). The AUROC score matrices are shown in Figure 5 (c).
27

---

presenting the results for TV score, input embedding, output embedding, and perplexity, respectively, In the figures, the Roman numeral I VIIII are represented as:
= high school mathematics
Il = high school physics
III = high school chemistry
IV high school biology
V high school geography
VI = high school government and polities
VII = high school psychology
VIII = high school statistics
We find that MS-Prob and PPL nearly fail the multiple-choice task and the output embedding is not as excellent expected, which is caused by the pattern collapse phenomenon.
Our method is comparable to the input embedding method and has very good absolute performances. For some far-shift OOD scenarios, e.g., mathematics saaeemmoaicssyyyhoooggyy (I-VII), mmysccs-polities (II-VI, etc., performances of the input embedding method and method basically the same, and there exists a reasonable range competing phenomena, e.g., our method performs more advantageously under physics-biology biology (II-IV), and the input embedding method is better under geggrphh (II-V). For some near-shift OOD scenarios, egg. matherattics-statistii (I-VIII,, where both domains essentially belong the category of math, method will be more well-performed, indicating that the embedding- basedd method produces performance degradation in fine-grained scenarios, while our method possesses stronger robustness.
Overall, our method is scalable and has greater advantages in fine-grained detection scenarios.
28

---

10D
100
66-19
57-3P
60.11
15-19
56 75
70.g0
59
43-47
39:06
34-45
39.49
33/99
40.08
34,50
GoAly
00:288
09.37
54,99
5G-33
40-45
41.8g
gh-00
03-21
39.30
40.00
30.72
42.005
123-9g
40-49
42.00
09.33
Bo
50.40
S3.50
34.93
53-12
31.81
11.04
52.95
52.28
10.35
58.77
50
50
G4:34
53.79
17
41.74
17.02
53.00
53-37
01.43
09
44-42
g0.03
04.g2
40.74
V1
02.14
49.5M
54.75
38.88
40.02
G1:36
09.09
G2.31
66.29
18.99
09.00
52.13
0044
00.54
65
64G1
51.23
13.09
63:48
09.75
R1.38
07-47
16.65
V18
5G.18
00.7%
31-45
51.8
G1.8
ViI
41.64
55.81
28.00
46.88
(XTA)
(N/A)
Maximum Probability
(b) Sequence Perplexity
100
100
79.34
91-40
88.a5
05.21
07.10
8R.5
75.0
7A-72
21.06
Ro:88
26.75
88.10
67.8
61.12
$6.45
87:00
Ro.84
Ra.06
8c.72
60.4
60.09
99-57
S5-42
00-42
69.62
66.50
47-3
$6.11
50
2R.0g
04:41
02.18
Re.01
8hvi2
90.04
Rt.31
70.06
V1
Rt.20
30.84
75-44
70.41
69.18
57.70
60.6g
08.78
V18
8c:07
87:78
73:00
00.50
00:59
$0.u2
85.06
43:44
90.72
$3.08
60.19
36.07
70.46
50-35
(V/A)
(c) Input Embedding
(d) Output Embedding
100
MS2-0
S6-12
99-99
92.89
$4,02
M6.06
AM
58 pp
89.0g
79.00
$0,g2
64777
02.50
0a.76
p6.cg
N7:01
$4.05
86.05
78040
No.tH
88
V1I
01.02
$2.54
81.21
89.72
68 277
03--1
91.
(
(c) TV Score (Ours)
Figure 5: AUROC score matrix in MMLU dataset of different OOD scores. Rows represent ID data, and columns represent OOD data.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
[Abstract, Section 1]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [Limitations]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
[Section 2, Appendix C]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [Section 4, Appendix E]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
[Section 4, Appendix E]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pullc//uuuii wwwiii/gguie///eeeeeeeeic CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [Section 4, Appendix E]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [Section 4, Appendix F]
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
32

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: [Section 4, Appendix E] Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpa//www...cpppppocccccccoaaaa
Answer: [Yes]
Justification: [Ethies Statement]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
[Ethics Statement]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
33

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
[Ethics Statement]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [Section 4, Appendix E]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
34

---

Answer: [NA].
Justification:
[Our paper does not introduce new assets.]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: [Our paper does not involve crowdsourcing experiments and research.]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "yxUaIozhz8", "reviewer": "Reviewer_SAjz", "summary": "This work discusses a novel method for out-of-distribution (OOD) detection in generative language models (GLMs), particularly in the context of mathematical reasoning tasks. The key insights are: 1) The high-density output space in mathematical reasoning leads to a \"pattern collapse\" that causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces, 2) GLMs exhibit early stabilization for in-distribution (ID) samples in mathematical reasoning, while OOD samples do not show this behavior.", "questions": "What is the underlying mechanism of the TV score method, and how does it relate to the observed \"pattern collapse\" in the output space? How can providing a more detailed analysis and visualization of the trajectory dynamics, as well as the intuition behind the choice of trajectory volatility as the detection metric, enhance the interpretability of the approach?\n\nHow does the computational complexity of the TV score method impact its real-world deployment, especially in resource-constrained environments? What steps can be taken to investigate the computational efficiency of the method and explore potential optimizations or approximations to improve its practical applicability?\n\nHow robust is the TV score method to adversarial attacks, and what are the critical considerations for the security of generative language models? What steps can be taken to evaluate the method's performance under different types of adversarial perturbations and develop strategies to improve its robustness?", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors propose a novel trajectory-based method called \"TV score\" that leverages the unique characteristics of the high-density output space in mathematical reasoning tasks to effectively detect OOD samples. This approach goes beyond the traditional OOD detection methods focused on uncertainty estimation and embedding distance measurement, which struggle in the challenging mathematical reasoning domain.\n\nRobust OOD detection is crucial for the real-world deployment of generative language models, as these models are susceptible to performance degradation when faced with out-of-distribution inputs. The authors' work addresses a practical and important problem in the field, as mathematical reasoning tasks are increasingly incorporated into language models with high-stakes applications.\n\nThe authors' analysis of the unique characteristics of the input and output spaces in mathematical reasoning tasks provides valuable theoretical insights into the challenges posed by this domain for OOD detection.", "weaknesses": "The underlying mechanism of the TV score method and its relationship to the observed \"pattern collapse\" in the output space is not fully explained. Providing a more detailed analysis and visualization of the trajectory dynamics, as well as the intuition behind the choice of trajectory volatility as the detection metric, would enhance the interpretability of the approach.\n\nThe computational complexity of the TV score method is not discussed, which is an important consideration for real-world deployment, especially in resource-constrained environments. Investigating the computational efficiency of the method and exploring potential optimizations or approximations would be valuable for improving its practical applicability.\n\nThe paper does not address the robustness of the TV score method to adversarial attacks, which is a critical consideration for the security of generative language models. Evaluating the method's performance under different types of adversarial perturbations and developing strategies to improve its robustness would be a valuable extension of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your constructive comments! We will respond to your concerns one by one.**\n\n---\n\n> W1 & Q1: The mechanism of TV score ... more analysis and visualization of trajectory ... intuition behind the choice of trajectory volatility ...\n\n### 1. Mechanism of TV score and its relationship to \"pattern collapse\"\n\nThe relationship of TV score and \"pattern collapse\" are not directly related, but indirectly transitioned through the theoretical intuition of Section 2.1. Refer to **\"General Rebuttal: Motivation line from \"pattern collapse\" to \"early stabilization\" and TV score\"** for details.\n\n### 2. Intuition behind the choice of trajectory volatility\n\nThe Intuition is our Hypothesis 1 (line 104): The \"pattern collapse\" in mathematical reasoning scenarios leads to more significant differences among different samples' trajectories compared to traditional generative tasks (we have modeled and proved this intuition from a theoretical perspective in Section 2.1)\n\n### 3. Detailed analysis and visualization of trajectory dynamics\n\nRefer to **\"General Rebuttal: Universality of \"Early Stabilization\" phenomenon in mathematical reasoning (Expanded visualization of Figure 3 in the paper)\"** for detailed responses.\n\n---\n\n> W2 & Q2: The computational complexity is not discussed ...\n\nThanks for your consideration. After obtaining all outputs $\\boldsymbol{y_l}$ of each layer, there are two main steps to obtain the final scores:\n\n(1) Get ID Data Information: Fitting Gaussian distribution $\\mathcal{G}_l$ = $\\mathcal{N}(\\boldsymbol{\\mu}_l, {\\boldsymbol \\Sigma}_l)$ for $L$ layers\n\n${\\boldsymbol \\Sigma}_l$ is a diagonal matrix because $d$ embedding dimensions are independent, so we only need to compute the mean and variance of each dimension of all $n$ ID sample embeddings.\n\n* Compute mean ${\\boldsymbol{\\mu}}_l$: $d(n-1)$ addition operations and $d$ multiplication operations are required;\n* Compute variance ${\\boldsymbol \\Sigma}_l$: $dn+d(n-1)=d(2n-1)$ addition operations and $dn+d=d(n+1)$ multiplication operations are required.\n\nWe fit $L$ Gaussian distributions, so the number of addition and multiplication operations are both $\\mathcal{O}(Ldn)$.\n\nWe report the computation time of fitting Gaussian distribution in the ID dataset MultiArith ($n$=600):\n| time (s) |\n| --- |\n| 1.132 |\n\n*Note: this part is a one-time event and only needs to be performed once.*\n\n(2) Get OOD Data Information: Compute TV Score\n\nFor $k=0$, we need to compute the Mahalanobis Distance between OOD sample and ID distribution (Eq 6). Since ${\\boldsymbol \\Sigma}_l$ is a diagonal matrix, it involves only simple vector multiplication and does not require matrix multiplication. Specifically, it requires $d + (d-1) = 2d-1$ addition operations and $2d$ multiplication operations. We have $L$ layers, so the numbers of addition and multiplication operations are both $\\mathcal{O}(Ld)$.\n\nFor $k>0$, for each increase in $k$ by 1, $L$ Gaussian distribution differences, $L$ embedding differences, and $L$ MD computations need to be performed. The increasing numbers of addition and multiplication are both $\\mathcal{O}(Ld)$.\n\nWe sample 1000 cases to compute TV score. Time is as below:\n\n|k|time mean (s)|time std (s)|\n|-|:-:|:-:|\n|0|0.0016|0.0001|\n|1|0.0033|0.0001|\n|2|0.0049|0.0002|\n|3|0.0066|0.0002|\n|4|0.0082|0.0002|\n|5|0.0098|0.0003|\n\n**After complexity analysis and experimental results, it is clear that our method is efficient and can be flexibly deployed in realistic scenarios.** *This is one of our strengths, especially compared to some probability-based metrics such as perplexity, which require time-consuming softmax exponential computation.*\n\n---\n\n> W3 & Q3: The paper does not address the robustness ...\n\nThanks for your consideration. However, prior works on OOD detection did not take this into account, so the robustness of OOD detection is not a well-defined question. Despite this, we try our best to design experiments to verify this point.\n\nWe make the following assumption and goal: For a perturbed ID sample, the sample still belongs to the ID sample in realistic scenarios, but models may misidentify it as an OOD sample. We need to avoid this misidentification.\n\nWe refer to [1] for perturbation method to input data in language models:\n\n* Paraphrasing: We generate one paraphrased input by querying ChatGPT using the prompt in [1];\n* Dummy Tokens: We randomly select tokens that marginally influence the original meaning and append them to the input. Such tokens could be newline characters, tab spaces, ellipses, or supplementary punctuation marks.\n\nWe compare two settings:\n\n* Original: Results in our paper\n* Perturbation: We replace ID samples in the test set with the perturbed text. We report results under two perturbations.\n\n||Llama2-7B||||GPT2-XL||||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||Far-shift OOD||Near-shift OOD||Far-shift OOD||Near-shift OOD|\n||AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|\n|Original|98.76±0.11|5.21±0.98|92.64±0.39|28.39±1.38|93.47±0.08|24.10±0.95|94.86±0.23|13.82±0.36|\n|Perturbation w/ Paraphrasing|97.94±0.12|5.75±1.00|91.88±0.42|29.28±1.41|93.12±0.09 |24.67±0.97|94.10±0.24|15.02±0.39|\n|Perturbation w/ Dummy Tokens|98.54±0.11|5.54±0.98|92.43±0.40|29.16±1.41|93.47±0.08|24.10±0.95|95.01±0.22|12.78±0.34|\n\n**Our method can largely defend against some perturbations in realistic scenarios, showing its strong robustness.** *We conjecture that this is because our method considers a large amount of information in the middle layer, whereas probability- or embedding-based methods only consider a single layer in the output/input, which makes our method better resistant to the influence of some random factors in the output layer, such as overconfidence, and therefore more robust.*\n\nAs for more explorations, we leave it for future work.\n\n[1] SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models. EACL, 2024.\n\n---\n**We expect these responses to address your concerns, and look forward to your more positive comments.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2G1CxKx5y", "reviewer": "Reviewer_KF5B", "summary": "This paper studies the OOD problem in GLMs under mathematical reasoning and found that the the patter collapse phenomena in the output space. The trajectory violation score is proposed to distinguish the ID and OOD samples. A thorough evaluation shows that the proposal can outperform traditional algorithms under offline detection, online detection, and quality estimation.", "questions": "1.  What is the internal relationship between the pattern collapse (in Figure 1) and the early stabilization (in Figure 3)?", "limitations": "The limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(+) This paper is well structured and well-written.\n(+) The authors conduct a thorough evaluation of the proposed methods", "weaknesses": "(-) The input data for the empirical study in Figure 3 is not introduced. Thus, whether it can reflect the scenario of mathematical reasoning is not clear. It lacks theoretical analysis. \n(-) The relationship of the pattern collapse (in Figure 1) and the early stabilization (in Figure 3) is not clearly illustrated. \n(-) The datasets for experiments are quite limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your constructive comments. We will respond to your concerns one by one.**\n\n---\n\n> W1: The input data for the empirical study in Figure 3 is not introduced. Thus, whether it can reflect the scenario of mathematical reasoning is not clear.\n\nThanks for pointing this out, and sorry for the missing experimental setup (i.e., input data source) for Figure 3.\n\n### 1. Setup Description of Figure 3\n\nThe ID data curve is the average of all samples in the MultiArith dataset, and the OOD data curve is the average of all samples in the five domains of the MATH dataset (Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus). These are consistent with our settings for the ID and OOD datasets in the experimental setup (Section 4.1). We will add this setup in the updated version. \n\n### 2. More evidences about \"it can reflect the scenario of mathematical reasoning\"\n\nPlease refer to **\"General Rebuttal: Universality of \"Early Stabilization\" phenomenon in mathematical reasoning (Expanded visualization of Figure 3 in the paper)\"** for detailed responses and evidences.\n\n\n---\n\n> W2 & Q1: The relationship of the pattern collapse (in Figure 1) and the early stabilization (in Figure 3) is not clearly illustrated.\n\nPlease refer to **\"General Rebuttal: Motivation line from \"pattern collapse\" to \"early stabilization\" and TV score\"** for detailed responses.\n\nThanks for pointing out the vague elaboration on the motivation line. The pattern collapse (in Figure 1) and the early stabilization (in Figure 3) are not directly related, but indirectly transitioned through the theoretical intuition of Section 2.1. To summarize, the \"pattern collapse\" leads to more significant trajectory differences across different samples, and the source of the trajectory differences between ID and OOD samples is the \"Early Stabilization\" phenomenon.\n\nIn detail, the function of each section is:\n\n* In Section 1, we **find the \"pattern collapse\" (in Figure 1)** in the output space;\n* In Section 2.1, we introduce the intuition: **the presence of \"pattern collapse\"** causes the convergence of the trajectory endpoints of different samples, **leading to significant trajectory differences across samples  (As stated in Lines 104 Hypothesis 1)**. We justify this intuition through theoretical modeling and proving.\n* We already have the intuition that trajectories can be a good measure, but it is still unknown what kind of difference exists between the trajectories of the ID and OOD samples. Thus, in Section 2.2, we conducted empirical experiments to observe the trajectory volatilities of different scenarios and discover the phenomenon of **\"early stabilization\" (in Figure 3), which is the root cause of the trajectory differences.**\n\nWe will state this motivation line at the start of Section 2 in the updated version.\n\n---\n\n> W3: The datasets for experiments are quite limited.\n\nThanks for pointing out the limited datasets. We illustrate this in terms of both data type and data size:\n\n### 1. Data Type\n\nAs for the data type, **we have collected 10 of the most commonly used datasets** in the LLM mathematical reasoning research for our experiments, and **they cover six types of mathematical tasks and all levels of difficulties from elementary school to college**. Due to space limits, we reported the average results for each setting in the main text, and the results on each dataset have been shown in Appendix E (Tables 8-11). Overall, our dataset types have been as rich as possible.\n\n### 2. Data Size\n\nData size is a limitation as we mentioned in Section Limitation. However, this is caused by the particular nature of the mathematical reasoning research field, and is not a subjective constraint on us, for the following reasons:\n* Compared to traditional text generation tasks such as summarization and translation, mathematical reasoning tasks did not receive much attention prior to the era of LLMs, and thus there are few general-purpose datasets. At the same time, automated construction of mathematical datasets needs to be based on complex rules, resulting in complex mathematical tasks requiring significant and time-consuming human intervention[1,2], making diversity and size severely limited. \n* After the advent of the Chain-of-Thought technique[3], mathematical reasoning started to receive more attention from the NLP field, but some of the more complex mathematical tasks are usually built for LLM evaluation[4], such as TheoremQA (600 samples)[5], SAT-Math (220 samples)[6], MMLU-Math (974 samples)[7], making the large-scale training sets not specifically constructed. \nAll these factors make the dataset size in the field of mathematical reasoning much smaller than traditional generative tasks.\n\nOn the other hand, **we are the first to study OOD detection on mathematical reasoning, and the current data type is sufficient to demonstrate the generality of our method.** If new large-scale mathematical reasoning datasets become available, our method can be further applied.\n\n[1] Solving general arithmetic word problems, EMNLP 2015.\n\n[2] Measuring mathematical problem solving with the math dataset, NeurIPS 2021.\n\n[3] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022.\n\n[4] MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, ICLR 2024.\n\n[5] TheoremQA: A Theorem-driven Question Answering Dataset. ACL 2023.\n\n[6] Agieval: A human-centric benchmark for evaluating foundation models. NAACL 2024.\n\n[7] Measuring massive multitask language understanding. ICLR 2021.\n\n---\n**We expect the above responses to address your concerns, and look forward to your more positive comments.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lcgt2MxZFl", "reviewer": "Reviewer_A7h4", "summary": "This paper presents a trajectory-based method for OOD detection in the mathematical reasoning setting. OOD detection is extensively studied in the text setting and image setting. The main motivation of this work is claimed to that mathematical reasoning poses significant challenges to embedding-based methods due to its high-density feature of output spaces, but this feature causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces. The proposed method uses trajectory volatility for OOD detection in mathematical reasoning. Experiments are conducted to validate the performance of the proposed method.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper studies OOD detection in the mathematical reasoning setting, which is less studied compared to text and image setting.  \n\nThis paper uses examples to illustrate the motivation and key idea. This improves the accessibility of this paper.", "weaknesses": "The motivation is not strong and convincing. Figure 1 illustrates two challenges with respect to the input space and out space respectively. It is not convincing, and more evidence or data analysis should be provided to make these two challenges solid. I do not buy that pattern collapse is generally hold for all types of mathematical reasoning tasks. The example is just a special case. The output can take any value in the real line, and the collapse probability not that high. \n\nThe idea of the proposed method is not convincing. I am not convinced that early stabilization is generally held for mathematical reasoning problems. More evidence should be provided to justify it. \n\nThe writing is also not precise. For example, in Equation (1), the domain of f is not specified. The notation \\phi is not defined. \n\nAlso, the experiment is not solid. There is no experiment to justify that the performance improvement is due to addressing the challenges in the input space and output space posed in the Introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your constructive comments! We will respond to your concerns one by one.**\n\n---\n\n> W1: The motivation is not strong and convincing. Figure 1 ...\n\n### 1. Input Space\n\nAs for the phenomenon that embedding varies less in different domains of input space: We have discussed it in Section 5 and conducted experiments in Appendix G.2 to demonstrate this.\n\n### 2. Output Space\n\nPlease refer to **\"General Rebuttal: Existence and Universality of \"Pattern Collapse\" phenomenon in mathematical reasoning\"** for detailed responses and evidences.\n\nWe must claim a key fact: **For generative language models (GLMs), they model real numbers or mathematical expressions not in a mathematical sense, but based on discrete token sequences after tokenization. Thus, the collapse occurs at the token level, not at the full mathematical expression level**. Due to the autoregressive generative nature of GLMs, the collapse phenomenon occurs during the prediction of each token.\n\nWe agree with your opinion that the number of values on the real line is infinite in a mathematical sense. However, **after tokenization, they contain only 0-9 number tokens and a limited number of special symbols**, such as decimal points, slashes, root signs. This means that two largely different expressions in the mathematical sense may cover many of the same tokens.\n\nWe emphasize two key conclusions from the statistic data presented in General Rebuttal:\n\n* Existence: **The average token duplication rate is up to 99% on all math tasks, and even a staggering 99.9% on some simple arithmetic tasks**; In contrast, **the token duplication rate on the text generation task is only about 60%, with about 2000 different types of token**, and still increasing as the total number of tokens increases. *These data and comparisons demonstrate that pattern collapse occurs in mathematical reasoning and not in text generation.*\n\n* Universality: **Token repetition rate exceeded 97% on all seven math tasks of different difficulties and types.** *This demonstrates the universality of \"pattern collapse\" in various mathematical reasoning tasks.*\n\n---\n\n> W2: I am not convinced that early stabilization is generally held for mathematical reasoning problems. More evidence should be provided to justify it.\n\n### 1. Setup Description of Figure 3\n\nThe ID data curve is the average of all samples in the MultiArith dataset, and the OOD data curve is the average of all samples in the five domains of MATH dataset (Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus). These are consistent with our settings for the ID and OOD datasets in the experimental setup (Section 4.1). We will add this setup in the updated version. \n\n### 2. More evidences about \"early stabilization is generally held for mathematical reasoning problems\"\n\nPlease refer to **\"General Rebuttal: Universality of \"Early Stabilization\" phenomenon in mathematical reasoning (Expanded visualization of Figure 3 in the paper)\"** for detailed responses and evidences.\n\n---\n\n> W3: The writing is also not precise. ...the domain of f is not specified. The notation \\phi is not defined.\n\nThanks for pointing out some questions about paper writing. \n\n* The domain of $f(\\boldsymbol{x}, \\boldsymbol{y}, \\boldsymbol{\\theta})$ is any value $(\\boldsymbol{x}, \\boldsymbol{y}, \\boldsymbol{\\theta})$ sampled from the joint probability distribution $P_{\\mathcal{X} \\times \\mathcal{Y} \\times \\Theta}$ defined on the $\\mathcal{X} \\times \\mathcal{Y} \\times \\Theta$, where $\\mathcal{X}, \\mathcal{Y}, \\Theta$ are the input space, the output space, and the parameter space, respectively, as defined in lines 70-75. We did not specify this because there are no special values that need to be excluded. \n\n* The notation \"\\phi\" does not appear in our paper.\n\nWe will improve our presentation in updated version.\n\n---\n\n> W4: The experiment is not solid. There is no experiment to justify that the performance improvement is due to addressing the challenges posed in the Introduction.\n\n### 1. Response to \"There is no experiment to justify that performance improvement is due to addressing the challenges ...\"\n\n* First, **the two \"challenges\" are two objectively existing phenomena in mathematical reasoning scenarios. However, they make existing embedding-based methods that cannot be applied to mathematical reasoning, so we call them \"challenges\" for existing methods, not for us**. As stated in lines 38-39, *“However, embedding-based methods encounter challenges under mathematical reasoning scenario”*. These two challenges are simply introduced to explain why existing methods cannot be applied to mathematical reasoning; \n\n* Second, because existing methods are hindered by these two phenomena, we need to find new methods to circumvent encountering them. Thus, **we aim to circumvent the two challenges faced by existing methods, not to address them**. As stated in lines 47-48, *“Therefore, we transform our perspective from the static embedding representation to the dynamic embedding shift trajectory in the latent space\"*, **we are not improving on existing methods, but rather two different research ideas.**\n\n* Third, these two phenomena are challenges for existing methods, but opportunities for us: we utilize them to design trajectory-based methods (Section 2-3).\n\n### 2. Reasons for performance improvement\n\nA complete explanation of why our method yields performance improvement is given in Section 2 (motivation part). \n\nRefer to **\"General Rebuttal: Motivation line from \"pattern collapse\" to \"early stabilization\" and TV score\"** for details.\n\n### 3. Response to “The experiment is not solid”\n\nWe have performed rich dataset experiments, scalable experiments, ablation analyses, and failure analyses in Sections 4-5 and Appendices E-G to demonstrate the solidity of our experiments.\n\n---\n\n**We expect these responses to address your concerns, and look forward to your more positive comments.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4UXaLkwH8E", "reviewer": "Reviewer_oehe", "summary": "This work studied the ODD detection in mathematical reasoning, which presents a new measurement, TV score, based on the observed pattern collapse property and early stabilization of GLMs. It seems the first discussion on the OOD detection in mathematical reasoning.", "questions": "1. Is it true that the embedding dimension should be fixed across layers to compute the embedding difference between neighboring layers?\n2. To better support the claim of choosing $k \\leq 5$, it would be beneficial to provide a quantitative analysis of over-smoothing occurrence when $k$ is set to a larger value.\n3. For testing the ODD performance, could you report the evaluation results in terms of AUPR and F1?\n4. Which datasets were used for the analysis in Figure 3?\n5. There are some grammatical mistakes that need fixing. Here are a few examples:\n   (a) \"we need ..., then computer ...\" in Line 150.\n   (b) The sentence in Lines 161-162.\n   (c) \"Outliers ..., then lead ...\" in Lines 164-165.", "limitations": "The authors adequately claimed the limitations of this work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This appears to be the first discussion on OOD detection in mathematical reasoning. The authors clearly explain why the trajectory works in an understandable and empirical manner. The manuscript is well-organized, and the methodology is both simple and effective.", "weaknesses": "1. Additional metrics commonly used for ODD should be included for evaluation.\n2. The writing requires improvement.\n3. Further insight into the potential impact of over-smoothing on the setting of the critical parameter $k$ is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your constructive comments! We will respond to the weaknesses and questions you raised in the following areas:**\n\n---\n\n> More Metrics (W1 & Q3)\n\nThank you for suggesting richer evaluation metrics about AUPR and F1. The results are below:\n\n| Method | Llama2-7B | | | |  GPT2-XL | | | |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | \n| | Far-shift OOD | | Near-shift OOD | | Far-shift OOD | | Near-shift OOD |\n| | AUPR | F1 | AUPR | F1 | AUPR | F1 | AUPR | F1 |\n| MS-Prob | 74.22±1.20 | 68.42±1.39 | 53.30±1.65 | 66.86±1.06 | 81.35±0.98 | 71.78±0.98 | 66.26±0.99 | 65.45±0.85 |\n| MC-Drop | 57.86±0.34 | 54.87±1.92 | 40.87±0.66 | 59.78±1.29 | 65.13±1.09 | 62.34±1.38 | 63.24±0.87 | 61.12±1.33 |\n| PPL | 78.89±0.40 | 72.25±0.87 | 62.73±0.98 | 75.58±1.11 | 71.32±1.21 | 75.60±0.74 | 62.64±0.59 | 60.95±1.16 |\n| I-Emb | 79.63±1.17 | 69.08±1.11 | 54.72±1.89 | 61.92±0.99 | 86.69±0.37 | 88.42±0.29 | 78.82±0.62 | 69.29±1.09 |\n| O-Emb| 60.58±1.25 | 56.06±1.65 | 39.84±0.81 | 60.05±0.94 | 78.24±0.86 | 85.34±0.41 | 75.39±0.75 | 68.83±1.06 |\n| TV score (ours) | **98.81±0.16** | **93.17±0.49** | **90.73±0.42** | **80.43±0.54** | 98.04±0.08 | 98.86±0.09 | **89.91±0.65** | **78.15±0.76** |\n| w/ DiSmo (ours) | 96.19±0.06 | 85.79±0.78 | 80.57±0.72 | 78.98±0.65 | **98.28±0.06** | **99.63±0.07** | 84.04±0.70 | 76.03±0.94 |\n\n**On both metrics, our approach still maintains a significant lead.**\n\n---\n\n> Discussion about k (W3 & Q2)\n\nThanks for suggesting a more detailed discussion of $k$. In Section 5 and Appendix G.1, we have conducted ablation studies for cases of $k \\leq 5$. To support our claim of choosing $ k \\leq 5$ and the evidence about the over-smoothing phenomenon when $k$ is too large, we report more AUROC results in Llama2-7B and GPT2-XL as we continue to increase $k$ values:\n\n| $k$ value | Llama2-7B | | GPT2-XL | |\n| --- | --- | --- | --- | --- |\n| | Far-shift OOD | Near-shift OOD | Far-shift OOD | Near-shift OOD\n| 0 | 98.76 | 92.64 | 93.47 | 94.86 |\n| 1 | 94.71 | 87.98 | 95.55 | 94.08 |\n| 2 | 94.66 | 85.39 | 96.54 | 94.19 |\n| 3 | 89.57 | 76.47 | 95.32 | 93.44 | \n| 4 | 82.20 | 58.66 | 95.17 | 92.09 |\n| 5 | 79.52 | 49.25 | 94.26 | 92.18 |\n| 6 | 57.65 | 47.89 | 93.01 | 82.65 | \n| 7 | 55.27 | 47.16 | 90.28 | 76.50 |\n| 8 | 58.63 | 48.34 | 82.11 | 76.18 |\n| 9 | 54.10 | 51.28 | 73.89 | 68.22 | \n| 10 | 52.93 | 49.72 | 59.66 | 57.91 |\n\nWhen $k>5$, the AUROC value on Llama2-7B shows a sharp drop and stabilizes around 50. The AUROC value on GPT2-XL remains high, but also shows a sharp decrease when $k$ increases close to 10. This is because GPT2-XL has 1.5 times more layers than Llama2-7B, the trajectory information is richer and the noise will be relatively more. \n\nOverall, as $k$ continues to increase, the useful trajectory volatility information is gradually blurred even though more noise is erased, causing over-smoothing. Therefore, $k \\leq 5$ is a better trade-off range, this is why we choose $k \\leq 5$.\n\n---\n\n> Experimental Setup about Figure 3 (Q4)\n\nThe ID data curve is the average of all samples in the MultiArith dataset, and the OOD data curve is the average of all samples in the five domains of the MATH dataset (Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus). We will add the setup in the updated version.\n\n---\n\n> Method Detail: Fixed Embedding Dimension (Q1)\n\nThe value of the embedding dimension is arbitrary and does not need to be fixed, but the embedding dimensions of neighboring layers should be equal, otherwise they cannot be subtracted.\n\nFor language models, the output dimension of the hidden layer is the embedding dimension set in the model configuration (e.g., 4096 for Llama2-7B and 1600 for GPT2-XL), so there is no problem of inconsistent dimensionality.\n\n---\n\n> Paper Writing (W2 & Q5)\n\nThank you for pointing out the mistakes in our writing, we will correct them in the updated version.\n\n---\n\n**We expect the above responses to address your concerns, and look forward to your more positive comments.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
General Rebuttal: Motivation line from "pattern collapse" to "early stabilization" and TV score
---


To summarize, our motivation line is: 

**(Section 1, Figure 1): We find "pattern collapse" in the output space**

**-> (Section 2.1, Theoretical Intuition and Proving): The "pattern collapse" leads to more significant trajectory differences across different samples**

**-> (Section 2.2, Empirical Experiments): The source of the trajectory differences between ID and OOD samples is the "Early Stabilization" phenomenon**

**-> (Section 3): The "Early Stabilization" makes our trajectory-based detection method effective.**

---

General Rebuttal:  Existence and Universality of "Pattern Collapse" phenomenon in mathematical reasoning
---

### 1. Why "pattern collapse"? **Tokenization is the key**

**For generative language models (GLMs), they model real numbers or mathematical expressions not in a mathematical sense, but based on discrete token sequences after tokenization. Thus, the collapse occurs at the token level, not at the full mathematical expression level.**

Due to the autoregressive generative nature of GLMs, the collapse phenomenon occurs during the prediction of each token.

### 2. More cases

For mathematical expressions that are very different in the mathematical sense, **after tokenization, they all contain only 0-9 number tokens and a limited number of special symbols**, such as decimal points, slashes, root signs, curly brackets. They make up a very small percentage of the vocab:

* 5517 -> ['▁', '5', '5', '1', '7']
* 21.59 -> ['▁', '2', '1', '.', '5', '9']
* 71/91 -> ['▁', '7', '1', '/', '9', '1']
* -\\sqrt{3255} -> ['▁', '-\\', 'sqrt', '{', '3', '2', '5', '5', '}']
* y^4-2y^3+7y^2+y-5 -> ['▁', 'y', '^', '4', '-', '2', 'y', '^', '3', '+', '7', 'y', '^', '2', '+', 'y', '-', '5']
* x^2/x^5 -> ['▁', 'x', '^', '2', '/', 'x', '^', '5']

### 3. "pattern collapse" is generally hold for all types of mathematical reasoning tasks

After explaining the tokenization behind "pattern collapse", we now demonstrate the universality of "Pattern Collapse" phenomenon in mathematical reasoning.

> Setup

To demonstrate the universality of “pattern collapse” across various tasks of mathematical reasoning, we conduct the following statistical experiment:
We **categorize the mathematical tasks into various types across different domains and difficulties, then count the token type number and token duplication rate corresponding to each category, and the vocab coverage**. We also test translation and summarization tasks by taking samples with the same token size as the mathematical reasoning dataset for a clear comparison.

We use Llama2 tokenizator (Vocab size = 32000). The computation metric is:

* Token Duplication Rate = 1 - token type number / token number

* Vocab Coverage = token type number / Vocab size

> Statistics Data

***{The statistics data comparisons are shown in PDF Table 2.}***

> Analysis

From the results, we can conclude that:

* Existence: The **average token duplication rate is up to 99% on all math tasks**, and even **a staggering 99.9% on some simple arithmetic tasks**; In contrast, the **token duplication rate on the text generation task is only about 60%, with about 2000 different types of token**, and still increasing as the total number of tokens increases. These data and comparisons demonstrate that pattern collapse occurs in mathematical reasoning and not in text generation.

* Universality: Token repetition rate **exceeded 97% on all seven math tasks** of different difficulties and types.

> Conclusion

These evidences demonstrate that the **"pattern collapse" occurs on generally all types of mathematical reasoning tasks**.

---

General Rebuttal: Universality of "Early Stabilization" phenomenon in mathematical reasoning (Expanded visualization of Figure 3 in the paper)
---
> Setup

**We present detailed visualization of the "early stabilization" phenomenon (detailed version of Figure 3 in the paper), which contains comparisons of trajectory volatility (embedding differences between neighboring layers) curves and sample standard deviation (color shading) between ID and OOD samples.**

The language model is Llama2-7B (32 layers, so 31 neighboring layer-pairs).  The ID data is the MultiArith dataset (Arithmetic domain, primary difficulty), and the OOD data consists of 10 datasets from different tasks and difficulties (identical to the setup in Section 4.1):

* Near-shift OOD:  GSM8K, SVAMP, AddSub, SingleEq, SingleOp (Arithmetic domain;  middle-school difficulty)
* Far-shift OOD: MATH-Algebra/Geometry/Counting_and_Probability/Number_Theory/Precalculus (algebra, geometry, counting-and-prob, number-theory, precalculus domain); university difficulty)

> Visualization and Corresponding Statistics Data

***{The visualization figures of all datasets are shown in PDF Figure 1.}***

***{The average volatility statistics for layers 1-31 (full layers), 20-31, and 26-31 on each dataset are shown in PDF Table 1.}***


> Analysis

From the detailed visualizations and statistical data, we can conclude that:

* Dataset Level: 
    * ID samples all show the "Early Stabilization" phenomenon compared to OOD math problems of different types and difficulties, because the average volatility in 26-31 layers of the ID dataset is significantly lower than that of OOD datasets;
    * The mid-to-late volatilities on the far-shift OOD dataset are more dramatic due to the greater deviation from the distribution of the ID dataset.
* Sample Level: Trajectory volatility can vary somewhat across samples, but in general, ID samples have less trajectory volatility than OOD samples, and in the mid-to-late layers, ID samples generally complete the main inference, while OOD samples still have large learning volatility overall.

> Conclusion 

**These detailed visualizations and analyses can demonstrate the generalizability of the "early stabilization" phenomenon in mathematical reasoning scenarios.**

Meta-review: This paper proposes a OOD detection method for large language models in mathematical reasoning. The problem of OOD detection in the context of mathematical reasoning tasks has not been investigated although robust OOD detection is important for real-world deployment of LLMs. The originality of the problem setting is high. The proposed method based on trajectory volatility is reasonable. The extensive experimental results demonstrate the effectiveness of the proposed method well. There are several points that should be improved in this paper, which include the mathematical writing, and clarity of the motivation. The paper should be revised incorporating the results presented in the rebuttal and the discussion.
Decision: Accept (poster)