Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning

Authors: yiming  wang
pei  zhang
baosong  yang
derek f wong
zhuosheng  zhang
rui  wang
shanghai  jiao
tong  university
tongyi  lab
  nlp
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-25 05:41:31

Abstract: Real-world data deviating from the independent and identically distributed (\textit{i.i.d.}) assumption of in-distribution training data poses security threats to deep networks, thus advancing out-of-distribution (OOD) detection algorithms. Detection methods in generative language models (GLMs) mainly focus on uncertainty estimation and embedding distance measurement, with the latter proven to be most effective in traditional linguistic tasks like summarization and translation. However, another complex generative scenario mathematical reasoning poses significant challenges to embedding-based methods due to its high-density feature of output spaces, but this feature causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces. Hence, we propose a trajectory-based method TV score, which uses trajectory volatility for OOD detection in mathematical reasoning. Experiments show that our method outperforms all traditional algorithms on GLMs under mathematical reasoning scenarios and can be extended to more applications with high-density features in output spaces, such as multiple-choice questions.

Main paper: 
 introduction: 
 The rapid development of generative language models (GLMs) [40, 41, 5, 2, 50] has empowered them to fit diverse and challenging datasets, showing strong generalization over in-distribution (ID) test data satisfying the independent and identically distributed (i.i.d.) assumption. However, unconstrained inputs in real-world settings frequently trigger distributional drifts, called out-ofdistribution (OOD) data. In such scenarios, model performance often deteriorates unexpectedly, yielding harmful outcomes. Thus, OOD detection [38, 4] is critical in safeguarding model security.A highly scalable detection method must not rely on specific OOD data distributions, so simulating scenarios where OOD data is unavailable is more promising. Most existing research mainly focuses on vision and text classification tasks [12, 30, 55, 26, 48] . In contrast, studies addressing OOD detection on GLMs remain relatively niche, despite the more severe risks associated with OOD perils in GLMs due to the potential for error propagation in autoregressive generated sequences [43] . Existing methods on GLMs only focus on traditional text generation scenarios like summarization and translation, and these methods did not step outside the research framework of uncertainty estimation [33, 31] and embedding distance measure [43] . Among these, [43] has demonstrated that embeddingbased methods are currently the only optimal solution for text generation scenarios, they determine whether a new sample is ID or OOD by calculating the Mahalanobis Distance [22] between the new sample embedding and ID embedding distribution in the static input or output space.Recently, mathematical reasoning has emerged as a challenging generative task and a crucial benchmark for evaluating model abilities. However, it presents unique phenomena in both input and output spaces that render embedding-based methods inapplicable, as shown in Figure 1 : Figure 1 : Embedding projection and cases of input and output spaces under mathematical reasoning and text generation scenarios. We select MATH [6] dataset for mathematical reasoning and OPUS [49] for text generation, each with four diverse domains. Different colors represent different domains, with lighter and darker shades indicating input and output. We use SimCSE [9] for sentence embeddings and UMAP [34] for dimensionality reduction. Appendix B shows detailed settings and examples.• Input space of mathematical reasoning exhibits vague clustering features across various domains, in contrast to the more defined clusters observed in text generation. This indicates that embedding may struggle to capture the complexity of mathematical questions. • Output space of mathematical reasoning exhibits high-density characteristics with significant overlap between different domains. we call this phenomenon Pattern Collapse. Since the output is mathematically symbolic [53, 37] , it compresses the search space, increasing the likelihood of overlap between questions from disparate domains (As cases of Figure 1a ). More importantly, the sequence tokenization used in GLMs allows for substantial token sharing among mathematically distinct expressions, as these mathematical tokens are primarily drawn from digits 0-9 and finite special symbols such as decimal points and square roots, rather than diverse linguistic elements. These scalar outputs lack distinctive features associated with specific domain distributions.More discussion of the two phenomena, especially the "pattern collapse", will be presented in Section 6. Given the limitations of traditional methods, we aim to explore innovative OOD detection solutions in mathematical reasoning scenarios. We transform our focus from static embedding space to the dynamic embedding trajectory, this motivation stems from a theoretical insight, as shown in Figure 2 : "pattern collapse" causes the convergence of the trajectory endpoints of different samples, leading to significant trajectory differences across samples. In Section 2.1, we model and prove this hypothesis to elucidate the intuition of using trajectories as a measure. Subsequently, in Section 2.2, we perform empirical experiments to investigate the underlying causes of trajectory differences between ID and OOD samples. Our findings reveal a phenomenon we term Early Stabilization, wherein GLMs achieve primary reasoning in later stages for ID samples, a pattern not observed in OOD samples. This observation provides direct evidence for the rationale behind using trajectory as a measure.Based on these analyses, in Section 3, we propose the Trajectory Volatility detection algorithm (TV Score) for mathematical reasoning. Then in Section 4 and 5, we conduct extensive OOD detection experiments with diverse datasets and GLMs to validate the effectiveness of our method. We also tackle the challenging scenario of OOD quality estimation, which raises higher precision demands on the OOD scores. Results indicate that our method surpasses all traditional algorithms for GLMs under the mathematical reasoning scenario. Additionally, we demonstrate the extension of our method to a broader range of tasks with high-density features in output spaces, such as multiple-choice questions.Problem Statement: OOD Detection on GLMs. We start by formalizing GLMs. Let X and Y be the input and output spaces with P X and P Y be the marginal distributions for respective space, and P X ,Y is the joint data distribution defined over X × Y. GLMs are trained given input sequence x = x 1 x 2 ...x t ∼ P X of length t to autoregressively generate the next token in the corresponding output sequence y = y 1 y 2 ...y n ∼ P Y of length n over the likelihood model p θ (y|x) = n i=1 p θ (y i |y ≺i , x), where each x i and y i are taken from vocabulary V and θ is sampled from the parameter space Θ. The "pattern collapse" phenomenon only exists in mathematical reasoning scenarios, where two samples initially distant in distance will converge approximately at the endpoint after undergoing embedding shifts, and does not occur in text generation scenarios. This produces a greater likelihood of trajectory variation under different samples in mathematical reasoning.Assume that P X ,Y denote a distribution sufficiently different from P X ,Y , the goal of OOD detection in GLMs is to find a score function f (x, y, θ) for each sample and a threshold ϵ, which may rely on the features of X , Y, and Θ, to achieve a high discrimination accuracy goal:EQUATION2 Dynamic Embedding Trajectory Distinguishes ID and OOD Samples in Mathematical ReasoningWe begin by defining the embedding trajectory and its volatility features. Given a sample s, we put its input to the language model p θ with L layers, and the output sequence consists of T tokens. For the t-th token, we denote its output embedding at layer l as h t l , with each embedding a d-dimensional vector. Following the definitions in [43, 51] , we define the average embedding y l = 1 T T t=1 h t l as the sentence embedding at layer l. Then, the embedding trajectory is formed as a progressive chain of these embeddings:y 0 → y 1 → • • • → y l → • • • → y L-1 → y L .To measure the change magnitudes of the embedding trajectory in the latent space, we define two types of trajectory volatilities:• Dimension-independent volatility V I (s) ∈ R d : For sample s, we first obtain the embedding difference vector |y l -y l-1 | between each adjacent-layer pair l -1 and l. V I is the average of all differences across layers, it captures the local trajectory changes across individual dimensions:EQUATION• Dimension-joint volatility V J (s) ∈ R: For sample s, we first obtain the L2-norm of the embedding difference ||y l -y l-1 || 2 between each adjacent-layer pair l -1 and l. V J is the average of all such differences across layers, it captures the global changes in the trajectory:EQUATION)In this section, we will clarify our motivation: Why Dynamic Embedding Trajectory As The Measure? This stems from the phenomenon of "pattern collapse" in the output space, where the endpoints of different sample trajectories converge to a high-density region. This constraint potentially increases the likelihood of trajectory differences across samples. We will model and prove this theoretical intuition in Section 2.1 through the Dimension-independent volatility. After gaining insight into trajectory differences, we specifically explore the differences between ID and OOD sample trajectories. We will empirically investigate this in Section 2.2 through the Dimension-joint volatility. 
theoretical intuition: trajectory differences with higher likelihood: 
 Figure 1 has illustrated the "pattern collapse" phenomenon in the output space in mathematical reasoning scenarios. We abstract this phenomenon in Figure 2 , which compares the trajectory trend between different samples in the mathematical reasoning and text generation scenarios: In mathematical reasoning, when initial points of two trajectories are separated by any distance in the input space, they typically converge to a significantly closer distance in the output space after undergoing an embedding shift. However, in text generation, outputs from different samples may not exhibit this same convergence. This finding inspires the following theoretical intuition: Hard constraints on trajectory endpoints in mathematical reasoning allow for a higher probability of trajectory differences under different samples, as expressed by the key hypothesis:Hypothesis 1 In scenarios characterized by pattern collapse in the output space, the probability of trajectory volatility differences across samples increases.We now formalize this hypothesis. Assume that the output space is a Gaussian distribution N (c, Σ 2 ), so the output embedding y L ∼ N (c, Σ 2 ). For two different samples s i and s j , their dimensionindependent volatilities are V I (s i ) and V I (s j ), respectively. We want to show that when pattern collapse exists, the probability of V I (s i ) -V I (s j ) ̸ = 0 will increase. According to the pattern collapse features under different scenarios, we can constraint the y L as follows:• For mathematical reasoning with pattern collapse, Σ → O, so we approximate that y L ≡ c;• For text generation without pattern collapse,Σ = diag(δ 1 , δ 2 , • • • , δ D ) ̸ = O, so y L ̸ ≡ c.With such formalized constraints, we model our Hypothesis 1 as a main theorem:Theorem 2.1 (Main Theorem) We assume that {y l } L l=1 are all independent variables sampling from vector space R d . For different samples s i and s j , their embedding sets are {[y i ] l } L l=1 and {[y j ] l } L l=1 , respectively. The likelihood of trajectory volatility differences between s i and s j under mathematical reasoning scenarios is higher than that under text generation scenarios, which means:E {[yi] l } L l=1 , {[yj ] l } L l=1 ∼ U (R d ) {V I (s i ) -V I (s j ) ̸ = 0|[y i ] L , [y j ] L ≡ c} > E {[yi] l } L l=1 , {[yj ] l } L l=1 ∼ U (R d ) V I (s i ) -V I (s j ) ̸ = 0|[y i ] L , [y j ] L ∼ N (c, Σ 2 )Due to space limits, we present the Complete Proof in Appendix C. This demonstrates that when pattern collapse occurs, the probability of trajectory volatility differences across samples increases.The theoretical analysis in Section 2.1 provides an intuition for using embedding trajectory as the measure for distinguishing between different samples. However, the specific trajectory differences between ID and OOD samples remain unclear. Thus, we investigate this empirically in this section.We select eleven mathematical reasoning datasets: one for ID data and ten for OOD data. The MultiArith dataset serves as the ID data, while the OOD datasets include GSM8K, SVAMP, AddSub, SingleEq, SingleOp, and MATH. The latter encompasses five tasks across various mathematical domains: Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus. This selection includes varying task types and levels of difficulty. We first train a base Llama2-7B (32 layers) using the ID training set, then inference on the ID test set and all OOD test sets. Details regarding the datasets, model, and implementation can be found in Section 4.1.We measure the dimension-joint volatility for all samples from each ID and OOD test set. Specifically, we compute each adjacent-layer change magnitude ||y l -y l-1 || 2 , and connect all values as a change curve, with higher value means the higher volatility. Figure 3 shows ten curve comparisons, with each sub-figure including the ID data and one OOD data. We find that the change magnitude slightly until the 20th layer. After 20 layers, for ID data, the change magnitude is again suppressed after a few layers of inference, while for OOD data, the magnitude is maintained at a relatively high level.We term this phenomenon "Early Stabilization": For ID data, GLMs largely complete their reasoning in the mid-to-late stages, and simple adjustments are sufficient after that. However, for OOD data, GLMs can still not complete accurate reasoning at a later stage. They thus can only randomly switch to a specific output pattern, i.e., the scalar mathematical expression pattern. This provides strong evidence that using embedding trajectory for OOD detection may be effective.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: [Limitations] Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community.Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
tv score: trajectory volatility score for ood detection: 
 In Section 2.2, we have identified a significant difference in embedding trajectory volatilities between ID and OOD samples in mathematical reasoning. We now aim to leverage this phenomenon to develop a lightweight OOD detection solution tailored for mathematical reasoning scenarios.Inspired by static embedding methods that use the ID sample embedding cluster as the reference to measure the Mahalanobis Distance (MaDis) [22] , we similarly aim to use the ID sample trajectory cluster as the reference to measure the difference between them and a new sample trajectory. While measuring the difference between an embedding and an embedding cluster only requires a simple MaDis calculation, quantifying the difference between a trajectory and a trajectory cluster is less intuitive. Thus, we seek a transition idea starting from the static space Gaussian assumption. First, we obey the assumption under static embeddings [3, 43] to fit all ID embeddings at each layer l to a Gaussian distribution G l = N (µ l , Σ l ). Next, for a new sample with y l be its embedding at layer l, we map it to its MaDis f (y l ) : R d → R with G l as follows:EQUATIONFinally, we treat the MaDis differences |f (y l ) -f (y l-1 )| between adjacent layer-pairs l -1 and l as the adjacent-layer volatility of the new sample, and average all adjacent-layer volatilities as the final trajectory volatility score (TV Score S) of this sample:S = 1 L • L l=1 |f (y l ) -f (y l-1 )| . (TV Score)The anticipated trend is that when the two adjacent embeddings after MaDis mapping exhibit a greater difference, the embedding change between the two layers is more volatile compared to ID data. This enables us to identify new samples with trajectory volatility features that significantly deviate from those of the ID samples, thus increasing the likelihood that they are OOD samples.Furthermore, as a trajectory, outliers in the trajectory may significantly impact feature extraction [21, 28] . To mitigate this, we explore higher-order differential smoothing techniques to enhance trajectory smoothness. We first define the k-order embedding ∇ k y l and Gaussian distribution∇ k G l = ∇ k N (µ l , Σ l )for all l ≤ L -k based on the backward difference:EQUATIONwhereC i k = k! i!(k-i)! . Similarly, we mapping ∇ k y l to its MaDis ∇ k f (y l ) : R d → R with ∇ k G l : ∇ k f (y l ) = ∇ k y l - k i=0 (-1) k+i C i k µ l+k ⊤ k i=0 C i k Σ l+k -1 ∇ k y l - k i=0 (-1) k+i C i k µ l+k . (6)Following the definition of TV Score, we define the trajectory volatility score after differential smoothing (TV Score w/ DiSmo ∇ k S) as follows:∇ k S = 1 L • L l=1 ∇ k f (y l ) -∇ k f (y l-1 ) . (TV Score w/ Dismo)The Algorithmic Process and Computational Complexity are detailed in Appendix D.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
setup: 
 Dataset Selection. For the ID dataset, we use the MultiArith [44] , which consists of Math Word Problems on arithmetic reasoning. For the OOD datasets, we intuitively introduce two types of detection scenarios following [43] : (i) Far-shift OOD scenario, we select the MATH [11] dataset with five domains of algebra, geometry, counting and probability, number theory, and precalculus; (ii) Near-shift OOD scenario, we select five independent datasets: GSM8K [6] , SVAMP [39] , AddSub [13] , SingleEq [18] , and SingleOp [18] . We consider the ID data negative(-) and the OOD data positive(+). Refer to Appendix E.1 for basic information and OOD features of these datasets. Implementation. To measure the application value of our method used in cutting-edge GLMs, we use Llama2-7B [50] and GPT2-XL (1.5B) [5] as our backbones for ID dataset training. Refer to Appendix E.3 for training details. However, there exists uncertainty about the data used in the pre-training phase, especially for Llama2 because its data is closed-source. Some research [57, 59] have confirmed the absence of data leakage in Llama2 for the MATH and GSM8K datasets, we still conduct pre-experiments to examine the rationality of the OOD data selection rigorously. Our criterion is the claim that a dataset can be categorized as OOD if it exceeds the capabilities of the base model, as proposed in prior studies [47, 27] . Results of the pre-experiments are shown in Appendix E.4, and they can confirm that these datasets can be considered as OOD data for the two GLMs.Baseline. We compare our method with five training-free baselines where OOD training data are unavailable. We refer to the latest survey [20] to select them for the scarcity of OOD detection methods on GLMs: (1) Maximum Softmax Probability (Prob.) [12] ; (2) Monte-Carlo Dropout [8] ; (3) Sequence Perplexity [3] ; (4) Input Embedding [43] ; (5) Output Embedding [43] . Refer to Appendix E.5 for details. Additionally, we set the smoothing order k ranges from 1 to 5 for TV Score w/ DiSmo and report the highest among them when with smoothing.Evaluation. We divide the OOD detection evaluation into two scenarios: (1) Offline Detection, which classifies whether samples from a given list belong to OOD. For each collection {D in , D out }, we report the AUROC [35] and FPR95 metrics. The former represents the area under the ROC curve and the latter represents the value of FPR at 95% TPR. (2) Online Detection, which utilizes the offline detection results to calculate an optimal classification threshold for a direct determination of whether new samples belong to OOD. We introduce the metrics in the corresponding result sub-sections.Offline Detection. Table 1 presents the results of the offline detection scenarios.• Performance Analysis: In the far-shift OOD setting, our average performance surpasses 98 (Llama2-7B) and 96 (GPT2-XL) under the AUROC metric, surpassing the optimal baseline by 10+ points. Moreover, our performance stands at an impressive 5.21 (Llama2-7B) and 9.89 (GPT2-XL) under the FPR95 metric, representing a remarkable 80%+ reduction compared to the optimal baseline, far surpassing all baseline methods. In the near-shift OOD setting, the robustness of our method is even more impressive. All of the baseline methods show significant performance degradation, especially in Llama2-7B, with the AUROC metric decreasing below 60 and the FPR95 elevating above 80. However, our method maintains excellent performances, with AUROC scores surpassing 90 and FPR95 below 30. This indicates that for more fine-grained OOD detection scenarios, our method demonstrates greater adaptability.• Model Analysis: Comparing performances of Llama2-7B and GPT2-XL, we find two phenomena: (i) Results on GPT2-XL are more stable, performance differences between GPT2-XL on farand near-drift settings are not significant, while Llama2-7B shows a significant performance Table 1 : AUROC and FPR95 results of the Offline Detection scenario. Underline and bold denote SOTA among all baselines and all methods, respectively. We report the average results under each setting in the main text, results of each dataset are shown in Table 11 and 12 degradation (mainly for baselines) on near-shift setting; (ii) the DiSmo technique is more effective on GPT2-XL, which suggests that there are more anomalous learning tendencies in latent spaces of small models, and the smoothing helps to minimize these anomalies.(Appendix F). Model Llama2-7B [50] GPT2-XL [5] Method Metric Far-shift OOD Near-shift OOD Far-shift OOD Near-shift OOD AUROC ↑ FPR95 ↓ AUROC ↑ FPR95 ↓ AUROC ↑ FPR95 ↓ AUROC ↑ FPR95 ↓ Max SoftmaxIn addition, we conduct significant tests (Details are shown in . We find that our methods almost pass all significance tests, while the embedding-based methods have the lowest pass rate among baselines, suggesting that their results are more susceptible to sampling error. We also find that the performance of differential smoothing fluctuates greatly in different settings. Therefore, we conduct the ablation of the smoothing order k. Results and analyses are shown in Appendix F.1.Online Detection. In this part, we utilize the TV score for online OOD discrimination. For each collection {D in , D out }, we obtain a detector and computer the optimal cut-off τ i of Youden Index, which is at the point in the AUROC curve where TPR -FPR is maximum. Then for all OOD samples s ∈ D out -D out , we donate t as the sampling size and computer the discrimination accuracy:EQUATIONIn addition, the discrimination accuracy should vary less under different data collections, reflecting the discriminator's robustness. Therefore, we denote the Robustness metric as sampling variance.Table 2 presents the results in Llama2-7B. Compared to the embedding-based methods, our TV score obtains about an average of 20-point accuracy improvement in both far-shift OOD and near-shift OOD settings, and on some datasets, such as Cnt.&Prob, our TV score achieves more than 40 points of improvement. These all imply that TV Score can perform online discrimination of OOD samples more accurately. In addition, our TV score also possesses stronger robustness, which means that in real scenarios, we can find the optimal threshold more consistently in the face of different accessible ID and OOD data, reducing the potential riskiness due to uncontrollable data acquisition.5 Generalizability Exploration 
beyond detection: ood quality estimation: 
 In this part, we utilize the TV score for generative quality estimation (QE). For text generation, the QE performance is usually measured by calculating the correlation coefficient between automatic scores and human ratings. However, QE in mathematical reasoning scenarios is not a well-defined problem. For one mathematical question, its answer is either right or wrong, the intermediate state does not exist. For example, when the correct answer is 12.5, it is difficult to judge which is better between generated answers of 1.25 and 13.6. The human approach may be to judge by comparing the difference value or similarity like Rouge [24] and BertScore [60] between the generated answer and the correct answer, which is unfair to the machine because there is a lot of randomness in the intermediary process of computation, and the solution pattern of machines is case-based [14] , so it is not suitable to judge the machine-generated results with customized mathematical rules.Therefore, we use the binary direct matching1 to compare the model-generated answers with the correct answers. Considering the open-ended output of the GLMs, we give a loose matching condition, i.e., as long as the correct answer is included in the generated answer by the model, the generated answer is recognized as correct and the matching score is 1, otherwise the matching score is 0. We compute the Kendall rank correlation coefficient τ [45] and Spearman rank correlation coefficient [36] between each OOD score and the matching score.Table 3 : OOD Quality Estimation: Kendall's τ and Spearman correlation between various OOD scores and benchmark quality metric binary matching. Each column shows the correlation when ID and OOD samples are merged. Underline denotes the SOTA among all baselines, and bold denotes the SOTA among our methods. We report the average results under each setting in the main text, results of each dataset are shown in Table 13 and 14 (Appendix F). Table 3 presents the results. For Llama2-7B, when compared with Kendall correlation, the correlation improvement of TV scores over SOTA baselines reaches up to 100% under both far-shift and near-shift OOD settings. Compared with Spearman correlation, TV scores demonstrate a correlation enhancement over SOTA baselines by up to 100% under far-shift OOD setting and 30% under near-shift OOD setting. GPT2-XL also demonstrates excellent performance. These findings indicate that our TV scores not only facilitate the binary discrimination of ID and OOD samples but also substantially reflect the quality and precision of generated mathematical reasoning.Model Llama2-7B [50] GPT2-XL [5] Method Metric Far-shift OOD Near-shift OOD Far-shift OOD Near-shift OOD Kendall ↑ Spearman ↑ Kendall ↑ Spearman ↑ Kendall ↑ Spearman ↑ Kendall ↑ Spearman ↑ Max SoftmaxApart from mathematical reasoning, our method also has a wider range of potential applications that can be extended to any task where the output space exhibits the pattern collapse property. An example would be multiple-choice questions, which is a popular evaluation tool in the era of large language models and also display the pattern collapse property due to the limited output space being confined to the "ABCD" four options. To verify the generalizability of our method, we conduct experiments using the multiple-choice dataset MMLU [10] , and our method also outperforms all traditional algorithms in this setting. Results and analyses are shown in Appendix F.4. 
rethink inapplicability of static embedding methods: 
 In Figure 1 , we find that the static embedding space for both input and output fails to capture distinct features across different domains. This renders traditional embedding-based methods unsuitable for mathematical reasoning tasks. Experimental results in Section 4 also verify this disadvantage. In this section, we conduct a detailed analysis of these phenomena, and reveal the root cause of the poor performance of static embedding methods in mathematical reasoning scenarios.The input space in mathematical reasoning exhibits vague clustering features compared to text generation scenarios as shown in Figure 1 . We speculate that as semantic representations, embeddings cannot accurately measure mathematical expressions in the mathematical sense. We begin by considering a counter-intuitive toy example. We construct a benchmark expression "A: 2+3+4=?" and three diverse expressions: "X: 8-0+9=?", "Y: xˆ2*yˆ3*xˆ4=?", and "Z: 234*2+345*4+456*4-243*3=?".From the mathematical sense, X is ID for A, whereas Y and Z are OOD for A since the difficulty and knowledge used to solve Y and Z are largely different from A compared to X. Table 4 presents the cosine similarity between each of them and A. Notably, X exhibits the lowest similarity to A, primarily due to having fewer token sharing with A. This is inconsistent with human perception of mathematics. To demonstrate this phenomenon, we use one of five domains from the MATH dataset as the ID and the remaining four domains as OOD. We rotate this setting across five iterations. Under each ID-OOD pair, we allocate half of the ID samples to compute the ID embedding distribution, while the other half and all OOD samples constitute the test samples. The Mahalanobis distance is then calculated between embeddings and the ID distributions for each sample input, yielding the OOD score. We employ SimCSE [9] to generate sentence embeddings. Table 5 (a) presents the AUROC score matrix, in nearly half of the settings, the AUROC value hovers around 50, i.e., completely random. This suggests that embeddings can not distinguish between domains in a mathematical sense.The output space in mathematical reasoning exhibits a unique high-density characteristic, which we call "pattern collapse". As shown in Figure 1 , the distribution of output embeddings across various domains in the mathematical reasoning scenarios is significantly concentrated and indistinguishable.Two reasons cause this phenomenon: (1) Expression level: Output space of mathematical reasoning is scalar [37] , resulting in a compressed search space with a higher probability of overlap in two widely divergent questions. For example, "1 + 3 =" and " tokenization, two expressions that are very different in the mathematical sense (e.g., Two numbers that are far apart on the real number line) may share many tokens because regular mathematical expressions are only taken from 0-9 number tokens and a limited number of special symbols, such as decimal points, slashes, and root signs. Thus, the collapse occurs at the token level during the autoregressive prediction of each token. EQUATIONWe also test translation and summarization tasks by taking samples with the same token size as the mathematical reasoning dataset for a clear comparison. Table 6 presents the statistics data, we find that: (i) The average token duplication rate was 98.9% on all math tasks, and even a staggering 99.9% on some easy arithmetic tasks; In contrast, the token duplication rate on the text generation task is only about 60%, with about 2000 token types, and still increasing as the total number of tokens increases. These data and comparisons demonstrate that pattern collapse occurs on mathematical reasoning and not on text generation. (ii) Token repetition rate exceeded 97% on all seven math tasks of different difficulties and types. From these conclusions, we can demonstrate that the "pattern collapse" occurs on generally all types of mathematical reasoning tasks.A straightforward approach to addressing "pattern collapse" in the output space is to leverage chainof-thought (CoT) techniques [54, 17, 62, 52, 61] to expand the output space size. Likewise, we adopt the solution steps associated with each sample in the MATH dataset as the output and employ SimCSE to derive embedding representations. The experimental setup aligns with Section 4.1, and the results are shown in Table 5 (b). We note a similar phenomenon as in Table 5 (a), i.e., the detection accuracy under different ID-OOD pairs varies greatly, and thus the detection randomness is more pronounced. This suggests that despite that CoT expands the output space size, the output answer is still essentially related to the difficulty and digit of mathematical reasoning, and the semantic embedding representation cannot reflect these features accurately. 
conclusion: 
 We propose the TV Score, a lightweight OOD detection method for mathematical reasoning, which distinguishes between ID and OOD samples by the embedding trajectory volatility in the latent space. We identify bottlenecks in OOD detection for mathematical reasoning and prove them empirically and theoretically. Experiments show that our method substantially outperforms all traditional algorithms, and can be extended to more application scenarios beyond mathematical reasoning.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: [Section 4, Appendix F] Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: [Ethics Statement] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: [Section 4, Appendix E] Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: Our limitation mainly lies in the relatively small sizes of datasets used in our experiments. Due to the difficulty of collecting and labeling mathematical reasoning data, dataset sizes in this field are generally small, mostly in the hundreds or thousands, making it difficult to obtain millions of data for training and reasoning as in translation or summarization tasks. To address this, we adopt test sampling to reduce the randomness under small-scale testing and mitigate the data imbalance.Table 7 : Examples of inputs/outputs from different domains in the mathematical reasoning scenario.Suppose that f is a function and f -1 is the inverse of f . If f (1) = 2, f (2) = 6, and The product of the proper positive integer factors of n can be written as n (ax+b)/c , whereEQUATIONx is the number of positive divisors n has, c is a positive integer, and the greatest common factor of the three integers a, b, and c is1. What is a + b + c? 1 Domain 4: Precalculus Simplify 1 1 -tan 2 x + 1 1 -cot 2 x .1 Table 8 : Examples of inputs/outputs from different domains in the translation scenario.The earth yielded grass, herbs yielding seed after their kind, and trees bearing fruit, with its seed in it, after their kind; and God saw that it was good. The White House Coronavirus Task Force was established on January 29.于是地发生了青草，和结种子的菜蔬，各从其 类；并结果子的树木，各从其类；果子都包着 核。神看着是好的。Task Force)是成立于2020年1月29日的以应对美 国COVID-19疫情的工作组。 Embedding Interpolation. We assume theF i (x) = [F i1 (x), F i2 (x), ..., F id (x)]⊤ : R → R d with d independent components fits the L coordinates, representing a continuous learning trajectory for s i , so [y i ] l = F i (l). F i (x) is taken from the functional space X, so {F i (l)} L l=1 are all independent variables. We constrain each component function of this F i (x) to satisfy the m-order (m ≥ 3) derivability property. Under this setting, the definition of dimension-independent trajectory volatility (Eq. 2) equates toEQUATIONModeling. We observe the "pattern collapse" phenomenon in the output space in Figure 1 . We abstract this phenomenon in Figure 2 , which compares the trajectory trend between different samples in the mathematical reasoning and text generation scenarios.EQUATIONAccording to the pattern collapse property under different tasks, we can constraint the endpoint embedding F i (L) in the output space:• For mathematical reasoning with pattern collapse, Σ → O, so we approximate that F i (L) ≡ c;• For text generation without pattern collapse,Σ ̸ = O, so F i (L) ̸ ≡ c.With such constraints, we model the main theorem:Theorem C.1 (Main Theorem) For different samples s i and s j , the likelihood of variations in trajectory volatility under mathematical reasoning scenarios is higher than that under text generation scenarios, which means:E {Fi(l)} L l=1 , {Fj (l)} L l=1 ∼ U (R d ) {V (s i ) -V (s j ) ̸ = 0|F i (L), F j (L) ≡ c} > E {Fi(l)} L l=1 , {Fj (l)} L l=1 ∼ U (R d ) V (s i ) -V (s j ) ̸ = 0|F i (L), F j (L) ∼ N (c, Σ 2 ) ,where U(R d ) denotes the uniform distribution defined in d-dimensional real number space.Next, we move on to the formal proofs. We begin with some propositions and lemmas that will be useful in the main theorem.Proposition C.1 (Lagrange Remainder Term) In the Taylor Expansion expressionf (x) = f (a) + df dx (x -a) + 1 2! • d 2 f dx 2 (x -a) 2 + • • • + 1 n! • d n f dx n (x -a) n + R n+1 (x), the remainder R n+1 (x) has the following property:|R n+1 (x)| = 1 (n + 1)! • d n+1 f dc n+1 • (x -a) n+1 ≤ M (n + 1)! • (x -a) n+1 ,where c ∈ [a, x] or c ∈ [x, a], and M = supd n+1 f dξ n+1 : ξ ∈ [a, x] or ξ ∈ [x, a] > 0.Proof. We consider the case of a < x with a > x identically. We use the Fundamental Theorem of Calculus (FTC) for the most basic expansion of f (x):EQUATIONContinue to use the FTC to expand the derivatives in integrals:EQUATIONTherefore, the generalized remainder is known asEQUATIONWe let m n+1 = min t∈[a,x]EQUATIONAccording to the Lagrange's Mean Value Theorem, there must exist a number c ∈ [a, x] withEQUATIONthis gives usEQUATIONand the equation on the left side of the proposition is proved completely. The inequality on the right-hand side is clearly established by the Lagrange's Mean Value Theorem. □ Lemma C.1 (Error Bound for the Midpoint Rule) Suppose that f (x) is a m-th (m ≥ 2) order differentiable function on the interval (-∞, +∞), andK = sup d 2 f dx 2 : x ∈ [a, b] ∈ R, then b a f (x) dx -(b -a)f a + b 2 ≤ K 24 (b -a) 3Proof. We do the first order Taylor Expansion for f (x) at the midpoint x = a+b 2 of the intervalEQUATIONFollowing Theorem C.1, we do the inequality scaling as below:b a R 1 (x) dx ≤ b a K 2! x - a + b 2 2 dx = K 6 b -a 2 3 - a -b 2 3 = K 24 (b -a) 3 (18) □ Lemma C.2 (Differential-Integral Error Order Estimation) Suppose that f (x) is a m-th (m ≥ 3) order differentiable function on the interval (-∞, +∞) and K i = sup d i f dx i : x ∈ (-∞, +∞) < +∞ (i = 1, 2, • • • , m), then on the closed interval [a, b], we have A := L i=1 f a + b -a L i -f a + b -a L (i -1) - b a df dx dx ≤ R(L) ∼ o( 1 L ).Proof. For each sub-interval a + b-a L (i -1), a + b-a L i , we perform a Taylor Expansion at the midpoint x = a + (i -1 2 ) to obtain the following expression:EQUATIONFollowing Lemma C.1, we do the inequality scaling for the first-order derivative summation:EQUATIONEQUATIONFinally, we formally prove our main theorem C.1 in this sub-section, thereby verifying the Hypotheses 1 in the main paper. Since the d components of V (s i ) -V (s j ) are independent of each other, we only consider the d-th dimensional component and the rest of the components V d (s i ) -V d (s j ) can be proved in the same way.Proof. According to Lemma C.2, we can approximateEQUATIONNow we want to remove the absolute value of the integrated function. Due to the continuity of the first-order derivative, there must be several zero points that are not extreme points, dividing the domain of function definition into several open intervals with alternating constant positive and negative function values. We first define such a set of zeros on the domain of definition D = [1, L]:EQUATIONFor all zero points i t ∈ X i or j t ∈ X j , they must satisfy the following properties to ensure that they are not extreme points:EQUATIONand j t is the same as i t . Therefore, in the sub-interval (1, i 1 ), (i 1 , i 2 ), • , (i p , L), F id (x) is alternately constant positive and constant negative, and the same as F jd (x).We assume that the first interval (1, i 1 ) and (1, j 1 ) are constant positive intervals (the same can be proven for constant negative intervals). In this setting, we can continue to simplify Eq.22:EQUATIONSince F id (x) and F jd (x) are taken from the functional space, {F id (1), {F id (i k )} p k=1 } and {F jd (1), {F jd (j k )} q k=1 } can be seen as independent variables. We letEQUATIONthen we can rewrite Eq.25 to the matrix form:EQUATIONwhere A is the coefficient matrix, and x is the unknown variable vector:EQUATIONWe let i p ≡ j q (mod 2), and return to the condition in the mathematical expectation on both sides of the inequality in the main theorem C.1 for the following categorical discussion:• If F id (L) = F jd (L) = c d , then c = 0, so V d (s i ) -V d (s j ) = 0 =⇒ Ax = 0; • If F id (L), F jd (L) ∼ N (c d , δ 2 d ), then c ̸ ≡ 0, so V d (s i ) -V d (s j ) ̸ = 0 =⇒ Ax = -c.We denote N (A) as the solution space of {x|Ax = 0} and P (A) as the solution space of {x|Ax = -c}. For x ∈ P (A), x = x p or x = x p + x n where x p is the special solution and x n is the solution in zero space, i.e., x n ∈ N (A). Since rank(A) = 1 < p + q + 2, the special solution x p exists, so the size of the solution space P (A) is larger than N (A). In this case, when the variable x is sampled from the real number space, its probability of being in N (A) is smaller, namely its probability of not being in N (A) is larger. This is exactly equivalent to the form of the proof of Main Theorem C.1, so the proof is complete. □During the proof of the main theorem, we can unlock a hidden conclusion due to the embedding interpolation: GLMs with a larger number of hidden layers may achieve more stable detection performance, i.e., discrepancies in embedding volatility of different samples will be more obvious.In Lemma C.2, we have proved the upper bound of differential-integral error order estimation is the equivalent infinitesimals of 1/L. Actually, when L → +∞, we haveEQUATIONwhich is clear according to the definition of Riemann Sum. This means that when the value of L increases, the differential-integral approximation error will be reduced, so the conclusion of the main theorem will be more accurate. In our experiments, we use GPT2-XL (48 layers) and Llama2-7B (32 layers) as training backbones and find that the average performance of GPT2-XL is higher and more stable than that of Llama2-7B, while the number of layers of GPT2-XL is 1.5 times that of Llama2-7B. This further validates the correctness of our extended conclusions.The pseudo-code of our TV Score computation pipeline is shown in Algorithm 1.Input: L: The number of hidden layers N : The size of ID dataset k: the smoothing differential order of TV score {y l } 1≤l≤L : the average hidden embedding of the OOD sample in each layer {[ ŷl ] i } 1≤l≤L,1≤i≤N : the average hidden embedding of all N ID samples in each layer 1: for l ← 1 to L do 2:Fitting ID samples {[ ŷl ] i } 1≤l≤L,1≤i≤N to Gaussian distribution G l = N (µ l , Σ l ) 3: f (y l ) ← (y l -µ l )⊤ (Σ l ) -1 (y l -µ l ) 4: end for 5: for i → 1 to k do 6:for l ← 1 to L do 7:∆ (i) G l = N (µ (i) l , Σ (i) l ) ← N ( i t=0 (-1) i+t C t i µ l+i , i t=0 C t i Σ l+i ) 8:end for 9:f (i) (y l ) ← y l (i) -µ l (i) ⊤ Σ (i) l -1 y l (i) -µ l (i)10: end for 11: for i → 1 to n do 12:TV i ← average f (i) (y l ) 1≤l≤L 13: end for Output:{TV i } 1≤i≤kAs for the computational complexity, once we have all y l , our TV score requires only scalar addition and multiplication. During the ID distribution fitting phase, both operations are O(Ldn), where n is the dataset size. During the score computation phase, both operations are O(Ldk). In practice, the computation time is measured in milliseconds.For the ID dataset, we use the MultiArith [44] , which consists of Math Word Problems on arithmetic reasoning. For the OOD datasets, we intuitively introduce two types of detection scenarios following [43] : (i) Far-shift OOD setting, we select the MATH [11] as the OOD data, which spans across distinct mathematical domains encompassing algebra, geometry, counting and probability, number theory, and precalculus. It contains college difficulty level math problems, whereas MultiArith has only elementary school difficulty, and thus can be considered as sourced from far-different distributions; (ii) Near-shift OOD setting, we select five arithmetic reasoning datasets as the OOD data: GSM8K [6] , SVAMP [39] , AddSub [13] , SingleEq [18] , and SingleOp [18] , they all consist of Math Word Problems like the MultiArith but require different reasoning hops and knowledge points for solving the problems, and thus can be considered as sourced from near-different distributions. In addition, we present the data sizes and examples of all ID and OOD datasets, as shown in Table 9 .For the ID dataset MultiArith, we find that every 100 consecutive samples show a similar quadratic operation pattern (e.g., samples with id 0-100 are a mixture of addition and subtraction, and id 100-200 are a mixture of addition and multiplication). Therefore, we divide it into 6 subsets (6*100) in order. In each subset, we take the first 60 as training samples and the last 40 as test samples.We train Llama2-7B [50] and GPT2-XL [5] models on the training split of MultiArith. Llama2-7B is trained with AdamW optimizer [29] for 10K steps and 8 batch sizes in 4-card RTX 3090 (2 per card). The learning rate is set to 1e-5, the warmup step to 10, and the maximum gradient normalization to 0.3. GPT2-XL is trained for 3K steps and 128 batch sizes in a single RTX 3090, and other configurations are the same as Llama2-7B.In this part, we examine the rationality of the OOD data selection, ensuring that the OOD data distribution significantly differs from the pre-trained data distribution and has not been fully learned during the pre-training phase. Some research [57, 59] have confirmed the absence of data leakage in Llama2 for MATH and GSM8K datasets, we still conduct experiments and analyses to ensure this. For GPT2-XL, We can determine this from the time dimension. GPT-2 was released in 2019, but the MATH and GSM8k datasets were released in 2021, so they are unlikely to appear in the pre-training data. However, for Llama2-7B, due to the closed-source data, we cannot confirm which data the model used in the pre-training phase, so we cannot fully determine whether the selected dataset was OOD for the model from a data perspective.Therefore, we argue that a dataset can be considered OOD when it is beyond the capability of the base model as claimed by prior work [47, 27] . We test all ten datasets we select as the OOD data in the pre-trained Llama2-7B and GPT2-XL model, and Table 10 shows the results. We find that GPT2-XL cannot handle any of the ten mathematical reasoning tasks, and Llama2-7B performs with very low accuracy. Therefore, from a capability standpoint, we can ensure that these datasets are OOD for these two GLMs.Let x represent the input sequence and y the output sequence, with lengths denoted as n x and n y respectively. In addition, we assume that p(•; θ) represents the GLM parameterized by θ that has been trained in ID dataset D, outputting a sequence of softmax probabilities. We compare some training-free baseline methods as below:• Maximum Softmax Probability [12] :EQUATION• Monte-Carlo Dropout [8] :EQUATIONwhere q(θ) ≈ p(θ|D). EQUATION• Input Embedding [43] :EQUATIONwhere µ x and Σ x represent the mean and variance of the Gaussian distribution associated with the first-layer hidden state. • Output Embedding [43] :EQUATIONwhere µ y and Σ y represent the mean and variance of the Gaussian distribution associated with the final-layer hidden state. Figure 4 : Smoothing order k analysis: k ranges from 0 -5 (k = 0 corresponds to the original TV Score). The upper part is for the OOD detection scenario and the lower part is for the OOD quality estimation scenario; the left part is for the far-shift OOD datasets and the right part is for the near-shift OOD datasets.In the main experiments, we found that differential smoothing is not as effective as the basic TV score, with excellent results occurring on only a few datasets. Figure 4 visualizes the results for the OOD detection scenario with no smoothing (k = 0) and smoothing order k 1-5. We find that there is a significant effect of differential smoothing in two cases: (1) very good performance without smoothing, e.g., the precalculus dataset, where the AUROC results are almost close to 100%; and (2) significantly poor performance without smoothing, e.g., the AddSub dataset, where the FPR95 results on all other near-shift OOD datasets are 20 below, the results on this dataset are more than 80, when differential smoothing helps to eliminate some of the noise features and helps better detection.In addition, for the case of k > 0, the peak performance basically occurs in the case of k = 1 or 2, which indicates that when k is too large, the phenomenon of over-smoothing tends to occur. Too much useful feature information is lost, leading to a decrease in detection accuracy. Our method has a wider range of application scenarios beyond mathematical reasoning. To verify generalizability, we choose the multiple-choice quizzing task, which has the same "pattern collapse" property as mathematical reasoning, since the output space is limited to the "ABCD" four options.We select the MMLU dataset [10] and choose eight domains among it: high school mathematics, high school physics, high school chemistry, high school biology, high school geography, high school government and politics, high school psychology, high school statistics. We test eight rounds, each using one of the domains as the ID dataset and the remaining seven domains as the OOD datasets.We We find that MS-Prob and PPL nearly fail on the multiple-choice task and the output embedding is not as excellent as expected, which is caused by the pattern collapse phenomenon.Our method is comparable to the input embedding method and has very good absolute performances.For some far-shift OOD scenarios, e.g., mathematics-psychology (I-VII), physics-politics (II-VI), etc., performances of the input embedding method and our method are basically the same, and there exists a reasonable range of competing phenomena, e.g., our method performs more advantageously under physics-biology (II-IV), and the input embedding method is better under physics-geography (II-V). For some near-shift OOD scenarios, e.g., mathematics-statistics (I-VIII), where both domains essentially belong to the category of math, our method will be more well-performed, indicating that the embedding-based method produces performance degradation in fine-grained scenarios, while our method possesses stronger robustness.Overall, our method is scalable and has greater advantages in fine-grained detection scenarios. Justification: [Abstract, Section 1] Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Although direct matching is the most accurate solution, it suffers from two issues: (i) Generated answers may include much noisy content, increasing the matching difficulty; (ii) Performances on GLMs of mathematical reasoning is poor, which unbalances the positive and negative samples and increases the randomness.xdx =" are both "4"; (2) Token level: We usually categorize different mathematical reasoning domains from a mathematical sense. However, GLMs model real numbers or mathematical expressions not in a mathematical sense, but based on the average embedding of all discrete token embeddings after sequence tokenization. Through 


Decision: This paper proposes a OOD detection method for large language models in mathematical reasoning. The problem of OOD detection in the context of mathematical reasoning tasks has not been investigated although robust OOD detection is important for real-world deployment of LLMs. The originality of the problem setting is high. The proposed method based on trajectory volatility is reasonable. The extensive experimental results demonstrate the effectiveness of the proposed method well. There are several points that should be improved in this paper, which include the mathematical writing, and clarity of the motivation. The paper should be revised incorporating the results presented in the rebuttal and the discussion.

Reviews: review: paper_summary: paper_summary: This work discusses a novel method for out-of-distribution (OOD) detection in generative language models (GLMs), particularly in the context of mathematical reasoning tasks. The key insights are: 1) The high-density output space in mathematical reasoning leads to a "pattern collapse" that causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces, 2) GLMs exhibit early stabilization for in-distribution (ID) samples in mathematical reasoning, while OOD samples do not show this behavior.
strength_weakness: strength_weakness: The underlying mechanism of the TV score method and its relationship to the observed "pattern collapse" in the output space is not fully explained. Providing a more detailed analysis and visualization of the trajectory dynamics, as well as the intuition behind the choice of trajectory volatility as the detection metric, would enhance the interpretability of the approach.

The computational complexity of the TV score method is not discussed, which is an important consideration for real-world deployment, especially in resource-constrained environments. Investigating the computational efficiency of the method and exploring potential optimizations or approximations would be valuable for improving its practical applicability.

The paper does not address the robustness of the TV score method to adversarial attacks, which is a critical consideration for the security of generative language models. Evaluating the method's performance under different types of adversarial perturbations and developing strategies to improve its robustness would be a valuable extension of this work.
questions: questions: What is the underlying mechanism of the TV score method, and how does it relate to the observed "pattern collapse" in the output space? How can providing a more detailed analysis and visualization of the trajectory dynamics, as well as the intuition behind the choice of trajectory volatility as the detection metric, enhance the interpretability of the approach?

How does the computational complexity of the TV score method impact its real-world deployment, especially in resource-constrained environments? What steps can be taken to investigate the computational efficiency of the method and explore potential optimizations or approximations to improve its practical applicability?

How robust is the TV score method to adversarial attacks, and what are the critical considerations for the security of generative language models? What steps can be taken to evaluate the method's performance under different types of adversarial perturbations and develop strategies to improve its robustness?
limitations: limitations: NA
paper_summary: paper_summary: This paper studies the OOD problem in GLMs under mathematical reasoning and found that the the patter collapse phenomena in the output space. The trajectory violation score is proposed to distinguish the ID and OOD samples. A thorough evaluation shows that the proposal can outperform traditional algorithms under offline detection, online detection, and quality estimation.
strength_weakness: strength_weakness: (-) The input data for the empirical study in Figure 3 is not introduced. Thus, whether it can reflect the scenario of mathematical reasoning is not clear. It lacks theoretical analysis. 
(-) The relationship of the pattern collapse (in Figure 1) and the early stabilization (in Figure 3) is not clearly illustrated. 
(-) The datasets for experiments are quite limited.
questions: questions: 1.  What is the internal relationship between the pattern collapse (in Figure 1) and the early stabilization (in Figure 3)?
limitations: limitations: The limitations are adequately addressed.
paper_summary: paper_summary: This paper presents a trajectory-based method for OOD detection in the mathematical reasoning setting. OOD detection is extensively studied in the text setting and image setting. The main motivation of this work is claimed to that mathematical reasoning poses significant challenges to embedding-based methods due to its high-density feature of output spaces, but this feature causes larger discrepancies in the embedding shift trajectory between different samples in latent spaces. The proposed method uses trajectory volatility for OOD detection in mathematical reasoning. Experiments are conducted to validate the performance of the proposed method.
strength_weakness: strength_weakness: The motivation is not strong and convincing. Figure 1 illustrates two challenges with respect to the input space and out space respectively. It is not convincing, and more evidence or data analysis should be provided to make these two challenges solid. I do not buy that pattern collapse is generally hold for all types of mathematical reasoning tasks. The example is just a special case. The output can take any value in the real line, and the collapse probability not that high. 

The idea of the proposed method is not convincing. I am not convinced that early stabilization is generally held for mathematical reasoning problems. More evidence should be provided to justify it. 

The writing is also not precise. For example, in Equation (1), the domain of f is not specified. The notation \phi is not defined. 

Also, the experiment is not solid. There is no experiment to justify that the performance improvement is due to addressing the challenges in the input space and output space posed in the Introduction.
questions: questions: See weakness.
limitations: limitations: See weakness.
paper_summary: paper_summary: This work studied the ODD detection in mathematical reasoning, which presents a new measurement, TV score, based on the observed pattern collapse property and early stabilization of GLMs. It seems the first discussion on the OOD detection in mathematical reasoning.
strength_weakness: strength_weakness: 1. Additional metrics commonly used for ODD should be included for evaluation.
2. The writing requires improvement.
3. Further insight into the potential impact of over-smoothing on the setting of the critical parameter $k$ is necessary.
questions: questions: 1. Is it true that the embedding dimension should be fixed across layers to compute the embedding difference between neighboring layers?
2. To better support the claim of choosing $k \leq 5$, it would be beneficial to provide a quantitative analysis of over-smoothing occurrence when $k$ is set to a larger value.
3. For testing the ODD performance, could you report the evaluation results in terms of AUPR and F1?
4. Which datasets were used for the analysis in Figure 3?
5. There are some grammatical mistakes that need fixing. Here are a few examples:
   (a) "we need ..., then computer ..." in Line 150.
   (b) The sentence in Lines 161-162.
   (c) "Outliers ..., then lead ..." in Lines 164-165.
limitations: limitations: The authors adequately claimed the limitations of this work.

Comments: title: Thanks for reviewing & Invite feedback
Thanks for recognition
Thanks for recognition
Thanks for response & Invite open discussion
Thanks for reviewing & Invite feedback
comment: Dear Reviewer SAjz,

With the reviewer-author discussion period coming to an end, we really hope that we can get your feedback, and that our responses adequately address your concerns.

Best,

Authors
Thank you for your reply. I have no further questions.
Dear Reviewer SAjz,

We want to express our sincere appreciation for your efforts and time spent reviewing our work and the constructive comments.

During the rebuttal period, we have provided a detailed response to address all the concerns mentioned by you about **mechanism, complexity, and robustness**. With the reviewer-author discussion period ending in one day, we kindly invite you to give some feedback on our responses, and we really hope our responses adequately address your concerns.

Best,

Authors
Thank you for supporting our work!
Thanks for your efforts to address my concerns. I stay positive for this work.
Glad to see that we have addressed most of your concerns and thanks for your recognition of our work!
Thanks for the clarification, and my concerns have been mostly addressed. As it is a very important research problem and many illustrations and experiments have been supplemented, I'm happy to increase my score.
Thanks for your response. Regarding *"consolidate the "Pattern Collapse" phenomenon, experiment finding, evaluation, etc."*, we have already provided a detailed response in the rebuttal stage, including clarification, statistics, visualizations, and quotations. Here we present some key evidence and conclusions again:

---

### 1. "Pattern Collapse" phenomenon (Evidence: Fact Clarification & Statistics)

First, **we have clarified that the language model's modeling of mathematical expressions is based on tokenization**, so "pattern collapse" occurs at the discrete token level, not at the level of mathematical sense. Understanding "pattern collapse" in terms of the real number line is misconceived, and it goes against the way language models are modeled.

Second, **we have given statistics** on seven different types of math tasks with different domains and difficulty levels, and have compared them to two classic text generation tasks, translation and summarization.
We mainly focus on the number of token types and the duplication rate, which can reflect how much the model collapses at the token level when predicting. We again present the results as follows:

|Types of Tasks|token number in dataset |token type number in dataset |Token Duplication Rate|Vocab Coverage|
|---|---|---|---|---|
|***Mathematical Reasoning***|||||
|Arithmetic (primary difficulty)|16136|14|99.9%|0.04%|
|Arithmetic (middle-school difficulty)|5663|16|99.7%|0.05%|
|Algebra|5234|107|98.0%|0.33%|
|Geometry|2615|75|97.1%|0.23%|
|Counting and probability|2524|43|98.3%|0.13%|
|number theory|2395|71|97.1%|0.22%|
|precalculus|3388|84|97.5%|0.26%|
|*Average* | *5422* | *58* | *98.9%*| *0.18%*|
|***Text Generation***||||
|Translation|2500 |1065 | 57.4% | 3.32% |
||5000 | 1832| 63.3% | 5.10% |
||10000 | 2980 | 70.2% | 9.31% |
|| 15000 | 3494 | 76.7% | 10.61% |
|*Average* | *5833* | *1959* | *66.4%*| *6.12%*| 
|***Summarization***|||| 
||2500 |1265 | 49.4% | 4.01% |
||5000 | 1970 | 60.6% | 6.16% |
||10000 | 3192 | 68.0% | 9.98% |
||15000 | 3876 | 74.1% | 12.11% |
|*Average* | *5833* | *2142* | *63.2%* | *6.69%*| 

**The average token duplication rate is up to 99% on all math tasks, and even a staggering 99.9% on some simple arithmetic tasks; The token duplication rate exceeded 97% on all seven math tasks of different difficulties and types.
This demonstrates that the "pattern collapse" occurs on generally all types of mathematical reasoning tasks.**

---

### 2. "Early Stabilization" finding (Evidence: Visualization)

In our GENERAL REBUTTAL, **we have given detailed visualizations** of trajectories across ten datasets of different domains and different difficulty levels in mathematical reasoning. 

We have also given the average volatility statistics for layers 1-31 (full layers), 20-31, and 26-31 on each dataset corresponding to the visualizations as follows:

|Dataset |1-31 layers |20-31 layers | 26-31 layers|
| --- | --- | --- | --- |
| *ID Dataset* | | | | |
| MultiArith | 6.53 | 14.84 | 10.89 |
| *Near-shift OOD Dataset* | | | | |
| GSM8K | 8.60 | 20.55| 26.43 |
| SVAMP | 8.02 | 18.82 | 24.68 |
| AddSub | 8.72 | 20.54 | 27.24 |
| SingleEq | 8.14 | 19.26 | 22.42 |
| SingleOp | 7.50 | 17.17 | 21.62 |
| *Far-shift OOD Dataset* | | | | |
| MATH-Algebra | 8.83 | 21.35 | 31.50 |
| MATH-Geometry | 10.00 | 25.27 | 34.14 |
| MATH-Count_and_Prob | 10.30 | 25.77 | 33.70 |
| MATH-Number_Theory | 9.40 | 23.10 | 33.86 |

**In all ten datasets, the phenomenon of "early stabilization" is significantly present, and it can be sufficiently demonstrated that "early stabilization" is universal in mathematical reasoning.**

---

### 3. Experiment Justification (Evidence: Quotation)

**We have explained your misunderstanding** about the purpose of our paper by quoting from the original text:

* The two "challenges" are faced by existing embedding-based methods that cannot be applied to mathematical reasoning, not by us. *As stated in lines 38-39, “However, embedding-based methods encounter challenges under mathematical reasoning scenario”.*

* We aim to circumvent the two challenges faced by existing methods, not to address them. We are not improving on existing methods, but rather two different research ideas. *As stated in lines 47-48, “Therefore, we transform our perspective from the static embedding representation to the dynamic embedding shift trajectory in the latent space".*

---

We strongly respect your concerns about the evidence we gave, and an open discussion always helps us to recognize our limited considerations. Unfortunately, we have not received targeted details of your concerns, can you tell us what it is about our evidence that you do not recognize? Your proposed valuable issues will largely help us to improve our paper quality, and we look forward to a more open discussion with you, thanks!
Many thanks for the clarification.  After reading the response, most of my concerns still remain such as consolidate the "Pattern Collapse" phenomenon, experiment finding, evaluation, etc.  This paper presents some interesting idea, but it needs many ideas to make it solid. Given this, I like to maintain my score.
Dear Reviewers,

We want to express our sincere appreciation for your efforts and time spent reviewing our work and the insightful and constructive comments.

During the rebuttal period, we have provided a detailed response to address all the concerns mentioned by all reviewers. 
With the reviewer-author discussion period half over, we kindly invite you to give some feedback on our responses, and we really hope our responses adequately address your concerns.

Best,

Authors