Title: Token Embeddings Violate the Manifold Hypothesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: token embedding

manifold hypothesis

statistical hypothesis test

point cloud
Abstract: A full understanding of the behavior of a large language model (LLM) requires our grasp of its input token space.
If this space differs from our assumptions, our comprehension of and conclusions about the LLM will likely be flawed.  We elucidate the structure of the token embeddings both empirically and theoretically. We present a novel statistical test assuming that the neighborhood around each token has a relatively flat and smooth structure as the null hypothesis. Failing to reject the null is uninformative, but rejecting it at a specific token $\psi$ implies an irregularity in the token subspace in a $\psi$-neighborhood, $B(\psi)$. The structure assumed in the null is a generalization of a manifold with boundary called a \emph{smooth fiber bundle} (which can be split into two spatial regimes -- small and large radius), so we denote our new hypothesis test as the ``fiber bundle hypothesis.'' By running our test over several open-source LLMs, each with unique token embeddings, we find that the null is frequently rejected, and so the evidence suggests that the token subspace is not a fiber bundle and hence also not a manifold. As a consequence of our findings, when an LLM is presented with two semantically equivalent prompts, if one prompt contains a token implicated by our test, the response to that prompt will likely exhibit less stability than the other.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript  introduces an analysis aimed at verifying if the points of a dataset can be meaningfully considered as contained in a manifold or in a fiber bundle. The statistical test is applied for each points separately, and is based on a null model in which the manifold is locally flat and uniformly populated. The approach is applied to the analysis to token emebeddings in four large language models, and it allows identifying, for each model, between 40 and 70 tokens which violate the manifold hypothesis with high statistical confidence. A smaller number of tokens violates even the fiber bundle hypothesis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: focusing on the geometric properties of the data set in the neighborhood of each point separately is interesting and brings insight on the properties of each single token. Indeed, the tokens violating the manifold hypothesis are listed in Supp Mat, and one can attempt to interpret why these specific tokens correspond to \"singularities\"\nWeaknesses:\n1) (minor) the geometric criterion at the basis of the approach (eq 1) is very similar to the approaches used from the seventies onward to estimate the fractal dimension of a dataset (for example the correlation dimension). It is well known that one can obtain  deviations from  a linear behavior of the log-log plot of the volume versus radius just because the probability density of the data on the manifold varies quickly. \n2) (major): the number of violations detected by the approach is very small, of the order of 0.1 % (according to table 1). Therefore, these results do not seriously challenge the idea that data representation are contained in a relatively low-dimensional manifold, also because, to the best of my knowledge,  nobody ever claimed that this manifold is a clean and smooth \"Riemanian\" variety. The fact that this manifold is not smooth everywhere is not a surprise.  \n3) (major): The first claim of section \"Experimental results\" is that  \"the local dimension is highly variable\".  Leaving aside the fact that even this result is not really novel, one  would like to see some numerical data supporting this claim.\n\nQUESTIONS:\n1) how does the intrinsic dimension  (ID) vary across the dataset?\n2) how does the values of the ID obtained by a linear fit compare with those obtained by other ID estimators? For example you should consider the Danco Estimator, or the approach https://www.stat.berkeley.edu/~bickel/mldim.pdf\n3) can you exclude that  the violations of the manifold hypothesis are due to variations of the density?\n4) if one would eliminate from the vocabulary the (few) \"singular\" tokens, substituting those by synonyms, would the data manifold become everywhere smooth?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose three hypothesis tests designed to assess the geometric structure of embedding subspaces associated with individual tokens. Specifically, they aim to determine whether a subspace can be characterized as (1) a manifold, (2) a fibered manifold, or (3) a fiber bundle. These tests are grounded in Theorem 1, which describes how the volume of a ball of radius $r$, centered at a point $\\psi$, changes for small versus large $r$. The authors apply their methodology to analyze token embeddings produced by large language models (LLMs).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* The paper addresses a timely and relevant problem.\n* The theoretical framework appears sound and well-motivated.\n* The conclusions challenge prevailing assumptions in the community.\n\n## Weaknesses\n\n* The paper's presentation is disorganized and often unclear.\n* The experimental section lacks detail and depth.\n* The empirical analysis is limited in scope.\n\n## Review Summary\n\nThis work introduces a novel and theoretically grounded approach for testing the manifold hypothesis in token embeddings derived from LLMs. The conclusions are intriguing and potentially impactful, as they challenge commonly held beliefs about the geometry of learned representations.\n\nHowever, the paper suffers from several major weaknesses. The organization and clarity of presentation are lacking---key intuitions and methodological explanations are buried in the appendix rather than integrated into the main narrative. Moreover, the experimental validation is limited both in scope and detail. The authors focus exclusively on LLM embeddings, ignoring other widely used embedding techniques such as Word2Vec or GloVe. Furthermore, while the proposed tests appear general, their empirical application is shallow, raising concerns about their validity and robustness.\n\nDespite the paper’s potential, I do not recommend it for publication in its current form. I suggest a **borderline reject**.\n\nThat said, I acknowledge that I may have missed or misunderstood certain aspects of the paper, and my confidence in this assessment is low. I would be open to revising my recommendation if the authors are able to address the concerns outlined below.\n\nRegardless of the final decision, I can see the potential of this work and I encourage the authors to continue their research in this direction.\n\nQUESTIONS:\n# Major Comments\n\n**1. Presentation and Organization**\nThe paper lacks clear structure, which significantly hinders readability. For example, Figure 3 is said to encapsulate the core methodology, yet its interpretation is non-trivial without additional context. The necessary intuition is relegated to the appendix, where it is unlikely to be seen early by readers. I strongly recommend relocating this explanation to the main text and integrating it into the broader narrative.\n\n**2. Volume Estimation in Figure 3b**\nIt is unclear how volume is defined or computed in Figure 3b. Lines 120–125 suggest that it may correspond to the number of tokens within a ball. If this is correct, the authors appear to be using token count as a proxy for manifold volume. This is a strong and potentially unjustified assumption. Can the authors provide a theoretical or empirical rationale for this estimation method?\n\n**3. Slope Thresholding**\nThe hypothesis tests seem to hinge on detecting slope changes in volume scaling. Does this involve setting a specific threshold? If so, how was this threshold determined? Why was this particular value chosen over others? The paper should clarify the methodology for threshold selection and discuss its sensitivity.\n\n**4. Experiments on Specific Token Subsets**\nThe authors’ conclusion---that LLM embeddings do not lie on a single coherent manifold---is intriguing, but more granular analysis is needed. For example, are certain semantic subsets (e.g., years from 1900–2000, color names, or month names) manifold-like? Exploring such structured subsets could strengthen the argument and help determine whether the negative results reflect global structure or just aggregate noise.\n\n**5. Comparison to Other Embedding Methods**\nThe empirical evaluation is limited to LLM embeddings. However, other widely used embedding techniques, such as Word2Vec and GloVe, also produce token representations. Applying the proposed tests to these methods could provide important comparative insights and help clarify whether the results are specific to LLMs or generalizable across embedding strategies.\n\n**6. Test Validity and Robustness**\nThe paper lacks validation of the proposed tests on synthetic data. For example, if the authors apply their methodology to points sampled from known manifolds (e.g., spheres, tori), do the tests recover the expected structure? How do noise and sampling density affect test accuracy? Exploring such controlled settings would provide valuable evidence for the tests’ robustness and reliability.\n\n---\n\n# Minor Comments\n\n* **Equation Formatting**: Equation (1) contains an unclosed parenthesis: $(e_*v)(B_r(\\phi)\\color{red})\\color{black}=\\dots$\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the presence of irregularities and singularities in the structure of token embedding spaces used by large language models. Treating the embedding space as a geometric object, the authors evaluate its conformity to the manifold hypothesis and the more general fiber bundle structure. They introduce statistical tests to identify tokens whose embeddings violate these hypotheses, revealing cases where the space is locally inconsistent or singularities. The authors also provide a theoretical justification for why such singularities can persist through the model, even with extended context, potentially affecting downstream behavior. Finally, they analyze patterns of geometric irregularity across multiple models, identifying patterns among tokens which violate the hypothesis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe paper presents an original and novel perspective on the geometry of token embedding spaces in large language models. By framing the embedding space through the lens of the manifold hypothesis and evaluating deviations via both manifold and fiber bundle tests, the authors introduce a methodology for identifying structural irregularities in token representations. This line of inquiry opens up potential avenues for systematically characterizing LLM embedding geometry and I appreciate the use of empirical analysis in topological and geometric theory is commendable.\n\nWeaknesses\nWhile the contribution is conceptually intriguing, the paper suffers from several issues that limit accessibility and clarity. Many of the key mathematical terms — such as manifold, fiber bundle, and singularity — are introduced without sufficient formal definition or intuitive explanation. This makes it difficult for readers unfamiliar with differential geometry (such as myself) to follow the central arguments. Additionally, some theoretical results, particularly Theorem 1 and Theorem 2, are introduced and referenced before they are clearly stated.\n\nFrom a presentation standpoint, several figures are difficult to interpret. I'm having a hard time understanding the significance of Fig1 and Fig2 bottom, the flowcharts in Fig2. \n\nFinally, a central concern is the scale and significance of the results. If interpreted table1 correctly, fewer than 70 tokens (out of ~30K–50K vocabulary) fail the manifold hypothesis test, and the number of fiber bundle violations, while higher, still represents a relatively small fraction (again if I understand table 1 correctly). While the authors argue these irregularities are meaningful, it's unclear whether these patterns are consistent across training runs or simply artifacts of randomness in high-dimensional embedding spaces. This raises questions about the practical implications and generality of the findings.\n\nI would like to continue this conversation to clarify some of the points in the rebuttal phase.\n\nQUESTIONS:\nQ1: Could the authors point to works in literature which previously suggested adding more tokens to the context help reduce singularities ? \n\nQ2: Could you please elaborate on the significance of Fig 2 bottom ? \n\nQ3: There are a number of works which have discussed a souce of potential geometric similarities across models to be due to shared dataset and next token distributions. Some references are Lee et al  2024 (Shared Global and Local Geometry of Language Model Embeddings) and  Zhao et. al 2024 (Impliciteometry of Next Token Prediction) . Do the authors have any thoughts on this, specially regarding how some of the singular tokens are shared among models such as Mistral and GPT ? Could this all be an artifact of the training data and the next token distribution ?\n\nLIMITATIONS:\nAddressed", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the structure of the token embedding space used as input for Large Language Models (LLMs). It demonstrates that this space does not possess the smooth structure typically assumed by the manifold hypothesis or the more general fiber bundle hypothesis.\n\nThe authors propose a novel statistical test to detect irregularities in the neighborhood of specific tokens and report that such irregularities are frequently observed for many tokens.\n\nFurthermore, the paper emphasizes that its primary contribution is a \"methodology for hypothesis testing on embedding points.\"\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nThis paper proposes a new statistical test, the \"fiber bundle hypothesis,\" to elucidate the structure of the token embedding space in Large Language Models (LLMs).\n\nIt offers the direct test of whether token subspaces are manifolds, a topic for which prior research had not presented rigorous hypothesis testing.\n\nThe authors conducted tests on four different open-source LLMs and found \"strong evidence that token subspaces are not manifolds.\" They also reported results indicating that tokens exhibiting irregularities often include word beginnings, word fragments, non-printable characters, and whitespace characters.\n\n\nWeakness\n\nAs the authors state, if the main point of this paper is the proposal of a \"methodology for hypothesis testing on embedding points,\" then in addition to the proofs of Theorems 1 and 2, experimental validation is needed to demonstrate the validity of this method. \nAlthough experimental results using the proposed method are shown, there is no verification of whether these results are \"valid.\" A comparative evaluation with results from other analysis methods is necessary. Alternatively, analysis using, for example, synthetic data where the ground truth is known, is also an option.\n\nWhile rejecting regularity does imply singularity, it does not afford us any evidence as to the type of singularity.\n\nQUESTIONS:\nMuch of language data often possesses semantic hierarchy within its context, and understanding how this hierarchy is represented in latent space is considered a crucial research topic. I believe Robinson+2025 is one significant prior work in this area; would it be possible to conduct an analysis and discussion regarding hierarchy similar to what is done there?\n\nLIMITATIONS:\nThe actual code and data are not publicly available. This makes it impossible to quickly verify the effectiveness of this method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by the widespread but implicit assumption that token embeddings in large language models (LLMs) lie on a smooth manifold, this paper investigates whether that assumption holds in practice. It introduces a statistical testing framework to detect geometric irregularities in the embedding space, based on the notion of local neighborhoods deviating from smooth fiber bundle structures.\n\nThe main methodological contribution is a local hypothesis testing procedure that assesses whether individual token neighborhoods exhibit manifold-like behavior. The authors demonstrate that many tokens reside in locally irregular regions, that is, their neighborhoods violate the smoothness and flatness assumptions central to the manifold hypothesis.\n\nThe authors show that these irregularities are not mitigated by context window: even when used within context, such tokens introduce instability in model behavior. As a result, semantically similar queries may yield significantly different outputs depending on subtle syntactic differences, with potentially serious implications for robustness and interpretability in LLMs.\n\nThe authors test their methodological/theoretical approach on four open-source models: GPT‑2, Llemma7B, Mistral‑7B, and Pythia‑6.9B.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** \n\nFrom my perspective, the paper has the following strengths:\n\n- **Significance:** The paper presents a well-defined motivation, namely, irregularities in the embedding space can lead to unstable model responses, which is undesirable. This justifies the need for statistical testing to identify tokens that lead to such unstable responses, especially given the current scale of LLM usage.\n\n- **Originality:** To the best of my knowledge, the paper offers an original perspective on the problem (subject to the presence of prior work and the distinctions discussed by the authors in Section 1.2). In particular, I appreciate the authors’ effort to emphasize the distinction of their approach from dimensionality reduction techniques, which could otherwise raise questions about overlap.\n\n- **Soundness:** The authors take a reasonable and systematic approach by examining the structural properties of the embedding space in practice. The experiments are logically structured, intuitive, and well-designed to support the methodological results of the paper.\n\n**Weaknesses**\n\nBelow, I provide a high-level summary of weaknesses. Please refer to the “Question” section for a detailed list of comments, where I distinguish between major and minor questions/comments.\n\nWhile somewhat subjective, I found that the paper occasionally **lacks clarity**, largely due to its narrative structure and presentation style:\n\n- **Mathematical formalism**:  I appreciate the formal approach taken in the theoretical sections. However, the paper would benefit from more consistent and clearly defined mathematical terminology. Some readers may not have sufficient background in topology, differential geometry, or related areas. Given space constraints, some definitions could be moved to an appendix with appropriate references from the main text.\n\n- **Structure**:  The structure of the paper is somewhat convoluted. There are abrupt transitions and forward references without sufficient contextual introduction, which makes the logical flow harder to follow.\n\n- **Hypothesis test formulation**:  Although the main methodological contribution is a statistical test, it is not clearly or explicitly formulated. A standard hypothesis testing format, when authors are stating the null hypothesis (H₀) and alternative hypothesis (H₁), would significantly improve clarity.\n\n- **Visual representation**:  While I appreciate the theoretical focus, the paper would benefit from additional and/or clearer/more intuitive visualizations. For instance, schematic or 3D representations of manifolds and fiber bundles (with both small and large radii) could enhance understanding. The existing figures are a step in this direction, but they remain difficult to interpret.\n\n- **Results interpretation**:  Some of the results and conclusions could be explained in greater detail. Certain claims would benefit from additional justification or clarification.\n\nI would like to emphasize that these weaknesses do not indicate any fundamental flaws. It is evident that the authors have already considered many of these points in preparing the submission. Please find my precise questions/comments/requests (both major and minor) in the section “Questions.”\n\nQUESTIONS:\n# Major Questions/Comments\n\nI will consider improving the clarity score and overall score if the following points are addressed:\n\n---\n\n## 1. Structure\n\n**a)** Section 1.1: Please enumerate the contributions (statistical test, context, propagation, etc.) with reference to the specific parts of the text.  \n\n**b)** Please consider restructuring Section 2. As mentioned in the \"Weaknesses\" section, it is a bit difficult to start discussing the methods and theorems, which are said to be explained in the next section, even though they have only been briefly introduced by that point (e.g., in Figure 2), and lack accompanying explanation. Theorem 1 is referenced but not yet formulated until the following section. One suggestion is to merge Sections 2 and 3.1, but this is not the only possibility.  \n\n**c)** Please consider adding a subsection after line 184, as this is where you begin discussing singularity propagation through the transformer.\n\n---\n\n## 2. Mathematical formalism\n\nI am aware that it is a non-trivial task to align a notion of a *“minimal but sufficient set of definitions”* with the expectations of the reader. That said, I have listed several terms that I believe are important to define explicitly. Terms such as *embedded reach*, *curvature*, *volume form*, *radius*, *pushforward*, *homeomorphism*, and *diffeomorphism* are used without explanation. \n\nAdditionally, it would be helpful to formally define *cusp point* and *pinch point* (unless they are supposed to be used informally), in which case I would appreciate a brief explanation.\n\n---\n\n## 3. Hypothesis test formulation\n\nWhile the testing framework is described, clarity would be improved by explicitly stating the hypotheses in standard form:\n\n- Null hypothesis (H₀): …  \n- Alternative hypothesis (H₁): …\n\nIdeally, this should be added at the end of Section 3.1, just before the discussion on limitations of the contextual window to solve irregularity. It would also be beneficial to clearly formulate the computed statistics (you may use additional algorithmic clauses if needed). The main point is to provide a clear and concise test description.\n\n---\n\n## 4. Visual illustration design\n\nSchematic diagrams or 3D (compared to 2D) illustrations of manifolds and fiber bundles (with small and large radii) could significantly enhance material accessibility and clarity. Within the same figure, you could also visually explain concepts such as *base space*, *radius*, *volume*, *cusp point*, *pinch point*, etc. \n\nThis would help readers build intuition, which can be leveraged when interpreting Figure 1. Additionally, this visual could potentially replace Figure 3a (left). While I acknowledge the authors’ effort in the current manuscript, I personally found the visuals somewhat difficult to interpret and believe they reduce overall clarity. \n\n---\n\n## 5. Results interpretation\n\nSome results and conclusions would benefit from more detailed explanations and clarification:\n\n**a)** Line 236: How were these general trends obtained? Were they based on manual inspection? Please elaborate and consider updating the manuscript.  \n\n**b)** Lines 200–306: Similarly, how were these results derived? Were they also from manual inspection? Please clarify and consider updating the manuscript.\n\n---\n\n## 6. Methodological clarity\n\nThe authors introduce a new test that has no existing history of successful application in prior work and apply it directly to real-world data (i.e., the embedding space of open-source LLMs), effectively without rigorous control over the data.\n\nWould it be better to include a small experimental section using synthetic manifolds or fiber bundles, where the underlying structure is known by controlled generation? This would allow you to validate the test in a controlled setting and make your claims more convincing. I would like authors to consider incorporating such experiments, or to know their opinion on why they might disagree.\n\n\n\n# Minor Questions/Comments\n\nQuestions/comments that are minor compared to the previous set, but I encourage the authors to consider them as they can further improve the quality of the paper:\n\n1. Line 12: “Failure to reject null hypothesis…” is a repetition of a very similar phrase above.\n\n2. Line 29: I wonder if this statement can be supported by prior work. While it intuitively seems to be true, it would be interesting to know if anyone has studied and quantified it.\n\n3. Line 59: Define the neighborhood  $B(\\psi)$.\n\n4. Line 71: Should it be “global token embedding space” instead of “global token embedding”? Otherwise, please ensure that the term *embedding* is not used interchangeably to mean a single instance, an embedding function, or an embedding space.\n\n5. Figure 1: Is there any particular reason why t-SNE is used for the large-scale view and PCA for the fine-grained structure? Why not build a three-dimensional embedding with either t-SNE or PCA and then zoom in? I could not find clarification.\n\n6. Line 99: “In our test case, the basic assumption…” — I agree that LLMs operate in a token subspace rather than the full latent space. However, what *test case* is referred to, and why is this assumption made? I believe this is just perhaps inaccurate phrasing, so please resolve the confusion.\n\n7. Figure 1 reference: The first reference to Figure 1 appears in the text at Line 215 (three pages later). Consider either referencing it earlier or moving the figure closer to where it is discussed.\n\n8. Figure 2: Why does the pipeline at the top not differentiate between small and large radii?\n\n9. Figure 3b: Should the label be “Reject fiber bundle (small radius)” instead of “Reject manifold (small radius)” for the first slope change in the green curve?\n\n10. Figure 3b: The blue curve appears to more closely resemble the top image in Figure 3a, which would suggest rejecting the fiber bundle as the slope increases. Why is that? Please make the figures coherent or provide an explanation.\n\n11. Figure 3a (left side): This part of the figure is a bit counterintuitive. Please consider improving the visualization (e.g., as suggested in pt. 4. Visual Illustration design in Major Comments section above) to better show the fiber bundle and connect it to the slope change dynamics in volume/radius. Additionally, even in this 2D depiction, $r_3$ seems off — it's unclear what is being represented.\n\n12. Line 142: There is ambiguity in the notation/terminology. Is $T$ the embedding or the embedding subspace?\n\n13. Line 145: Am I correct that $\\psi$ is used solely as an element of $\\mathbb{R}^l$, and not in relation to $T$? Otherwise, please resolve the confusion.\n\n14. Theorem 1: There is a bracket inconsistency in the LHS (induced volume of $B_r(\\psi)$).\n\n15. Notation $T^w$: Please briefly introduce this notation. You use $T^{w-1}$ first, and only later define it as a sequence of tokens.\n\n16. Line 201: Please check whether the reference is correct. Section 4.1 is the “Limitations” section, which seems like a mistake.\n\n17. Line 218: What does “chain of sausages” mean?\n\n18. Lines 221–224: Consider moving this part to a footnote.\n\n19. Table 1: Could you please elaborate on why all quartiles for LLaMA-7B are exactly 4096? Is there an explanation for these results?\n\n20. Line 257: It is unclear what is meant by “dependency between large and small scale variability.”\n\n21. Line 270: It would be helpful to have a formal definition of *singularity* if this is a specific type of irregularity. In Line 51, it is stated that the token subspace in LLMs is “singular,” meaning not a manifold. However, the term “singular” and its variants are later used without clarification.\n\n22. Line 323: Would it be possible to at least list possible *post hoc methods* and discuss the implications of determining that?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}