Title: Continuity and Isolation Lead to Doubts or Dilemmas in Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: transformers

continuity

representational collapse
Abstract: Understanding how Transformers work and how they process information is key to the theoretical and empirical advancement of these machines. In this work, we demonstrate the existence of two phenomena in Transformers, namely _isolation_ and _continuity_. Both of these phenomena hinder  Transformers to learn even simple pattern sequences. Isolation expresses that any learnable sequence must be isolated from another learnable sequence, and hence some sequences cannot be learned by a single Transformer at the same time. Continuity entails that an attractor basin forms around a learned sequence, such that any sequence falling in that basin will collapse towards the learned sequence. Here, we mathematically prove these phenomena emerge in all Transformers that use compact positional encoding, and design rigorous experiments, demonstrating that the theoretical limitations we shed light on occur on the practical scale.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives and demonstrates two phenomena that a decoder-only transformer with compact positional encoding exhibits. The first is continuity: given two similar tokenized sequences, the next token prediction is similar for one sequence as for the other sequence, if the sequences have a small enough Hamming distance. Their theoretical claim was examined by looking at LM’s sensitivity to small changes in the prompt, one on zero fundamental sequence and the other on a syntax verification task, in which the model needs to say whether the syntax is correct or not. They observed that most models do not change their predictions when there is a small change in syntax, supporting the continuity argument. The second is isolation, which is about no such transformer can learn all sequences of infinite length, for example, all periodic sequences, because they have a relative hamming distance of 0.  In practice, they demonstrate the effect of isolation on sequences with a small number of differing samples, supporting that decoder-only transformers with compact positional encoding learn a limited set of periodic sequences.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n\n I appreciated the mathematical rigor of the paper, and the theoretical claims are well-defined. The idea is original and connects analysis with understanding transformer behavior. There is a nice connection between the theoretical derivations and practical experiments. \n\n__Weaknesses:__\n\n My issue with this theoretical idea is that it is too much of a corner case. The authors claim that the lack of these abilities “*severely* constrains their (even mild) intelligent behavior”, which is a bit unsubstantiated. It is also bizarre to assume intelligence depends on the learnability of infinite sequences. I think the evaluation examinations are too tangential to their actual use cases.  Pragmatically, few give infinite sequences to transformers, but just missing this aspect of behaving properly with respect to infinite or very long sequences does not disqualify a transformer from being intelligent. The paper lacks an argument or a demonstration on why this can be a concerning characteristic of the transformer. \n\nThe title is not appropriate to its content, as it is unclear what these doubts and dilemmas refer to, the wording deviates from the main findings of the experiments.\n\nSomething is missing between the introduction and the contributions. I suggest putting related work before the contribution. \n\nLine 114, 115 on Page 3 is broken. \n\n321-322 is missing a comma\n\nsection 1.3’ s title does not match its content\n\nQUESTIONS:\nLine 89, what is CPE transformer? \n\nWhat is the accuracy of the model in the syntax prediction error detection task? \n\nIs the model sensitivity dependent on sequence length?\n\nI guess figure 2’s emphasis is on the left figure, do we know in closer detail, the message is if there is a small number/proportion of differing signals, then the prediction results are insensitive, and the right plot suggests that the prediction results are sensitive when there is a substantial differing portion of signals, but you cannot see very well from the left side of figure 2. (How come there are still some models that are insensitive to a big portion of differing signals?) \n\nHow does the sensitivity result depend on the sequence length? I know it is 190 in the experiment, but what happens for sequence length in the common token range?\n\nWhat is the accuracy of the model in the syntax prediction error detection task?\n\nLIMITATIONS:\nYes, limitations are sufficiently discussed in section 5 of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe this paper really on two simple (yet non-trivial) observations:\n- Continuity: a Transformer outputs can not change too much when its prompt does not change much\n- Isolation: as a consequence, a Transformer can not learn simultaneously infinite sequences that are equal on a dense set, sooner or later it will have to make next token prediction error on one of the two sequences.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**:\n- In essence, the paper shows that Transformers sees infinite sequences from the Besicovitch space (which is much smaller than the Weyl or Cantor spaces).\n- The results do not see too toyish, but of practical relevance at scales in which Transformers operate in practice.\n- It could explain the difficulty to do long-context with transformers, and justify the use of CoT, memory layers, tools...\n- The paper is really well written.\n\n**Weaknesses**:\n- It would be nice to have a finer characterization of the modulus of continuity in Theorem 1. In particular, I assume that the MLP layers F could introduce big variations that may make this work irrelevant in practice (e.g. I won't be surprised if someone show that one can modify a single token in a prompt to lead to completely different outputs for any decently long sequences, and that the conflict with this paper would be due to the modulus of continuity of the MLP layers).\n- It may be the case that the results are brittle, and do not really explain as many practical observations on long context as one may want to believe. In particular, I am not sure that the theoretical tools used in this paper could explain the observations made on the RULER benchmark (see question below).\n\nQUESTIONS:\nIt has been shown that hybrid models between Mamba and Transformers work better than Transformers for long context (https://github.com/NVIDIA/RULER). Do you have any intuition of what could explain these observations? Would your work allow you to shed some lights on that matter? (Maybe Mamba models suffers a representation collapse at a lower speed)\n\nDoes your study make you bullish on approaches such as https://arxiv.org/pdf/2501.19399?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe core of this paper is to deepen the understanding of the mechanism of Transformer by conceptualizing and analyzing two phenomenon: \n\n1)\tIsolation expresses that any learnable sequence must be isolated from another learnable sequence, and hence some sequences cannot be learned by a single Transformer at the same time. \n\n2)\tContinuity entails that an attractor basin forms around a learned sequence, such that any sequence falling in that basin will collapse towards the learned sequence.\n\nThe authors provide not only the mathematic proof, but also the careful designed experiment to verify the claim.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1.  Overall, this paper is well written.\n2.  The paper offers a valuable perspective on analyzing the Transformer architecture.\n\nWeakness:\n1.  The paper appears to be poorly positioned. While the title suggests a focus on large language models (LLMs), the main content primarily discusses Transformers. Although LLMs are typically built on Transformer architectures, the two are not synonymous, and this distinction is not clearly addressed.\n2. Several prior works [1,2,3] have explored the fundamental limitations of the general Transformer architecture, but this paper does not sufficiently engage with or contextualize those contributions.\n3.  The findings of this paper are limited to pattern discovery in purely numerical sequences. It remains unclear whether these results can be generalized to natural language. In most cases, natural language understanding does not rely on the kinds of patterns examined in the paper. Moreover, natural language carries significantly richer semantic content than the simplified tasks considered here.\n\n[1] Faith and Fate: Limits of Transformers on Compositionality\n\n[2] Limits of Transformer Language Models on Learning to Compose Algorithms\n\n[3] On Limitations of the Transformer Architecture\n\nQUESTIONS:\nCan similar findings be applied to RNNs? Additionally, could you suggest possible architectural designs that might address the limitations identified in this paper?\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers provides two interesting fundamental limitations in the learning capability of transformers, namely continuity and isolation. Continuity shows that prompts the are close enough always produces probability output that are also close. On the other hand, isolation shows that for any two infinitely long sequences that are learnable by a transformer, if the two sequences differ at infinitely many places, then they must be far enough in Hamming distance. The paper also shows experimental analysis showing the importance of the limits in learning in several synthetic as well as some practical scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper theoretically illustrates two fundamental limitations in the learning capability of transformers\n\n- The paper also shows experiments illustrating the continuity and isolation phenomenon on synthetic tasks such as number sequence generation as well as practical applications such as syntax verification tasks.\n\n- The paper is well written in terms of the clarity of the results and simple experiments supporting and illustrating the claims.\n\nWeaknesses: \n- The theorems are all claims for infinite sequences. Hence, it is difficult to judge how applicable they are practically beyond the illustrated synthetic examples. Is it possible to show more convincing practical usecases such as the code-syntax related.\n\n- For Fig. 3 the pattern does not seem to follow same as in Fig. 2 for continuity. Please explain if I am misunderstanding the figures.\n\n- As also mentioned in the limitations, the analysis lacks consideration of chain-of-thought reasoning generated by recent reasoning models. Is it possible that these reasoning traces allow different output after reasoning, hence, not be limited by the results in the theorem. Can we check it empirically on some of the synthetic datasets using some reasoning models?\n\nQUESTIONS:\nPlease also see the weaknesses.\nAdditional questions:\nLine 223: \"Middle positions result in lower sensitivity, and early tokens yield almost no sensitivity at all\"\nQ: does the theory explain this observation?\n\nLine 196: \"same last token\"\nWhy is this condition in theorem 1 important, please explain intuitively.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}