Title: Architectural and Inferential Inductive Biases for Exchangeable Sequence Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Exchangeability

Inductive Bias

Epistemic uncertainty

Multi-step inference

Transformer architecture
Abstract: Autoregressive models have emerged as a powerful framework for modeling exchangeable sequences---i.i.d. observations when conditioned on some latent factor---enabling direct modeling of uncertainty from missing data (rather than a latent). Motivated by the critical role posterior inference plays as a subroutine in decision-making (e.g., active learning, bandits), we study the inferential and architectural inductive biases that are most effective  for exchangeable sequence modeling. For the inference stage, we highlight a fundamental limitation of the prevalent single-step generation approach: its inability to distinguish between epistemic and aleatoric uncertainty. Instead, a long line of works in Bayesian statistics advocates for multi-step autoregressive generation; we demonstrate this "correct approach" enables superior uncertainty quantification that translates into better performance on downstream decision-making tasks. This naturally leads to the next question: which architectures are best suited for multi-step inference? We identify a subtle yet important gap between recently proposed Transformer architectures for exchangeable sequences (Müller et al., 2022; Nguyen & Grover, 2022; Ye & Namkoong, 2024), and prove that they in fact cannot guarantee exchangeability despite introducing significant computational overhead.  Through empirical evaluation, we find that these custom architectures can significantly underperform compared to standard causal masking, highlighting the need for new architectural innovations in Transformer-based modeling of exchangeable sequences.


Peer reviews:
comments: SUMMARY:
The paper tackles both the inferential and architectural inductive biases that matter when using Transformer sequence models on exchangeable data such as i.i.d. rows in a table.
On the inference side, it proves that the common one-step prediction heuristic systematically discards information.
On the architecture side, the authors show that recently proposed “exchangeable” masking schemes for Transformers is suboptimal.

STRENGTHS AND WEAKNESSES:
## Strength
1. This paper tackles sequential modeling from a unique perspective. They show how the inference routine affects performance. I think this is a nice and meaningful perspective for the ML community.
2. The paper is well organized, it is easy to find information and follow.

## Weakness
1. Typo: line 223 (undertainty), appendix D.2 (exchangeablity), line 15 (infintely).
2. After proving existing masks insufficient, the paper did not provide a corresponding solution that satisfies both Property 1 and Property 2.

QUESTIONS:
1. Is C-permutation invariant the same as conditional permutation invariant?
2. Based on my understanding, the proposed theory does not apply to most time series data due to exchangeable property, is that correct?
3. Can the authors provide some concrete example on how one apply either one-step or multi-step inference? A simple real world example would help a lot for understanding the paper.
4. Can the authors discuss how strong is the assumption $\hat{P_\phi} = \mathbb{P}$ ?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the optimal inferential and architectural inductive biases for modeling exchangeable sequences with autoregressive models, particularly for downstream decision-making tasks. The authors make two primary contributions: \n- First, they argue that the prevalent use of one-step inference is insufficient as it fails to distinguish between epistemic and aleatoric uncertainty. They advocate for multi-step autoregressive generation, demonstrating both theoretically and empirically that it provides superior uncertainty quantification, leading to better performance in active learning and bandit tasks. \n- Second, the paper critically examines existing Transformer architectures designed to enforce exchangeability. The authors prove that these architectures only guarantee a weaker property of \"conditional permutation invariance\" and not full exchangeability. Through empirical evaluation, they show these specialized architectures are computationally inefficient and perform worse than a standard Transformer architecture.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper makes a clear and significant argument for using multi-step inference in decision-making contexts, which is supported using both theory and experiments.\n2. It provides a valuable and timely critique of prior architectural work, demonstrating that simpler causal models are more efficient and effective than complex \"permutation-invariant\" transformer designs that don't fully guarantee exchangeability\n\nWeaknesses:\n1. While the paper critiques existing architectures, it does not propose a novel, efficient architecture that can guarantee full exchangeability.\n2. The experiments, while effective, are limited to synthetic data; validation on real-world datasets would further strengthen the claims.\n\nQUESTIONS:\n1. Given that standard causal models outperform specialized ones, what is the path forward for incorporating exchangeability as an inductive bias? \n2. Multi-step inference is more computationally expensive. How should practitioners manage the trade-off between its superior performance and its higher computational cost in real-world applications?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reveals a potential flaw of using Transformers trained with one-instance generation loss for posterior inference. The work shows that the current approaches to building and using sequence models for exchangeable data on regression tasks are often theoretically unsound and practically suboptimal.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n(S1) The paper's primary strength is its formal and clear-headed analysis. The distinction drawn between \"conditional permutation invariance\" and true \"exchangeability\" is a particularly insightful and valuable contribution that clarifies a subtle but critical misunderstanding in prior work.\n\n(S2) The paper provides compelling evidence that specialized attention mechanisms, despite being designed to handle exchangeability, can fail in practice. This finding serves as an important and actionable \"negative result\" for researchers, cautioning against the pursuit of complex architectural fixes that may not be theoretically sound.\n\n(S3) By demonstrating the fundamental mismatch between autoregressive models and exchangeable data, the work implicitly questions the community's default choice of Transformers for this problem class. It forces a necessary and critical re-evaluation of whether adapting ordered models is a more fruitful path than developing architectures inherently suited to permutation-invariance.\n\nWeakness\n\n(W1) The paper's motivation is well-grounded in showing that current Transformer-based approaches are flawed. However, it does not sufficiently justify why the community should continue to focus on adapting Transformers for this task at all, rather than exploring more natural alternatives like Deep Sets or other architectures explicitly designed for permutation-invariant data.\n\n(W2) The paper's critique, while sharp, is focused on a small and specific set of recent works. To broaden its impact and appeal to a general audience, the paper would benefit from contextualizing its findings within a wider survey of methods for modeling exchangeable data, thereby making a stronger case for the novelty and significance of its specific contributions.\n\n(W3) While using Gaussian Processes as a case study is a reasonable starting point, the paper's claims about modeling general tabular data would be substantially more convincing with a broader empirical evaluation. Demonstrating the identified issues and the benefits of multi-step inference on a variety of well-known, real-world tabular datasets is essential to validate the practical relevance of the findings.\n\nQUESTIONS:\n(Q1) The distinction between epistemic and aleatoric uncertainty is a key justification for multi-step inference. However, this approach introduces practical challenges. Could you provide an analysis of the computational overhead of this method? How does the inference time scale with the \"imagination horizon\" (the length of generated sequences) and the size of the model? Following this, how should one determine the optimal imagination horizon? Is there a risk that a poorly chosen horizon could lead to noisy or biased uncertainty estimates, and how sensitive are the downstream task improvements to this choice?\n\n(Q2) Following (W3), the experiments mainly demonstrate your points on synthetic data. How do you expect these findings to translate to large-scale, noisy, real-world tabular datasets where features may have complex dependencies? Would the performance gap between single-step and multi-step inference become more or less pronounced in such settings?\n\n(Q3) The paper convincingly demonstrates the limitations of applying autoregressive models to exchangeable data. This raises a fundamental question: what is the rationale for focusing on adapting Transformers for this task, rather than exploring architectures inherently designed for permutation-invariance, such as Deep Sets, or other graph-based neural networks?\n\n(Q4) What would be equivalent problem/strategy when number of steps becomes large?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAutoregressive models such as Transformer-based model architectures are increasingly used for exchangeable sequence modeling, but common approaches rely on **one-step inference**, which fails to distinguish between epistemic uncertainty and aleatoric uncertainty.\n\nThis limitation negatively impacts decision-making tasks.\n\nThe paper did multiple analyses to show that \n- multi-step inference in training is essential for reliable uncertainty quantification and downstream task performance;\n- existing architectural approaches exchangeability are flawed and computationally expensive without providing benefits;\n- designing transformer architectures that achieve full exchangeability while remaining computationally efficient remains an open challenge.\n\nSTRENGTHS AND WEAKNESSES:\n## Reviewer Expertise Disclaimer\nThis work falls outside my core area of expertise. While I have some familiarity with sequence modeling and uncertainty quantification, I encourage the area chairs to weigh this review accordingly, especially regarding technical novelty and empirical depth.\n\n## Strengths\n1. **Theoretical and Empirical Analysis**: The paper provides a thorough theoretical critique, in the context of uncertainty quantification, of one-step inference and supports its arguments with empirical results demonstrating the advantages of multi-step inference in specific tasks.\n2. **Architectural Insights**: It offers a detailed examination of transformer architectures concerning **exchangeability**, contributing to the understanding of their limitations and guiding future architectural designs.\n\n## Limitations and Concerns\n1. **Limited scope of novelty**: The paper's core critique seems novel but only in a narrow context. Broad limitations of single-step inference and standard Transformer architectures have been explored in broader reasoning and planning contexts (e.g., Shojaee et al., 2025; Valmeekam et al., 2023; Creswell et al., 2022). While the paper’s domain (exchangeable sequences) is more narrow, the broader concern has precedent.\n2. **Uncertainty quantification is conceptually important but underused in practice, especially in sequential decision making**: While uncertainty quantification is theoretically valuable, the claim that it is a prerequisite for intelligent agents is overstated. While uncertainty quantification is indeed valuable for certain types of decision-making, it is not a universally accepted prerequisite for intelligent behavior. Classical control systems, expert systems, and many successful RL algorithms function without explicit UQ mechanisms, despite the presence of partial observability and noisy environments.\n3. **Limited generality of the scope**: Although the paper frames its contributions in terms of “decision-making,” the empirical evaluations are limited to multi-armed bandits and active learning, and especially in exchangeable settings. These are important, but relatively constrained, problem settings.\n\nCombined, the scope of novelty and impact of the paper seems relatively limited for a NeurIPS publication.\n\n## Recommendation\nDue to my distance from the specific subdomain, I do not make a definitive accept/reject recommendation. I encourage a review from a domain expert who can more fully evaluate the novelty and technical depth of the paper.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}