Title: REOrdering Patches Improves Vision Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: long sequence model

tokenization
Abstract: Sequence models such as transformers require inputs to be represented as one-dimensional sequences. In vision, this typically involves flattening images using a fixed row-major (raster-scan) order. While full self-attention is permutation-equivariant, modern long-sequence transformers increasingly rely on architectural approximations that break this invariance and introduce sensitivity to patch ordering. We show that patch order significantly affects model performance in such settings, with simple alternatives like column-major or Hilbert curves yielding notable accuracy shifts. Motivated by this, we propose _REOrder_, a two-stage framework for discovering task-optimal patch orderings. First, we derive an information-theoretic prior by evaluating the compressibility of various patch sequences. Then, we learn a policy over permutations by optimizing a Plackett-Luce policy using REINFORCE. This approach enables efficient learning in a combinatorial permutation space. _REOrder_ improves top-1 accuracy over row-major ordering on ImageNet-1K by up to 3.01% and Functional Map of the World by 13.35%.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nApplying a sequence model on image patches is an increasingly common paradigm in modern computer vision. This paper explores the impact of patch ordering on performance when full self-attention is dropped in favor of more efficient (but permutation-sensitive) alternatives like Mamba, Longformer, and Transformer-XL. This paper finds that simple alternatives to the usual row-major patch order (such as column-major and hilbert order) yield significantly different image classification performance on ImageNet-1K and Functional Map of the World. However, the optimal ordering varies per dataset-model pair and is only weakly correlated with compressibility, so the paper proposes a novel RL formulation to learn the best patch ordering by optimizing a Plackett-Luce distribution over permutations with a REINFORCE objective.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Novel premise and RL formulation**. To the best of my knowledge, this is the first paper that explores patch ordering for vision models. As vision models are scaled, the proposed ideas may become increasingly relevant. The resulting RL formulation to optimize over a distribution of permutations is also elegant and can serve as a template, even for other use cases.\n2. **Clarity**. The paper is very well-written and does a great job of prefacing its contributions with bits of essential background, such as on Mamba, Longformer, Transformer-XL, and the Pluckett-Luce distribution.\n\n**Weaknesses**\n\n1. **Unclear takeaways**. The paper does a great job of probing the impact of patch ordering and bringing to light that the typical row-major order may not always be optimal. However, it remains unclear what the actual optimal patch ordering may be. Some extra analysis may help pick out the signal from the noise here. Questions I have in this direction:\n   - On lines 205-208, the paper notes that Mamba/ARM doesn’t perform well with many orderings, likely because of its fixed scan directions (forwards, backwards, up, down). Can you visualize these scan orders? I can imagine what the forwards and backwards scan orders may look like, but what about up and down? If those scan orders contain many spatial discontinuities, that may explain the poor results and render many of those results uninterpretable. If so, can you adapt the scan orders or the patch orderings appropriately, to induce better performance?\n   - Compression percent is only weakly correlated with performance, and the sign of the correlation also seems to depend on the model. Can you do a study on the robustness or noisiness of this metric? Why is Hilbert ordering less compressible than column ordering? This is surprising, as the paper points out. Could it be a sign of a noisy measure of compressibility? Are there other metrics that may show stronger correlations, e.g. using JPEG compression?\n   - What do the patch orderings that are learned through RL look like? Do they converge to any patterns, by model or by dataset? Similarly, in Figure 4, why should the patches related to the “keyboard” class move to the end of the patch order? Was that trained on a single image? If not, what’s special about the “keyboard” class such that it dictates the optimal patch order across the entire dataset?\n2. **Missing evaluation in the long sequence regime**. For all experiments in the paper, resolution is set to 224x224 input images, a regime where O(n^2) full self-attention is viable. In this regime, patch ordering is not really a consideration, since full self-attention is permutation invariant. Results would be more persuasive if tested on a truly long sequence regime, such as higher resolution images or videos.\n\nAddressing these weaknesses may improve my rating.\n\nQUESTIONS:\nI’ve embedded most of my questions and suggestions for the authors within the Strengths and Weaknesses section.\n\nOne other miscellaneous suggestion: Consider training the random permutation baselines for each model-dataset pair and including them in Figure 2. The paper slates this for future work, but if done, it’ll better contextualize the performance of the 6 explicitly considered orderings.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how the order of image patches affects the performance of vision models. They especially focus on processing images as long sequences like Transformer-XL, Mamba, and Longformer. While standard vision transformers such as ViT are not sensitive to patch ordering due to full self-attention, newer efficient models introduce inductive biases that make them more sensitive to the sequence of input patches. They show that simply changing the patch order from the common row-major format to alternatives like column-major, spiral, or Hilbert curves can lead to noticeable differences in accuracy depending on the model and dataset. By this experiment, they figured out that the order of patch is non-trivial. To solve this, they propose REOrder, a method that learns task-specific patch orderings using reinforcement learning. REOrder models the patch order as a distribution over permutations using the Plackett-Luce model, and optimizes it with reinforcement learning and Gumbel-top-k sampling. The learned ordering consistently improves classification performance across models and datasets, showing that rethinking patch order is a promising direction for enhancing long-sequence vision models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Insightful approach: The paper convincingly identifies and analyzes a practical inductive bias in long-sequence vision models, sensitivity to patch ordering, which has been largely underexplored. This sheds light on a subtle but impactful limitation in common transformer variants and challenges the assumption that input ordering is inconsequential. It's a fresh perspective that could influence how future vision architectures are designed and trained.\n\n2. Generalizable Solution: The proposed method, REOrder, is easy to be plugged into many transformer-based models without architectural changes. The use of a Plackett-Luce distribution with REINFORCE and Gumbel-top-k sampling is well-motivated and makes the optimization over permutations tractable. It’s a rare case where a relatively simple reinforcement learning setup yields consistent and non-trivial gains across diverse architectures and datasets.\n\n3. Clarity of paper and their experiences: The experiments are quite comprehensive covering different patch orderings, multiple models. Furthermore, their supplementary further show their training details which further demonstrates their effectiveness and robustness. \n\nWeakness\n\n1. Global ordering limits flexibility: REOrder learns a single patch permutation shared across all inputs for each model-dataset pair. This approach may not capture per-image structural differences, especially in diverse datasets like ImageNet. The paper does not discuss this limitation or possible dynamic extensions.\n\n2. Lack of interpretability or analysis of learned orderings: Beyond a single illustrative example, the paper does not analyze what kinds of patterns the learned orderings reflect or why they help. There is little insight into how orderings vary across classes, datasets, or models, which limits understanding and generalizability.\n\n3. Limited analysis on the impact of patch ordering priors: While the paper uses six predefined patch orderings as priors, it lacks a deeper analysis of how the choice of initial ordering affects REOrder’s performance. In some cases, REOrder appears to achieve substantial gains even when starting from a weak prior (e.g., Snake for Mamba with 'Functional Map of the world' dataset), but this is not explicitly discussed. Furthermore, there is no comparison against random patch orderings, which would help clarify how much the method relies on having a reasonable starting point. Without this, it’s difficult to assess the robustness and sensitivity of REOrder to the choice of prior.\n\nQUESTIONS:\n1. It would strengthen the paper to include an analysis or ablation of per-image or per-class adaptive ordering strategies. Such results could help validate whether a single global ordering is sufficient or whether dynamic ordering could further improve performance.\n\n2. Beyond the example shown in Figure 4, could you provide a broader analysis of the learned patch orders across classes or datasets? This would help readers understand what structural or semantic properties the ordering policy captures.\n\n3. It would be helpful to include experiments that evaluate REOrder when initialized from random or very weak patch orderings. This would clarify how much the method relies on having a good prior and whether it remains robust across different starting points.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This manuscript presents an interesting question; whether the ordering of patches affects the performance of variants of ViT.
It is well known that the vanilla self-attention is permutation-invariant. However, some variants of ViT do not feed all the low-level patches. This would make a limited window of tokens like limited perceptive field.

The authors empirically showed that the ordering matters in Longformer, Mamba and Trransformer-XL.
To alleviate (ot to utilize) this phenomenon, The authors propose RL-based solution.

STRENGTHS AND WEAKNESSES:
This paper presents an interesting problem and empirically shows the significance of ordering in some models.
The RL-based approach shows significant improvement in Mamba model.
The evaluation is clean and somewhat through.

However, the proposed method basically work only for Mamba.  In case of Mamba, as well discussed in Section 3.3, the architecture is not permutation-invariant from the beginning. It is also well know that Mamba is sequence-dependent.
It may not be proper to the ordering of patched in Mamba and ViT-variant together.

Based on the limited benefit/cost of the proposed method in ViT-variants, I don't think this method would be adopted to ViT-based models.

QUESTIONS:
It is not clear why RL-based method is not effective for Longformer.
It would be better to explore nontent-aware dynamic re-ordering method.

LIMITATIONS:
This method is effective Mamba models.
Though, the authors argue that there is a hidden opportunity of improving performance using better reordering, the   RL method did not add up extra gain over fixed ordering in ViT-variant.
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem that the design of patch ordering in autoregressive vision models has a significant impact on performance. Based on the hypothesis that conventional fixed patch ordering does not guarantee optimality, the authors propose a method to dynamically determine the ordering using reinforcement learning. By applying the proposed method to the Mamba architecture, for instance, the authors demonstrate significant performance improvements on the FMoW dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The description is clear and well-structured throughout the paper.\n\n2. The issue of the effect of patch order on autoregressive image models is novel and interesting in that it has not been fully explored.\n\n3. The proposed method provides a framework for dynamically optimizing the patch order based on information-theoretic initialization and reinforcement learning, and is applicable to implementations. In particular, the approach of designing the initial order in terms of information compression is unique and suggestive.\n\nWeaknesses \n\n1. Concerns about insufficient training in the experimental setting:\nFigure 2 shows only performance differences and omits absolute accuracy. On the other hand, line 194 of the text states that ViT achieves only 37.5% in Top-1 Accuracy for ImageNet-1k. This is extremely low compared to the performance of standard ViT (in the upper 70% range in the original ViT paper), probably due to the lack of convergence caused by the short training period of 100 epochs. In evaluating the effectiveness of the proposed method, a comparison under more fully trained conditions would be desirable, and it is difficult to draw strong conclusions from the current experimental results. This concern is similar for the other architecture, the other data (FMoW).\n\n2 Basis for selection of baseline models is unclear:\nTransformer-XL, Longformer, and Mamba are selected, but the reasons for their selection and their relationship to this method are not clearly presented. breaking Permutation-invariance While there are many variants of Transformer, why these three were chosen and what comparative insights can be gained from them?\n\n3. Ambiguous positioning with the technical context:\n Although the proposed method focuses on sequential image models, many efficient and high-performance approaches have already been proposed for application to image tasks using Mamba and other methods, such as 2D extensions of SSM and bi-directional processing. In such a technological context, it is somewhat unclear what practical and theoretical contributions this study has to offer.\n\n4. Overlearning Concerns for Patch Order:\n Even if the proposed method is effective in a particular pre-training task, it is unclear whether it will generalize to out-of-distribution data and downstream tasks. This is a major practical concern. In particular, additional experiments on out-of-distribution generalization performance and transition learning tasks are desirable to eliminate the possibility that the patch order is overfitting the training data.\n\nQUESTIONS:\n1 Can you please explain why ViT's accuracy (37.5%) on ImageNet-1k is significantly lower and how this affects your assessment of the proposed method's effectiveness?\n\n2 Could you elaborate on the reasons for selecting Transformer-XL, Longformer, and Mamba as baselines and the significance of each comparison?\n\n3 In light of recent developments such as the 2D extension of Mamba and bidirectional processing, what do you think is the practical standing of this research?\n\n4 Have you verified the performance of the proposed method on out-of-distribution data or downstream tasks to rule out the possibility of overlearning to a particular order? Or do you have any future plans to do so?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the effect of the patch order on the performance of transformer-influenced image classification models that are not equivariant to permutations of patches (together with positional embeddings).\n\nThe paper presents the hypothesis that the standard row-major patch ordering might not be the optimal one and proposes a policy learning algorithm that learns the patch ordering in parallel with the classifier (for some number of epochs before freezing the policy). The policy learning uses the REINFORCE algorith with a policy model with the number of parameters corresponding to the number of patches (Plackett-Luce model).\n\nThe paper tests one equivariant architecture, ViT, and 3 non equivariant ones, Linformer, Transformer-XL, and Mamba (ARM). First, we see that changing the patch order of the model degrades the performance of non-equivariant models. For example, ImageNet-1k top-1 accuracy drops most on the Mamba model (around 6%) when a trained model is tested on different patch orders (row-major to column-major, spiral, ...).\n\nTraining the model with the proposed algorithm improves the performance with respect to the sub-optimal patch orderings, and the learned model performs similarly to the model trained with the row-mayor patch ordering.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The source code for reproducing the experiments is provided will publicly available.\n2. The paper is written well, has clear hypotheses, and presents an original investigation.\n3. The research questions of the paper make sense.\n\n**Weaknesses**\n\n1. Validation experiments could be improved.\n\t1. If I understand correctly, the main paper does not include experiments with supervised classification training from the start with a learned patch ordering and with a random ordering, but only concurrently with permutation learning. Experiments singling out the effect of the learned permutation would be valuable.\n\t2. An experiment with a random patch ordering is included in the appendix but not mentioned in the main paper. I think that the substantially worse performance on a random patch ordering is a valuable data point.\n2. Clarity could be improved.\n\t1. What is $M_\\text{TXL}$?\n\t2. I understand that all the Mamba experiments use ARM (Ren at al.), but I am not sure. Please consider clarifying it.\n\t3. Figure 2:\n\t\t1. Is the y-axis relative change or absolute (percentage points)? ()\n\t\t2. Does Figure 2 use a single training run?\n\t\t3. L190: \"We evaluated top-1 accuracy on validation sets, estimating the Standard Error of the Mean (SEM) using a non-parametric bootstrap method with 2,000 resamples.\" What is the sizes of the samples? Is the standard error of the mean 0 if the size equals the whole validation dataset?\n\t4. Figure 3. Why is the ImageNet-1k top-1-accuracy of the ViT (ViT-B/32?) baseline (Row) only around 38%?\n3. The discussion about REOrder experiments (Section 6.4 and Conclusion) could be improved.  \n\t1. I think that there is too much focus on the results of improving patch orderings that are worse than the default row-major ordering (Hilbert curve and diagonal), while what matters most is the improvement over row-major.\n\t2. How does the validation accuracy vary over different training runs?\n\t3. The percentages can be ambiguous. I suggest saying \"pp\" (percentage points).  \n4. Presentation could be improved (easy to address).  \n\t3. Not all appendix sections are referenced in the main paper, while some are quite valuable (for example, Appendix A).  \n\t4. Some more model details could be presented in the main paper for better information availability and sanity-checking. For example, it takes some searching to find out that \"ViT\" is ViT-Base and that the patch size is 32×32 (if I am correct). Also, is the Mamba in the experiments ARM?  \n\t5. Figure 3: The y-axes of the ViT experiments is a bit broken. Some numbers occur on multiple ticks.  \n\t6. Missing details:  \n\t\t1. For how many epochs is the patch ordering learned? (What is the value of $M$?)  \n\t\t2. What is the shape of the temperature schedule?  \n\nMinor suggestions:\n- The notation seems inconsistent: some matrices are not in bold.\n\nQUESTIONS:\n**Questions:**\n\n1. Please address weakness points 1 - 3.\n2. Why is the patch size selected to be 32×32?\n\nI hope that the authors will be able to address all the important weaknesses. I apologize for any errors on my part.\n\nLIMITATIONS:\nThe paper comments on some limitations in the appendix. Additional limitations seem to be the lack of experiments with fixed previously learned patch orderings and low baseline accuracies.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-04"}