Title: RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autoregressive Pretraining

Mamba

Remote Sensing

Foundational Models
Abstract: Recent advances in self-supervised learning for Vision Transformers (ViTs) have fueled breakthroughs in remote sensing (RS) foundation models. However, the quadratic complexity of self-attention poses a significant barrier to scalability, particularly for large models and high-resolution images. While the linear-complexity Mamba architecture offers a promising alternative, existing RS applications of Mamba remain limited to supervised tasks on small, domain-specific datasets. To address these challenges, we propose RoMA, a framework that enables scalable self-supervised pretraining of Mamba-based RS foundation models using large-scale, diverse, unlabeled data. RoMA enhances scalability for high-resolution images through a tailored auto-regressive learning strategy, incorporating two key innovations: 1) a rotation-aware pretraining mechanism combining adaptive cropping with angular embeddings to handle sparsely distributed objects with arbitrary orientations, and 2) multi-scale token prediction objectives that address the extreme variations in object scales inherent to RS imagery. Systematic empirical studies validate that Mamba adheres to RS data and parameter scaling laws, with performance scaling reliably as model and data size increase. Furthermore, experiments across scene classification, object detection, and semantic segmentation tasks demonstrate that RoMA-pretrained Mamba models consistently outperform ViT-based counterparts in both accuracy and computational efficiency. The source code and pretrained models have be released at https://github.com/MiliLab/RoMA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper seems to present novel designs and self-supervised pretraining strategies for foundation models in remote sensing, including the use of Mamba to avoid the quadratic bottleneck of transformer-based architectures, a patch rotation augmentation mechanism and an unclear multiscale prediction mechanism. However, the rather poor quality of writing makes it difficult to understand the actual contributions of the paper.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors seek to improve the efficiency of foundation models in remote sensing. This is important to handle the very large images typical of this field.\n- Experimental results seem promising with good metrics across different tasks compared to existing methods and significant runtime and memory savings\n\nWeaknesses:\n- The paper is very difficult to read due to poor writing quality. This makes it difficult to properly understand what the contributions of the work are and their technical details. There are several mistakes in the use of English (examples: caption of figure 2, lines 154-156, ...). A few examples of poor writing quality are: repetitions in lines 75-80; paragraph \"Advantage of Autoregressive Pre-training of Mamba\" does not explain any advantages, only the fact that a causal autoregressive model such as Mamba matches the setting of a causal autoregressive prediction task used for training. \n- Several sections are unclear., particularly section 3.2.3. How are patches tokenized? what's the ordering that is decided for next-token prediction? It is said that the Mamba-based encoder (which is never explained what it is) generates key and values for a causal cross-attention operation. This is quite confusing. Not only it is not clear how it works, but also, this operation would have the classic quadratic complexity in the number of tokens of the transformer, which the authors claim is a main limitation of existing RS foundation models (lines 5,33,157). The \"multi-scale\" part of the prediction is also unclear: how are the multiple scales generated and processed? Neither the text nor fig.3 are illuminating on this aspect.\n- The novelty of using a rotation augmentation with angular embedding for the pretraining method seems fairly limited\n\nQUESTIONS:\nThe paper might have some interesting contributions from a technical standpoint but the presentation is too confusing. The authors should be significantly more careful in explaining the technical details of the method, clearly highlighting how every step of the proposed design and methodology works.\n\nLIMITATIONS:\nDespite what is reported in the checklist, limitations are not discussed with the conclusions of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose rotation-aware multi-scale autoregressive learning (RoMA), a self-supervised pretraining method designed for Mamba-based architectures applied to remote sensing. The overall objective is to propose a first Mamba-based foundation model for remote sensing improving the scalability for high-resolution imagery compared to ViTs. To this aim, RoMa pretraining integrates a rotation aware pretaining objective and a multi-scale token prediction objective. Experiments have been performed for classification, change detection and semantic segmentation including ablation studies to evaluate technical choices on RoMA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\t\n- The overall submission is well written and well structured. \n\t\n- Exploring a Mamba-based architecture as a remote sensing foundation model is relevant to tackle the ViT limitation of complexity bottleneck in attention layers w.r.t. large input size.\n\t\n- The MAE pretraining strategy is commonly employed to pretrain remote sensing foundation model and the authors justify well in 3.2.1. its limitation, especially for a Mamba-based architecture, compared to an autoregressive setting.\n\t\n- The adaptative rotation encoding strategy serves to mitigate a well known problem in remote sensing and includes learnable angle embeddings. \n\n- A proper analysis (Figure 5) shows that a Mamba-based architecture along the proposed pretraining strategy scales well with the data volume and model size highlighting the potential of the work for future large-scale applications.\n- Exhaustive experiments show better results compare to competing methods on well explained settings, including ablation studies to highlight the impact of each component and choices in RoMA.\n\nWeaknesses\n\t\n- The literature on rotation-invariant training for remote sensing, especially object detection, is rich [1, 2, 3] and could have been included to better understand if this strategy has already been used for pretraining or only for supervised learning.\n\t\n- The multi-scale prediction strategy is similar to the multi-level loss adopted in Scale-MAE [4], one  would appreciate a clear comparison to state the novelty of the proposed strategy defined as a contribution.\n\t\n- The loss function is computed in the pixel space whereas literature in MAE (see reference in related work) focuses the loss in the token space. One would appreciate to clearly state why the loss is not applied in the token space. Is this choice possible within the given framework? If yes, what would be the difference in the training process in terms of performances?\n\t\n- Even if the selected methods are relevant for comparisons in Table 2, one may appreciate comparisons with more recent approaches that already shown the limitation of MAE-only based pretraining. In particular, one may include CROMA [5] pretrained with a combination of MAE and contrastive learning, or AnySat [6]  showcasing a JEPA framework for remote sensing foundation models. Both have shown impressive results and would be relevant competing methods.\n- Experiments comparing performance with competing method would gain robustness if standard deviations would be included, e.g. by simply changing the initialisation of the linear prob.\n\t\nI would like to emphasize that this submission is already of high quality and could have a significant impact on the community. I would consider increasing my rating if the weaknesses are properly considered.\n\nReferences:\n\n[1] J. Han et al., Redet: A rotation-equivariant detector for aerial object detection. In CVPR 2021.\n\n[2] X. Yang and J. Yan. Arbitrary-oriented object detection with circular smooth label. In ECCV 2020.\n\n[3] J. Han et al., Align deep features for oriented object detection. In IEEE Transactions on Geoscience and Remote Sensing 2022.\n\n[4] C. J. Reed et al., Scale-MAE: A scale-aware masked autoencoder for multiscale geospatial representation learning. In ICCV 2023.\n\n[5] A. Fuller et al., CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders. In NeurIPS 2023\n\n[6] G. Astruc et al., AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities. In CVPR 2025.\n\nQUESTIONS:\nQuestions, comments and typos:\n\t\n- Figure 1: please increase the size of all the text within the figure.\n\t\n- Line 19, 71, 335: It is mentioned that extensive experiments have been conducted on \"object detection\". Shouldn't it be \"change detection\" instead (Line 305)?\n\t\n- Related work: Please consider referring to CROMA, a foundation model mixing masked autoencoder and contrastive learning [5].\n\t\n- Related work: Please consider referring to AnySat, a foundation model based on JEPA which could be considered as a MIM method [6].\n\t\n- Figure 3, caption: What does \"high-value patches\" mean here? Not fined in the caption.\n\t\n- Figure 4: \"LBP\" is not defined within the figure.\n\t\n- Line 226: Missing hyperlink to Section 1.\n\t\n- Line 239: \"the autoregressive process compares them with their corresponding tokens in the original image for learning\", it is not clear if the comparison correspond to the loss computation, which seems to be at the token level in this sentence, whereas clearly explained as at the pixel level in equation 2.\n\t\n- Table 3 (b): Why selecting the LBP feature descriptor instead of HOG, where HOG shows better performance?\nPlease consider moving the figures either at the top or bottom of each page for better readability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-21"}

comments: SUMMARY:
The paper introduces Mamba into the field of remote sensing imagery, utilizing autoregressive pretraining to construct a foundational model. It proposes a rotation-aware pretraining mechanism and multi-scale token prediction. The contributions of this study are as follows:
1)	This paper introduce RoMA, the first self-supervised autoregressive pretraining framework for Mamba architectures in remote sensing, enabling efficient scaling to high-resolution RS imagery. 
2)	It propose a dynamic rotation-aware mechanism that integrates adaptive region cropping and angle-aware embeddings. By guiding the model to predict angular variations during autoregressive learning, it effectively addresses rotational diversity and sparse target distributions, enhancing rotation-invariant feature learning.
3)	It design a multi-scale prediction objective that addresses the extreme variations in object scales, enabling the model to learn robust object representations for downstream tasks.

STRENGTHS AND WEAKNESSES:
Strengths : The idea is novel, and the theoretical reasoning is correct.
Weaknesses: Some parts of the text are unclear.

QUESTIONS:
1. Remote sensing targets are typically small. Could the autoregressive pre-training method lead to the neglect of local information, thereby resulting in insufficient learning of small targets?  
2. How can we ensure that the selection of top k tokens includes the target? If the target is not in a relatively central position, it may still lead to the situation shown in Figure 4(b).  
3. In Figure 3, were larger blocks reconstructed using different orders?
4. Was angle prediction included? How does angle embedding function within the network?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds a Mamba-based pretrained foundation model that addresses the quadratic computational complexity bottleneck of Transformer-based models, enabling more efficient processing of high-resolution remote sensing imagery. It introduces a rotation-aware pretraining strategy that combines adaptive cropping with angular embeddings to better handle sparsely distributed objects with arbitrary orientations, and proposes a multi-scale token prediction objective to cope with large variations in object scale. Furthermore, extensive experiments demonstrate the scalability of the proposed foundation model, showing that its performance improves consistently with increased training data and model parameters, highlighting its potential as a strong foundation model for remote sensing tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.It pioneers the use of the Mamba architecture in self-supervised learning for remote sensing, addressing the inefficiency of traditional Transformer-based models when handling high-resolution imagery.\n\n2.By leveraging the linear computational complexity of Mamba, the proposed framework achieves significant improvements in scalability and computational efficiency, which are critical for large-scale remote sensing applications.\n\n3.The paper introduces well-motivated and technically sound components—including a rotation-aware pretraining strategy and multi-scale token prediction—that are carefully designed to address the unique challenges of remote sensing data, such as orientation variability and scale diversity.\n\n4.Extensive experiments across multiple downstream tasks (scene classification, object detection, and semantic segmentation) demonstrate consistent performance gains over ViT-based models, supporting the generality and robustness of the proposed method.\n\n5. The work is aligned with the development of scalable and general-purpose RS foundation models, and the authors provide scaling law analyses that further validate its long-term potential.\n\n\nWeaknesses:\n\n1. It remains unclear how well the proposed RoMA framework generalizes to different types of remote sensing imagery such as SAR or hyperspectral data.\n\n2.The authors claim that the proposed model has significant advantages in handling high-resolution remote sensing images; however, to my knowledge, remote sensing imagery typically has resolutions much higher than 1248×1248. The paper should consider evaluating on higher-resolution data to more convincingly demonstrate its superiority in high-resolution scenarios.\n\nQUESTIONS:\n1.While the proposed framework shows promising results on standard optical RS datasets, could the authors comment on or provide evidence regarding the applicability of RoMA to other types of remote sensing data such as SAR or hyperspectral imagery? Since these modalities have very different data characteristics, it is important to clarify whether the method generalizes beyond the current experimental scope.\n\n\n2.The paper claims advantages in handling high-resolution images. However, the highest resolution used in the experiments appears to be 1248×1248, which is relatively modest compared to typical operational remote sensing images that can reach tens of thousands of pixels per side. Could the authors either include experiments on higher-resolution datasets or clarify the limitations of the current approach in truly large-resolution scenarios?\n\nLIMITATIONS:\nDespite its contributions, this work has a few limitations. First, the generalizability of the RoMA framework to other visual domains or non-optical remote sensing modalities, such as SAR or hyperspectral imagery, remains unverified. These modalities exhibit different spatial and spectral properties that may challenge the current design. Second, although the framework is designed to handle high-resolution images, the experiments are conducted on images up to 1248×1248 resolution, which may not reflect the challenges posed by ultra-high-resolution remote sensing data commonly used in real-world applications. Future work could explore extending RoMA to broader data modalities and scaling to much higher image resolutions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}