RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing
Authors: Fengxiang Wang
Yulin Wang
Mingshuo Chen
Haotian Wang
Hongzhen Wang
Haiyan Zhao
Yangang Sun
Shuo Wang
Di Wang
Long Lan
Wenjing Yang
Jing Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Autoregressive Pretraining
Mamba
Remote Sensing
Foundational Models
Primary area: deep_learning

Abstract: Recent advances in self-supervised learning for Vision Transformers (ViTs) have fueled breakthroughs in remote sensing (RS) foundation models. However, the quadratic complexity of self-attention poses a significant barrier to scalability, particularly for large models and high-resolution images. While the linear-complexity Mamba architecture offers a promising alternative, existing RS applications of Mamba remain limited to supervised tasks on small, domain-specific datasets. To address these challenges, we propose RoMA, a framework that enables scalable self-supervised pretraining of Mamba-based RS foundation models using large-scale, diverse, unlabeled data. RoMA enhances scalability for high-resolution images through a tailored auto-regressive learning strategy, incorporating two key innovations: 1) a rotation-aware pretraining mechanism combining adaptive cropping with angular embeddings to handle sparsely distributed objects with arbitrary orientations, and 2) multi-scale token prediction objectives that address the extreme variations in object scales inherent to RS imagery. Systematic empirical studies validate that Mamba adheres to RS data and parameter scaling laws, with performance scaling reliably as model and data size increase. Furthermore, experiments across scene classification, object detection, and semantic segmentation tasks demonstrate that RoMA-pretrained Mamba models consistently outperform ViT-based counterparts in both accuracy and computational efficiency. The source code and pretrained models have be released at https://github.com/MiliLab/RoMA.

RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing
Fengxiang Wang', Yulin Wang", Mingshuo Chen 3, Haiyan Zhao², Yangang Sun?,
Shuo Wang', Hongzhen Wang² Di Wang45; Long Lan', Wenjing Yang Jiig Zhang
I College of Computer Science and Technology, National University of Defense Technology, China 2 Tsinghua University, China Beijing University of Posts and Telecommunication, China School of Computer Science, Wuhan University, China Zhongguaneun Academy, China
88
60
CPPUI Menooyy (Mi1))
Vix VIl
Our vrr
86
Ars
55
PPS
Manba
1301
82
AID
UCM /Scea. Class.
Secn. Class,
Resolution
Revelution
Figure 1: Comparison of ViT (pretrained with MAE (2) and our Mamba model (pretrained with
RoMA) in scene classification, change detection, and semantic segmentation. Mamba outperforms ViT while being more computationally and memory efficient for high-resolution images. Notably, Mamba-B achieves 1.56x faster inference and reduces GPU memory usage by 78.9% on 1 1248 x 1248 resolution images (6084 tokens per image) on a single NVIDIA 4090 GPU (batch size 2).
Abstract
Recent advances in self-supervised learning for Vision Transformers (ViTs) have fueled breakthroughs in remote sensing (RS) foundation models. However, the quadratic complexity of self-attention poses significant barrier to scalability, par- ticularly for large models and high-resolution images. While the linear-complexity Mamba architecture offers a promising alternative, existing RS applications of Mamba remain limited supervised tasks on small, domain sppecifi--seiieceee datasets, To address these challenges, we propose RoMA, a framework that enables scalable self-supervised pretraining Mamba-based RS foundation models using large- scale, diverse, unlabeled data. RoMA enhances scalability for high-resolution images through tailored auto-regressive learning strategy, incorporating two key innovations: 1) rotation-aware pretraining mechanism combining adaptive crop- ping with angular embeddings to handle sparsely distributed objects with arbitrary orientations, and 2) multi-scale token prediction objectives that address the extreme variations in object scales inherent to RS imagery. Systematic empirical studies validate that Mamba adheres RS data and parameter scaling laws, with perfor- mance scaling reliably as model and data size increase. Furthermore, experiments across scene classification, changing detection, and semantic segmentation tasks demonstrate that RoMA Rpretrrrred Mamba models consistently outperform ViT- based counterparts in both accuracy and computational efficiency. The source code and pretrained models were released at RoMA
1 Introduction
Over the past decade, advancements in remote sensing (RS) technology and more efficient data acquisition have significantly enhanced applications in ecosystem monitoring ( and natural disaster management ( These applications rely on crucial model capabilities such as scene classification (5)
*Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

object detection [6], change detection [7]. and semantic segmentation 8 However, training solely on limited task sask-specific data restricts the scale and generalizability of current RS deep learning models.
Recent breakthroughs in self-super learning (SSL) 12 9 have led to the development of RS 14
that offer robust feature representations and excel across various remote sensing tasks, However, many of these tasks involve high-resolution such as the 4,000 4,00 pixel images in the DOTA dataset for object detection. Most RSFMs rely on Vision Transformer -biT)- attention architectures, whose quadratic complexity limits their practicality on high- righluesouution data. To overeome this challenge. researchers are exploring pretraining RSFMs on architectures with linear complexity, with Mamba 17 emerging as promising alternative.
foundation models (RSFMs) [T0 12
13
15 16
The Mamba architecture is well-regarded in remote sensing for its efficient inference with high- resolution images in downstream tasks [8] [9]. However, current Mamba-based studies are limited to small-scale training datasets, restricting their exposure to diverse remote sensing data. This contrasts with trends in ViT-based RSFMs, which use self-supervised pretraining harness extensive unlabeled data. Therefore, exploring self-supervised learning for Mamba to harness large-scale remote sensing data-and thereby compete with ViT-presents promising, yet underexplored, direction.
Autoregressive pretraining [20] 21 offers a principled solution to Mamba's 's sequence continuity challenges by representing images as 1-D sequences and employing next-token prediction. Its causal token dependencies naturally align with Mamba's 's unidirectional tinear-time scanning, preserving spatial coherence without the disruptions introduced by masking. While this approach has been successfully applied to natural images [20 21 22]. RS images present unique challenges that remain largely unaddressed. We highlight three key challenges: (1) RS images often contain sparsely distributed foreground objects amid complex backgrounds. (2) Unlike objects in natural images, which typically maintain fixed orientations due to gravity. overhead RS images feature objects at varying orientations. (3) The wide range of object sizes in RS images complicates the extraction of effective representations. These challenges naturally lead the question of whether Mamba-base based RSFMs can scale efficiently with both increasing model size and larger data volumes mirroring the performance improvements observed in self-supervised pretrained VIT architectures [23 24].
To address these challenges, we propose Rotation-aware Multi-scale Autoregressive learning (RoMA), framework that enables scalable self-super pretraining of Mamba-based RSFMs using large- scale, diverse, unlabeled data. Specifically, RoMA enhances scalability for high-resolution images through tailored autoregressive learning strategy, incorporating two key innovations: (1) rotation- aware pretraining mechanism combining adaptive cropping with angular embeddings to handle sparsely distributed objects with arbitrary orientations. By identifying key regions for rotation augmentation, it enhances rotation-invariant representation learning. Additionally, it embeds angle information during rotated cropping and requires the model to predict angular changes during autoregressive pretraining, further reinforcing votation-invarian visual representations; and (2) multi- scale token prediction objectives that address the extreme variations in object scales inherent RS imagery. By aggregating predicted token information across multiple spatial scales, this strategy helps Mamba capture more complete and structurally robust object representations during autoregressive pretraining.
Building on RoMA, we investigate its potential for pretraining Mamba-based RSFMs. Through systematic empirical studies, we confirm that Mamba aligns with RS data and parameter scaling laws, exhibiting reliable performance improvements model and data size increase. Additionally, experiments across scene classification, changing detection, and semantic segmentation tasks show that RoMA- -retrrirt Mamba models consistently surpass its ViT-based counterparts in both accuracy and computational efficiency.
The contributions of this study follows:
(1) We introduce RoMA, the first self-supervised autoregressive pretraining framework for Mamba architectures in remote sensing, enabling efficient scaling agery. RoMA validates that Mamba-based RSFMs follow scaling laws, achieving consistent performance gains with larger models and datasets.
to high-resolution RS im-
(2) We propose a dynamic rotation-aware mechanism that integrates adaptive region cropping and angle-aware embeddings. By guiding the model to predict angular variations dur-
2

---

ing autoregressive learning, it effectively addresses rotational diversity and sparse target distributions, enhancing rotation-invarian feature learning.
(3) We design multi-scale prediction objective that addresses the extreme variations in object scales, enabling the model to learn robust object representations for downstream tasks.
2 Related Work
Remote Sensing Foundation Models. While vast amounts of RS data exist, much of it remains unlabeled and thus inaccessible for supervised learning [25). Recently, self-supervised learning frameworks have been employed to learn representations for tasks such as scene classification,
object detection, and semantic segmentation, with methods falling into generation- based [2] and contrastive learning-based -ranin [26] categories. Notably, GASSL, (27 and CACo (2X utilize spatio- temporal information, while SeCo [29] focuses on multiple Earth locations at different timestamps. Beyond representation learning, rotation-aware detection has also been widely studied in RS. ReDet introduces
backbone, CSL [31 addresses angle boundary issues by turning regression into fine-grained classification, and S2A-Net [32 enhances detection accuracy by aligning features with rotated anchors. Most recent work in RS has primarily focused on Masked Image Modeling (MIM), categorized by general image knowledge [33. large parameter scales [34,, praii-temporia information (II), and multi-sensor data [35 36 37 (8), with multi-scale methods
30
13 14 [5] improving performance. A recent study further explores plain ViT as a remote sensing foundation model by introducing Rotated Varied-Size Attention (RVSA) mechanism to better handle arbitrarily oriented objects [39]. MA3E
44) incorporates angle factors into MIM training. In parallel, multimodal foundation models have emerged to bridge heterogeneous RS modalities. CROMA 4 combines contrastive radar radar-optical pretraining with masked reconstruction to learn rich multimodal RS representations, while AnySat [42] adopts JEPA-based joint-embedding framework with scale-adaptive encoders to unify various resolutions, scales, and modalities. Despite these advances, most methods focus on ViT-based RSFMs and MIM pretraining, while the Mamba-based autoregressive models remain unexplored.
Vision Mamba in Remote Sensing. Recently, the Mamba architecture has excelled in NLP and has been adapted to the vision domain to address visual problems. Vision Mamba (Vim) [7] uses Vim blocks, consisting entirely of Mamba layers, with forward and backward scanning to model bidirectional representations. Vmamba (43 incorporates Visual State Space (VSS) blocks, combining Mamba and 2D convolution layers, supported by Swin Transformer [4) llkee pyramid architecture. The vision Mamba architecture has also expanded into remote sensing, producing various Mamba- based projects, categorized into four types: classification, detection, segmentation, and others. For classification, SSMamba 45 and SpectralMamba [8] handle hyperspectral data, while RSMamba [19] and RSCaMa [47] focus on change detection. Segmentation methods include Samba (48) and RS- RS-Mamba [49], which use Mamba alone. Despite the growing research in RS using Mamba, current work limited to supervised training on small-scale datasets, not fully exploiting the vast RS data.
46) focuses on visible light. Detection methods like ChangeMamba
Self-Supervised Learning in Vision. Inspired by the success of self seff-super learning in NLP, visual self-supervised learning methods are thriving in three main categories: contrastive learning
26 [5 autoregressive learning 220 |1 [2] and Masked Image Modeling (MIM) [9] 2] Current
research on MIM focuses on regression targets and masking strategies. Various targets include
discrete tokens [53] HOG features
15 deep features (55) and frequencies (56), have already
been explored. However, MIM methods often face training issues while pretraining the Mamba architeture [22 Recently, autoregressive pretraining in the visual domain has been explored. Most works, like VAR (21). have explored the application of autoregression in image generation. We are more focused on the work of autoregression self-supervised pretraining. iGPT [20 first highlighted the potential of autoregressive pretraining as a general self-supervised visual representation strategy.
SAIM [5T] RandSAC (57) and AIM [52] explored further. These works mainly focus on pretraining the ViT series and have not explored pretraining the Mamba series. ARM [2 firstly explored the
compatibility of autoregressive pretraining with Mamba on the ImageNet (58 dataset, but has not considered the specific issues of the RS field, like the rotation-invariant representations and various sizes information in RS images Notably, while ARM has explored models of different sizes on natural images, it has not evaluated Mamba's autoregressive pretraining performance across varying data
3

---

scales. In the RS field, we are the first to establish the relationships between Mamba's pretraining performance with data volume, and model size
3 Method
Ingut Image
Putches Sequences
Ensxder Type
Decader Type
Oatput
Chas
Patching
K.
oSS
Quass Sequiirce
Nett Tiken Preilitivn
MAR
Masking
VII Encxder
MAE Decnder
Figure 2: Comparison between our autoregressive pretraining strategy and the standard MAE method. (I) RoMA encodes all patches using Mamba encoder, whereas MAE encodes only randomly sampled subset. (2) RoMA predicts the next token in a sequence to capture continuity, while MAE only reconstructs masked patches.
3.1 Preliminaries
Autoregressive Model. Considering sequence of discrete tokens (æ1, 22 ..... IN), where In E [S] an integer from a vocabulary of size S. The next-token autoregressive model posits that the probability of observing the current token zn depends only on its preceding tokens (æ1, ......... #n-1). This unidirectional token dependency assumption enables the factorization of the sequence æ's likelihood follows:
p(11, T2, AN) = II p(zn I1, T2,..... fnn11)
Training an autoregressive model Ps involves optimizing Ps (2 [i, #2, :2, over dataset. This process, known as the next-token prediction, allows the trained Ps to generate new sequences.
MAE-based Pretraining of Mamba. Previous work (TO T6
15 K3 primarily used MAE-based methods for pretraining Remote-Sensien Fundamental Models (RSFMs), where ViT is served as their visual backbones often.
3.2 RoMA: Rotation-aware Multi-scale Autoregressive learning
We propose the RoMA autoregressive pretraining framework for the Mamba architecture in RS field. Specially, as shown in the Figure extend the iGPT 20) series with KV cache-based prediction method. The Namba-Encoder processes the entire image to compute the Key and Value for all tokens. Then calculate the learnable query vector from the K Key and Value, and compute the loss between the Query and the target ground truth. Building on the autoregressive pretraining structure for natural images, RoMA introduces two key contributions: an adaptive rotation encoding strategy and multi-scale prediction strategy.
3.2.1 Auto-regressive Pre-training of Mamba RS Imagery
Disadvantages of MAE-based Pre-training of Mamba. First, compare properties of general visual pre-training tasks and RS tasks, together with reflection on why MAE-based pre-training are suboptimal choice towards RS imagery data:
Explosion of visual tokens on high-resolution RS data. As informed by Section 1, high- solution RS imagery exhibits numerous visual tokens. In contrast, the quadratic complexity of ViT-based MAT pre-training protocols is computational infeasible towards increasing visual tokens in high-resolution RS tasks (see detailed comparisons on speed, GPU usage and accuracy in Figure.[].

---

Appe
Mutti-Scale
Detaut
Concer
ML
Lx
Key and Valte Seguences
Single Seale
203371
Query Sequeuers
Next Token Prodiction
Figure 3: Overview of the RoMA Pretraining Pipeline. The input image is first divided into patches, and high-value patches are selected for random rotation using the Adaptive Rotation Encoding Strat- egy. These patches then tokenized and processed by the Mamba encoder. The encoded features undergo autoregressive next-token prediction, followed by a multi-scale strategy that computes loss at different scales for gradient updates.
Disruption of MAE on Visual Tokens in Mamba's Architecture. To be specific, MAE learns semantic representations by a meet--aasinnteenene renesteconstructing pipeline. However, observe that the mask operation in MAE disrupts between adjacent tokens, i.e., breaking sequential orders among tokens. As illustrated Figure the mask operation conflicts with the linear scanning operation embodied in Mamba, which aggregates temporally related tokens but not randomly related tokens.
Advantage of Autoregressive Pre-training of Mamba. Autoregressive pre-training aligns naturally with the sequential nature Mamba's 's architecture, which processes input tokens in a temporally ordered manner. Specifically, autoregressive modeling constructs image patches sequentially and predicts the next token based on previous context, mirroring Mamba's 's token-by-token scanning mechanism. This architectural alignment facilitates more coherent temporal dependencies and better token transition modeling. allowing Mamba to learn more structured and semantically meaningful representations. Therefore, the autoregressive training paradigm not only complements Mamba's design but also enhances its ability to model spatial continuity and visual context in remote sensing imagery.
3.2.2 Adaptive Rotation Encoding Strategy
Pre-training is critical for RS Data. As shown in Figure 4 RS images contain redundant airport runway pixels, and varying airplane angles lead different postures and shapes. RS images often contain redundant airport runway pixels, while airplanes appear at various orientations, leading to different postures and shapes. Such directional diversity has been extensively addressed in supervised change detection through rotation-equivariant rotation-invarian However, Autoreeessseeeee pretraining for natural images does not consider uneven, sparse information distribution and rotational invariance in RS images. For instance, as shown in Figure
designs (30
RS images contain redundant airport runway pixels, and varying airplane angles lead to different postures and shapes. These unique characteristics prompt encoder can learn high knowledge density features with rotational invariance. In RoMA, we outline an adaptive rotation encoding strategy to enhance autoregressive pretraining for remote sensing, RoMA omits explicit angle prediction. Instead, angle embeddings introduce directional priors that help the model leamn invarrnnn representations during pre-training, without supervision from angle labels.
rethink how the
1. Split the input image.
Associate each patch x! with a score.
3. Selecting the patch (16x 16) with the highest score.
Compute all 96x 96 candidate boxes containing the patch and select the with the highest
5

---

value.
5. Compare the 96x96 patch to the image-wide patch mean. If it exceeds the mean, select it; otherwise, proceed to 64x64 patches until surpasses the mean.
We then detail each step outlined above for rotation-invarian encoding strategy: (1) Step of ARES: We split the input image. Rx into N = (H W)/p² non-overlapping patches x E RNx((²²), where p is the patch size, W) ) the size of the input image, and C' is the number of channels; (2) Step 2 of ARES: We associate each patch x2 with a score, computed via efficient feature descriptor F, c.g., LBP [59], and then select the token with the highest values; (3) Step 3 of ARES: Let to denote the selected token. Then, centered on the patch represented by top. we expand its size to obtain a larger, more suitable region. (4) Step 4 of ARES: After extracting square token region with edge length of L = (96, 64, 32). compute its average feature value and compare with the average feature value of each patch in the original image; (5) Step 5 of ARES: If token, has higher average feature value, it is identified and proceeds to the rotation step. Otherwise, a smaller L is selected, and step 4 is reapplied. This process repeats up three times until region with higher average feature value than the original image is reached.
high-value region
Migrating Information Loss. The selected patch is cropped to generate diverse rotated remote sensing scenes, while potential information loss the
Retation
Adaptive Path Selecting
edge pixels might occur. To mitigate
this, follow MA3E [40] and apply center cropping to retain the inscribed square (marked in yellow) within the largest inscribed circle. This region, oriented in any direction, replaces the original scene and introduces explicit angular variations. In addition po- sitional embeddings, we also incor- porates learnable angle embeddings shared across patches within the ro- tated crop, i.e., served as implicit cues, aiding the model perceive angu- lar changes while distinguishing them from the background.
OC
Sclettvee Racch Candidiee Foach
16x 16
96x 96
61
(a)
(b)
Figure 4: Illustration of the Adaptive Rotation Encoding Strategy. (a) Pipeline of the Adaptive Rotation Encoding Strategy. LBP refers to Local Binary Pattern. (b) Random patch selection for rotation without adaptive selection. The random approach in (b) disrupts object information in the RS image.
Finally, the Adaptive Rotation Encod-
ing Strategy processes the image before feeding it into the Mamba-based encoder for representation learning. RoMA follows the standard Mamba architecture [22] without modifications, focusing purely on pretraining Mamba for RS field. While architectural improvements could enhance performance and efficiency, RoMA prioritizes pretraining strategies, leaving further Mamba optimizations to future research.
3.2.3 Multi-scale Prediction Strategy
Images are continuous 2D signals. To apply autoregressive pretraining via next-token prediction, two steps are required: (1) Convert images into discrete tokens. (2) Define them as a mie-dimensional sequence for unidirectional modeling. Methods like iGPT (20] and VAR (21) tackle these challenges by slicing images into segments and arranging them into a feature sequence in specified one- dimensional order.
However, discussed in Section directly applying this method to RS images fails to consider key factors. Unlike natural images, which focus on visual semantic understanding, RS images focus on surface measurement information (60). Arbitrarily disrupting spatial relationships in RS images leads to fundamentally different interpretations. For example, token (+5)) and its neighbors (((+1.)) re (f.f+)) are closely related due to planar surface measurements. As seen in Figure 2 the token (1.))

---

SpaahevViguFi)
SpaceNat Yoar3)
55
55
82
77
74
45
Dus Viler
Thea Witmz
Volane
Madd Parancsrs
Modd Picxietess
(Thouarch)
(Milkan))
(a) Data Volume Scaling
(h) Model Size Scaling
Figure 5: Scaling with Data Volume and Model Size. Each experiment was conducted three times, and the average was reported the final result. (a) We showcase the Mamba-Base model's performance on three downstream tasks after RoMA pretraining with different data scales. (b) We compare the performance of various Mamba model sizes on three downstream tasks, all pretrained with 4 million data using RoMA. Details on pretraining and downstream task configurations are provided in Section
representing the part of plane is closely related to vertical neighboring tokens and some distant vertical tokens. The unidirectional flattening autoregressive methods compromises surface information representation in RS images. Therefore, we introduce multi-scale prediction strategy to mitigate the effects of unidirectional flattening.
The Mamba-bas encoder generates key and value feature representations for each token. During decoding, we apply cross-attention with token-level causal masking to sequentially predict tokens, ensuring each token relies only on previously observed ones. The decoder performs autoregressive prediction at the token level. After the decoder generates the reconstructed outputs, the autoregressive module aligns them with the corresponding regions of the original image for supervision. The reconstruction quality optimized using the mean squared error (MSE) loss between the predicted and original pixel values.
Building on MSE loss function, we concatenate token representations from each decoder block at higher scale following a predefined raster order. A fully connected multi-layer perceptron (MLP) then reconstructs the pixel values of the next block. Notably, I& E 16x16 while a higher scale, spatial aggregation results in yn. E R is the pixel size value of larger scale. The formula is as follows:
K
A N
(()) K
[0,(0; - xx112
Σ
(0: yn
where 8 represents the network parameters, ground truth pixel value of the n-th cluster block, and ÿr, (0: V/en) denotes the reconstructed value based on and preceding tokens (yen), is the number of all tokens in an image, I& denotes the ground truth pixel value of the k-th token, and (6; (xcx) is the reconstructed value based the
the number of cluster blocks in an image, Yn the
network parameters (0) and preceding tokens (x=c)) in the sequence. The parameter À regulates the contribution of the MSE loss from higher-scale cluster blocks to the overall loss.
3.3 Scaling Mamba-based RSFMs
To investigate the scaling potential of the self- supervised pretrained Mamba architecture for developing powerful RSFMs. With RoMA, we analyze the relationships between Mamba's 's per- formance with model parameters, and data scale. The scalability of ViT architectures pretrained with MAE has been well established, demon- strating performance gains with increasing data volume and model size [23] 24). However, no prior work has systematically examined whether the Mamba architecture follows similar trend
Table The configuration of different architec- ture variants.
Model
Block
Width
Depth
Param.(M)
VIT-T
Attention-MI.P
192
12
5.7
Mamba-T
Mumba+MIPP
192
12
5.3
ViT-S
Atiention-Mil
384
12
Mamba-S
Mamba+MI.P
384
12
21
VIT-B
Attention+ML. ML.
768
12
86
Manbu-B
Mambu+MLP
768
12
VIT-L
Attention-MLP
1024
24
307
Mamba-L
Mamba+MLP
1024
24
297
RS field. For the first time, we explore Mamba's 's scaling behavior in RS domain using the RoMA pretraining method.
7

---

Table Results for scene classification, change detection, and semantic segmentation. "TR" represents the ratio of training data. indicates results from MA3E 40 and MTP [T.. f denotes our reproduction with their official code.
Scene
Change Delection
Semantic Segmentation
Classilicatio
Methods
Publication
Backhone
Params
AID 62
UCM 63
OSCD
64
SpaceNetvl 65
OA(TR= 50%)
OA(TR=509)
mF1
Natural Image pretraining
MoCo v3
50
ICCV'21
VITB
86M
78.72
38.34
DINO
JCCV'21
VIT-B
86M
78.51
40.04
MAE 2
CVPR 22
VIT-B
86M
84.21
52.75
SimMIM
CVPR'22
VIT-B
86M
83.19
51.48
LoMaR
66
Arxiv'22
VII-B
86M
82.26
51,89
MixMAE
67
CVPR
Swin-B/WIH
88M
81.53
50,63
ARM
ICLR'25
Mamba-B
85M
81.14
50.41
47.28
77.89
Image pretruining
SeCo
29
10CV'21
ResNet-50
25.6M
78.26
47.45
47.67
77.09
C
CVPR'23
ResNet-50
77.81
40.53
52.11
77.94
SatMAE
|1
NIPS'22
VIT-L
307M
55.10
34.28
52.76
78.07
ScaleMAE
13
ICCV'23
VIT-L
307M
48.46
28.19
GFM 35
1CCV'23
Swin-B
REM-
80.58
49.73
RVSA 10
TGRS'23
ViT-B-RVSA
86M
84.06
50.86
50,28
79.56
SaMMA+++
CVPR'24
VIT-L
307M
85.98
MA3E +
40
ECCV'24
VIT-B
86M
85.86
55.69
RoMA
Mamba-B
85M
87.36
59.45
55.63
79.50
Scaling with Data Volume: Mamba shows a clear performance boost on downstream tasks as the pretraining data volume grows. We pretrain the Mamba-Base model with RoMA across various data scales and evaluate its performance in the downstream tasks. As illustrated in Figure 55 larger datasets lead to significant improvements. Mamba-based RSFMs exhibit no significant performance bottlenecks across a broad pretraining data scale from 62.5K to 4M, achieving data learning capa- bilities on par with ViT-based RSFMs. We look forward to future advancements of data volume in remote sensing, where larger datasets can further enhance Mamba-based RSFMs through our RoMA pretraining framework.
Scaling with Model Size: Mamba's 's performance also improves with increasing model size. We conduct extensive pretraining on four model variants- -Tiny, Small, Base, and followigllo the configurations in our code. As shown in Figure 5b] larger models consistently achieve superior results on downstream tasks. Although Mamba-Large surpasses Mamba-Bat Base AID dataset, its performance gain remains limited, likely due to insufficient pretraining. With only 300 epochs on million samples, the training may not be adequate for 297M-parameter model. Due to experimental constraints, we did not extend pretraining to 800 epochs in MAE. The OSCD and SpaceNet experiments are ongoing, with updates to follow, However, these results do not alter our key findings: Mamba-basee RSFMs pretrained with RoMA demonstrate performance gains as model parameters scale. While this growth remains inconclusive in more large-scale experiments, we anticipate future research will further explore Mamba's 's scaling potential.
4 Experiments
We pretrain Mamba extensively using RoMA and assess its effectiveness across diverse downstream tasks. Finally, we conduct thorough ablation studies on RoMA's design choices.
Pretraining Setup. Our pretraining experiment setup largely follows ARM [22.. We train both the Mamba-B on the OpticalRS-4N [16). We adjust the input image to a size of 196 196, with a patch size of 16, using the Adam optimizer and cosine learning rate scheduler. The initial learning rate is set to 1.5e-4, and batch size is set 256, with a epoch 400.
Downstream Tasks. We further evaluated RoMA across three key downstream tasks: scene clas- sification, changing detection, and semantic segmentation. In addition to benchmarking against ViT-based RSFMs, we compared RoMA with other pretraining methods for natural images. These encompass methods leveraging contrastive learning and generative learning and autogressive pre- training approaches, ARM [22 The Mamba-B architectures strictly adhere to the simplest Mamba design from ARM, without any modifications, allowing us to exhaustively test the advantages of the RoMA pretraining framework.
8

---

Table Ablation study on the design choices of RoMA with Mamba-B backbone. We report the top-1 accuracy (%). The default settings of RoMA are highlighted in grey.
(b) Feature Descriptor in Adaptive
(a) Main ablation. Adaptive Ro- Rotation Encoding Strategy. Local (c) Selecting Patch Size in Adaptive tation Encoding Strategy (ARE) Binary Pattemn (LBP) measurement Rotation Encoding Strategy. Three and Multi-scale Prediction Strategy outperforms the Wavelet Transform layers the most effective choice. (MSP) significantly improve RoMA, and Histogram of Oriented Gradi-
ents (HOG).
ART
MSP
AID
Festure Descripue
AlB
0A (IR--20%)
CA (ZR0-58%
OA (TR=200%
OA (TR=55PS
78,00
76.90
Wordkt HDG
LBP
71.94
71.42
71.38
17AM
18-32
96
71.23
70.48
96-64-32
71.70
(f) Various Scales choices in Multi-
(d) Threshold for patch selection Coefficient A. The variation of scale Prediction Strategy. Experi- in the Adaptive Rotation Encoding the coefficient À in Multi-scale Pre-ments were conducted using the stan- Strategy is based on the image's diction Strategy. balances autore-dard 16x16 patch as baseline, with reconstruction and sparsity additional combinations multiples of 2-6 6.
overall average computed from the Feature Descriptor.
regularization.
Thrsheld
Coefiicint
AID
Alb
(TIb=20%)
OA (TR:-00)
PIAEN HOOK
0111 14
0.5% Aeg.
Amg-
71
68.30
74.18
72.39
21.31
7749
28.25
1432
1.%% Aigg
71.20
75000
7811
Scene Classification. By freezing the model's parameters and fine-tuning only the final fully connected (FC) layer. linear probing effectively demonstrates its feature extraction ability. Since
full-parameter fine-tuning already achieves over 99% performance on classification datasets, like
AID 622 we prefer linear probing rather than fine tuning. We use two scene classification datasets: AID and UCM (63. with training details, including the train split ratio, following [T0 13.
Evaluation is based on overall accuracy (OA). The results in Table show RoMA's competitive
performance compared to other pretraining methods.
Change Detection. We used the OSCD [64] dataset consisting of RGB images for change detection.
Following previous works [6], we kept the experimental setups consistent, using UNet (68) as the decoder. On the OSCD dataset, our method outperforms ARM (22 and other methods that overlook
rotational invariance and information sparsity and varying object size issues in RS.
Semantic Segmentation. We further evaluate the pretrained model on semantic segmentation tasks, using common remote sensing datasets: SpaceNetv1 1 [65] Our implementation follows [61), using UperNet [69] as the segmentation framework. However, in pixel-level tasks, Mamba- based RSFMs show less pronounced advantages compared to other downstream tasks. We attribute this to RoSA's autoregressive pretraining, which prioritizes multi-scale patch-based targets over pixel-level prediction.
Ablation Study. Due to resource constraints, we conducted the ablation experiments using the MillionAID [70] dataset and trained for 400 epochs. Table Bajseensss the performance of RoMA's two main contributions, with ARM as the bascline. The experiment shows a significant gain in feature extraction capability. Tables 3bl and 3c adding the Adaptive Rotation Encoding Strategy on the bascline. Table 3b]pprriiinnngg the effects of different Feature Descriptors. We believe the Feature Descriptor method can be further optimized without affecting the Adaptive Rotation Encoding Strategy's 's effectiveness. Table 3e evaluates patch sizes for rotation, showing that various sizes improve performance. Tables 3d] Be) and 3f examine the Multi-scale Prediction Strategy. The parameter and threshold selections are shown in Tables [3d and [3e] while Table 3r presents the performance of aggregating spatial features from multiple scales. Our results indicate that adding excessive multi-scale information doesn't 't guarantee improved performance; instead, using only large-scale aggregated information along with the original 16x16 patch data yields better results.
Table Peak GPU memory usage (MB) across different input resolutions.
Table 5: Inference speed (samples/sec) across different input resolutions.
Resolation
768
1021
1248
1530
2018
3072
4096
Resolation
768
1024
1248
1520
2048
3072
4096
RoMA-Baw
2693
4504
6336
9434
16934
37485
66357
RoMA-Base
15.91
11.43
7.86
4.37
11726
21531
52106
00M
00M
0OM
ViT-Base
22.11
9.94
4.99
2.57
0OM
0OM
0OM

---

Further Analyses
Scalability to Ultra- Ultra-iigh-esoolution Inputs. To further examine ROMA'S scalability, we evaluated it on inputs ranging from 768 x 768 to 4096 x 4096. Both RoMA-Baa and ViT-Base were tested for GPU memory usage and inference speed on a single NVIDIA A100 (batch size = 1). As shown in Tad 5 RoMA scales stably up to 4096 x 4096, while ViT-Base fails beyond 2048 x 2048. These results further verify RoMA's computational efficiency and suitability for ulrr-hiig-ressolution remote sensing imagery.
Ability to Learn Small Targets. To further analyze RoMA's ability capture local information, we evaluate its performance on small-object categories in the iSAID dataset [71]. We compare UperNet [99 with different backbones, following the RingMo [72] fine-tuning protocol. As shown in Table Our method achieves the highest overall mloU and also shows notable improvements on small-object classes, especially Small Vehicle (average width 15 pixels), where RoMA surpasses all other backbones. These results suggest that the proposed adaptive region cropping strategy effectively increases the visibility of small foreground objects during pretraining, allowing the model to learn more discriminative representations.
Table 6: Fine-tuning performance on iSAID 71
(following RingMo (72) settings).
Method
Backbone
mloU
Ship (33 px)
Small Vehicle (15' px)
Swimming Pool (341
Plane (53' px)
UperNet
IMP-ResNet-50
61.9
65.9
48.8
44.5
83.8
UperNet SeCo-ResNet-50
57.2
63.9
44.8
9,3
83.3
UperNet RSP-ResNet-50
61.6
64.2
47.5
43.8
82.8
UperNet
ViT-B-RVSA
63.8
46.7
85.6
UperNet
VITAE-B-VSSA
63.5
69.6
51.9
47.5
85,4
UperNet
RingMo
67.2
73.5
51.2
48.9
85.7
UperNet
RoMA-B (Ours)
67.4
73.8
53.7
51.8
86.0
Region Selection. To further validate the effectiveness of the top-k token selection mechanism, we conducted an additional experiment on the SpaceNet V1 (65) building segmentation dataset. This ex- periment aims to examine whether the proposed adaptive region cropping can consistently capture target regions compared with random cropping under the same resolution settings. As shown in Table 7 our method achieves a foreground capture accuracy of 75.09%. significantly higher than the 38.56% of random cropping. These results demonstrate that the top-k strategy effectively preserves informative regions, leading to more reliable semantic representations for autoregressive modeling.
Effectiveness of Top-k
Table 7: Foreground object capture rate on SpaceNet V1 (65).
Cropping Strategy
Accuracy (%)
Random Cropping
38.56
Top-h Adaptive Reglon Selection (Ours)
Feasibility of Token-space Reconstruction Loss. To evaluate the flexibility of RoMA, we compared pixel- space and token-space reconstruction losses. Following BETT [53] and CAE [73], the token-space loss was com- puted using a frozen CLIP 74] teacher, where the student predicted its latent features through lightweight MIM head. As shown in Table g token-space loss consistently outperforms pixel-space loss on UCM and AID, indicating its stronger semantic representation capability and the adaptability of the RoMA framework to diverse pretraining objectives.
Table 8: Comparison between pixel- space and token-space reconstruction loss.
Pretraining Loss Type
UCM-S5 (%)
AID-55
Base (Pisel Space)
$2.39
RoMA-Baa MAAA-BaseT (Token Space)
$6.67
$1.72
6 Conclusion
We introduce RoMA, the first self-supervised autoregressive framework to scale Mamha-based foun- dation models for RS. By leveraging large-scale, diverse, unlabeled data, RoMA enables scalable self-supervised pretraining of Mamba-based RS models. Extensive experiments across scene classifi- cation, changing detection, and semantic segmentation show that RoMA-pretrained Mamba models consistently outperform ViT-based counterparts. Additionally, these models achieve significant efficiency improvements. reducing GPU memory consumption by 78.9% and accelerating inference speed by 1.56x at 1,248 x 1,248 resolution, while maintaining linear computational scaling. Our findings also provide new insights into Mamba's scaling laws, demonstrating consistent performance gains with increasing data volume and model size, highlighting its potential for large-scale Earth ob- servation tasks. Limitations. The current RoMA framework is evaluated mainly on optical imagery, and future work will extend it to multi-source remote sensing data (c.g., SAR and hyperspectral).
10

---

Acknowledgementss: This work was partially supported by the National Natural Science Foundation of China (No. 62372459, No.62376282 and No. 624B2109).
References
[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[2] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 15979- 15988, IEEE, 2022.
[3] Nathalie Pettorelli, Henrike Schulte to Bühne, Ayesha Tulloch, Grégoire Dubois, Cate Macinnis-Ne Ng. Ana M Queirós, David A Keith, Martin Wegmann, Franziska Schrodt, Marion Stellmes, et al. Satellite remote sensing of ecosystem functions: Opportunities, challenges and way forward. Remote Sensing in Ecology and Conservation, 4(2):11-9) 1-33, 2018.
[4] Olalekan Mumin Bello Yusuf Adedoyin Aina. Satellite remote sensing as tool in disaster management and sustainable development: Towards a synergistic approach. Procedia-Social and Behavioral Sciences, 120:365-317 373, 2014.
[5] Liang Huang, Fengxiang Wang. Yalun Zhang. and Qingxia Xu. Fine-grained ship classification by combining enn and swin transformer. Remote Sensing, 14(13):3087, 2022.
[6] Wentong Li. Yijie Chen, Kaixuan Hu, and Jianke Zhu. Oriented reproints for aerial object detection. In CVPR, pages 1829-1838, June 2022.
[7] Curtis E Woodcock, Thomas R Loveland, Martin Herold, and Marvin E Bauer. Transitioning from change detection monitoring with remote sensing: A paradigm shift. Remote Sensing of Environment, 238:111558: 2020.
[8] Xiaohui Yuan, Jianfang Shi, and Lichuan Gu. A review of deep learning methods for semantic segmentation of remote sensing imagery. Expert Systems with Applications, 169: 14417, 2021.
[9] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, pages 9653 9663, 2022.
[10] Di Wang. Qiming Zhang, Yufei Xu, Jing Zhang, Bo Du, Dacheng Tao, and Liangrei Zhang, Advancing plain vision transformer toward remote sensing foundation model. IEFE Transactions on Geoscience and Remote Sensing, 61:1-15, 2023.
[11] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong Hc, Marshall Burke, David Lobell, and Stefano Ermon. Satmac: Pre-training transformers for temporal and multi-spectral satellite imagery. Advances in Neural Information Processing Systems, 35:197-211, 2022.
[12] Fanglong Yao, Wanxuan Lu, Heming Yang. Liangyu Xu, Chenglong Liu. Leiyi Hu, Hongfeng Yu, Nayu Liu, Chubo Deng, Deke Tang, Changshuo Chen, Jiaqi Yu, Xian Sun, and Kun Fu. Ringmo-sense Remote sensing foundation model for spatiotemporal prediction via spatiotemporal evolution disentangling. IEEE Trans. Geosci. Remote. Sens., 61:1-21, 2023.
[13] Colorado J. Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp. Kurt Keutzer. Salvatore Candido, Matt Uytendaelee and Trevor Darrell. Scale-MAE: A scale-aware masked autoencoder multiscale geospatial representation learning. In ICCV, pages 4065 4076. IEEE, 2023.
[14] Maofeng Tang, Andrei Cozma, Konstantinos Georgion, and Hairong Qi. Cross-scale mae: tale of multiscale exploitation in remote sensing. In Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS 223 Red Hook, NY, USA, 2023. Curran Associates Inc
[15] Mubashir Noman, Muzammal Nascer, Hisham Cholakkal, Rao Muhammad Anwer, Salman H. Khan, and Fahad Shahbaz Khan. Rethinking transformers pre-training for multi spectrll satellite imagery. In CVPR, pages 27811-22199 77811927819. IEEE, 2024.
[16] Fengxiang Wang. Hongzhen Wang, Di Wang. Zonghao Guo, Zhenyu Zhong, Long Lan, Jing Zhang, Zhiyuan Liu, and Maosong Sun. Scaling efficient masked autoencoder learning on large remote sensing dataset. arXiv preprint arXiv:2406.1 11333, 2024.

---

[17] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In ICML, 2024.
[18] Jing Yao. Danfeng Hong, Chenyu Li, and Jocelyn Chanussot. Spectralmamba: Efficient mamba for hyperspectral image classification. arXiv preprint arXiv:2404. 08489, 2024.
[19] Hongruixuan Chen, Jian Song. Chengxi Han, Junshi Xia, and Naoto Yokoya. Changemamba: Remote sensing change detection with spatiotemporal state space model. IEEE Transactions on Geoxcience and Remote Sensing, 62:1 2024.
[20] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In ICMI, volume 119 of Proceedings of Machine Learning Research, pages 1691-1703, PMIR, 2020.
[21] Keyu Tian, Yi Jiang, Zehuan Yuan, BINGYUE PENG, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via sext-scale prediction. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 84839-84865. Curran Associates, Inc., 2024.
[22] Sucheng Ren, Xianhang Li, Haoqin Tu, Feng Wang, Fangxun Shu, Lei Zhang, Jieru Mei, Linjie Yang. Peng Wang. Heng Wang, Alan Yuille, and Cihang Xie. Autoregressive pretraining with mamba in vision. In ICLR, 2025.
[23] Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adoock, Armand Joulin, Piotr Dollar, Christoph Feichtenhofer, Ross Girshick, Rohit Girdhar, and Ishan Misra. The effectiveness of mae pre-pretraining for billion-scale pretraining. In ICCV, pages 5484-5494, October 2023.
[24] Cheng Ze Lu, Xiaojie Jin, Qibin Hou, Jun Hao Liew, Ming-Minn Cheng, and Jiashi Feng. Delving deeper into data scaling in masked image modeling. arXiv preprint arXiv:2305. 152488 2023.
[25] Di Wang, Jing Zhang, Bo Du, Gui-Song Xia, and Dacheng Tao. An empirical study remote sensing pretraining. IEEE Transactions on Geoscience and Remote Sensing, 61:1-20, 2023.
[26] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pages 9630 1400,9940, 2021.
[27] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lohell, and Stefano Ermon. Geography aware self-supervised learning. In ICCV, pages 10161-10170, 2021.
[28] Utkarsh Mall, Bharath Hariharan, and Kavita Bala. Change-awan sampling and contrastive learning for satellite images. In CVPR, pages 5261-5270, 2023.
[29] Oscar Mañas, Alexandre Lacoste, Xavier Giró-i-Nieto, David Vázquez, and Pau Rodríguez. Seasonal contrast: Unsupervised pre- Trainiinn from uncurated remote sensing data. In ICCV, pages 9394-9403. IEEE, 2021.
[30] Jiaming Han, Jian Ding, Nan Xue, and Gui-Song Xia. Redet: rotation equivarran detector for aerial object detection. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 2786-2795, 2021.
[31] Xue Yang and Junchi Yan. Arbirrayyorrented object detection with circular smooth label. In European conference computer vision, pages 677-694. Springer, 2020.
[32] Jiaming Han, Jian Ding, Jie Li, and Gui-Song Xia. Align deep features for oriented object detection. IEEE transactions geoscience and remote sensing, 60:1-11, 2021.
[33] Matías Mendieta, Boran Han, Xingjian Shi, Yi Zhu, and Chen Chen. Towards geospatial foundation models via continual pretraining. In ICCV, pages 16806-16816,, 2023.
[34] Keumgang Cha, Junghoon Sco, and Tackyung Lee. A billion-scale foundation model for remote sensing images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pages 1-17, 2024.
[35] Xin Guo, Jiangwei Lao, Bo Dang, Yingying Zhang, Lei Yu, Lixiang Ru, Liheng Zhong. Ziyuan Huang. Kang Wu, Dingxiang Hu, Huimei He, Jian Wang, Jingdong Chen, Ming Yang. Yongjun Zhang, and Yansheng Li. Skysense: A multi-moda remote sensing foundation model towards universal interpretation June 2024.
for earth observation imagery. In CVPR, pages
12

---

[36] Danfeng Hong, Bing Zhang. Xuyang Li, Yuxuan Li, Chenyu Li, Jing Yao, Naoto Yokoya, Hao Li. Pedram Ghamisi, Xiuping Jia, Antonio Plaza, Paolo Gamba, Jón Ati Benediktsson, and Jocelyn Chanus- sot. Spectralgpt: Spectral remote sensing foundation model. IEEE Trans. Pattern Anal. Mach. Intell. (188):227--2444 2024.
[37] Xuyang Li, Danfeng Hong, and Jocelyn Chanussot. S2mae: A spatial-spectral pretraining foundation model for spectral remote sensing data. In CVPR, pages 24088-24097 2024.
[38] Vishal Nedungadi, Ankit Kariryaa, Stefan Ochmcke, Serge Belongie, Christian Igel, and Nico Lang. Mmearth: Exploring multi-modas pretext tasks for geospatial representation learning. In ECCV, pages 164-182. Springer, 2024.
[39] Di Wang. Qiming Zhang, Yufei Xu. Jing Zhang, Bo Du. Dacheng Tao, and Liangpei Zhang, Advancing plain vision transformer toward remote sensing foundation model. IEEE Transactions on Geoscience and Remote Sensing, 61:1-15, 2022.
[40] Zhihao Li, Biao Hou, Siteng Ma, Zitong Wu, Xianpeng Guo, Bo Ren, and Licheng Jiao, Masked angle-aware autoencoder for remote sensing images. In ECCV, pages 360-278. Springer, 2025.
[41] Anthony Fuller, Koreen Millard, and James Green. Croma: Remote sensing representations with contrastive radar-optical masked autoencoders. Advances in Neural Information Processing Systems, 144:5566-5344 2023.
[42] Guillaume Astruc, Nicolas Gonthier, Clement Mallet, and Loic Landrieu. Anysat: One earth observation model for many resolutions, scales, and modalities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15330-19540, 2025.
[43] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 103031 1-33063.. Curran Associates, Inc., 2024.
[44] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 9992-10002. IEEE, 2021.
[45] Lingho Huang, Yushi Chen, and Xin He. Spectral-spatial mamba for hyperspectral image classification. Remote Sensing, 16(13), 2024.
[46] Keyan Chen, Bowen Chen, Chenyang Liu, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsmamba: Remote sensing image classification with state space model. IEEE Geoscience and Remote Sensing Letters, pages 1 1-5, 2024.
[47] Chenyang Liu, Keyan Chen, Bowen Chen, Haotian Zhang, Zhengxia Zou, and Zhenwei Shi. Rscama: Remote sensing image change captioning with state space model. IEEE Geusci. Remote, Sens. Lett.. 21:1-5, 2024.
[48] Qinfeng Zhu, Yuanzhi Cai, Yuan Fang, Yihan Yang. Cheng Chen, Lei Fan, and Anh Nguyen. Samba: Semantic segmentation of remotely sensed images with state space model. Heliyon, 10( 10119): 2024.
[49] Sijic Zhao, Hao Chen, Xueliang Zhang, Pengfeng Xiao, Lei Bai, and Wanli Ouyang. Rs-mamba for large remote sensing image dense prediction. IEEE Transactions on Geoscience and Remote Sensing, 62:1-14, 2024.
[50] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, pages 9735. Computer Vision Foundation IEEE, 2020.
[51] Yu Qi, Fan Yang, Yousong Zhu, Yufei Liu, Liwei Wu, Rui Zhao, and Wei Li. Exploring stochastic autoregressive image modeling for visual representation. Brian Williams, Yiling Chen, and Jennifer Neville, editors, AAAI, pages 2074-200 2081. AAAI Press, 2023.
[52] Alaacidin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Vaishaal Shankar, Alexander Toshev, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. In /CML, ICML'24. JMLR.org, 2024.
[53] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, Beit: Bert pre-training of image transformers. In ICLR, 2021.
13

---

[54] Chen Wei, Haoqi Fan. Saining Xie, Chao- Wu, Alan Yuille. and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, pages 14668-14678, 2022.
[55] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image hert pre-training with online tokenizer. In ICLR, 2021.
[56] Hao Liu, Xinghua Jiang, Xin Li, Antai Guo, Yiqing Hu, Deqiang Jiang, and Bo Ren. The devil is in the frequency: Geminated gestalt autoencoder for self-supervised visual pre-training. In AAAI, volume 37, pages 1649-16500 2023.
[57] Tianyu Hua, Yonglong Tian, Sucheng Ren, Michalis Raptis, Hang Zhao, and Leonid Sigal. Self-supervision through random segments with aautoregressive coding (randsac). In ICLR, 2022.
[58] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pages 248-255, IEEE, 2009.
[59] Marko Heikkilä, Matti Pietikäinen, and Cordelia Schmid. Description of interest regions with local binary patterns. Pattern recognition, 42(3):425-446, 2009.
[60] Mi Zhang, Bingnan Yang, Xiangyun Hu, Jianya Gong, and Zuxun Zhang. Foundation model for generalist remote sensing intelligence: Potentials and prospects. Science Bulletin, 69(23))3652-3656 2024.
[61] Di Wang, Jing Zhang. Minqiang Xu, Lin Liu, Dongsheng Wang, Erzhong Gao, Chengxi Han, Haonan Guo, Bo Du, Dacheng Tao, and Liangpei Zhang. Mip: Advancing remote sensing foundation model via multi-task pretraining, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pages 1-24, 2024.
[62] Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xinoqiang Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 5567):3965-3981, 2017.
[63] Maryam Mehmood, Ahsan Shahzad, Bushra Zafar, Amsa Shabbir, and Nouman Ali. Remote sensing image classification: A comprehensive review and applications. Mathematical problems in engineering, 2022(1):588095,, 2022.
[64] Rodrigo Caye Daudt, Bertr Le Saux, Alexandre Boulch, and Yann Gousscau. Urban change detection for multispectral earth observation using convolutional neural networks. In IGARSS 2018-2018 IEEE International Geoxcience and Remote Sensing Symposium, pages 2115-2118. IEEE, 2018.
[65] Adam Van Fiten, Dave Lindenbaum, and Todd M Bacastow. Spacenet: A remote sensing dataset and
challenge series. arXiv preprint 1rXiv:1807.01232. 2018.
[66] Jun Chen, Ming Hu, Boyang Li, and Mohamed Elhoseiny. Efficient self-supervised vision pretraining with local masked reconstruction. arXiv preprint arXiv:2206.00790. 2022.
[67] Jihao Liu, Xin Huang, Jinliang Zheng, Yu Liu, and Hongsheng Li. Mixmae: Mixed and masked autoencoder for efficient pretraining of hierarchical vision transformers. CVPR, pages 6252-6261, 2023.
[68] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-IICC 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[69] Tete Xiao, Yingeheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pages 418-434, 2018.
[70] Yang Long, Gui-Song Xia, Shengyang Li, Wen Yang, Michael Ying Yang. Xiao Xiang Zhu, Liangpei Zhang, and Deren Li. On creating henchmark dataset for acrial image interpretation: Reviews, guidances and million-aid aid. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 14:4205-4234 2021.
[71] Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. isaid: A large-scale dataset for instance segmentation in aerial images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 28-37, 2019.
[72] Xian Sun, Peijin Wang. Wanxuan Lu, Zicong Zhu, Xiaonan Lu, Qibin He, Junxi Li, Xuee Rong. Zhujun Yang, Hao Chang, Qinglin He, Guang Yang. Ruiping Wang. Jiwen Lu, and Kun Fu. RingMo: A remote sensing foundation model with masked image modeling. IEEE Trans. Geosci. Remote. Sens., 61:1-22, 2023.
14

---

[73] Xiaokang Chen, Mingyu Ding. Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. International Journal of Computer Vixion, 132(1):208-22) 2024.
[74] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction accurately reflect our contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations and future directions at the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We provide analysis for each theoretical result presented in the paper, including formulas where necessary and visualizations when appropriate.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental results are reproducible. Detailed information is provided in the appendix. We also plan to open-source our dataset and code to further contribute to the community.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of the dataset and experimental setup the paper. Due to time constraints and submission size limits, the data and code cannot be included at this stage, but we commit to open-sourcing all datasets and code as soon as possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guide http:/guuideeeoeeuuuiiieeee (oodeSubmissionoool1
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
cilaps.cc/publle http://ullccgguddeeeeeeeee httpa////gudeee//eeeuubbsiissee
http:/ddeeeeeeeeeeeuuaasiiiooaaa www.amiisssooooiiiioooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All results are reproducible, with details in the appendix. We will also open- source dataset and code support the community.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted extensive experiments in strict accordance with prior work in the field and commit open-sourcing our code ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Similar previous work, we used 16-24 A100 GPUs computing resources. Variations in computational resources do not affect our experimental results. Detailed information is provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: hwtparrpww..ccpppla/ccccccccca.a
Answer:
[Yes]
Justification:
We strictly adhered to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the limitations of our work and directions for future research the final section of the paper.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We did not collect any potentially sensitive data from the internet. All provided data comply with established protocols the remote sensing imagery community, ensuring there are no security concerns.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All data, models, and API utilized in this paper are publicly available and appropriately cited. The license and terms of use are properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide a new dataset, which will be released in the future for use by the research community.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We only used LLMs to assist with writing, complying with the LLM policy. For benchmark evaluation, MILMs were employed as assessment tools, following widely accepted evaluation protocol in this field.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22


Reviews:
[{"review_id": "m37bLH5IFL", "reviewer": "Reviewer_ezvU", "summary": "This paper builds a Mamba-based pretrained foundation model that addresses the quadratic computational complexity bottleneck of Transformer-based models, enabling more efficient processing of high-resolution remote sensing imagery. It introduces a rotation-aware pretraining strategy that combines adaptive cropping with angular embeddings to better handle sparsely distributed objects with arbitrary orientations, and proposes a multi-scale token prediction objective to cope with large variations in object scale. Furthermore, extensive experiments demonstrate the scalability of the proposed foundation model, showing that its performance improves consistently with increased training data and model parameters, highlighting its potential as a strong foundation model for remote sensing tasks.", "questions": "1.While the proposed framework shows promising results on standard optical RS datasets, could the authors comment on or provide evidence regarding the applicability of RoMA to other types of remote sensing data such as SAR or hyperspectral imagery? Since these modalities have very different data characteristics, it is important to clarify whether the method generalizes beyond the current experimental scope.\n\n\n2.The paper claims advantages in handling high-resolution images. However, the highest resolution used in the experiments appears to be 1248×1248, which is relatively modest compared to typical operational remote sensing images that can reach tens of thousands of pixels per side. Could the authors either include experiments on higher-resolution datasets or clarify the limitations of the current approach in truly large-resolution scenarios?", "limitations": "Despite its contributions, this work has a few limitations. First, the generalizability of the RoMA framework to other visual domains or non-optical remote sensing modalities, such as SAR or hyperspectral imagery, remains unverified. These modalities exhibit different spatial and spectral properties that may challenge the current design. Second, although the framework is designed to handle high-resolution images, the experiments are conducted on images up to 1248×1248 resolution, which may not reflect the challenges posed by ultra-high-resolution remote sensing data commonly used in real-world applications. Future work could explore extending RoMA to broader data modalities and scaling to much higher image resolutions.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1.It pioneers the use of the Mamba architecture in self-supervised learning for remote sensing, addressing the inefficiency of traditional Transformer-based models when handling high-resolution imagery.\n\n2.By leveraging the linear computational complexity of Mamba, the proposed framework achieves significant improvements in scalability and computational efficiency, which are critical for large-scale remote sensing applications.\n\n3.The paper introduces well-motivated and technically sound components—including a rotation-aware pretraining strategy and multi-scale token prediction—that are carefully designed to address the unique challenges of remote sensing data, such as orientation variability and scale diversity.\n\n4.Extensive experiments across multiple downstream tasks (scene classification, object detection, and semantic segmentation) demonstrate consistent performance gains over ViT-based models, supporting the generality and robustness of the proposed method.\n\n5. The work is aligned with the development of scalable and general-purpose RS foundation models, and the authors provide scaling law analyses that further validate its long-term potential.\n\n\nWeaknesses:\n\n1. It remains unclear how well the proposed RoMA framework generalizes to different types of remote sensing imagery such as SAR or hyperspectral data.\n\n2.The authors claim that the proposed model has significant advantages in handling high-resolution remote sensing images; however, to my knowledge, remote sensing imagery typically has resolutions much higher than 1248×1248. The paper should consider evaluating on higher-resolution data to more convincingly demonstrate its superiority in high-resolution scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1. While the proposed framework shows promising results on standard optical RS datasets, could the authors comment on or provide evidence regarding the applicability of RoMA to other types of remote sensing data such as SAR or hyperspectral imagery? Since these modalities have very different data characteristics, it is important to clarify whether the method generalizes beyond the current experimental scope.**\n\n**A1:** Thank you for your insightful question. In principle, RoMA can potentially be extended to other types of remote sensing data such as SAR and hyperspectral imagery.\n\nFor **SAR data**, although it differs from optical imagery in imaging mechanism (active microwave vs. passive optical), it still follows a similar top-down observation geometry. This allows the **rotation-invariant operations** and **multi-scale prediction strategies** in RoMA to remain applicable. Moreover, SAR images often exhibit high-intensity backscatter in regions corresponding to man-made structures (e.g., buildings, ships), which aligns well with RoMA's **adaptive cropping strategy** based on information density. In terms of channel compatibility, most SAR images contain 1–4 channels; for single-polarization data (e.g., VV), we can replicate the image into a 3-channel format to match the input requirements of the Mamba backbone.\n\nFor **hyperspectral data**, although it consists of a large number of spectral bands, the core components of RoMA, can be extended to 3D spatial-spectral cubes. Inspired by SpectralGPT [1], the hyperspectral image cube can be partitioned into spatial-spectral subcubes. We can then compute a 3D information density map by evaluating each subcube's spatial saliency and spectral variance. Based on this map, the most informative subcubes can be selected, and **adaptive cropping strategy** can be applied to their surrounding 3D regions. The **multi-scale prediction strategy** can also be extended by considering multiple scales in both spatial and spectral dimensions. For the **rotation operation**, unlike traditional 3D image volumes, hyperspectral cubes consist of two spatial dimensions and one spectral dimension without a well-defined geometric structure across all three axes. Therefore, we do not apply geometric 3D rotations to subcubes. Instead, we enhance data diversity using hyperspectral-specific augmentations such as spectral band shuffling. These operations preserve the semantic consistency of spatial-spectral mappings while improving the model's generalization ability.\n\nIn summary, while current experiments focus on standard optical datasets, the architecture and principles of RoMA are general enough to be extended to SAR and hyperspectral imagery.\n\n[1] SpectralGPT: Spectral Remote Sensing Foundation Model, IEEE TPAMI 2023.\n\n### **Q2. The paper claims advantages in handling high-resolution images. However, the highest resolution used in the experiments appears to be 1248×1248, which is relatively modest compared to typical operational remote sensing images that can reach tens of thousands of pixels per side. Could the authors either include experiments on higher-resolution datasets or clarify the limitations of the current approach in truly large-resolution scenarios?**\n\n**A2:** Thank you for the valuable question. To address the reviewer's concern, we conducted additional experiments on higher-resolution images using RoMA-Base and ViT-Base. Each model was evaluated over 200 forward passes to ensure statistical reliability. We report peak GPU memory usage (in MB) and average inference speed (samples per second) across a range of input resolutions. All experiments were conducted on a single NVIDIA A100 GPU with a batch size of 1. (Note: Figure 1 in the main text uses an RTX 4090 GPU with a batch size of 2.)\n\nPeak memory usage (MB):\n\n| Resolution  | 768  | 1024  | 1248  | 1520  | 2048  | 3072  | 4096  |\n| ----------  | ---- | ----- | ----- | ----- | ----- | ----- | ----- |\n| RoMA-Base   | 2693 | 4504  | 6526  | 9434  | 16934 | 37485 | 66357 |\n| ViT-Base    | 4229 | 11726 | 24531 | 52106 | OOM   | OOM   | OOM   |\n\nInference speed (samples/sec):\n\n| Resolution | 768   | 1024  | 1248  | 1520 | 2048 | 3072 | 4096 |\n| ---------- | ----- | ----- | ----- | ---- | ---- | ---- | ---- |\n| RoMA-Base  | 24.98 | 15.91 | 11.43 | 7.86 | 4.37 | 2.00 | 1.15 |\n| ViT-Base   | 22.11 | 9.94  | 4.99  | 2.57 | OOM  | OOM  | OOM  |\n\nThese results demonstrate the scalability of RoMA to higher-resolution inputs. In contrast, ViT-Base encounters out-of-memory (OOM) errors at resolutions beyond 2048×2048. This highlights RoMA's superior computational efficiency and its potential applicability to operational remote sensing scenarios involving ultra-high-resolution images.\n\n---\n**Overall, thank you again for your constructive feedback. If you have any further questions, please don’t hesitate to let us know, we would be happy to continue the discussion.**"}, {"review_id": "k5zcVWFp7f", "reviewer": "Reviewer_9Qb3", "summary": "The paper introduces Mamba into the field of remote sensing imagery, utilizing autoregressive pretraining to construct a foundational model. It proposes a rotation-aware pretraining mechanism and multi-scale token prediction. The contributions of this study are as follows:\n1)\tThis paper introduce RoMA, the first self-supervised autoregressive pretraining framework for Mamba architectures in remote sensing, enabling efficient scaling to high-resolution RS imagery. \n2)\tIt propose a dynamic rotation-aware mechanism that integrates adaptive region cropping and angle-aware embeddings. By guiding the model to predict angular variations during autoregressive learning, it effectively addresses rotational diversity and sparse target distributions, enhancing rotation-invariant feature learning.\n3)\tIt design a multi-scale prediction objective that addresses the extreme variations in object scales, enabling the model to learn robust object representations for downstream tasks.", "questions": "1. Remote sensing targets are typically small. Could the autoregressive pre-training method lead to the neglect of local information, thereby resulting in insufficient learning of small targets?  \n2. How can we ensure that the selection of top k tokens includes the target? If the target is not in a relatively central position, it may still lead to the situation shown in Figure 4(b).  \n3. In Figure 3, were larger blocks reconstructed using different orders?\n4. Was angle prediction included? How does angle embedding function within the network?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths : The idea is novel, and the theoretical reasoning is correct.\nWeaknesses: Some parts of the text are unclear.", "was_revised": true, "final_justification": "Thanks to the authors for the detailed response.  I will keep my scores.", "rebuttal": "### **Q1. Remote sensing targets are typically small. Could the autoregressive pre-training method lead to the neglect of local information, thereby resulting in insufficient learning of small targets?**\n\n**A1:** We agree that small object representation is a critical challenge in remote sensing. However, we believe that the potential loss of local information is not inherently caused by the autoregressive pretraining paradigm, but is more related to the choice of patch size and whether the target region is adequately included in the input.\n\nTo assess the ability of our RoMA model to capture small targets, we fine-tune RoMA-Base with UperNet [1] on the iSAID dataset [2], which contains a large number of small target categories. We compare its performance with several state-of-the-art models, reporting both the overall mIoU and the IoU for representative small-object categories, including Ship, Small Vehicle, Swimming Pool, and Plane. The average object widths for these categories are around 15-53 pixels.\n\nThe results are shown in the table below. Our method achieves the highest overall mIoU and also shows notable improvements on small-object classes, especially Small Vehicle (average width 15 pixels), where RoMA surpasses all other backbones. These results suggest that the proposed **adaptive region cropping strategy** effectively increases the visibility of small foreground objects during pretraining, allowing the model to learn more discriminative representations.\n\nFine-tuning performance on iSAID (following RingMo [3] settings):\n\n|Method|Backbone|mIoU|Ship ($33^2$ px)|Small Vehicle ($15^2$ px)|Swimming Pool ($34^2$ px)| Plane ($53^2$ px)|\n|---|---|---|---|---|---|---|\n|UperNet|IMP-ResNet-50|61.9|65.9|48.8|44.5|83.8|\n|UperNet|SeCo-ResNet-50|57.2|63.9|44.8|9.3|83.3|\n|UperNet|RSP-ResNet-50|61.6|64.2|47.5|43.8|82.8|\n|UperNet|ViT-B+RVSA|63.8|68.9|51.9|46.7|85.6|\n|UperNet|ViTAE-B+RVSA|63.5|69.6|51.9|47.5|85.4|\n|UperNet|RingMo|67.2|73.5|51.2|48.9|85.7|\n|**UperNet**|**RoMA-B (Ours)**|**67.4**|**73.8**|**53.7**|**51.8**|**86.0**|\n\n[1] Unified Perceptual Parsing for Scene Understanding, ECCV 2018.\n\n[2] iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images, CVPR Workshops, 2019.\n\n[3] RingMo: A Remote Sensing Foundation Model With Masked Image Modeling, IEEE TGRS 2022.\n\n---\n### **Q2. How can we ensure that the selection of top k tokens includes the target? If the target is not in a relatively central position, it may still lead to the situation shown in Figure 4(b).**\n\n#### **(1) How can we ensure that the selection of top k tokens includes the target?**\n\n**A2-(1):** This is an interesting question. To ensure that the selected tokens include the target, we incorporate a **three-stage safeguard mechanism** (see lines 183-195 of the main text):\n\n**(1) Coarse localization via information density:** We first compute an information density map using LBP, which highlights regions rich in texture and structure, often corresponding to land-cover targets. This enables us to localize potential object regions while filtering out most of the uninformative background.\n\n**(2) Candidate region selection**: To further ensure that land objects are included, we fix the size of candidate regions and generate all possible regions that fully contain the previously identified coarse localization points. Among these candidates, we select the region with the highest average information density, under the constraint that it must exceed the global average of the entire image. This improves the likelihood of capturing informative targets.\n\n**(3) Adaptive region shrinking**:  If **none** of the fixed-size candidate regions selected in step (2) satisfies the threshold, i.e., its average information density is **not higher than** the global average of the entire image, we reduce the region size and regenerate all candidate regions that still include the coarse localization point. We then repeat the selection process described in step (2). This adaptive refinement increases the likelihood of isolating smaller yet informative targets that may otherwise be diluted by surrounding background.\n\nThrough this three-stage safeguard, our framework maintains strong coverage of targets, mitigating the potential risk of information loss.\n\n#### **(2) If the target is not in a relatively central position, it may still lead to the situation shown in Figure 4(b).**\n\n**A2-(2):** Thank you for your question. To evaluate the performance of our proposed **adaptive region cropping strategy**, we conducted a quantitative analysis using the full SpaceNet V1 building segmentation dataset.\n\nSpecifically, we uniformly rescaled all images and their corresponding segmentation masks to a fixed resolution of 224×224, and divided each image into non-overlapping 16×16 patches. We then applied our adaptive region cropping strategy to each image to select a representative region.\n\nThe results show that random cropping yields only 38.56%, whereas our method achieves a **significantly higher foreground object capture rate of 75.09%**, indicating that in over three-quarters of the samples, the selected region overlaps with at least one labeled object in the segmentation mask. This demonstrates the effectiveness of our adaptive cropping strategy.\n\nForeground Object Capture Accuracy (SpaceNet V1):\n\n| Cropping Strategy  | Accuracy (%)  | \n| ----------  | ---- | \n| Random   | 38.56 | \n| Ours    | 75.09 | \n\nThis substantial improvement demonstrates that our adaptive cropping strategy effectively increases the likelihood of capturing foreground objects, which is crucial for training autoregressive models to learn semantically meaningful representations. While a 75% overlap rate does not guarantee full object inclusion, it still provides a much denser learning signal compared to random cropping, and we found it sufficient to yield superior performance over existing methods in downstream tasks.\n\nWe acknowledge that further improvement is possible. In future work, we plan to investigate more advanced region selection strategies, such as leveraging auxiliary weak annotations, to further enhance target coverage and representation learning.\n\n---\n### **Q3. In Figure 3, were larger blocks reconstructed using different orders?**\n\n**A3:** Thank you for your question. When reconstructing larger blocks, we did not employ different ordering strategies. Instead, we followed the **spatial layout of the original image patches**. For example, a 2×2 block would be arranged as:\n\n[1 , 2] &nbsp;&nbsp;&nbsp; &nbsp;[1 , 2]\n\n[5 , 6]     $\\Rightarrow$     [5 , 6]\n\n\n\nThis is consistent with the description in lines 241-242 of the main text:\n\n*\"Building on MSE loss function, we concatenate token representations from each decoder block at a higher scale following a predefined **raster order**\"*.\n\nWe appreciate your comment and will revise Figure 3 to more clearly reflect this ordering scheme for better clarity.\n\n---\n### **Q4. Was angle prediction included? How does angle embedding function within the network?**\n\n**A4:** Thank you for your question. RoMA does not include an explicit angle prediction task. Instead, angle embeddings are incorporated to provide **directional prior information**, which helps guide the network toward learning **rotation-invariant representations** during pre-training. These embeddings are injected alongside patch tokens to condition the model with orientation cues, without requiring supervision from angle labels. The effectiveness of this strategy has been demonstrated in MA3E [1].\n\n[1] Masked Angle-Aware Autoencoder for Remote Sensing Images, ECCV 2024.\n\n---\n**Thank you again for your constructive feedback. If you have any further questions, please don’t hesitate to let us know, we would be happy to continue the discussion.**"}, {"review_id": "JfSzhjiI8s", "reviewer": "Reviewer_tojz", "summary": "The authors propose rotation-aware multi-scale autoregressive learning (RoMA), a self-supervised pretraining method designed for Mamba-based architectures applied to remote sensing. The overall objective is to propose a first Mamba-based foundation model for remote sensing improving the scalability for high-resolution imagery compared to ViTs. To this aim, RoMa pretraining integrates a rotation aware pretaining objective and a multi-scale token prediction objective. Experiments have been performed for classification, change detection and semantic segmentation including ablation studies to evaluate technical choices on RoMA.", "questions": "Questions, comments and typos:\n\t\n- Figure 1: please increase the size of all the text within the figure.\n\t\n- Line 19, 71, 335: It is mentioned that extensive experiments have been conducted on \"object detection\". Shouldn't it be \"change detection\" instead (Line 305)?\n\t\n- Related work: Please consider referring to CROMA, a foundation model mixing masked autoencoder and contrastive learning [5].\n\t\n- Related work: Please consider referring to AnySat, a foundation model based on JEPA which could be considered as a MIM method [6].\n\t\n- Figure 3, caption: What does \"high-value patches\" mean here? Not fined in the caption.\n\t\n- Figure 4: \"LBP\" is not defined within the figure.\n\t\n- Line 226: Missing hyperlink to Section 1.\n\t\n- Line 239: \"the autoregressive process compares them with their corresponding tokens in the original image for learning\", it is not clear if the comparison correspond to the loss computation, which seems to be at the token level in this sentence, whereas clearly explained as at the pixel level in equation 2.\n\t\n- Table 3 (b): Why selecting the LBP feature descriptor instead of HOG, where HOG shows better performance?\nPlease consider moving the figures either at the top or bottom of each page for better readability.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\t\n- The overall submission is well written and well structured. \n\t\n- Exploring a Mamba-based architecture as a remote sensing foundation model is relevant to tackle the ViT limitation of complexity bottleneck in attention layers w.r.t. large input size.\n\t\n- The MAE pretraining strategy is commonly employed to pretrain remote sensing foundation model and the authors justify well in 3.2.1. its limitation, especially for a Mamba-based architecture, compared to an autoregressive setting.\n\t\n- The adaptative rotation encoding strategy serves to mitigate a well known problem in remote sensing and includes learnable angle embeddings. \n\n- A proper analysis (Figure 5) shows that a Mamba-based architecture along the proposed pretraining strategy scales well with the data volume and model size highlighting the potential of the work for future large-scale applications.\n- Exhaustive experiments show better results compare to competing methods on well explained settings, including ablation studies to highlight the impact of each component and choices in RoMA.\n\nWeaknesses\n\t\n- The literature on rotation-invariant training for remote sensing, especially object detection, is rich [1, 2, 3] and could have been included to better understand if this strategy has already been used for pretraining or only for supervised learning.\n\t\n- The multi-scale prediction strategy is similar to the multi-level loss adopted in Scale-MAE [4], one  would appreciate a clear comparison to state the novelty of the proposed strategy defined as a contribution.\n\t\n- The loss function is computed in the pixel space whereas literature in MAE (see reference in related work) focuses the loss in the token space. One would appreciate to clearly state why the loss is not applied in the token space. Is this choice possible within the given framework? If yes, what would be the difference in the training process in terms of performances?\n\t\n- Even if the selected methods are relevant for comparisons in Table 2, one may appreciate comparisons with more recent approaches that already shown the limitation of MAE-only based pretraining. In particular, one may include CROMA [5] pretrained with a combination of MAE and contrastive learning, or AnySat [6]  showcasing a JEPA framework for remote sensing foundation models. Both have shown impressive results and would be relevant competing methods.\n- Experiments comparing performance with competing method would gain robustness if standard deviations would be included, e.g. by simply changing the initialisation of the linear prob.\n\t\nI would like to emphasize that this submission is already of high quality and could have a significant impact on the community. I would consider increasing my rating if the weaknesses are properly considered.\n\nReferences:\n\n[1] J. Han et al., Redet: A rotation-equivariant detector for aerial object detection. In CVPR 2021.\n\n[2] X. Yang and J. Yan. Arbitrary-oriented object detection with circular smooth label. In ECCV 2020.\n\n[3] J. Han et al., Align deep features for oriented object detection. In IEEE Transactions on Geoscience and Remote Sensing 2022.\n\n[4] C. J. Reed et al., Scale-MAE: A scale-aware masked autoencoder for multiscale geospatial representation learning. In ICCV 2023.\n\n[5] A. Fuller et al., CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders. In NeurIPS 2023\n\n[6] G. Astruc et al., AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities. In CVPR 2025.", "was_revised": true, "final_justification": "The authors' submission is high quality and will have a significant impact in the field of remote sensing. Additionally to all strengths mentioned in my review, the authors provided additional experiments on token vs. pixel space losses, comparisons with robust foundation models, and provided standard deviations demonstrating their method's performance and robustness. The Mamba-based architecture is expected to significantly impact the community by shaping research towards larger and/or higher resolution inputs.", "rebuttal": "### Q1. Literature on rotation-invariant training could been included.\nWe will incorporate relevant citations and expand the discussion accordingly in the revised version of the paper. Specifically, lines 175-182 will be revised as follows (new content highlighted in bold):\n*RS images often contain redundant airport runway pixels, while airplanes appear at various orientations, leading to different postures and shapes. **Such directional diversity has been extensively addressed in supervised object detection through rotation-equivariant or rotation-invariant designs~\\cite{han2021redet, yang2020arbitrary, han2022align}.** However, autoregressive...*\n\n---\n### Q2. Difference between Scale-MAE with multi-scale prediction strategy\nWhile both RoMA and Scale-MAE involve multi-scale reconstruction strategies, their mechanisms and motivations are fundamentally different.\n\n**Scale-MAE** employs a Laplacian-pyramid-style decoder that progressively upsamples feature maps to produce multi-level representations. Each decoder stage is tasked with reconstructing the low-frequency or high-frequency component of the original image. The corresponding supervision consists of low-resolution and high-resolution versions of the input image, approximating its low- and high-frequency content, respectively.\n\nIn contrast, **RoMA's multi-scale prediction strategy** does not rely on progressive decoding. Instead, it forms **larger-scale features** by **concatenating decoder token representations that are spatially adjacent** in the original image layout. This process enables RoMA to effectively reconstruct the input at **multiple patch granularities**, such as 16×16 and 32×32 patch-level representations, in parallel.\nIn the revised version of the paper, we will provide a clearer comparison with Scale-MAE to highlight the novelty of RoMA's multi-scale design.\n\n---\n### Q3.Clearly state why the loss is not applied in the token space. Where possible within the given framework? \nWe fully acknowledge that applying reconstruction loss in token space, as opposed to pixel space, has been shown in prior works such as BEiT[1] and CAE[2] to improve semantic representation learning in MAE-style frameworks. This is largely because token-space loss operates on semantically rich latent features, rather than low-level raw pixel intensities.\n\nHowever, such approaches typically depend on a pre-trained tokenizer (e.g., dVAE[3], CLIP[4]) or an auxiliary feature encoder to provide supervision in the latent space. In our work, we deliberately adopt pixel-space reconstruction loss in order to isolate the contribution of our RoMA framework itself, without introducing additional variables such as tokenizer quality or auxiliary networks. This design choice ensures a fair and transparent comparison with existing baselines under a unified pretraining setting.\n\nThat said, token-space loss is fully compatible with our framework. To demonstrate this, we conducted a preliminary experiment comparing pixel-space and token-space reconstruction objectives. For the token-space setup, we adopt a frozen CLIP model as the teacher, and apply LayerNorm to normalize its output features. The RoMA framework serves as the student, augmented with a lightweight MIM head (fully connected layers) to predict the teacher’s latent representations. Both teacher and student receive the same full input image, and we compute the Smooth-L1 loss between their latent features across all spatial positions. The overall architecture is illustrated below:\n```\nInput ──► Teacher (Frozen CLIP) ─► LN ─┐\n        │                              ├─► Smooth-L1 Loss (all tokens)\nInput ──► Student ───► MIM Head ───────┘\n```\nThis setup enables supervision in the token space, allowing the student to align with the high-level semantic representations produced by a strong teacher encoder. Compared to pixel-level objectives, this strategy encourages the model to learn more abstract and transferable visual features during pretraining.\nDue to time constraints, we pretrained both versions (pixel-space and token-space) for 50 epochs, and transferred the resulting models to the UCM and AID remote sensing classification benchmarks, following the same protocol as in our main paper. The results are as follows:\n\n|Pretraining Loss Type|UCM-55(%)|AID-55(%)| \n|-|-|-| \n|RoMA-Base(Pixel Space)|52.39|80.34|\n|RoMA-Base(Token Space)|56.67|81.72|\n\nThese results confirm that token-space loss indeed yields stronger performance, further validating its effectiveness. More importantly, this experiment illustrates the flexibility and compatibility of the RoMA framework with different pretraining objectives.\n\nWe consider this a promising future direction. To maintain a focused and methodologically clean evaluation of our core contributions, we plan to include this experimental comparison, along with implementation details, in the supplementary material of the revised version.\n[1] BEiT: BERT Pre-Training of Image Transformers, ICLR.\n[2] CAE: Context AutoEncoder for Self-Supervised Representation Learning, IJCV.\n[3] DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors, NeurIPS.\n[4] Learning Transferable Visual Models From Natural Language Supervision, ICML.\n\n---\n### Q4. Comparisons with more recent approaches that already shown the limitation of MAE-only based pretraining, like CROMA and AnySat.\nWe acknowledge the importance of including recent advances such as CROMA and AnySat, and we will incorporate citations, discussions, and comparisons with these works in the revised version of the paper. Specifically, lines 98–102 of the first paragraph in the related work section will be revised as follows:\n\n*MA3E integrates remote sensing angle factors into the MIM training architecture for both the encoder and decoder. **Recent efforts have also explored hybrid pretraining strategies that combine reconstruction and contrastive objectives. CROMA \\cite{CROMA} jointly leverages masked autoencoding and cross-modal contrastive learning over spatially aligned SAR and optical imagery, aiming to improve both unimodal and multimodal representations. AnySat\\cite{AnySat}, in contrast, focuses on learning a universal model across heterogeneous inputs by incorporating contrastive learning into a joint embedding predictive architecture, demonstrating strong performance on multi-resolution and multi-modal datasets**. While these methods...*\n\nTo further address this point, we conducted preliminary comparisons. We have completed the evaluation of AnySat and CROMA on scene classification datasets using the same protocol as our main paper.  However, CROMA is tailored for multimodal SAR-optical inputs with a backbone designed for multispectral imagery. To adapt it, we added a 1×1 convolution to project the 3-channel RGB input into the 12-channel space required by CROMA, with the layer updated during linear probing.\n\n|Datasets|AID-55(%)|UCM-55(%)|\n|-|-|-|\n|CROMA|73.46|50.48|\n|AnySat|79.76|53.18|\n|RoMA-Base|87.36|59.45|\n\nThese results highlight the strong performance of our RoMA framework. We also note that AnySat and CROMA may not perform optimally in this setting, possibly due to its pretraining being oriented toward multi-resolution, multi-sensor data, whereas AID and UCM are single-modal RGB datasets. With more extensive fine-tuning and adaptation, its performance may improve further.\n\n---\n### Q5. Experiments about standard deviations, e.g. linear prob\nFollowing your advice, we conducted linear probe classification experiments **three times** using different random seeds for initialization, while keeping all other settings consistent with the protocol described in our main paper. The results are summarized in the table below.\n\n|Methods|AID-55|UCM-55|\n|-|-|-|\n|ARM|81.07±0.24|50.14±1.49|\n|SatMAE++|86.14±0.51|55.93±2.13|\n|RoMA-Base|87.25±0.16|59.95±1.25|\n\nAs shown, **RoMA exhibits consistently strong performance with low variance**, indicating better robustness compared to other competing methods.\nDue to time constraints, we report standard deviations for two representative baselines (ARM and SatMAE++). We will include standard deviation results for additional methods in the revised version.\n\n---\n### Q6. Typos and Minor Issues\n1. The font size in Figure 1 will be enlarged in the revised manuscript.\n2. The term \"object detection\" will be corrected to \"change detection\" in the revised manuscript.\n3. We will cite CROMA in the related work section and briefly discuss its relevance in the revised manuscript.\n4. We will include a citation to AnySat in the related work section and briefly discuss its connection to masked image modeling methods in the revised manuscript.\n5. The term \"high-value patches\" refers to image patches with high information density, as computed using the LBP metric described in lines 180-187. In the revised version, we will provide a clearer explanation of this term in the caption of Figure 3.\n6. In the revised version, we will include the full name of LBP (Local Binary Pattern) in the caption of Figure 4 for clarity.\n7. The missing hyperlink to Section 1 will be added in the revised manuscript.\n8. We acknowledge that the wording in line 239 may be misleading. To clarify, the comparison described indeed refers to the pixel-level loss computation, as explicitly defined in Equation 2. The phrase \"corresponding tokens in the original image\" simply refers to the decoder outputs being aligned with their corresponding image regions, rather than indicating token-level supervision. We will revise this sentence for clarity in the updated version.\n9. We apologize for the mistake: the accuracy values for HOG and LBP were mistakenly swapped in Table 3(b). This will be corrected in the revised version of the manuscript. We also appreciate your suggestion regarding figure placement, and we will adjust the layout so that figures appear at the top or bottom of each page to improve readability.\n\n---\n**Thank you again for your constructive feedback.**"}, {"review_id": "JomIXABTnn", "reviewer": "Reviewer_c7wu", "summary": "The paper seems to present novel designs and self-supervised pretraining strategies for foundation models in remote sensing, including the use of Mamba to avoid the quadratic bottleneck of transformer-based architectures, a patch rotation augmentation mechanism and an unclear multiscale prediction mechanism. However, the rather poor quality of writing makes it difficult to understand the actual contributions of the paper.", "questions": "The paper might have some interesting contributions from a technical standpoint but the presentation is too confusing. The authors should be significantly more careful in explaining the technical details of the method, clearly highlighting how every step of the proposed design and methodology works.", "limitations": "Despite what is reported in the checklist, limitations are not discussed with the conclusions of the paper.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The authors seek to improve the efficiency of foundation models in remote sensing. This is important to handle the very large images typical of this field.\n- Experimental results seem promising with good metrics across different tasks compared to existing methods and significant runtime and memory savings\n\nWeaknesses:\n- The paper is very difficult to read due to poor writing quality. This makes it difficult to properly understand what the contributions of the work are and their technical details. There are several mistakes in the use of English (examples: caption of figure 2, lines 154-156, ...). A few examples of poor writing quality are: repetitions in lines 75-80; paragraph \"Advantage of Autoregressive Pre-training of Mamba\" does not explain any advantages, only the fact that a causal autoregressive model such as Mamba matches the setting of a causal autoregressive prediction task used for training. \n- Several sections are unclear., particularly section 3.2.3. How are patches tokenized? what's the ordering that is decided for next-token prediction? It is said that the Mamba-based encoder (which is never explained what it is) generates key and values for a causal cross-attention operation. This is quite confusing. Not only it is not clear how it works, but also, this operation would have the classic quadratic complexity in the number of tokens of the transformer, which the authors claim is a main limitation of existing RS foundation models (lines 5,33,157). The \"multi-scale\" part of the prediction is also unclear: how are the multiple scales generated and processed? Neither the text nor fig.3 are illuminating on this aspect.\n- The novelty of using a rotation augmentation with angular embedding for the pretraining method seems fairly limited", "was_revised": true, "final_justification": "The work might present some interesting results by adapting Mamba to remote sensing foundation models but with modest methodological novelty. Clarity of manuscript writing remains a significant concern even after rebuttal.", "rebuttal": "### **Q1:**  Poor writing obscures the contributions and includes grammar issues, repetition, and unclear sections.\n\n**A1:** We sincerely appreciate the reviewer's feedback regarding writing clarity. We acknowledge that certain sentences may lack fluency or appear ambiguous. In response:\n\n- **For the caption of Figure 2**, we agree that the phrasing can be improved. The current version indeed contains a typo (\"different\" → \"difference\", \"autogresive\" → \"autoregressive\"). \n\n    In the revised version, we will rephrase it as:\n\n    *Comparison between our autoregressive pretraining strategy and the standard MAE method. (1) RoMA encodes all patches using a Mamba encoder, whereas MAE encodes only a randomly sampled subset. (2) RoMA predicts the next token in a sequence to capture continuity, while MAE only reconstructs masked patches.*\n\n- **For lines 154-156**, we acknowledge the original phrasing is grammatically flawed and may seem vague. In the revised version, we will rephrase this section for better clarity:\n\n    ***Limitations of MAE-based Pretraining for Mamba.** We begin by analyzing why MAE-based pretraining may be suboptimal for Mamba-based RSFMs, given the unique characteristics of RS imagery and the architectural mismatch between MAE and Mamba.*\n\n- **For lines 75-80**, which enumerate the contributions, the repetition noted by the reviewer may stem from syntactic redundancy. We will streamline the phrasing to make each contribution more concise, while preserving the technical depth. For example, we will replace the first contribution with\n\n    *(1) We introduce RoMA, the first autoregressive pretraining framework for Mamba in remote sensing, which supports high-resolution imagery and demonstrates consistent performance gains with increasing model and data scales.*\n\n    While the second contribution will be replaced by:\n\n    *(2) We propose a rotation-aware mechanism that combines adaptive region cropping with angular embeddings to enhance rotation-invariant representation learning. This design effectively handles orientation variability and sparsely distributed targets in remote sensing scenes during autoregressive pretraining.*\n\n- **Regarding the \"Advantage of Autoregressive Pre-training of Mamba\" paragraph**, our intent was to highlight the architectural alignment between Mamba and autoregressive modeling, as both employ sequential processing. We have revised this section to more clearly explain how this alignment brings practical benefits to the pretraining process. The paragraph will be revised to:\n\n    *Autoregressive pre-training aligns naturally with the sequential nature of Mamba's architecture, which processes input tokens in a temporally ordered manner. Specifically, autoregressive modeling constructs image patches sequentially and predicts the next token based on previous context, mirroring Mamba's token-by-token scanning mechanism. This architectural alignment facilitates more coherent temporal dependencies and better token transition modeling, allowing Mamba to learn more structured and semantically meaningful representations. Therefore, the autoregressive training paradigm not only complements Mamba's design but also enhances its ability to model spatial continuity and visual context in remote sensing imagery.*\n\n\n### **Q2:** Section 3.2.3 is unclear, especially regarding patch tokenization, token order, multi-scale prediction, and the role of the Mamba encoder.\n\n**A2:** We thank the reviewer for pointing out the ambiguity in Section 3.2.3. We appreciate the opportunity to clarify the technical design regarding patch tokenization, sequence ordering, the Mamba encoder, causal decoding, and the multi-scale prediction mechanism. \n\n**Notably, many of these technical components follow widely adopted practices in the community (e.g., ViT-style patching, raster ordering, and standard Mamba configuration). To avoid redundancy, we did not elaborate on them in detail in the manuscript. However, we acknowledge the need for clearer exposition and will revise the manuscript accordingly.**\n\nBelow, we provide a detailed response:\n\n- **Patch Tokenization**: Each input image is divided into non-overlapping patches (e.g., 16×16 pixels), which are then linearly projected into visual tokens via a trainable embedding layer, following the standard practice in ViT-based models [1].\n\n    [1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021.\n\n- **Token Sequence Ordering**: To support autoregressive modeling, we arrange the patch tokens into a 1D sequence using a raster-scan ordering (left to right, top to bottom), consistent with the Vision Mamba design [2].\n\n    [2] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model, ICML 2024.\n\n- **Mamba Encoder**: The backbone of RoMA is a Mamba-based encoder used for extracting image features, as stated in lines 216-217 of the manuscript.\n\n    Our contribution focuses on designing an effective **pretraining strategy tailored for remote sensing**, rather than altering the encoder’s architecture.\n\n- **Causal Cross-Attention**: During pretraining, we introduce a decoder to perform causal supervision via cross-attention. Specifically, the decoder comprises a sequence of virtual query tokens, each designed to predict a target patch token based only on encoder outputs from earlier positions (as defined by raster-scan order). This setup enables autoregressive learning while maintaining strict causality.\n\n    Although this cross-attention mechanism introduces quadratic complexity during pretraining (with respect to the decoder’s token length), we emphasize that:\n\n    > This decoder is only used during training to provide autoregressive supervision. It is discarded after pretraining. Thus, it does not affect the efficiency of the final foundation model at inference time.\n\n- **On Computional Complexity**: We respectfully believe there may be a misunderstanding. The foundation model we develop and ultimately deploy is the **Mamba encoder**, whose **linear complexity** [2] is one of its primary advantages:\n\n    $$\n    \\mathcal{O}(N \\cdot d)\n    $$\n\n    where $N$ is the number of tokens and $d$ is the hidden dimension. This contrasts with conventional ViT-based encoders used in many remote sensing foundation models, which exhibit quadratic complexity:\n\n    $$\n    \\mathcal{O}(N^2 \\cdot d)\n    $$\n\n    Hence, **our efficiency claims pertain strictly to the Mamba encoder**, and the decoder’s quadratic operations during training do not impact inference-time cost.\n\n    > **In summary**, RoMA's computational efficiency claims are based solely on the **Mamba encoder**, which is used at inference. The decoder contributes only during training and does not affect the model’s practical deployment cost.\n\n    [2] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model, ICML 2024.\n\n- **Multi-Scale Prediction Strategy** Thank you for pointing this out. We clarify that the multi-scale prediction in RoMA is implemented by aggregating spatially adjacent decoder tokens and reconstructing their corresponding higher-level patches in parallel. As described in lines 241-243 of the paper.\n\n    Specifically, we construct **larger-scale features** by **grouping and concatenating adjacent decoded tokens** based on their spatial positions (e.g., 2×2 neighboring tokens for a 32×32 patch if the token represents 16×16), and then map these features back to pixel space using an MLP. This enables RoMA to predict target pixels at **multiple spatial granularities** (e.g., both 16×16 and 32×32 patch levels) simultaneously, thereby enhancing its ability to model objects with diverse scales in RS images.\n\n    We will revise both the text and Figure 3 in the updated manuscript to make this multi-scale strategy more intuitive and visually clear.\n\n### **Q3:** The novelty of using a rotation augmentation with angular embedding for the pretraining method seems fairly limited\n\n**A3:**  Thank you for your comment. **Our contribution extends well beyond the use of rotation augmentation and angular embeddings.** Specifically, our goal is to address a fundamental challenge in current remote sensing foundation models: the inefficiency of ViT-based architectures in handling high-resolution inputs due to their quadratic complexity.\n\nTo address this, we propose RoMA, a novel autoregressive pretraining framework built on a Mamba-based encoder with linear computational complexity, making it inherently more scalable to high-resolution remote sensing imagery.\n\nTo support effective pretraining under the unique characteristics of remote sensing imagery, we introduce a series of tightly integrated strategies:\n\n- **(1) Sparsity of foreground objects:** We design an **adaptive region cropping strategy** to identify regions with high knowledge density, which often correspond to meaningful land objects. This ensures that the model focuses its learning capacity on informative regions.\n\n- **(2) Variation in object orientation:** We introduce **angular embeddings** that encode rotation angles into the token representations. This facilitates rotation-invariant feature learning and improves robustness to orientation diversity.\n\n- **(3) Extreme variation in object scales:** We propose a **multi-scale prediction strategy**, which reconstructs token representations at different spatial granularities (e.g., 16×16 and 32×32 patches), enabling the model to capture both fine-grained and coarse spatial patterns.\n\nThese components are not isolated techniques, but are integrated cohesively into an autoregressive pretraining pipeline, which aligns well with Mamba’s sequential architecture. As a result, we successfully construct a remote sensing foundation model based on the Mamba framework, tailored to both the computational efficiency and unique characteristics of remote sensing imagery."}]

Author rebuttal:

Meta-review: This paper presents a framework, RoMA, with the aim of working around the quadratic complexity of standard transformers such as ViT, in the context of remote sensing data, which includes high-resolution images. To this end, RoMA follows the Mamba architecture and shows how this can be adapted to a self-supervised learning scenario. Two key ingredients in achieving this are: (1) rotation-aware pretraining, and (2) multi-scale token prediction objectives.

Several strengths of this work are:
- bringing significant improvements in scalability and computational efficiency,
- showing the importance of this work in the context of remote sensing,
- exhaustive and convincing experimental validation and analyses.

After the discussion phase, the main weakness of the paper was identified as the lack of presentation clarity by reviewer c7wu, who gave a slightly lower rating of borderline reject. All the other reviewers were convinced by the paper, the rebuttal and the additional information provided in the discussion phase. Reviewer c7wu also acknowledged their low confidence in this topic (compared to the other reviewers) and was prepared to give the authors the benefit of doubt regarding the improvement of presentation clarity.

Thus, the area chair recommends the paper for acceptance. All the additional information and clarification presented in the rebuttal and the discussions should be included in the final version of the paper, with particular attention to presentation clarity.
Decision: Accept (poster)