Title: PhySwin: An Efficient and Physically-Informed Foundation Model for Multispectral Earth Observation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Foundation Model

Physics-Informed ML

Efficient Machine Learning

Multispectral Image

Remote Sensing
Abstract: Recent progress on Remote Sensing Foundation Models (RSFMs) aims toward universal representations for Earth observation imagery. However, current efforts often scale up in size significantly without addressing efficiency constraints critical for real-world applications (e.g., onboard processing, rapid disaster response) or treat multispectral (MS) data as generic imagery, overlooking valuable physical priors. We introduce PhySwin, a foundation model for MS data that integrates physical priors with computational efficiency. PhySwin combines three innovations: (i) physics-informed pretraining objectives leveraging radiometric constraints to enhance feature learning; (ii) an efficient MixMAE formulation tailored to SwinV2 for low-FLOP, scalable pretraining; and (iii) token-efficient spectral embedding to retain spectral detail without increasing token counts. Pretrained on over 1M Sentinel-2 tiles, PhySwin achieves SOTA results (+1.32\% mIoU segmentation, +0.80\% F1 change detection) while reducing inference latency by up to 14.4$\times$ and computational complexity by up to 43.6$\times$ compared to ViT-based RSFMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPhySwin uses a Swin v2 architecture with a novel MAE pretraining process that retains Swin sliding window self-attention without using inefficient mask tokens. Additionally, it introduces two \"physically\" based regularizations to smooth MAE reconstruction across the channel dimensions and bind them to realistic values.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper introduces three novel improvements that all show significant performance or throughput improvements. The proposed Refined MixMAE pretraining, which retains SW-MSA, is an elegant and natural extension of the original MixMAE pretraining. It fully retains the Swin v2 architecture while bringing noticeable improvements in fine-tuning performance at almost no training cost. The Token-Efficient Grouped Spectral Embedding is a very clever way of giving the patch encoder the ability to process very different bands in different ways at almost no cost.\n\nWeaknesses:\nThe results section is lacking in several aspects:\n- While the inclusion of change detection benchmarks is positive, the evaluation on semantic segmentation and classification is insufficient in both scope and comparative analysis.\n    - Multiple S2-exclusive benchmarks are missing, see GeoBench[1] and PANGAEA[2] for some classical EO benchmark.\n    - There is a lack of comparison to recent EO foundation models such as SatLas (Swin-based), Dofa, Croma, Galileo, Panopticon, AnySat, and TerraMind, many of which are already evaluated on GeoBench[1] and PANGAEA[2].\n- Classification results are reported only using accuracy, which is saturated (over 90% for both BigEarthNet and EuroSAT); additional metrics like F1 score should be included.\n- The evaluation heads used are only described in the supmat, a reference to the supmat should at least be included in 4.1.\n\nConsidering the proposed approach is only trained on and evaluated with S1 and S2, it can hardly be called a foundation model for EO. The massive multimodality of EO data is one of the main challenges toward creating a proper foundation model for EO. Moreover, PhySwin is both pretrained and evaluated on FMoW-S2 and BigEarthNet-S2, which is a clear limitation for properly evaluating the pretrained model's ability to adapt to slight domain shifts.\n\nWhile the regularizations proposed are logical and bring improvements, their physical reality is questionable. $L_{smooth}$ is independent with respect to the $\\Delta\\lambda$ between two bands, and $L_{bound}$ is a simple regularization to the $[0,1.2]$ interval.\n\nIn some tables, multiple results are **bold**. Instead, second-best results should be underlined in all tables.\n\nThe authors claim on line 288: \"pretraining data volume and quality, and token count correlate positively with FM performance,\" yet no experiments validate this claim.\n\n[1] GEO-Bench: Toward Foundation Models for Earth Monitoring, Lacoste et al.\n\n[2] PANGAEA: A GLOBAL AND INCLUSIVE BENCHMARK FOR GEOSPATIAL FOUNDATION MODELS, Marsocci et al.\n\nQUESTIONS:\n- Please include the F1 score for the classification tasks on BigEarthNet and EuroSAT. Without these metrics, the evaluation remains incomplete and may justify a lower score.\n- Please report results on additional S2-exclusive benchmarks (e.g., GEO-Bench, PANGAEA) and compare against recent EO foundation models. Addressing this point could lead to an increased review score.\n- The claim on line 288 asserts that pretraining data volume, quality, and token count correlate with performance. Please provide experimental evidence to support this statement or remove it. Failure to do so may lead to a reduced evaluation score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to leverage physical properties of spectral channels in multispectral remote sensing images as a pre-training prior for efficient Remote Sensing Foundation Models. The authors argue that architectural optimizations alone are not sufficient to obtain accurate and resource-efficient Foundation Models for Earth Observation (EO), thus prompting the use of physical properties as regularization: a research venue that the authors reckon has been overlooked in recent advancements in the field. The SwinV2 vision transformer model is chosen as backbone to all downstream tasks, since authors argue it is the most appropriate to handle multi-scale EO images.\nThe paper introduces three main contributions:\n- Physical grounding achivied with two _radiometric priors_ computed on the reconstructed images produced during MAE pre-training; these priors serve as a regularizing loss signal.\n- Introduction of a refined version of MixMAE for hierarchical transformers.\n- Image embedding approach that retains the spectral structure of multispectral images without increasing the number of tokens produced. Channels are grouped into spectral bands; each group is embedded through its dedicated mapping; the final token is formed by concatenating the group embeddings. Moreover the paper introduces _Spectral Group Masking_, where group embedding subspaces of tokens are randomly masked; authros argue it may encourage feature robustness.  \n\nThe proposed model is evaluated on semantic segmentation, change detection and land cover classification. Experiments and ablations show that incorporating physical priors during pre-training effectively produces sensible features for downstream tasks. As for inference performance, the proposed method scores significant advancements with respect to attention-based models in terms of FLOPS and throughput, while maintaining comparable task performance, effectively approaching the efficiency level of ConvNets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**:\n  - Integrating physics grounding with multispectral images is interesting, as each channel / spectral band can encode meaningful information for downstream tasks. \n  - The improvements on inference throughput obtained by applying the proposed token-efficient embedding technique is significant and adequately underlined in the experiments.\n  - Relevant qualitative examples are provided in the appendix to demonstrate strengths and shortcomings of the proposed approaches.\n\n**Weaknesses**:\n  - Ablations do not clearly isolate the effects of physical grounding from orthogonal architectural enhancements: looking at the performance reported in Tables 5 and 6, it seems that the ablation experiments on physics-informed losses were performed on models featuring both Spectral Group Masking and enhanced MixMAE (inferred by looking at the last row). It would be interesting to report the results of Table 5(a) obtained with a model pre-trained without refined MixMAE and Spectral Group Masking. I believe this would provide a fairer comparison with the results obtained by the current models.\n  - No experiments were conducted to assess the effectiveness of physics-informed loss as an extension to current state-of-the-art models pre-trained with MAE: I believe that it would be worth evaluating the effectiveness of physical grounding on current SOTA models pre-trained with MAE. Such addition would provide:\n    - a clear understanding of its impact on a broader list of setups;\n    - a relevant baseline to evaluate the effectiveness of both refined MixMAE and Spectral Group Masking, given that they operate independently from grounding.\n - Overall, I believe this work is significant, but it falls short in effectively articulating the true reasons behind its importance. The physical inspirations presented are unconvincing, as they consist of existing losses borrowed from other works without a clear connection to physics (i.e. smoothing or clipping the output of a model). Meanwhile, the architectural choices – arguably the primary drivers of the model's efficiency – are largely overlooked. A presentation that places greater emphasis on the architectural design would enhance the clarity of the paper and better highlight its contributions.\n\nQUESTIONS:\n**Suggestions**\n- For better clarity, please mark which results where replicated indipendently by you and which were reported from papers.\n- Even though fully masking spectral group feature subspaces is sensibly presented as a way to prevent the model from relying on any single spectral group, I think that it would be interesting to try masking single components of the embedding, for instance using a uniformly sampled mask, across spectral groups; such experiments could provide additional insights on the robustness of learned features and the modelling of relationships inside and across spectral groups.\n- I think the caption of figure 1(a) and 1(b) should be expanded with a more thorough explanation on what these two plot represent and how visualized data was produced.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new foundation model for multispectral remote sensing data which introduces three modifications with respect to previous work in its architecture/pre-training process: (1) it integrates radiometric physical constraints into the pre-training loss (bounding reflectance values to be on fixed range due to energy conservation), (2) it uses an efficient modified MixMAE scheme, which coordinates the masking patterns with the shifted windows in SWIN, (3) a spectral embedding technique that groups spectral bands and uses a lightweight embedding function per group that are then concatenated to create a single embedding vector for each spatial location, maintaining the final token count fixed, independent of the number of channels.\nThey compare this method with other foundation models for multispectral remote sensing data on four different types of downstream tasks: semantic segmentation (2 benchmark datasets), change detection (2 benchmark datasets), scene classification (2 benchmark datasets) and multi-label land cover classification (1 benchmark dataset), both in terms of accuracy metrics (appropriate for each task) as well as inference efficiency. They perform ablation studies to verify the effect of each of the modifications proposed also both in terms of accuracy metrics and on inference efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and describes very clearly what are the proposed improvements relatively to previous work and the rationale behind the changes.\n2. The proposed modifications seem technically sound.\n3. The experimental analysis and ablation studies are well designed. \n4. The accuracy results for semantic segmentation and change detection show that the model is on par or better than the other FMs compared.\n5. The efficiency results in terms of throughput in inference are very compelling.\n\nWeaknesses:\n1. The proposed changes are incremental and heavily build upon previous work. \n2. The claim of being “physically-grounded” is a little strong given that they are only introducing a bound on the reflectance value in the training loss). \n3. For the scene classification and multi-label land cover classification task, they say that the results are competitive “near-SOTA” in terms of accuracy, but in fact the differences are quite large in terms of percentage points (for BigEarthNet it is 4 percentage points), so it might not be worth having a faster model which is actually significant worse.\n4. There are many recent foundation models for remote sensing data (see https://github.com/Jack-bo1220/Awesome-Remote-Sensing-Foundation-Models) and they only compare against a few, leaving out ones which seem to have better results in terms of accuracy (DOFA, Prithvi-v2, DINO, AnySat,…), although it is difficult to compare because there are no standard benchmarks for this type of model (GeoBENCH is an attempt to establish that). \n5. Although there is a temporal aspect to remote sensing images, this paper completely ignores this, where most of the recent models take that aspect into account.\n\nQUESTIONS:\n1. Why have you ignored the temporal aspect of remote sensing images in pre-training? Do you think it is not important to model this? Would it be possible to modify your model to include multiple images as input?\n2. How have you selected the models to compare against? There are many other models (see link above) which have not been included such as DOFA, Prithvi-v2, DINO, AnySat.\n3. Thinking about a “physically-grounded” model for remote sensing, is bounding reflectance values to be on fixed range due to energy conservation the only physical constraint possible? \n4. Are there ways to trade-off accuracy versus efficiency in your model? What is the effect of the sizes of the network, number of parameters?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PhySwin, a foundation model developed for multispectral (MS) Earth observation (EO) imagery that addresses key limitations in current remote sensing foundation models (RSFMs), namely computational inefficiency and the neglect of physical/knowledge priors. The model incorporates three core innovations: physics-informed pretraining, which adds spectral smoothness and energy conservation constraints as regularizations; a refined MixMAE method adapted for the hierarchical SwinV2, utilizing shifted window attention to improve scalability and efficiency; and a token-efficient spectral embedding strategy that groups similar spectral bands and applies spectral group masking (MaskSpec) to retain spectral information without increasing token count. PhySwin is trained on +1M Sentinel-2 images, and it shows improved performance on key EO tasks such as semantic segmentation and change detection with computational complexity.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed model, PhySwin, is a novel advancement that integrates physically grounded learning in the training stage, incorporating radiometric constraints (bounded reflectance and spectral smoothness) to enhance generalization and interpretability. A masking strategy that combines spatial and spectral masking further strengthens feature resilience and richness. Its efficiency-focused architecture, built on SwinV2 and a refined MixMAE framework, delivers significant computational advantages. The model demonstrates enhanced performance across multiple Earth observation tasks, with gains such as +1.32% in segmentation mIoU and +0.80% in change detection F1 score. These benefits are validated through extensive experiments on diverse benchmarks. \nPhySwin some limitations too as it underperforms on all multi-label and scene classification tasks when compared to ViT-based baselines. Additionally, the study does not explore scaling up the model, leaving open the question of how larger PhySwin variants might perform on more complex or diverse tasks (while authors list this as a limitation). Finally, the paper does not report uncertainty estimates or statistical significance tests, making it difficult to evaluate the variability or robustness of the reported results.\n\nQUESTIONS:\n- Spectral smoothness is a novel way of using physical information in the model training. However, misregistration of various spectral bands can punish $L_{smooth}$. How can this be addressed? Is this something that we need to accept as a limitation?\n\n- L141-142: It’s unclear how the claim “tokens attend only to others from the same source image, as dictated by M” is enforced. Equation (3) governs the reconstruction loss, but it does not prevent cross-source attention during encoding. Did you explicitly create an attention mask using M to restrict token interaction within each source? If so, I suggest adding 1–2 sentences in the text to clarify how this constraint is implemented in the encoder. In addition, the dual reconstruction loss in Eq(3) allows the model to learn to reconstruct missing information from real images, it would be good to add this. \n\n- Why Table 4 is missing SkySense, despite SkySense showing high performance in some tasks (Tables 2, and 3)?\n\n- [Minor] Figure 3 is missing a color legend for labels, please add.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}