SAR2Earth: A SAR-to-EO Translation Dataset for Remote Sensing Applications

Authors: Jeongwon Ryu, Youngtack Oh, Minseok Seo
Primary area: datasets and benchmarks
Keywords: AI for social good, Dataset, SAR-to-EO translation, Benchmark

Abstract:
Electro-optical (EO) images are essential to a wide range of remote sensing applications. With the advent of data-driven models, the efficiency of EO image analysis has significantly improved, enabling faster and more effective outcomes in these applications. However, EO images have inherent limitations—they cannot penetrate cloud cover and are unable to capture imagery at night. To overcome these challenges, synthetic aperture radar (SAR) images are employed, as they can operate effectively regardless of weather conditions or time of day. Despite this advantage, SAR images come with their own difficulties: they are affected by speckle noise, complicating analysis, and existing algorithms developed for EO imagery are not directly transferable to SAR data. To address these issues, we introduce SAR2Earth, a benchmark dataset specifically designed for SAR-to-EO translation. By translating SAR images into EO-like representations, SAR2Earth allows the extensive range of algorithms developed for EO imagery to be applied effectively to SAR data. The dataset consists of 18 spatially aligned pairs of SAR and EO images, collected from 8 distinct regions encompassing both urban and rural. We provide comprehensive evaluations, detailed model analyses, and extensive experimental results. All codes and datasets will be made publicly available at https://sar2earth.github.io.

Decision: reject
Reviewer scores: 8, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729098039738

thread_type:
official_review

<thread_object>
creator: reviewer_lqkw

<content>
summary:
Electro-optical (EO) satellite images are used in many important applications of machine learning. But the Earth's surface can be blocked by cloud coverage and is not viewable at night. Synthetic aperture radar (SAR) do not have these two limitations. This paper introduces SAR2Earth, a dataset for SAR-to-EO image translation. It consists of 18 spatially registered pairs of SAR and EO images from 8 different regions. This paper also benchmarks image-to-image translation methods (introduced for natural imagery) on SAR2Earth, showing that the CUT method performs best. This paper qualitatively shows that EO foundation model predictions improve by translating a SAR image to an EO image, then providing the foundation model with this synthetic EO image — compared to directly providing the SAR image.

strengths:
The paper is well written and presented. Remote sensing is an important application area of ML. The introduced dataset — named SAR2Earth — is valuable to the remote sensing community, as it contains high-resolution SAR-EO pairs sampled from diverse regions and temporal differences. Running many baseline methods on SAR2Earth is a valuable contribution. The examples using SAR to EO translation to create synthetic EO images which are fed to foundation models is also a valuable contribution — and it shows that this research direction complements broader trends in ML.

weaknesses:
My main concern is that I do not believe this paper is valuable to share with the broader ICLR community. I believe it is more suited to RS-specific journals or workshops. The most similar work to this, that I am ware of, is a CVPR 2024 workshop paper by Low et al., that is cited. SAR2Earth seems to have more regional diversity and this paper performs more extensive (and valuable) experiments. As far as I can tell, Low et al. provide far more SAR-EO pairs than this submission. 

Other comments:
- It is hard to tell what the MAE / MSE/ PSNR metrics imply for a remote sensing practitioner. For example, are synthetic images with an MAE of 0.1 still useful? One potential experiment is to find a dataset with SAR-EO pairs, like BigEarthNet or EuroSat (there is a SAR version available). Then translate the SAR image to EO and try classifying it. The performance drops would be very interesting to me. 
- I cannot find a discussion on RS models that were pretrained to ingest SAR. There are many: Presto, CROMA, DOFA, SoftCon, DeCur, etc. If we have foundation models that can ingest SAR, wouldn't it be better to directly provide the SAR image, rather than a synthetic EO image that requires a SAR-to-EO translator?

questions:
It seems like this dataset only contains 18 SAR-EO pairs, can you confirm this is the case? If so, what is the bottleneck to scale your collection pipeline up to many thousands of pairs?

Please also see questions in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729435836513

thread_type:
official_review

<thread_object>
creator: reviewer_bjda

<content>
summary:
This paper introduces SAR2Earth, a benchmark dataset designed for the translation of synthetic aperture radar (SAR) images into electro-optical (EO) image representations. This dataset addresses the limitations of EO images, which cannot penetrate cloud cover or capture nighttime imagery, by leveraging the all-weather and day-night capabilities of SAR images.

strengths:
1. This article has strong practical applications and provides a broad platform for future research.
2. Their dataset have considered realistic conditions for practical applications.
3. This paper have provided extensive ablation studies from pre-processing to post-processing techniques.

weaknesses:
1. As a paper introducing a new dataset, this article provides too few qualitative results.
2. The number of images in the proposed dataset remains unknown, and details such as image numbers, the size of the training and test set are not disclosed.
3. The applications demonstrated in this paper are limited in section 5.6.

questions:
1. How to ensure the accuracy of the co-registration of SAR and EO? 
2. Why not compared diffusion-based approaches for SAR-to-EO translation? These methods are widely used (Line 134).
3. For SAR-to-EO pipeline proposed in this paper, how can the authenticity of the synthetic EO images be ensured, considering that generative models may introduce artifacts? If the model generates false features, how should they be handled?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730102901542

thread_type:
official_review

<thread_object>
creator: reviewer_pahi

<content>
summary:
This paper introduces SAR2Earth, a deep learning benchmark dataset for SAR to optical translation. It verifies the application of this dataset through various methods and perspectives. The content is comprehensive, and both data and code are openly shared.

strengths:
This paper constructs a high-resolution (<1m) SAR to optical translation dataset, covering both rural and urban scenes, including large-scale areas—a first in this field. The quality of the data and methods is commendable, with detailed data processing and sufficient experimental comparison. The introduction is clear, outlining the significance of the dataset, the construction process, and various factors influencing SAR to optical translation. This dataset holds significant potential in the SAR to optical translation domain.

weaknesses:
Although the paper presents a valuable dataset, several shortcomings remain:
1．	The connection between the background introduction and experimental results lacks coherence. The significance of SAR to optical translation is highlighted as solving cloud and fog issues, but the experiments fail to substantiate this claim. While the GeoChat analysis and SAM segmentation results are novel, the GeoChat testing method is questionable and too qualitative. Conversely, the OSM experiment or surface segmentation, which could better demonstrate the conversion's significance, is not included.
2．	The dataset review is not comprehensive, and its positioning is unclear. Since 2018, SAR to optical translation has seen significant advancements with many global datasets such as SEN12MS dataset for Sentinel-2, Planet-CR dataset for Planet, and SMILE dataset for Landsat. These were not mentioned. The main contribution of this dataset lies in its sub-meter resolution, which previous global datasets do not offer, enhancing urban (built-up) scene accuracy.
3．	The experimental discussion is not sufficiently in-depth. The paper discusses speckle noise but does not address the alignment issues with increased resolution using WGS84. The impact of these alignment errors and their solutions should be discussed, which is not encountered in lower-resolution datasets.

questions:
1.	The title mentions datasets and remote sensing applications, yet experimental applications are not clearly reflected. GeoChat and SAM are not true applications. SAR to optical conversion's main significance lies in interpreting scenes in complex weather, which is only mentioned in the background but not experimentally demonstrated.
2.	The dataset review is not comprehensive, and its positioning is unclear. What are the differences and advantages between your dataset and those medium-resolution global datasets?
3.	Is there a better way to deal with the errors of SAR and optical registration, not just WGS84 registration?
4.	Many existing models are compared in the experiments. Can these models handle noise directly within the network without pre-processing before model training?
5.	The time span between SAR and optical data is lengthy, sometimes exceeding a year. How can this limitation be narrowed? Seasonal differences and surface changes may affect the translation model's accuracy. How can this be mitigated?
6.	Can the model trained on this dataset be directly applied in specific scenarios, or how far is SAR to optical translation from real-world application? Which specific scenarios would benefit most?
7.	GeoChat verification seems biased towards optical images. Would describing SAR images show similar biases?
8.	The migration accuracy for urban and rural scenes is low. Does this issue extend to other scenes? Can dataset expansion and method changes address this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730450244946

thread_type:
official_review

<thread_object>
creator: reviewer_psfn

<content>
summary:
This paper introduces a novel public benchmark dataset called SAR2Earth which designed for SAR-to-EO translation. Both EO and SAR images have inherent limitations: EO images cannot penetrate cloud cover and capture imagery at night; SAR images are affected by speckle noise and complicating analysis. Meanwhile, existing large foundation models and generalization models trained on EO images do not perform effectively on SAR images. As a result, SAR-to-EO translation methods have been proposed to mind this gap. This paper proposes SAR2Earth dataset with co-registered SAR and EO images, which aims to support further research for SAR-to-EO translation. The intuition is the limitation of available SAR-to-EO datasets. 
The authors propose a SAR-to-EO pipeline and test the SAR2Earth dataset on various image-to-image translation methods with different preprocessing and postprocessing techniques. The experimental results demonstrate the potential value of this benchmark.

strengths:
•	This paper proposes a comprehensive benchmark dataset called SAR2Earth for SAR-to-EO translation. The SAE2Earth dataset is collected from 8 regions, with rural/urban domains, a wide range of temporal shifts from 1-month to 5-year, and high resolution images ranging from 0.15m to 0.6m.
•	To evaluate the performance of the dataset, this paper proposed a SAR-to-EO pipeline that integrates previous research and provided benchmark results on state-of-the-art image-to-image translation models.
•	The experimental results are comprehensive, detailed, and promising. The proposed public benchmark dataset can support future research in this domain.

weaknesses:
•	The dataset description lacks specific details like the number of images in the dataset and recommended dataset splits.
•	Compared with section 5, the SAR-to-EO pipeline in section 4 occupies only a small portion of the paper and lacks adequate explanation.
•	As the authors point out, this work did not leverage extra modalities, like OSM-based label maps or metadata. These extra information can provide better understanding for regional characteristics in SAR-to-EO translation.

questions:
•	Can you provide more information about the SAR2Earth dataset? E.g. a comparison of the number of images with previous datasets.
•	Can you provide more detail in section 4, e.g. a detailed flow chart? This may provide readers with a comprehensive understanding of the pipeline.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732204906480

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal

comment:
General Rebuttal


We appreciate the reviewers' thoughtful feedback and insightful suggestions, which guided us in refining our work. In response, we made several significant updates to the manuscript, which are summarized below.


1. We have updated **Section 3.2** and added **Appendix A** to include more detailed information about the SAR2Earth dataset.


2. We have added comparisons with cloud removal datasets in **Section 2.2 and Appendix B** to highlight the unique features of SAR2Earth.


3. Additional qualitative results have been added to **Figure 4**, and a detailed pipeline description has been updated in **Section 4.3 and Appendix B**.


4. New experimental results for Land Cover Segmentation and Visual Grounding have been introduced and updated in **Sections 5.1 and 5.6**.


For a detailed discussion of the changes, please refer to the specific responses to individual comments. Thank you again for helping us enhance the quality of this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311695098

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision and Discussion

comment:
Dear Program Chairs (PC), Senior Area Chairs (SAC), Area Chairs (AC), and Reviewers,

As the discussion phase concludes, we would like to express our sincere gratitude to all participants for their time and valuable insights. The thoughtful questions and feedback provided have been instrumental, and we have worked diligently to address the concerns raised.

We enhanced the dataset description with additional details on image counts, data splits, and regional diversity for greater clarity. Additionally, we introduced new experiments, such as Land Cover Segmentation and Visual Grounding, to demonstrate the effectiveness of SAR-to-EO translation. Finally, we emphasized SAR2Earth’s role as a benchmark dataset to advance SAR-to-EO technologies and support real-world applications and future research.

We are pleased that our responses addressed key concerns raised by the reviewers, resulting in positive feedback and improved scores. While challenges such as time discrepancies and high-resolution SAR-EO co-registration remain, our approach lays a strong foundation for further exploration in these areas.

The recognition of the dataset’s high-resolution data and challenging benchmarks is greatly appreciated. While reviewers Pahi and LqKw suggested it might fit better in another venue, we emphasize that the focus on "datasets and benchmarks" aligns closely with the objectives of this community. The constructive feedback provided has been invaluable, and we deeply appreciate the effort invested in reviewing this work.

We hope the revisions and explanations will be carefully considered during the final evaluations. Once again, our sincere thanks go to all reviewers for their contributions to improving this work.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new dataset, SAR2Earth, designed to benchmark SAR-to-EO image-to-image translation methods. The dataset consists of paired SAR and EO image patches extracted from 18 spatially aligned pairs of high-resolution (<1m) SAR and EO images, collected from 8 distinct regions spanning both urban and rural areas. The paper benchmarks several paired and unpaired methods for SAR-to-EO image-to-image translation, alongside SAR image denoising. While the primary focus of the paper is image-to-image translation, it also addresses image denoising as part of the pipeline, which has relevance for certain downstream applications.

Reviewers have recognized the dataset as a valuable resource, given its provision of high-resolution (<1m) SAR and EO image pairs. These pairs are not only beneficial for image-to-image translation but also for other tasks that require learning cross-modal representations in remote sensing.

However, the paper's scope appears somewhat diffuse. The benchmarking covers denoising, image-to-image translation, and other applications, which detracts from a more focused presentation on the dataset and the benchmarking of paired and unpaired image-to-image translation methods. Additionally, several SOTA methods that are highly relevant to this task, such as BicycleGAN, MUNIT, NICE-GAN, and Attn-CycleGAN, are not considered for the benchmark.

There are other issues in the paper that remain unclear. For instance, Table 1 states that SAR2Earth's sensor type is "Google Earth." If Google Earth was used to collect Sentinel-2 images, as suggested by the project's website (https://sar2earth.github.io/), it raises concerns about the feasibility of obtaining EO images at the same high resolution as Capella SAR images. Additionally, the temporal misalignment or discrepancies between SAR and EO images could lead to inaccuracies in representation learning. Moreover, the lack of SAR-EO co-registration is a significant issue for high-resolution paired images, especially for tasks like image-to-image translation, where accurate co-alignment is important.