DiffusionSat: A Generative Foundation Model for Satellite Imagery
Authors: Samar Khanna
Patrick Liu
Linqi Zhou
Chenlin Meng
Robin Rombach
Marshall Burke
David B. Lobell
Stefano Ermon
Venue: ICLR 2024 poster (2024)
Keywords: satellite images
generative models
diffusion models
computer vision
Primary area: generative models

Abstract: Diffusion models have achieved state-of-the-art results on many modalities including images, speech, and video. However, existing models are not tailored to support remote sensing data, which is widely used in important applications including environmental monitoring and crop-yield prediction. Satellite images are significantly different from natural images -- they can be multi-spectral, irregularly sampled across time -- and existing diffusion models trained on images from the Web do not support them. Furthermore, remote sensing data is inherently spatio-temporal, requiring conditional generation tasks not supported by traditional methods based on captions or images. In this paper, we present DiffusionSat, to date the largest generative foundation model trained on a collection of publicly available large, high-resolution remote sensing datasets .
As text-based captions are sparsely available for satellite images, we incorporate the associated metadata such as geolocation as conditioning information. 
Our method produces realistic samples and can be used to solve multiple generative tasks including temporal generation, multi-spectral superrresolution and in-painting. Our method outperforms previous state-of-the-art methods for satellite image generation and is the first large-scale _generative_ foundation model for satellite imagery.
The project website can be found here: https://samar-khanna.github.io/DiffusionSat/

Published as a conference paper at ICLR 2024
DIFFUSIONSAT: A GENERATIVE FOUNDATION MODEL FOR SATELLITE IMAGERY
Samar Khanna" Patrick Liu', Linqi Zhou', Chenlin Meng', Robin Rombach", Marshall Burke', David B. Lobell', Stefano Ermon 1.3
Stannford University. "Stability AI, CC Biohub "Correspondence to: samarkhanna.
[at] cs. cs.stanford.edt
ABSTRACT
Diffusion models have achieved state-of-the-ar results on many modalities includ- ing images, speech, and video. However, existing models are not tailored to support remote sensing data, which is widely used in important applications including envi- ronmental monitoring and crop-yield prediction. Satellite images are significantly different from natural images they be multi-spectrall irregularly sampled across time and existing diffusion models trained on images from the Web do not support them. Furthermore, remote sensing data is inherently natio-temppora, requiring conditional generation tasks not supported by traditional methods based captions or images. In this paper, present DiffusionSat, to date the largest generative foundation model trained on collection of publicly available large, high- resolution remote sensing datasets, As text- text-based captions are sparsely available for satellite images, we incorporate the associated metadata such geolocation conditioning information. Our method produces realistic samples and can used solve multiple generative tasks including temporal generation, superresolution given multi-spectral inputs and in-painting. Our method outperforms previous state-of-the- the-art methods for satellite image generation and is the first large-scale generative foundation model for satellite imagery. The project website can be found here: https: http://saarkKkkae whanna.hhnaaaa.. www.tg.thh...g.gg wttp:u..o../iffffiooosos..
INTRODUCTION
Diffusion models have achieved state of the art results in image generation (Sohl-Dickstein et al.)
2015 Ho et al. 2020: Dhariwal Nichol 2021; Kingma et al. 2021 Song & Ermon 2019 2020).
Large scale models such as Stable Diffusion Rombach al. (2022) (SD) have been trained on Internet-scale image-text datasets to generate high-resolution images from urov-idovided captions.
These diffusion-based foundation models, used priors, have led to major improvements in
variety of inverse problems like inpainting, colorization, deblurring (Luo et al. (202), medical image reconstruction (Khader al. 2023 Xie & Li 2022 and video generation (Blattmann al. 2023).
Similarly, there are a variety of high-impact MI. tasks involving the analysis of satellite images, such disaster response, environmental monitoring, poverty prediction. crop-yield estimation, urban planning and others (Gupta et al. 22019) / et al. [2021] Ayush et al. [202 (1bl)2200 2020 Jean et al. 2016
_020]Jean al. 2021 [ Rustowicz et al. 2019; Yeh et al. (021)) These tasks consist of important inverse problems, such as super- resolution (from frequent low resolution images to high resolution ones), cloud removal, temporal in-painting and more. However, satellite images fundamentally differ from natural images in terms of perspective, resolutions, additional spectral bands, and temporal regularity. While foundation models have been recently developed for discriminative learning satellite images Cong et al. (2022): Ayush ctt al. (2021a); Bastani et al. (2022), they are not designed to and cannot solve the inverse problems (eg: super-resolution) described above.
You al. 2017 Wang et al. 2018/ & Körner 2020]
To fill this gap. we propose DiffusionSat, a generative foundation model for satellite imagery inspired from SD. Using commonly associated metadata with satellite images including latitude, longitude, timestamp, and
distance (GSD), we train our model for single-image generation collection of publicly available satellite image data sets. Further, inspired from

---

Published as a conference paper at ICLR 2024
longtudd
timesop
addrion
lathude
GDD
timeriop embedding
con cincatenation
cloud Govtr
metardata enibedding
arcention
sinusokdal projection
comainad ambedding
MLP ambeddiing
morth
WAE Enooder
VAE Decoder
Soussarr
Stable Diffusion Denoising U-Net Eg
| Z
D
cenoikiingg loop with Cascarlee--in free guidenne (fiet lattorlate)
feow Lotellice inage recructional facility Lity the
Zn
Istant-space image
umbiedding
Figure I: Conditioning freely available metadata and using large, publicly available satellite imagery datasets shows DiffusionSat is powerful generative foundation model for remote sensing data.
NontrolNets Zhang & Agrawala 2023). design conditioning models that can easily be trained for specific generative tasks or inverse problems including super-resolution, in-painting, and temporal generation. Specifically, our contributions include:
1. We propose novel generative foundation model for satellite image data with the ability to generate high-resolution satellite imagery from numerical metadata as well text.
2. We design a novel 3D 3DD-conditioning extension which enables DiffusionSat demonstrate state-of-the-arr performance on super-resolution, temporal generation, and in-painting
3. We collect and compile global generative pre-training dataset from large, publicly available satelilte image datasets (see section 3.1
BACKGROUND
Diffusion Models Diffusion models are generative models that aim learn a data distribution PData from samples DickstDi et al., 2015 Ho et al. 2020; SSog & Ermon 2019 [Song et al. 2020b Song Ermon 2020). Given an input image æ PData we add noise to create a noisy input + oze, where (0, I) is Gaussian noise. and denote noise schedule parameterized by diffusion time t (higher leads to more added noise). The diffusion model eo then aims to denoise Tz, and is optimized using the score-matching objective:
E Ee- [lly ea (z
(I)
where the target y can be the input noise e, the input image or the "velocity" == agE oæ We can additionally condition the denoising model with side information e E R.D. which can be a class embedding, text, other images etc.
Latent diffusion models (LDMs) (Vahdat et al.) 2021 Sinha et al. 2021 Rombach et al. 2022 first
downsample the input x using VAE with encoder & and decoder D, such that (E
a reconstructed image. Instead of denoising the input image æ, the diffusion process is used on downsampled latent representation = E(x) This approach reduces computational and memory cost and has formed the basis for the popularly used StableDiffusion (SD) model (Rombach et al. (022).
3 METHOD
First, we describe our method for the following tasks interest: single-image generation, conditioned on text and metadata, multi-spectral superresolution, temporal prediction, and temporal inpainting.
3.1 SINGLE IMAGE GENERATION
Our first goal is to pre-train DiffusionSat to be able to generate single images given an input text prompt and/or metadata. Concretely, we begin by considering datasets such that each image
2

---

Published as a conference paper at ICLR 2024
RC is paired with associated text-caption Our goal is to learn the conditional data distribution p(x(T) such that we can sample new images X ~ p(-17).
LDMs are popularly used for text-to-image generation primarily for their strong ability to use text prompts. An associated text prompt 7 is tokenized, encoded via CLIP (Radford et al. (02T), and then passed to the DM co co(x;; via cross-attention (Vaswani et al. 2017) at each layer. However, while text prompts are widely available for image datasets such as LAION-5B -B (Schuhmann et al.) (022)) satellite images typically either do not have such captions, or accompanied by object-detection boxes, segmentation masks, or classification labels. Moreover, requiring such labels precludes the use of vast amounts of unlabeled satellite imagery. Ideally, we would like pretrain DiffusionSat on existing labelled and unlabelled datasets, without necessarily curating expensive labels.
To solve this challenge, we note that satellite images are commonly associated with metadata including their timestamp, latitude. longitude, and various other numerical information that are correlated with the image (Christie et al., 2018). We thus consider datasets where each image x RCCH** is paired with a text-caption as well as cheaply available numerical metadata k E R where M the number of metadata items. We thus wish to learn the data distribution p(x T, k). With good enough metadata k, we want to still sample an image of high quality even if T is poor missing.
We now turn to conditioning on One option is to naively incorporate each numerical metadata item kz. j E [1....,M) into the text caption with short description. However, this approach unnecessarily discretizes continuous- valuue covariates and can suffer from text-encoders' known shortcomings related to encoding numerical information (Radford al., 2021). Instead, we choose to
encode the metadata using the same sinusoidal timestep embedding eq. 2 used in diffusion models:
Project (ks, 20) sin (AS2- #)- PPoojec((;; 2i 1) = cos [ 3
(2)
where k is the metadata or timestep value, i is the index of feature dimension in the encoding, d is the dimension, and 5 = 10000 is large constant. Each metadata value k2 is first normalized to a value between 0 and 1000 (since the diffusion timestep / E {0..... 1000)), and is then projected via the sinusoidal encoding. different MLP for each metadatum encodes the projected metadata value identically to the diffusion timestep (Ho et al. 2020) as follows eq. (3):
fe, (kyy = MLP ([Pro ject (ki,, O),... Project (kij, d)))
(3)
where fo, represents the learned MLP embedding for metadata value ky. corresponding to metadata type j (eg: longitude). Our embedded is then fe, (k,) E RD, where is the embedding dimension. The M metadata vectors are then added together m fo, (k1) + ................................ + fo (kM)), where m RR.. which is then also added with with the embedded timestep t sa(() E RP,, so that the final conditioning vector c = m
To summarize, we first we first encode an image x E RC using the SD variational autoencoder
{(x) E C'' xw" Gaussian noise is then added to the latent image features to give us 2, = a,z + ogE (see section 2 The conditioning vector created from embedding metadata and the diffusion timestep, as well as the LeDeeeebedded text caption r' = (T) ). are passed through a DM co Ez(; T', c) to predict the added noise. Finally, the VAE decoder D upsamples the denoised latents to full resolution (fig.
(VAE) (Rombach et al. 2022 Esser et al. 2021) latent representation
Lastly, we initialize the encoder E, the decoder D, the CLIP text encoder To, and the denoising unet co all with SD 2.1's weights. We only update the denoising UNet co and the metadata and timestep embeddings fa, during training to speed up convergence using the rich semantic information in the pretrained SD weights. During training, we also randomly zero out the metadata vector m with probability of 0.1 to allow the model to generate images when metadata might be unavailable or inaccurate, A similar strategy employed to learn unconditional generation by Ho et al. (2020).
Single Image-Text- Mexa-Mat Datasets There is no equivalent of a large, text-image dataset (eg: LAION (Schuhmann et al. 2022)) for satellite images. Instead, we compile publicly available annotated satellite data and contribute large, high-resolution generative dataset for satellite images. Detailed descriptions on how the caption is generated for each dataset are in the appendix. (i) fMoo: Function Map of the World (fMoW) Christie al. (2018) consists of global, high-resolution (GSD 0.3m-1.5m) MAXAR satellite images, each belonging to one of 62 categories. We crop each image to 512x512 pixels. The metadata consider include longitude, latitude, GSD (in meters), cloud
3

---

Published as a conference paper at ICLR 2024
tempore liger
6.00 (1)
9pm
baredparaaeee sie, Timaaiood XVG
addd ton
concafanna
attomson
ESKA 040. 2012, 1,00
weights
Eil.sionear transttle copy
sanellite
place of Norship Lkratee
cituncidsl projection embeddingg
530.11 50.64 11570
Inpur
Figure 2: DiffusionSat flexibly extends to variety of conditional generation tasks. We design 3D version of ControlNet (Zhang Agrawala| 2023) which accept sequence of images. Like regular ControlNets, 3D ControlNet keeps a trainable copy of SD weights for the downsampling and middle blocks. Latent image features are reshaped to combine the batch and temporal dimensions hefore being input to these layers. The output of each SD block is then passed through temporal layer (top right), which re-expands the temporal dimension before passing the latent features though 3D convolution (initialized with zeros) and temporal, pixel-wise transformer. The metadata associated with each input image projected as in fig.
cover (as a fraction), year, month, and day. To generate a caption, we consider the semantic class and the country code. (ii) Satlas: Satlas Bastani al. 2022
large-scale, multi-task dataset of NAIP and Sentinel-2 satellite images. For our dataset, we use the NAIP imagery in Satlas-small, roughly of the same size IMoW. We the same metadata as in item (iii) SpaceNet: Spacenet Van Etten iet al. (2018) 2021) is a collection of satellite image datasets for tasks including object detection, semantic segmentation and road network mapping. We consider subset of Spacenet datasets, namely Spacenet v1, Spacenet v2, and Spacenet v5. We use the same metadata as earlier.
3.2 CONTROL SIGNAL CONDITIONAL GENERATION
Single-image DiffusionSat can generate a high reshuuooolttin satellite image given associated prompt and metadata, but it cannot yet solve the inverse problems described in section [ To leverage its pretrained weights, use it prior for conditional generation tasks which do encompass inverse problems such as super-resolution and in-painting. Thus, we now consider generative tasks where we can additionally condition on control signals (eg: sequences of satellite images) E RT'CC' xw" with associated metadata k, RR single caption and target metadata k R Here, C², H', and W' reflect the possible difference in the number of channels, height, and width, respectively, between the conditioning images and the target image. The goal is sample X p(-|s; k,; T; T; k). where X is sample conditioned on the control signal for given caption 7 and given metadata k.
Temporal Generation Recent works for video diffusion have proposed using 3D convolutions and temporal attention (Blattmann et al., 20233 Wu et al.) 2022 Zhou et al.) (022). while others propose using existing 2D UNets and concatenating temporal frames the channel dimension (Voleti et al. 2022 An et al. (023). However, sequences of satellite images differ in a few key ways from frames of images
video: (i) there is high variance in the length of time separating images in the sequence, while frames in video data are usually separated by a fixed amount of time (fixed frame rate) (ii) the length of time between images can be on the order of months or years, therefore capturing wider range of semantic information than consecutively placed frames in video (eg: season, human development, land cover). (iii) there is a sense of "global time" across locations. Even if one compares satellite images across different countries terrains, patterns may be similar if the
4

---

Published as a conference paper at ICLR 2024
generic caption, motadota
fiee metadata, coondiattss Fronce USA
fixed caption & metadets, vanving to
fixed caption metadota varving resouation low high resoutiio
satellste image
a satellite tmage of stadius 43,80% BO7E
satellite Jaoge of an
satellite image cf an Deuscmiott park Austrolia GBD: 1
45 09hL, 122
Augasi
Jaruaay
CSC: 0.5m
sinuciide projection leamned MI embediinng
incopporeting metadata
Figure 3: Here generate samples from single-image DiffusionSat. We see that changing the coordinates from a location in Paris one in USA changes the type of stadium generated, with American foothall and baschall more likely to appear the latter location. Additionally, for locations that receive snow, DiffusionSat accurately captures the correlation between location and season. However, naively incorporating the metadata into the text caption results in poorer conditioning flexibility across geography and season, (eg: with winter and summer time images produced for both August and January, a lack of "zooming in" when lowering the GSD).
year is known to be 2012 as opposed to 2020 (especially for urban landscapes). This is not the case for video data, where "local" time across frames sufficient to provide semantic meaning.
Usually, sequences of satellite images have fewer images than frames in natural image videos. As such, generating long sequences of images is less useful than conditioning on existing satellite imagery to predict the future or interpolate in the past (He et al. 2021 Bastani et al. (202).. Thus, we introduce our novel conditioning framework shown in fig. 2 to solve the inverse problem of frame-by- frame conditional temporal prediction. Unlike 2D ControlNet, we use 3D cero-convolutions between each StableDiffusion block (Zhang & Agrawala 2023). Our temporal attention layers, similar to VideoLDM (Blatmaan al. 2023), further enable the model to condition temporal control signals. We introduce a learned parameter or for each block to "mix" the output of the temporal attention layer to prevent noise from early stages in training from affecting our pre-trained weights (fig. (2)
key advantage of our approach is the ability to provide each item in the control sequence s with its own associated metadata, which is done similarly as in fig. [ by projecting each metadatum individu- ally and embedding it with an MLP. The embedded metadata for each image is then concatenated with its image and passed through the 2D layers of the ControlNet. DiffusionSat thus invariant to the ordering of images in the control sequence since each image's temporal position is solely determined by the timestamp in its metadata. single DiffusionSat model can then be trained to predict images in the past and future, or interpolate within the temporal range of the sequence.
Super-resolution with multi-spectral input Unlike in section 3.2 our input is a sequence s of lower resolution (GSD) images than the target image and can contain differing number of channels. The output of the model is still a high-resolution RGB image, as before.
Temporal Inpainting The task functionally equivalent to 3.2 except the goal to in-paint corrupted pixels (eg: from cloud cover, flooding, fire-damage) rather than predict new frame in
EXPERIMENTS
We describe the experiments for the tasks in section Implementation details are in appendix A.
5

---

Published as a conference paper at ICLR 2024
Method
SSIMI
PSNR1
LPIPS]
MSEJ
Method
FIDJ
ISt
CLIPF
Pix2Pix
0.1374
8.2722
0.6895
0.1492
SD1
117.74
6.42
17.23
DBPN
0.1518
11.8568
0.6826
0.0680
SD 2.1f
37.99
7.42
16.59
SD 0.1671
10.2417
0.6403
0.0962
SD 2.1 $
24.23
7.60
18.62
SD + CN
0.1626
10.0098
0.6506
0.1009
Ours
15.80
6.69
17.20
Ours 0.1703
10.3924
0.6221
0.0928
Table 1:
Single-image 512x512 genera- tion on the validation set of fMoW. refers finetuned SD 2.1 without any metadata information. I refers to incorporating the metadata in the text caption.
Tahle 2: Image sample quality quantitative results on fMoW superresolution. DBPN refers to [Haris al. (20T8), Pix2Pix from [sola et al. (017). Our method beats other super-resolution models for multispectral data.
For single image generation, we report standard visual-quality metrics such as FID (Heusel et al. 2017), Inception Score (IS), and CLIP-score (Radford et al. 2021). For conditional generation, given reference ground-truth image, we report pixel-quality metrics including SSIM (Wang et al. 2004), PSNR, LPIPS (Zhang et al. 2018) with VGG (Simonyan & Zisserman 2014) features. As noted in Gong et al. (2021) and He al. (202I), LPIPs is more relevant perceptual quality metric used in evaluating satellite images. Our metrics are reported on a sample size of 10,000 images.
4.1 SINGLE IMAGE GENERATION
We first consider single-image generation, as the task that DiffusionSat is pre-trained on. We compare against a pre-trained SD 2.1 model Rombach al. (2022), SD 2.1 model finetuned on dataset with our captions, but without metadata, and finally y a SD 2. model finetuned on our dataset with the metadata included in the caption (see table D. We find that including the metadata, even within the caption, is better than a caption formed from just the labels of satellite images. This is reflected in better FID scores, which measure visual quality. We expect that the text-metadata model t does better in terms of CLIP score given its more highly descriptive caption. However, treating metadata numerically, in DiffusionSat, further improves generation quality and control, as seen fig.
4.2 CONTROL SIGNAL CONDITIONAL GENERATION
We now use single-imas DiffusionSat effective prior for the conditional generation tasks of super-resolution, temporal generation/prediction, and in-painting. We describe the dataset for each task and demonstrate results using our 3D conditioning apporach on Texas-housing super-resolution, fMoW super-resolution using F-o-tSentinel multispectral inputs, temporal generation on the fMoW- temporal dataset, and temporal inpainting on the xBI natural disaster dataset. DiffusionSat achieves state of the art LPIPs and close to optimal performance on the SSIM and PSNR metrics as well.
flMoW Superresolution Using the dataset provided in Cong et al. 2022)) , we create fMo Sentinel-IMoW-RGB dataset with paired Sentinel-2 (10m-60m GSD) and fMoW (0.3-1.5m GSD) images at each of the original fMoW-RGB locations. Given all 13 multi-spectral bands of the Sentinel-2 image (here T = 1), we aim to reconstruct the corresponding high resolution RGB image. Super-resolution (section 4.2) given low-resolution (10m-60m) multi-spectral input is especially difficult, since most fMo W-RGB images are <Im GSD. We find that DiffusionSat once again outperforms strong super-resolution baselines, such as SD (which has shown to (table 2 fig. 4 We further note that while methods such as DBPN (Haris et al. 2018) yield strong PSNR7SSIM, these metrics don't reflect human perception and favor blurriness over sharp detail (Zhang et al. 2018 Saharia et al. 2022b).
Texas Housing Superresolution The dataset for this task is introduced by Spatial Temporal Superresolution (STSR) (He et al.) 2021) and contains 286717 houses built between 2014 and 2017 in Texas. Each location consists of two high-resolution images from NAIP (GSD Im) and 2 low- resolution images from Sentinel-2 (GSD 10m). A high resolution image at time t and corresponding low-resolution images at times t and t' form the control signal s, and the task is to reconstruct the other high resolution image at time L'.

---

Published as a conference paper at ICLR 2024
304
08P
Figure 4: Generated samples from (MoW-Senting superresolution validation set. The conditioning image is the Sentinel-2 multispectral (MS) image represented here as SWIR, NIR, RGB. The desired output is the high-resolution (HR) EMoW-RGB image. Our method is able to capture lne-grained details better than other baselines, even when the low-resolution MS image lacks detail. SD tends to "hallucinate" details.
We also perform an ablation on the efficacy of pretraining on our single image datasets against finetuning directly on SD weights. We find significant improvement from DiffusionSat pretraining. and and from using the 3D ControlNet (across all metrics) over simply stacking the images in the channel dimension and using 2D ControlNet (table
1' 1'>1
1' <t
Model
SSIM
PSNR1
LPIPS
SSIM
PSNRT
LPIPST
Pix2Pix
0.5432
20.8420
0.4243
0.3909
17.9528
0.4909
cGAN Fusion
0.5976
21.5226
0.3936
0.4220
17.8763
0.4726
DBPN
0.5781
21.4716
0.5101
0.4572
18.9330
0.5910
SRGAN
0.5361
21.1968
0.5261
0.4221
18.9772
0.5694
STSR (EAD)
0.6470
22,4906
0.3695
0.5225
19.7675
0.4275
STSR (EA64)
0.6570
22.5552
0.3764
0.5338
19.8547
0.4342
SD 3D ControlNet
0,4747
17.8023
0.4166
0.3458
16.1467
0.4351
Ours + ControlNet
0.5403
20.3982
0.3874
0.4657
1007
0.3652
Ours + 3D ControlNet
0.5982
21.0299
0.3247
0.4825
18.4604
0.3534
Table 3: Sample quality results Texas housing validation data. t' represents generating an image in the past given future HR image, and t the task for generating future image given a past HR image.
IMoW Temporal Generation Many locations in fMoW (Christie et al. 2018 contain multiple images across time. For our experiments, if < 4, we add copies of the latest image pad the sequence to 4 images. Given sequence s of conditioning images, DiffusionSat can predict another image at any desired target time by appropriately adjusting the target metadata k, (section 3.2) Since prior works aren't It designed to predict an image any given target time, we consider tasks where the target image is chronologically prior or later than the first image
Our experiments show that DiffusionSat outperforms STSR and MCVD (Voleti et al. 2022). as well as regular SD with our 3D ControlNet. Quantitative and qualitative results are in table 4 and fig. 5 respectively. These reveal DiffusionSat's improved ability over the baselines to capture the target date's 's season (eg: snow, terrain color, crop maturity) as well as development of roads and buildings. Other models, lacking the ability to reason about metadata covariates, often simply copy input image in the conditioning sequence as their generated output. In A.3.I we showcase DiffusionSat's novel ability generate sequences of satellite images without prior conditioning images
7

---

Published as a conference paper at ICLR 2024
t'
t' <t
Model
SSIM
PSNR
LPIPS!
SSIM
PSNRI
LPIPS
STSR (EAD) (He et al. 2021)
0.3657
13.5191
0,4898
0.3654
13.7425
0,4940
MCVD Volet al. 2022
0.3110
9.6330
0.6058
0.2721
9.5559
0.6124
SD + 3D CN
0.2027
11.0536
0.5523
0.2218
11.3094
0.5342
DiffusionSat + CN
0.3297
13.6938
0.5062
0,2862
12.4990
0.5307
DiffusionSat 3D CN
0.3983
13.7886
0.4304
0.4293
14.8699
0.3937
Table Sample quality quantitative results on EMo-rreppral validation data. t represents generating an image in the past given a future image, and t' < the task for generating a future image given a past image.
MCVD
62 SDCN
Dur
2012--01
2011-00
2308-00
201047
Oura
60 CH
STSI
MOVD
2016.00
2012-00
2015-00
20.0-C9
Shaf
Sirllie Flatric wi
Figure 5: Generated samples from the MMo--temporl validation set, for temporal prediction. The columns in the center are ground-truth images from temporal sequence. the right, we see generated samples future- peuureprediction task. The goal is to generate the image marked by the date in red, given the other images (to its left) conditioning signals. Similarly, for the past- prediction task on the left, the goal is to predict the image marked by the date in blue given the 3 images its right. DiffusionSat leverages pretrained weights to capture seasonal changes and predict human development better than the haselines. Images hest viewed zoomed in.
In-painting Rather than artificially corrupt input images, we use the xBD dataset (Gupta et al. 2019) which is subset of the x View-2 (Lam al., 2018) challenge to assess damage caused by natural disasters. Since each location carries pre- and post-disaster satellite image, we consider the in-painting task of reconstructing damaged areas in the post-disaster image, or introducing destruction to the pre-disaster image, where T = 1. We demonstrate qualitative results in fig. o DiffusionSat' capability to reconstruct damaged roads and houses for variety of disasters including floods, wind, fire, earthquakes etc will be important for disaster response teams to identify access routes and assess damage. We also show that DiffusionSat can add damage from different natural disasters, which can be useful for forecasting or preparing areas for evacuation.
5 RELATED WORK
Diffusion Models Diffusion models (Ho et al. 2020: Song et al. 2020b Kingma ct al.
202 l)
have
recently dominated the field of generative modeling, including application areas such as speech (Knng
et al. 2020, Popov et al. 202T). 3D geometry (Xu et al.] 2022 Luo & Hu 2021 Zhou et al. 2021).
and graphics (Chan et al.| 2023 Poole et al., 2022 Shue et al. 2023). Besides ad advancements in the theoretical foundation, large-scale variants built latent space (Rombach et al. 2022 Saharia
et al. 2022a Ho et al. 2022) have arguably been the most influential. With these foundation models
came slew of novel applications such subject customization (Ruiz et al.] 2023 Liu al. 2023
Kumari et al., (023) and text-to-3D generation (Poole et al.) [202)] Lin et al.] [2023; Wang et al.
2023). Many works have also demonstrated these models' impressive adaptation capabilities through
8

---

Published as a conference paper at ICLR 2024
p4M6 paGE
balora
and kiuze
peod past
beffor
altar
prod furure
Troddng
fooding
Euppoos
Dacocos
Figure 6: Inpainting results. The two columns marked "before" and "after" represent ground truth images. The "pred past" column is generated by conditioning on the "after" image, and the "pred future" column likewise by conditioning on the "before" image. DiffusionSat successfully reconstructs damaged roads and houses from floods, fires, and wind, even when large portions the conditioning image are masked by clouds or damage.
finetuning. For example, ControlNet (Zhang & Agrawala 2023). T2IAdapter (Mou et al. 2023). and InstructPix2Pix (Brooks et al. 2023), which add additional trainable parameters, have proven to be highly successful in adding control signals to the pre-trained diffusion networks.
Generative Models for Remote Sensing Image super-resolution is well studied for natural image
datasets (Dong et al. 20155 Ledig et al. 2017: Saharia et 2022h) Haris et al. (018)) Rombach et al.
2022
Generative Adversarial Networks (GANs) Goodfellow al. (2014) such as SR-GAN Ledig
et al.
(017) are among the most popular remotc-sensing super-resolution methods (Wang et al.
2020
et 2019 Gong et al. 2021 Corneiise al.
2022 Bastani al. 2023 Rabbi et al.
2020).
Other methods have tailor-man made convolutional architectures for Sentinel-2 image-input
superresolution (Razzak al., 2023; Tarasiewicz et al., 2023). More recently, Seatial-Temporal
Super Resolution (STSR) He et al. 1 uses a conditional-pixx synthesis approach to condition on a combination of high and low resolution images to generate high-resolution image at an earlier or later date. In general, these models lack the flexibility and generality of latent-diffusion models across a variety of tasks and datasets, and can suffer from unstable training (Kodali et al.) 2017). Our work aims to address these shortcomings by proposing single approach based on pretrained LDMs that can flexibly translate to downstream generative tasks via our novel conditioning mechanism.
6 CONCLUSION
In this work, we provide DiffusionSat, the first generative foundation model for remote sensing data based on the latent-diffusion model architecture of StableDiffusion Rombach et al. (2022). Our approach consists of two components: (i) single-image generation model that can generate high-resolution satellite data conditioned on numerical metadata and text captions (ii) novel 3D) control signal conditioning module that generalizes inverse problems such as multi-spectral input super-resolution, temporal prediction, and in-painting.
For future work, would like to explore expanding DiffusionSat to even larger and more diverse satellite imagery datasets. Testing the feasibility of DiffusionSat on generating synthetic data (Le) et al. 2023) might also augment existing discriminative methods to scale larger datasets. Lastly, investigating faster sampling methods or more efficient architectures will enable easier deployment use of DiffusionSat in cesttrce-contrrained settings.

---

Published as a conference paper at ICLR 2024
We hope that DiffusionSat spurs future investigation into solving inverse problems posed by remote-sensing data. Doing so would unlock societal benefits to important applications including object detection given super-resolved Sentinel-2 images (Shermeyer & Van Etten (019)), crop- phenotyping (Zang et al., 2020), ecological conservation efforts (Boyle et al., 2014] Johanneen et al. 2007). natural disaster (eg: landslide) hazard assessment (Nichol et al. al.]] 2006), archaeological prospection (Beck et al.) (007). urban planning Li et al. (2019); (0011)1XXia Xiao et al. (2006); Piyoosh & Ghosh (2017) and precise agricultural applications (Gevaert el al. 2015).
ACKNOWLEDGEMENTS
This research is based upon work supported in part by the Office of the Director of National Intelli- gence (ODNI) Intelligence Advanced Research Projects Activity (IARPA), via 2021-201 1000004, NSF(#1651565) ARO (W911NF-21-1-0125)) ONR (N00014-23- 1--2199... CZ Biohub. HAL The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes withstitst any copyright annotation therein.
REFERENCES
Jie An, Songyang Zhang, Harry Yang. Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent- shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint vrXiv:2304.08477, 2023.
Kumar Ayush, Burak Uzkent, Marshall Burke, David Lobell, and Stefano Ermon. Generating inter- pretable poverty maps using object detection in satellite images. arXiv preprint arXiv:2002.01612, 2020.
Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell. and Stefano Ermon. Geography-awan supervud learning. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 10181-10190, 2021a.
Kumar Ayush, Burak Uzkent, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Efficient poverty mapping from high resolution remote sensing images. In Proc. AAAI Conf. Artif. Intell, volume 35, pp. 12-20, 2021b.
Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. Sat- las: A large-scale, multi-task dataset for remote sensing image understanding. arXiv preprint arXiv:2211.15660. 2022.
Favyen Bastani, Piper Wolters, Ani Kembhavi, Jon Borchardt, Arnav Chheda, Aaron Sarnat, and Michael Schmitz, 2023. URL httpt::ttss/sttttl aller. ai/superres
Anthony Beck, Graham Philip, Maamoun Abdulkarim, and Daniel Donoghue. Evaluation of corona and ikonos high resolution satellite imagery for archaeological prospection in western syria. antiquity, 81(311):161-177 2007.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprint arXiv:2304.08815 2023.
Sarah A Boyle, Christina M Kennedy, Julio Torres, Karen Colman, Pastor E Estigaratiia and Noć U de la Sancha. High-resolution satellite imagery is an important yet underutilized resource in conservation biology. PLoS One, (1)xe8690 2014.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Marshall Burke, Anne Driscoll, David Lobell, and Stefano Ermon. Using satellite imagery to understand and promote sustainable development. Science, 371 (6535)))ea 2021.
10

---

Published as a conference paper at ICLR 2024
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. arXiv preprint arXhv:2304.02602: 2023.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In CVPR, 2018.
Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi- spectral satellite imagery. Advances in Neural Information Processing Systems, 35:197-211, 2022.
Julien Cornebise, Ivan Oršolić, and Freddie Kalaitzis. Open high-resolution satellite imagery: The worldstrat dataset-with application super-resolution. Advances in Neural Information Processing Systems, 35:25979-2559 25991, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis, Advances in neural information processing systems, 14:8780-8794 2021.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2): 295-307, 2015.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Caroline M Gevaert, Juha Suomalainen, Jing Tang, and Lammert Kooistra. Generation of spectral- temporal response surfaces by combining multispectral satellite and hyperspectral uav imagery for precision agriculture applications. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 8(6):314031166, 2015.
Yuanfu Gong, Puyun Liao, Xiaodong Zhang, Lifei Zhang, Guanzhou Chen, Kun Zhu, Xiaoliang Tan, and Zhiyong Lv. Enlighten-get for super resolution reconstruction in mid-resolution remote sensing images. Remote Sensing. 13(6):1104, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston. xbd: A dataset for assessing building damage from satellite imagery. arXiv preprint urXiv:1911.09221 2019.
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back- pooj-projection networks for super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1664-1673, 2018.
Yutong He, Dingjie Wang, Nicholas Lai, William Zhang, Chenlin Meng, Marshall Burke, David Lobell, and Stefano Ermon. Spatial-temporal super-resolution of satellite imagery via conditional pixel synthesis. Advances in Neural Information Processing Systems, 34:27903-27915 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update nule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruigi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint
2022.
11

---

Published as a conference paper at ICLR 2024
Xiao Huang, Di Zhu, Fan Zhang, Tao Liu, Xiao Li, and Lei Zou. Sensing population distribution from satellite imagery via deep learning: Model selection, neighboring effects, and systematic biases. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 5137-5151, 2021.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Computer Vision and Pattern Recognition (CVPR), 2017 IEFE Conference on, 2017.
Neal Jean, Marshall Burke, Michael Xie, W Matthew Davis, David B Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty. Science, 353((301))790-17 2016.
Kasper Johansen, Nicholas C Coops, Sarah E Gergel, and Yulia Stange. Application of high spatial resolution satellite imagery for riparian and forest ecosystem classification. Remote sensing of Environment, 110(1):29-44 44, 2007.
Firas Khader, Gustav Müller-Franzes,, Soroosh Tayebi Arasteh, Tianyu Han, Christoph Haarburger, Maximilian Schulze-Hagen, Philipp Schad, Sandy Engelhardt, Bettina Baeßler, Sebastian Foersch, et al. Denoising diffusion probabilistic models for 3d medical image generation. Scientific Reports, 13(1):7303, 2023.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in nearal information processing systems, 34:21696-21707,, 2021.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv: /t0///055007700 07211,, 2017.
Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint 2XXi::0099099660 2020.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1931-1941, 2023.
Darius Lam, Richard Kuzma, Kevin McGee, Samuel Dooley, Michael Laielli, Matthew Klaric, Yaroslav Bulatov, and Brendan McCord. xview: Objects in context in overhead imagery. arXiv preprint arXiv: 1802.008000 2018.
Van Anh Le, Varshini Reddy, Zixi Chen, Mengyuan Li, Xinran Tang, Anthony Ortiz, Simone Fobi Nsutezo, and Caleb Robinson. Mask conditional synthetic satellite imagery. arXiv preprint arXiv:2302.04305, 2023.
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6681-4690. 2017.
Xinghua Li. Zhiwei Li, Ruitao Feng. Shuang Luo, Chi Zhang, Menghui Jiang, and Huanfeng Shen. Generating high-quality and high-resolution scamless satellite imagery for large-scale urban regions. Remote Sensing, 12(1):81, 2019.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pp. 300-3000, 2023.
Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05 2023.
Shitong Luo and Wei Hu. Diffusson probabilistic models for 3d point cloud generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
12

---

Published as a conference paper at ICLR 2024
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Refusion: Enabling large-size realistic image restoration with latent- space diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1680-1691, 2023.
Rose M Rustowicz, Robin Cheong, Lijing Wang, Stefano Ermon, Marshall Burke, and David Lobell. Semantic segmentation of crop type in africa: A novel dataset and analysis of deep learning methods. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 75 82, 2019.
Wen Ma, Zongxu Pan, Feng Yuan, and Bin Lei. Super-resolution of remote sensing images via dense residual generative adversarial network. Remote Sensing, 11(21):2578, 2019.
Jorge Andres Chamorro Martinez, Laura Elena Cué La Rosa, Raul Queiroz Feitosa, Ieda Del' Arco Sanches, and Patrick Nigri Happ. Fully convolutional recurrent networks for multidate crop recognition from multitemporal image sequences. ISPRS Journal of Photogrammetry and Remote Sensing, 171:188-201, 2021.
Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie T2i-adapter: Learning adapters to dig out more controllable ability for texl-to-image diffusion models. arXiv preprint arXiv:230..0845. 2023.
Janet E Nichol, Ahmed Shaker, and Man-Sing Wong. Application high-resolution stereo satellite images to detailed landslide hazard assessment. Geomorphology, 76(1-2):68-75, 2006.
Atul Kant Piyoosh and Sanjay Kumar Ghosh. Semi-automatic mapping of anthropogenic impervious surfaces in an urban/suburban area using landsat 8 satellite data. GlScience Remote Sensing, 54 (4):471-494, 2017.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. /4988, 2022.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599 8608. PMLR, 2021.
Jakaria Rabbi, Nilanjan Ray, Matthias Schubert, Subir Chowdhury, and Dennis Chao. Small-object detection in remote sensing images with end-to-end edge-enhanced gan and object detector network. Remote Sensing, 12(9):1432, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Muhammed T Razzak, Gonzalo Mateo -arteoG Gurvan Lecuyer, Luis Gómez-Chove Yarin Gal, and Freddie Kalaitzis. Multi-spectral multi-image super-resolution of sentinel-2 with radiometric consistency losses and its effect on building delineation. ISPRS Journal of Photogrummetry and Remote Sensing, 195:1-13, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVÉ Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Mare Rußwurm and Marco Körner. Self-attention for raw optical satellite time series classification. ISPRS journal of photogrammetry and remote sensing, 169:421 2020.
13

---

Published as a conference paper at ICLR 2024
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713-4726, 2022b.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.084000 2022.
Jacob Shermeyer and Adam Van Etten. The effects of super-resolution object detection perfor- mance in satellite imagery. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 11875-20886, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409.1555 2014.
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems, 34: 12533-12548, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 2212-12448233 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456, 2020b.
Tomasz Tarasiewicz, Jakub Nalepa, Reuben A Farrugia, Gianluca Valentino, Mang Chen, Johann A Briffa, and Michal Kawulok. Multitemporal and multispectral data fusion for super-resolution of sentinel-2 images. IEEE Transactions on Geoscience and Remote Sensing, 2023.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Neural Information Processing Systems (NeurIPS), 2021.
Adam Van Etten, Dave Lindenbaum, and Todd M Bacastow. Spacenet: A remote sensing dataset and challenge series. arXiv preprint arXiv:1807.01232. 2018.
Adam Van Etten, Daniel Hogan, Jesus Martinez Manso, Jacob Shermeyer, Nicholas Weir, and Ryan Lewis. The multi-temporal urban development spacenet dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6398-6407, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
14

---

Published as a conference paper at ICLR 2024
Vikram Voleti, Alexia Jolicoeur-Martineuu and Christopher Pal. Masked conditional video diffusion for prediction, generation, and interpolation. arXiv preprint arXiv:2205.0985 2022.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https
//github. com htt.:/ugggiinggg faceeee/ge/ee//e / di ffueess 2022.
Anna X Wang, Caelin Tran, Nikhil Desai, David Lobell, and Stefano Ermon. Deep transfer learning for crop yield prediction with remote sensing data. In Proceedings of the 1st ACM SIGCAS Conference Computing and Sustainable Societies, pp. 1-5, 2018.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint 2rx.i:2305.16213 2023.
Zhongyuan Wang, Kui Jiang, Peng Yi, Zhen Han, and Zheng He. Ultra-dense gan for satellite imagery super-resolution. Neurocomputing, 398:328-331 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612 2004.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212. 1565, 2022.
Jieying Xiao, Yanjun Shen, Jingfeng Ge, Ryutaro Tateishi, Changyuan Tang, Yanqing Liang, and Zhiying Huang. Evaluating urban expansion and land use change in shijiazhuang, china, by using gis and remote sensing. Landscape and urban planning, 75(1-2)x69-8) 2006.
Yutong Xie and Quanzheng Li. Measuremeent-condiitonee denoising diffusion probabilistic model for under-sampled medical image reconstruction. In International Conference Medical Image Computing and Computer-Assiste Assisted Intervention, pp. 655-664. Springer, 2022.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint XXXiv:200300222. 2022.
Christopher Yeh, Chenlin Meng, Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, David B Lobell, and Stefano Ermon. Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learning. arXiv preprint urX1v:2111.04244, 2021.
Jiaxuan You, Xiaocheng Li, Melvin Low, David Lobell, and Stefano Ermon. Deep gaussian process for crop yield prediction based on remote sensing data. In Thirty- First AAAI conference on artificial intelligence, 2017.
Chongyuan Zhang, Afef Marzougui, and Sindhuja Sankaran. High-resolution satellite imagery applications in crop phenotyping: an overview. Computers and Electronics in Agriculture, 175: 105584, 2020.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.055443 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 586-595, 2018.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv. Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through voxnt-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5826-5835, 2021.
15

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
TRAINING DETAILS
We list implementation details for our experiments in this section. All models are trained on half- precision and with gradient checkpointing, borrowing from the Diffusers (von Platen et al. 2022) library.
Single-Image DiffusionSat We 8 NVIDIA A100 GPUs. The text-to-image models trained with a batch size of 128 for 100000 iterations, which we determined was sufficient for convergence. We choose constant learning rate of 2e-6 with the AdamW optimizer. We train two variants- one for images of resolution 512x512 pixels, and one for 256x256 pixels.
For sampling, we use the DDIM (Song et al. 2020a) sampler with 100 steps and guidance scale of 1.0. We generate 10000 samples on the validation sets of fMoW-RGB.
Super-resolution We use the 512 single-image DiffusionSat model as our prior. We train our ControlNet Zhang & Agrawala (2023) by upsampling the conditional multi-spectral image 256x256 pixels, which we found to work better than conditioning on 64x64 conditioning images. We use NVIDIA A A100 GPUs, and train the model for 50000 iterations with learning rate of Se-5 using the AdamW optimizer. We drop Sentinel bands B1, B9, B10, which we find to not be useful, similar to Cong al. (2022). We use the same sampling configuration as above.
Texas Housing We use the 256 single-image DiffuionSat model as our prior. We train our 3D ControlNet on sequences of the HR image and the two LR Sentinel-2 images. We use 4 NVIDIA A100 GPUs, and train the model for 50000 iterations with learning rate of 5e-5 using the AdamW optimizer. The sampling configuration is the same as above.
fMMoW Temporal We use the 256 single-image DiffusionSat model W-temporal dataset. the ControlNet on sequences of at-most conditioning images on the fMoW-tempon
prior. We train 3D
pick one of the conditioning images and copy it over until the sequence is padded to length. We avoid samples where there is only image per location. We train using 4 NVIDIA A 100 GPUs, for 40000 iterations with learning rate of 4e-4 using the AdamW optimzer. The sampling configuration matches the ones above.
location has less than 3 images,
A.2 DATASETS
A.2.1 CAPTIONS AND METADATA
The text captions are dependent on the metadata fields available for each dataset. For the captions below, fields denoted in angle brackets below filled in using the metadata for each example. Some sections of each caption, denoted by square brackets below, are randomly and independently dropped of caption instances at 10% rate. We label datasets from the same satellite sources with the same image type (e.g., both Texas Housing and Satlas use NAIP images, so both are labelled as Msat lattlas" images).
Dataset fMoW
Caption
"a [fnow] satellite image "of <object>] [in <country>]
SpaceNet Satlas
"a [spacenet] satellite image [of <object>] [in <city>]"
"a [satlas] satellite image [of <object>>"
"a [satias] satellite image lof houses]
Texas Housing
[built <year_buiith (covering <nun_acres> acres]" ma [frow] satellite image (<before/after>) being affected by <disaster_type) natural disaster"
xBD
Table 5: Captions created for each dataset type based on available label information.
Besides the captions, we also incorporate numerical metadata from fields. Each field was normalized based on high and low reference values: mporm m/(high- low)) x scale, where scale is scaling
16

---

Published as a conference paper at ICLR 2024
field lon lat
description
min
max
longitude, in degrees longitude, in degrees ground sampling distance proportion pixels with cloud cover
-180
180
-90
90
gad
0
10
cloud_ _cover
1
year
year of the satellite image
1980
2100
month
month of the year day of the month
12
day
0
31
Dataset
Inage
Caption
Metadata
lon: -76.781
17. 98
25d: 0.941
satellite inage dealership in United States of America
fnow
cloud_ cover:
year: 2010
month: 10
day: 6
lon: 78.
lat: 85.648
satellite inage buildings
gsd:
satlas
cloud_cover: cover::
yeaz: 2013
month: day: 22
lon: -43.636
lat: -22.892
spacenet satellite image 144 buildings covering an area of 9288.166 squared meters in Rio
0.793
spacenet
cloud_covers 9
year: 1980
month: day: 0
Figure Sample captions and pre-normalization metadata fMoW, Satlas, SpaceNet datasets.
factor of 1000, such that low maps to 0 and high maps to scale. The fields are summarized in Figure
We include examples of both the captions and numerical metadata in Figure g
A.3 TEMPORAL GENERATION
In this section, we provide further results for the temporal generation task, demonstrating the powerful capabilities of DiffusionSat.
A.3.1 SEQUENCE GENERATION
We first demonstrate how we can generate temporal sequences of satellite images unconditionally i.e. without any prior conditioning image, unlike section 4.2 To do so, we first generate satellite image using single-image DiffusionSat given caption and desired metadata for our image. We then apply our novel 3D-conditioning ControlNet, already trained for temporal generation, on the first image to generate the next image in the sequence, given years/months/days into the future or past). We now re-apply the 3D-conditioning ControlNet on the first 2 generated images to get the third image of the sequence. Repeating this procedure,
desired metadata (eg: how many
17

---

Published as a conference paper at ICLR 2024
able to regressiisy sample sequences of satellite images given desired metadata properties. Generated samples using this procedure are shown in fig. 9
Norckmaa
Bock
In
CRTL
Figure 9: Auto- regressively generated sequences of satellite images given caption (for the sequence) and desired metadata (per image). The image sampled from single-imag -mage DiffusionSat is outlined in red. On the left, we generate sequences backwards time (i.e.: into the past) given the first generated image. For example, for the park in Japan (2nd column from the left), the metadata, from the bottom to the top image, is: (1.07, 2014, 5, 25), (1.62, 2013, 6, 17), (1.17,2 201 3, 17), (1.17, 2010, 11, 8). The metadata is in order (GSD, year, month, day). We omit listing the latitude and longitude, since that remains the same for the sequence, but it is inputted metadata, descrihed in fig. On the right, generate images forwards in time (i.e. into the future) given the first generated image outlined red. For example, for the crop field Chile, metadata, from the bottom to the top image, is: (1.03, 2016, 2. 23), (1.03, 2015. 9. 17). (0.97, 2014, 10, 12). (1.17, 2014, 8, 13). As we can see, our model generates realistic sequences that reflect trends in detail and development both forwards and backwards in time.
Our results demonstrate novel way of generating arbitrarily long sequences of satellite images- our conditioning mechanism can flexibly handle both conditional and unconditional generation. The generated samples reflect season and trends in development (eg: past images have fewer structures, future images usually have more detail).
A.4 GEOGRAPHICAL BIAS
Concerns about bias for the outputs of machine learning models are natural given the large, potentially biased datasets they are trained on (Huang et al. 2022)) We perform an evaluation of the generation
quality of single-image DiffusionSat across latitude and longitude around the globe in fig. T0 and fig. 11
Our results show no particular favoritism for location, even though one would expect better generation quality for regions in North America and Europe. We would still like to point out few caveats:
(i) FID or LPIPs scores may not the most informative metric towards estimating bias in sample quality. We use as measure of generation quality given lack of better alternatives for the novel problem of estimating geographical bias generative remote sensing models.
(ii) The FID scores are dependent on sample size, and so while the scores might be evenly distributed, it still remains the case that there are far more dataset samples from developed regions of the world, and a dearth of images for large swaths (eg: across Africa). Even so, for severely biased model we would expect poorer generation quality for data-poor regions of the world.
(iii) We estimate only one angle of bias. Bias may still exist along different axes, such as generating types of buildings, roads, trees, crops, and understanding the effects of season. We leave this investigation future work.
18

---

Published as a conference paper at ICLR 2024
Generated sample FID
80
600
60
500
40
400
Lattude
300
20
440
200
100
-150
100
Longitude
Figure 10: FID scores of ingle-image DiffusionSat prompted on 10k samples of the fMoW-RGB validation set for coordinates around the world.
Generated sample lpips scores
0.76
0.74
20
Lattude
0.72
0
0.70
60
0.66
80
0.64
-150
-100
-50
50
100
150
Longitude
Figure 11: L.IPs scores of super-resolution DiffusionSat prompted on 10k samples of the fenW-Sentiee- MGW-RGB validation set for coordinates around the world.
19


Reviews:
[{"review_id": "dhxv7TMOOP", "reviewer": "Reviewer_tstN", "summary": "The paper presents a diffusion based remote sensing model able for the following generative downstream tasks: (i) temporal image generation, (ii) multispectral image super resolution, and (iii) image in-painting.\n\n\nThe work is novel as it represents the first diffusion-based remote sensing model. This is interesting given the multi-spectral nature of remote sensing data and the \"image caption\" adaptations required for the diffusion backbone. Further, the authors adapted ControlNet architecture to a 3d ControlNet architecture to serve their task.", "questions": "**(Q1)**: Question for clarification, in your abstract you have mentioned 3 downstream tasks (temporal image generation, multispectral image super resolution, and image in-painting. In your methods section you have mentioned 5 downstream tasks (single-image generation, conditioned on text and metadata, multi-spectral superresolution, temporal prediction, and temporal inpainting). Do you summarize those 5 into the 3 in the abstract? And if yes, why?\n\n**(Q2)**: The \"Experiment\" section presents results for the following downstream tasks:\n\"4.1 Single Image Generation\", \"4.2 Conditional Generation\". Then in \"4.2 Conditional Generation\" results are shown for \"super resolution\" and \"temporal prediction\". Finally, the \"in-painting\" section is missing. This is not in sync with the previously mentioned downstream tasks. Could you clarify the structure of the experiment section?\n\n**(Q3)**: In Table 2, results are presented and compared to other generative methods. Why do you not provide this comparison for Table 1 and Table 3, where results are only compared against non-adapted stable diffusion? Is there any reason for this, if yes, it might be missing in the text body of section 4. I would also recommend to cite the methods you compare to in Table 2.\n\n**(Q4)**: In Fig. 4, which part of the left image is damaged and later in-painted? I see some color-corrections done on the middle image and the ground truth. Is the \"damage\" done on some specific RGB (or multispectral) channels?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**(S1):** this work presents a novel diffusion-based approach for remote sensing data. It is great to see people extending diffusion to remote sensing data as it is of complex nature given its multi-spectral composition.\n\n**(S2):** this work outlines multiple generative downstream tasks for remote sensing which do go beyond \"simple\" image generation. This is important because in remote sensing we have no shortage of data and therefore actually not much demand for \"simple\" image generation.", "weaknesses": "**(W1)**: the presented work is very domain specific i.e., remote sensing data. It would be interesting to see if this approach is able to generalize to other datasets of similar multi-spectral data.\n\n**(W2)**: I am confused by the \"4. Experiment\" section. It is not always straightforward to link the tables and images to the different generative downstream tasks presented. And it seems that some results are missing (i.e., the In-painting section/paragraph is missing entirely)? This point also goes hand in hand with my question Q1, Q2, Q3 below: you mention different downstream tasks in the abstract and the method section, and then the results are not consistently reporting on these downstream tasks (as it seems to me, but maybe I misunderstand something). This is the strongest weakness as it renders the evaluation of the methods to be very difficult for the reader of the submitted paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0pKyA0jIDc", "reviewer": "Reviewer_ZvAY", "summary": "This work proposes a generative foundation model, i.e., DiffusionSat, for remote sensing data based on the latent-diffusion model architecture of StableDiffusion. Conditioning on freely available metadata as well as generated captions on large, publicly available\nsatellite datasets makes DiffusionSat a powerful and flexible generative model. Further, a novel 3D ControlNet which allows DiffusionSat to generalize to multi-spectral superresolution, temporal prediction, and in-painting is designed.", "questions": "1. The proposed foundation model for satellite images is not novel and significant since there are many foundation models in the remote sensing area, such as  [1-2].\n[1] A Billion-scale Foundation Model for Remote Sensing Images. arXiv:2304.05215.\n[2] Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model. TGRS, 2022.\n\n2. The authors claim that \"while foundation models have been recently developed for discriminative learning on satellite images, no such foundation model exists for generative tasks. \" Please explain why the existing foundation models are not applicable to the generative tasks.\n\n3. The methodology of training the proposed foundation model is not novel and the authors only apply the stable diffusion model to the satellite images. Although the authors use some new conditional inputs in the stable diffusion framework, it is not novel enough for publication in this conference.\n\n4. What is the purpose of conducting research on the generative foundation model? Please make a discussion for this point.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "This work proposes a generative foundation model for remote sensing data based on StableDiffusion. The proposed foundation model produces realistic samples and can be used to solve multiple generative tasks including temporal generation, multi-spectral superrresolution and in-painting.", "weaknesses": "1. The necessity and motivation of designing the generative foundation model are not clear and convincing.\n2. The methodology of training the proposed foundation model is not novel since the whole framework is a combination of stable diffusion and ControlNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OYyoqmI9fh", "reviewer": "Reviewer_DCYE", "summary": "Authors propose DiffusionSat which is a Stable Diffusion (SD) inspired generative model to generate satellite imagery. The imagery generation can be conditioned satellite imagery metadata. To be able to train this model authors had to generate a large-scale satellite imagery pretraining dataset by combining multiple open datasets, generating text descriptions, and adding metadata available. Work was done to encode the conditioning metadata properly. The pretrained model was tested in multiple downstream tasks we competitive performance.", "questions": "1. It is very difficult to see the inpainted damage referred to in Figure 4. Please consider highlighting the area where damage has been inpainted. \n2. Abstract missing dot after \"remote sensing datasets\"\n3. No text under \"In-painting\" subsection within the experiments section.\n4. Share more examples of generated images", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper tackles an important problem with understady on the computer vision field with impotant possitive societal benefiting applications\n* Novel incorporation of additional metadata and problem setup and can spin off a new line of work in geospatial ML\n* The generated dataset used for this study can be very useful in other applications.", "weaknesses": "* **Results for certain tasks are missing or incomplete.** The paper mentions that they show state-of-the-art results for super-resolution, temporal generation, and in-painting. However, only a single qualitative example is provided as result. Also multiple other relevant approaches have been proposed. Superresolution results just compare the proposed approach with Stable Diffusion baseline but ignores the line of work done in the field including [1,2] and others.\n* **The paper write up needs work to improve cohesion.** The related work section could be merged into the background. The task description coud be joint with the results since bothe are very brief.\n* **Overall evaluation and results comparison.** There are multiple other approaches for satellite image generation including conditional that this approach does not compare against. [1]  If the focus wants to be on using this model as foundational model for other geospatial ml tasks, then comparison with other self-supervised and/or foundational models is important.\n\n\nReferences:\n1. Tarasiewicz, Tomasz, et al. \"Multitemporal and multispectral data fusion for super-resolution of Sentinel-2 images.\" IEEE Transactions on Geoscience and Remote Sensing (2023).\n2. Razzak, Muhammed T., et al. \"Multi-spectral multi-image super-resolution of Sentinel-2 with radiometric consistency losses and its effect on building delineation.\" ISPRS Journal of Photogrammetry and Remote Sensing 195 (2023): 1-13.\n3. Le, V. A., Reddy, V., Chen, Z., Li, M., Tang, X., Ortiz, A., ... & Robinson, C. (2023). Mask Conditional Synthetic Satellite Imagery. arXiv preprint arXiv:2302.04305.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K8VTfodTrL", "reviewer": "Reviewer_tt9n", "summary": "The authors propose a stable diffusion variant, called DiffusionSat, designed for satellite image synthesis tasks. The proposed approach leverages text captions, an assortment of satellite image metadata (longitude, latitude, GSD, etc.), and sequences of images to address various overhead image problems, such as varying spatial resolution (GSD), cloud cover, and seasonality.  DiffusionSat shows strong performance across multiple tasks (super-resolution, temporal generation, and in-painting) on three repurposed datasets.", "questions": "My initial rating is weak accept. The proposed approach is interesting and has novelty in its formulation. The manuscript is well written and the results are compelling. Ultimately, I think this paper will have a positive impact on the community.\n\nFor the fMoW super-resolution task, are the super-resolved images multi-spectral or just RGB? The text seems to suggest that the model takes in a 13 band Sentinel-2 image and outputs a reconstruction of a 3 band WorldView-3 image from fMoW. Is the reconstruction limited to just 3 bands (R, G, B)? If so, then the task isn't actually multi-spectral super-resolution.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Tackles an important and interesting problem, view synthesis in remote sensing.\n\n- Well-written, clear motivation.\n\n- Method has the potential to positively impact the remote sensing community.\n\n  DiffusionSat operates on a sequence of satellite images (with varying spatial resolution) and applies attention across time via a temporal pixel-transformer. This allows the approach to leverage an image sequence to address the shortcomings that may be present in any individual image (e.g., clouds). While the underlying temporal transformer was introduced by VideoLDM (Blattmann et al., 2023), its application to remote sensing problems is still valuable.  Further, the proposed method allows satellite image synthesis to be controlled using metadata and text. For example, synthesizing images with seasonality (summer grass vs winter snow), location (France vs.  USA), and ground sample distance.\n\n- The authors introduce a new temporal generation task using satellite image datasets. This task targets common issues with using satellite image sequences, namely that the interval between images is not fixed and that frames can have varying GSD.\n\n- Evaluation is fairly extensive.", "weaknesses": "- Limited technical novelty. \n\n  The proposed method reduces to a straightforward way of embedding metadata and then conditioning a stable diffusion model on said metadata. The temporal transformer component is from VideoLDM. \n\n- Quantitative performance across multiple tasks is strong, but not convincingly better than existing approaches. While the proposed method does achieve SOTA performance on some metrics on some benchmarks, it doesn't show consistent enough performance across the board to substantiate the claim that the method is SOTA on all these tasks, as the introduction indicates.\n\n- The approach and evaluation are limited for multi-spectral imagery (MSI).\n\t\n  This work does not address the issue of varying image band GSD (e.g., Sentinel-2 bands) in MSI, despite claiming to support MSI synthesis. For various remote sensing products, image bands may not all be the same GSD. Some effort should be taken to recognize this occurrence, since 10m -> 1.5m super-resolution is far easier than 20m->1.5m (Sentinel-2 bands 5, 6, 7, 8a, 11, and 12) or 60m -> 1.5m (Sentinel-2 bands 1, 9, and 10) [1] super resolution.\n\n  [1] https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi/resolutions/spatial\n\t\n  The MSI super-resolution problem aims to synthesize 3-band WV3 imagery from 13-band Sentinel-2 imagery. While this may be useful in some scenarios, it does leave out a fairly significantly problem, namely, generating all spectral bands. As such, the method does not really perform MSI synthesis, but rather can support MSI inputs for RGB super-resolution.\n\n- Figure 4 is not informative as the disaster-related damage is difficult to identify. Consider highlighting damaged areas, or use a more clear example.\n\n- Minor editing problems: leftover question mark (page 4), repetitive text (bottom of pages 1 and 3), typo and highlighting on page 8, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)