SatDiffMoE: A Mixture of Estimation Method for Satellite Image Super-resolution with Latent Diffusion Models

## Abstract

Abstract During the acquisition of satellite images, there is generally a trade-off between spatial resolution and temporal resolution (acquisition frequency) due to the onboard sensors of satellite imaging systems. High-resolution satellite images are very important for land crop monitoring, urban planning, wildfire management and a variety of applications.
It is a significant yet challenging task to achieve high spatial-temporal resolution in satellite imaging.
With the advent of diffusion models, we can now learn strong generative priors to generate realistic satellite images with high resolution, which can be utilized to promote the super-resolution task as well.
In this work, we propose a novel diffusion-based fusion algorithm called SatDiffMoE that can take an arbitrary number of sequential low-resolution satellite images at the same location as inputs, and fuse them into one high-resolution reconstructed image with more fine details, by leveraging and fusing the complementary information from different time points. Our algorithm is highly flexible and allows training and inference on arbitrary number of low-resolution images.
Experimental results show that our proposed SatDiffMoE method not only achieves superior performance for the satellite image super-resolution tasks on a variety of datasets,
but also gets an improved computational efficiency with reduced model parameters,
compared with previous methods.

## 1 Introduction

Satellite imaging is a very useful technique for monitoring the natural phenomena and human activities on the surfaces of the Earth.
Lots of applications rely on satellite images such as crop monitoring, weather forecasting, urban planning, wildfire management and so on  [DiffusionSat, sustainabledevelopment, poverty1, poverty2, toefl, sat1, sat2, sat3].
However, the acquisition of satellite images can be very expensive and the spatial and temporal resolution (the frequency that a satellite image is captured) may be limited due to the physical constraints of sensors [DiffusionSat].
In addition, the high temporal resolution may come with trade-off in spatial resolution. Recent advance in satellite imaging technology enables us to capture the same area with a high-revisit frequency, but the spatial resolution is often limited. For instance, two Sentinel-2 satellites with resolutions from 10m to 60m can capture all land surfaces every five days [WorldStrat, satmae, sat4, sat5], but to perform land crop monitoring or wildfire management, this resolution is not sufficient.
On the other hand, some very high-resolution satellite images such as SPOT6 or WorldView can have resolution better than 1.5m [fmow], but it is extremely difficult to collect those images for a large area, and these images cannot be captured as frequently as the Sentinel images. The limited temporal resolution severely limits downstream applications in urban planning, object detection, or continuous monitoring of crop or vegetation covers.

To solve the aforementioned challenges, super-resolution algorithms have been introduced to predict the high-resolution (HR) satellite images from a bunch of corresponding low-resolution (LR) satellite images [fusion0, refusion1], so that to obtain more fine details.
For example, given the low-resolution satellite images (10m) from Sentinel-2 of a specific location, the super-resolution algorithms are developed to predict the high-resolution (1.5m) SPOT6 satellite image of the same location at a specific time.
Nevertheless, solving remote sensing super-resolution problems is still an open problem. One challenge in remote sensing is that low-resolution and high-resolution images often come from different sensors, and may maintain very different image features due to the large imaging modality gap resulted from different sensors [fusion0].
Moreover, since the low-resolution and high-resolution images are usually acquired at different time points as demonstrated in Fig. 1, a lot of atmospheric disturbance may pose additional challenges for modeling the sensor imaging process.
Therefore, unlike the natural images, in the satellite images, the down-sampling process can be extremely difficult to model.

In order to tackle this challenging task of remote sensing super-resolution, considering that the acquisition of satellite images often comes with multiple revisits (a collection of satellite images at the same location but at different time stamps) or with multiple spectral bands,
we hypothesize that fusing multiple low-resolution images with different spectrums or at different time stamps may provide complementary information to the model so as to benefit the super-resolution task.
With such motivation, existing works [WorldStrat] have introduced a recursive fusion module that takes the concatenated LR images as inputs and applies a residual attention model for outputting the HR image.
DiffusionSat [DiffusionSat] introduces a 3D ControlNet architecture that fuses LR images of different spectral bands to reconstruct the corresponding HR image. However, these works require a fixed number of LR images or require an absolute timestamp for each LR and HR, which is often not feasible and flexible at inference time in real practice because it is challenging to find a fixed amount of paired LR images for each HR image.

In this paper, we propose a novel diffusion-based method for solving the satellite image super-resolution problem as demonstrated in Fig. 1.
Our contribution can be summarized as below:

Figure: Figure 1: An overview of our proposed method SatDiffMoE.
Refer to caption: extracted/5666614/figures/intro.jpg

- •
We propose a novel diffusion-based fusion algorithm for satellite image super-resolution that can take an arbitrary number of time series low-resolution (LR) satellite images as input, and fuse their complementary information to reconstruct high-resolution (HR) satellite images with more fine details.
- •
Specifically, we introduce a new mechanism to train a latent diffusion model using the paired LR and HR images (from the same location but with different time stamps), particularly being aware of the relative time difference between corresponding LR and HR images, to capture the time-aware mapping distribution of LR to HR images
- •
At inference time, by leveraging the trained time-aware diffusion model, we propose a novel approach to fuse the information from time series LR images, by estimating the center of reverse sampling trajectories of different LR images using a perceptual distance metric, so as to align the semantics from various LR images for super-resolution task.
- •
We achieve the state-of-the-art performance on a variety of datasets for satellite image super-resolution.
Moreover, our method demonstrates an improved computational efficiency with reduced model parameters compared with previous methods

## 2 Background

#### Latent diffusion models

Diffusion models consists of a forward process that gradually add noise to a clean image, and a reverse process that denoises the noisy images. The forward model is given by $x_{t}=x_{t-1}-\frac{\beta_{t}\Delta t}{2}x_{t-1}+\sqrt{\beta_{t}}\Delta t\omega$ where $\omega\in N(0,1)$. When we set $\Delta t\rightarrow 0$, the forward model becomes $dx_{t}=-\frac{1}{2}\beta_{t}x_{t}dt+\sqrt{\beta_{t}}d\omega_{t}$, which is a stochastic differential equation. The solution of this SDE is given by

$$ $dx_{t}=(-\frac{\beta(t)}{2}-\beta(t)\nabla_{x_{t}}\log p_{t}(x_{t}))dt+\sqrt{ \beta(t)}d\overline{w}.$ (1) $$

Thus, by training a neural network to learn the score function $\nabla_{x_{t}}\log p_{t}(x_{t})$,
one can start with noise and run the reverse SDE to obtain samples from the data distribution.

Latent diffusion models (LDM) [rombach2022highresolution] have been proposed for faster inference and training with a reduced computational burden.
By applying an autoencoder to reduce data dimension, LDMs train the diffusion model in a compressed latent space, and then decode the latent code into signals.
This method enables high-quality high-resolution image synthesis benefited from its compressed latent space, which is an ideal fit for satellite images due to the large image size.
Nevertheless, it is still challenging to perform image restoration accurately with LDMs [ReSample, rombach2022highresolution].
Various works have tried to extend LDMs for high-dimensional or high-resolution signal synthesis, such as video generation [video1, video2, video3, video4, Video5, refusion2, refusion3, refusion4, refusion5]. However, few works apply LDMs for image fusion yet.
It is an open problem to sample high-resolution images conditioning on multiple similar low-resolution images with LDMs.

#### Single-image super-resolution

Single-image super-resolution (SISR) focuses on reconstructing the high-resolution image from one corresponding low-resolution image. In the era of deep learning, super-resolution problem has become a popular research question and many data-driven methods have been proposed. In past few years, state-of-the-art methods apply techniques such as Convolutional neural network (CNN), Generative Adversarial Network (GAN), and Transformers for restoring the high-resolution images [VAE, GAN, DBPN, MSRResNet, Pix2pix, Attentionallyouneed].
These methods usually learn the mapping from low-resolution image to high-resolution image through a data-driven way using the pair data to train the neural network.

However, many of these methods output blurry or inaccurate reconstructions since they are learning a direct mapping between LR images and HR images without considering the distribution of possible HR images [ReSample, PINER].
Due to the ill-posedness of the super-resolution problem, there may exist multiple HR images corresponding to one single LR image. A direct regression-based approach may let the network learn an average of all possible HR images, which leads to blurry output.
Diffusion models address this issue by learning a strong generative prior that can perform posterior sampling [chung:23:dps] instead of direct regression. This sampling method outputs realistic images, and leads to better image perceptual quality.
One line of work assumes the degradation operator is known, and focuses on inference-time posterior sampling with the diffusion prior without retraining [chung:23:dps, DDRM, DDNM, ReSample, Solve2]. The other line of work assume the degradation operator is unknown. They concat the LR image into the noise vector or conditional networks and then retrain or fine-tune the diffusion model [ControlNet, DiffusionSat]. Both lines of work show good ability to model complex high-resolution image distributions and outperform CNN approaches in image perceptual quality. However, all these works focus on obtaining HR images from a single LR image.

#### Multi-image super-resolution

The goal of multi-image super-resolution is to combine the information from multiple LR images to reconstruct one HR image.
In satellite imagery, different sensors have different resolutions. For instance, Sentinel-2 SITS has a resolution of 10-60m, but SPOT-6 or fMoW can have a resolution of less than 1.5m [DiffusionSat].
Given sequential low-resolution images collected at the same location but different times, the hypothesis is that performing multi-image super-resolution is able to combine the information of LR images at multiple times to obtain a more accurate and higher-quality HR image.
There are a couple of recent works in this venue [fusion0, WorldStrat, DiffusionSat].
For example, Cornebise et al. trained a network with a traditional autoencoder architecture, but modify the encoder to incorporate multiple images as input [WorldStrat].
HighRes-net [WorldStrat] adopted this idea to solve satellite image fusion problem with a network composed of an encoder, a recursive fusion network, and a decoder. However this approach does not include the temporal information of LR images
DiffusionSat [DiffusionSat] proposed to train a 3D ControlNet on top of a fine-tuned latent diffusion model that leverages multispectral bands for reconstruction. However, at training, the 3D ControlNet requires the same number of low-resolution images for each paired high-resolution images. In addition, at inference-time, the number of low-resolution images used for reconstruction must be the same as the training set.
Motivated by this, essentially different from all these previous works, we aim to propose a more flexible and robust fusion algorithm that can take an arbitrary number of low-resolution inputs with corresponding time information to generate the high-resolution satellite image.

## 3 Methods

Figure: Algorithm 1 SatDiffMoE: Satellite Image Fusion with Latent Diffusion Models

Figure: Figure 2: The framework of our proposed SatDiffMoE. In the training phase, we train a latent diffusion model for HR (high-resolution) image conditioning on a single LR (low-resolution) image and its relative time difference with the HR image. Then in the inference phase, we fuse the reverse sampling trajectories conditioning on each LR image of the same location. We can randomly select different trajectories for fusion, but output to a single image at the end.
Refer to caption: extracted/5666614/figures/framework.png

Instead of conditioning on multiple low-resolution images as a concatenated input [DiffusionSat], we propose a novel fusion algorithm: condition on each image and then fuse each score (the output of the conditional diffusion model) into a high-resolution image reconstruction.
First, in order to embed the temporal information, we introduce a new method that trains a conditional diffusion model using one LR image and the relative time difference (between the input LR image and the target HR image) as inputs to reconstruct the HR image.
Then we propose a novel inference-time algorithm that modifies the intermediate outputs in the reverse sampling procedure based on the assumption that HR reconstruction of LR images at the same location should look similar. Our method is illustrated in Fig. 2.

#### Training

We propose a novel conditioning mechanism for reconstructing HR image from one LR image and its corresponding time difference with the target HR image.
Firstly, we utilize Stable Diffusion [rombach2022highresolution], a pre-trained latent diffusion model for natural image generation, as the backbone model and fine tune the model on satellite images.
We also propose a novel method to add additional control to the generating process.

Let HR be the target high-resolution image we want to reconstruct, $\text{LR}_{i}$ denote the $i_{th}$ low-resolution image collected at the same location as the HR image, and $dt_{i}$ denotes the $i_{th}$ relative time difference between $\text{LR}_{i}$ and HR.
We observe that it is possible to train the model on the joint distribution of $p(\text{LR}_{i},\text{HR})$
However, during inference, the HR image is not available and we can only access the LR images.
As a result, let $F$ be the operator that masks out the HR component from the score $s_{\theta}(\text{LR}_{i},\text{HR}))$, and we propose a training objective:

$$ $\mathop{\arg\min}\limits_{\theta}\mathbb{E}_{\text{LR}_{i},\text{HR},t, \epsilon\in N(0,1)}||F(s_{\theta}((\text{concat}(\mathcal{E}(\text{LR}_{i}), \mathcal{E}(\text{HR}))_{t},t)-\epsilon)||^{2}_{2}$ (2) $$

where $\mathcal{E}$ is the pretrained image encoder of the LDM, $\text{concat}(\mathcal{E}(\text{LR}_{i}),\mathcal{E}(\text{HR}))_{t}$ is the output of applying the forward process of LDM on the concatenation of $\text{LR}_{i}$ and HR.
Through this designed training objective, we do not need to add any additional parameter to the pretrained stable diffusion model,
since the stable diffusion model can take arbitrary image resolution as input and output.

Then we propose a novel method that takes the relative time difference $dt_{i}$ between the $\text{LR}_{i}$ and HR as another conditional input to the LDM.
We make a clone of the time embedding network for the original Stable diffusion model, feed $dt_{i}$ into the cloned embedding network, and then we add the output from the time embedding and the output from the $dt_{i}$ embedding together as an overall embedding with positional encoding to the diffusion model.
We aim to reconstruct the same HR image regardless of what time $\text{LR}_{i}$ is taken at, so we add this relative time difference embedding $dt_{i}$ to offset the time difference of $\text{LR}_{i}$ and HR.
Then the final training objective becomes:

$$ $\mathop{\arg\min}\limits_{\theta}\mathbb{E}_{\text{LR}_{i},\text{HR},t, \epsilon\in N(0,1)}||F(s_{\theta}((\text{concat}(\mathcal{E}(\text{LR}_{i}), \mathcal{E}(\text{HR}))_{t},dt_{i},t)-\epsilon)||^{2}_{2}$ (3) $$

Compared to ControlNet, our proposed model is more computationally efficient with much fewer training parameters. We observe our method also converges much faster than ControlNet and achieves better reconstruction results, which is demonstrated in the experiments section below.

#### Inference

As discussed in the previous subsection, we want to estimate a single HR reconstruction based on $\text{LR}_{i}$ taken at different times.
Hence we can assume that the outputs conditioning on different $\text{LR}_{i}$ should be aligned semantically since they are reconstructing the same HR image.
Let $z_{t}(\text{LR}_{i},dt_{i})$ be a noisy sample at time $t$ during diffusion reverse sampling conditioning on $\text{LR}_{i}$ and $dt_{i}$.
Based on the assumption above, we expect $\mathbb{E}[z_{0}|z_{t}(\text{LR}_{i},dt_{i})]$ to be similar for each $i_{th}$ LR.
To achieve this, we propose a novel method that firstly finds the center $\bar{z}_{0}$ of the vectors of $\mathbb{E}[z_{0}|z_{t}(\text{LR}_{i},dt_{i})]$ for all i, and then updates each $z_{t}(\text{LR}_{i},dt_{i})$, so that $\mathbb{E}[z_{0}|z_{t}(\text{LR}_{i},dt_{i})]$ to be closer to the center than without updating.

We can find the desired center via optimization, where $d$ can be a specified distance function, and $N$ is the total number of low-resolution images. Note that $N$ can be different for different HR, which further demonstrates the flexibility of our method. For the distance function, we propose a novel approach that uses a convex combination of $l_{2}$ loss and LPIPS loss in order to prevent blurry outputs but still keep images close to each other.

$$ $\bar{z}_{0}=\mathop{\arg\min}\limits_{x}\sum_{i=1}^{N}d(x,\mathbb{E}[z_{0}|z_{ t}(\text{LR}_{i},dt_{i})])$ (4) $$

$$ $d(x,x_{i})=(1-\alpha)\ell_{2}(x,x_{i})+\alpha\text{LPIPS}(x,x_{i})$ (5) $$

where $\alpha$ is the weight for LPIPS loss.
Note that for computational efficiency, when computing $\bar{z}_{0}$, we by design choose not to sum up every $\text{LR}_{i}$, but randomly sample a batch from the set of $\text{LR}_{i}$ to compute the summation.
We observe in experiments this random batch selection strategy improves computational efficiency while not sacrificing performance,which may because of the redundancy information among LR images.
Similarly for the computational efficiency, such optimization update is not performed on every time step, but on every k steps.
In all of our experiments, we set $k=5$, which we find from empirical study would suffice the fusion strength while reducing the computational cost.

After obtaining $\bar{z}_{0}$, we propose to update all intermediate samples from the $\text{LR}_{i}$ to be closer to $\bar{z}_{0}$. Recall that in DDIM reverse sampling [DDIM], the reverse sampling can be decomposed by a clean image component and a noise component. We follow [DDS]’s approach that only updates the clean image component, and leaves the noise component intact. Specifically, let $\lambda$ be a hyperparameter balancing the original clean image component and $\bar{z}_{0}$, we can update the new clean component as:

$$ $\hat{z}_{0}(\text{LR}_{i},dt_{i})=(1-\lambda)\mathbb{E}[z_{0}|z_{t}(\text{LR}_ {i},dt_{i})]+\lambda\bar{z}_{0}$ (6) $$

Therefore, the overall reverse sampling step can be written as

$$ $\bm{z}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{z}_{0}(\text{LR}_{i},dt_{i})+\sqrt{ 1-\bar{\alpha}_{t-1}-\eta\delta_{t}^{2}}\bm{s}_{\theta}(\bm{z}_{t},\text{LR}_{ i},dt_{i},t)+\eta\delta_{t}\bm{\epsilon},\quad t=T,\ldots,0,$ (7) $$

The pseudo-code of our proposed algorithm is demonstrated in Alg. 1.

## 4 Experiments

**Table 1: Comparison of the LPIPS metrics for super-resolution on the fMoW dataset for different categories. Best results are in bold.**
|  | Airport | Amusement Park | Car Dealership | Crop Field | Educational Institution | Electric Substation |
| --- | --- | --- | --- | --- | --- | --- |
| WorldStrat [WorldStrat] | 0.723 | 0.732 | 0.747 | 0.738 | 0.733 | 0.736 |
| MSRResNet [MSRResNet] | 0.743 | 0.739 | 0.733 | 0.794 | 0.725 | 0.783 |
| DBPN [DBPN] | 0.763 | 0.740 | 0.728 | 0.783 | 0.726 | 0.750 |
| Pix2Pix [Pix2pix] | 0.621 | 0.652 | 0.652 | 0.645 | 0.647 | 0.643 |
| ControlNet [ControlNet] | 0.625 | 0.653 | 0.648 | 0.650 | 0.658 | 0.644 |
| DiffusionSat [DiffusionSat] | 0.623 | 0.647 | 0.637 | 0.649 | 0.652 | 0.639 |
| SatDiffMoE (Ours) | 0.579 | 0.626 | 0.600 | 0.608 | 0.612 | 0.606 |

We try to answer the following questions in this section:
(1) Can our proposed method achieve high-quality satellite image super-resolution results by fusing multiple time series low-resolution images?
(2) Is our proposed fusion module effective?
(3) Can our proposed method be more computationally efficient compared with previous methods?
To study these questions, we benchmark the super-resolution performance on two widely used satellite image datasets: the fMoW dataset and the WorldStrat dataset.

### 4.1 Datasets

#### WorldStrat

We take the paired LR-HR satellite image dataset from [WorldStrat]. Each area of interest contains a single SPOT 6/7 high-resolution image with five bands. We take the RGB band of the SPOT6/7 satellite images, which has a GSD of 1.5 m/pixel. The low-resolution images are taken from the Sentinel-2 satellites consisting of 13 bands. We only pick the RGB band from them. For each area of interest, we have 16 paired low-resolution images taken at different time. The resolution ranges from 10 m/pixel to 60m per pixel. We crop the high-resolution image into 192x192 patches, and the low-resolution image into 63x63 patches.

#### fMoW

Function Map of the World (fMoW) [fmow] consists of high-resolution (GSD 0.3m-1.5m) satellite images of a variety of categories such as airports, amusement parks, crop fields and so on. However, its temporal resolution is limited due to its high resolution. We use the metadata of timestamp for pairing low-resolution Sentinel-2 images.
Using the dataset provided in Cong et al. (2022), we create a fMoWSentinel-fMoW-RGB dataset with paired Sentinel-2 (10m-60m GSD) and fMoW (0.3-1.5m GSD) images at each of the original fMoW-RGB locations. Then, we use the bounding box provided by the metadata to extract relevant areas, and then crop the high-resolution images to patches of 512x512. For each high-resolution fMoW image, we find the corresponding Sentinel-2 images of the same location. We only take the RGB band of Sentinel-2 images and then apply the same cropping method as that for fMoW-RGB images.

### 4.2 Performance Benchmark

For super-resolution tasks on fMoW and WorldStrat datasets, we report perceptual quality metrics LPIPS to measure the perceptual similarity of the reconstructed image and the ground truth image, and FID to measure how realistic the reconstruction looks. We also report distortion metrics such as PSNR and SSIM for pixel-level similarity. Note that in satellite images, LPIPS is a more relevant metrics here as it measures the perceptual similarity, as mentioned in [DiffusionSat].

#### Implementation Details

**Table 2: Comparison of LPIPS and FID metrics for super-resolution on WorldStrat dataset and fMoW. Best results are in bold. Second best results are underlined.**
| Method | WorldStrat | fMoW |  |  |
| --- | --- | --- | --- | --- |
| LPIPS$\downarrow$ | FID$\downarrow$ | LPIPS$\downarrow$ | FID$\downarrow$ |  |
| WorldStrat [WorldStrat] | 0.481 | 139.3 | 0.736 | 426.7 |
| MSRResNet [MSRResNet] | 0.472 | 159.7 | 0.783 | 286.5 |
| DBPN [DBPN] | 0.475 | 122.6 | 0.750 | 278.2 |
| Pix2Pix [Pix2pix] | 0.427 | 93.90 | 0.643 | 196.3 |
| ControlNet [ControlNet] | 0.580 | 108.0 | 0.644 | 102.3 |
| DiffusionSat [DiffusionSat] | 0.561 | 92.97 | 0.638 | 102.9 |
| SatDiffMoE (Ours) | 0.418 | 88.12 | 0.606 | 115.6 |

We evaluate our algorithms on WorldStrat and fMoW LR-HR datasets. For each AOI (Area of Interest) of the WorldStrat dataset, we resize each LR image to $192\times 192$. We fix the prompt to be "Satellite Images" and compute the time difference between the low-resolution image and high-resolution image. Then, we add the time difference embedding network to the stable diffusion 1.2 and then fine-tune the stable diffusion 1.2 on the low-resolution and high-resolution pair of the WorldStrat dataset conditioning on the time difference. For each high-resolution image, we randomly select a low-resolution image from its 16 corresponding low-resolution images for training. After that, we get a latent diffusion model that takes a low-resolution image and the relative time difference as input and output the predicted high-resolution image.

For each AOI of the fMoW dataset, we resize each LR image to $512\times 512$ to align with the size of high-resolution image. We let the prompt to be "Satellite image of selected class from a direct overhead view", where selected class is the category name. We also use the same time difference embedding network and then fine tune the stable diffusion 1.2.

During inference, for both datsets we use 50 NFEs, and then perform optimization every 5 steps. More implementation details can be found in the supplementary materials.

For WorldStrat, we evaluate our algorithm on first 1000 images in the validation set. For fMoW, we select the first 100 images in the validation set from 6 classes: Airport, Amusement Pak, Car Dealership, Crop Field, Educational Institution, Electric Substation for evaluation.

#### Baselines

We consider six state-of-the-art baselines, WorldStrat [WorldStrat], MSRResNet [MSRResNet], DBPN [DBPN], Pix2Pix [Pix2pix], ControlNet [ControlNet], and DiffusionSat [DiffusionSat]. Both ControlNet and DiffusionSat are diffusion-based methods that takes LR as an additional condition for fine-tuned stable diffusion to predict the HR image. Both MSRResNet and DBPN are CNN-based methods that directly map the low-resolution image to the high-resolution image, and Pix2Pix is a GAN-based method. Both WorldStrat and DiffusionSat are fusion-based methods that fuse multiple low-resolution input to predict the HR image, while others take single low-resolution input and predict the HR image.

#### Results and Discussions

Figure: Figure 3: (Left) Super-resolution on fMoW dataset. (Right) Super-resolution on WorldStrat dataset.
Refer to caption: extracted/5666614/figures/demon_fmow.jpg

The LPIPS scores on six selected class in fMoW dataset are reported in Table. 1. We also report LPIPS and FID scores in Table 2 on both fMoW and WorldStrat datasets compared to the six baselines mentioned before. We observe that our algorithm largely achieves better or comparable performance in perceptual quality. Our method achieves state-of-the-art LPIPs score compared to all baselines and comparable FID scores. CNN-based baselines tend to perform poorly in perceptual quality, resulting in sub-par FID scores compared to diffusion-based methods. While ControlNet may show slightly better FID scores, but the LPIPs score is significantly worse than ours. We also show reasonable PSNR and SSIM scores, as demonstrated in the Appendix. Qualitatively, as demonstrated in Fig. 3, we also demonstrate that our method is able to capture fine-grained details. Compared to baselines, we are able to reconstruct both realistic images and accurate details.

**Table 3: Number of parameters and memory required.**
| Method | WorldStrat | fMoW |  |  |
| --- | --- | --- | --- | --- |
| Parameters (M) | Iterations for Training | Parameters (M) | Iterations for Training |  |
| ControlNet [ControlNet] | 1427 | 12500 | 1427 | 21400 |
| DiffusionSat [DiffusionSat] | 1428 | 12500 | 1428 | 20000 |
| Ours | 1068 | 800 | 1068 | 4400 |

**Table 4: Evaluating the Effectiveness of Relative Rime Difference and Fusion.**
| RelativeTimeDiff | Fusion | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$ |
| --- | --- | --- | --- | --- | --- |
|  |  | 12.52 | 0.122 | 0.565 | 102.3 |
| ✓ |  | 15.28 | 0.272 | 0.496 | 105.1 |
| ✓ | ✓ | 17.40 | 0.396 | 0.418 | 88.12 |

Computational Efficiency
We observe that the training phase of our proposed method requires much fewer parameters and iterations to converge than ControlNet and DiffusionSat, while achieving comparable or better reconstruction performance. We report the number of parameters and number of iterations of training in Table. 3. We train each model until the FID stops improving. Our method converges significantly (5-15 times) faster than ControlNet and DiffusionSat on both datasets. We also do not observe the "sudden convergence phenomena" in our training which is reported in ControlNet, which implies that our training may be more stable.
Our method also exhibits better computational efficiency compared to ControlNet. As demonstrated in Table. 3, our algorithm requires significantly less training time than ControlNet.

Figure: Figure 4: Super-resolution results on fMoW dataset by SatDiffMoE with and without fusion. We show SatDiffMoE without fusion by conditioning on the first LR image and SatDiffMoE with fusion by fusing the four LR images.
Refer to caption: extracted/5666614/figures/demon_fusion_fmow.jpg

**Table 5: Ablation study: number of images for fusion**
| Number of images for fusion | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LPIPS | 0.41124 | 0.3967 | 0.3920 | 0.3891 | 0.3877 | 0.3874 | 0.3867 | 0.3866 |
| Inference Time (s) | 16.00 | 17.11 | 18.82 | 20.64 | 22.45 | 24.25 | 25.99 | 27.81 |

### 4.3 Ablation Studies

We want to study the impact of the fusion module and the relative time difference embedding on the reconstruction quality. We are also interested in the number of images for fusion. Intuitively, with more LR images provided for fusion, we have more complementary details and can achieve better perceptual quality.

#### Effectiveness of relative time embedding and fusion

We demonstrate the performance of our method without RelativeTimeDiff and Fusion, our method with RelativeTimeDiff, and our full method on the WorldStrat validation dataset in Table. 4.
We found that both modules have significant positive impact to the perceptual quality (LPIPS score), while the fusion module improves both the LPIPS score and the FID score. On the other hand, we also find improvement in the distortion metrics when adding both modules. Qualitatively, as shown in Fig. 4, with our fusion algorithm, we can reconstruct super-resolved images with better quality.

#### Impact of number of images on fusion

We present the effect of increasing number of image for fusion on the reconstruction quality on the first 100 images in the validation set of WorldStrat dataset in Table. 5. We observe that the reconstruction quality improves significantly when adding more images for fusion when there are few images for fusion (i.e. fewer than 4), but the performance almost saturates when the number of images exceeds 10, and will trade-off between image quality and inference time. This observation validates our hypothesis that fusing information from multiple LR images improves the reconstruction quality and there exists a diminishing return on the number of LRs.

## 5 Conclusion

In this work, we present "SatDiffMoE", a novel framework for satellite image super-resolution with latent diffusion models. We first present a novel training mechanism that conditions on relative time difference of LR images and HR images. Then, we propose a novel inference-time algorithm that fuses the reverse sampling trajectory from inputs of different LRs at the same location but different times. Our method is highly flexible that can adapt to an arbitrary number of low-resolution inputs at test-time and requires fewer parameters than diffusion-based counterparts. One of our limitation is we do not impose physical measurement constraint in our reconstruction process, which we will leave as future work.

## Acknowledgments and Disclosure of Funding

The authors acknowledge support from Michigan Institute for Computational Discvoery and Engineering (MICDE) Catalyst Grant, and Michigan Institute for Data Science (MIDAS) PODS Grant.

## Appendix A Appendix / supplemental material

### A.1 Implementation Details

#### Data Preprocessing

WorldStrat
We obtain the RGB satellite images provided by [WorldStrat]. We follow the preprocessing steps given by [WorldStrat], that crop the HR images into patches of $192\times 192$ and LR images into corresponding pathes of the HR images of size $63\times 63$. For each HR image, there are 16 corresponding LR images of different time. We extract the RGB band of the HR images and the RGB band of every LR images. We then resize LR image into the size of $192\times 192$. We use the same training and validation split provided by [WorldStrat] and then form a training and validation set. We then extract the timestamp from the metadata and compute the $dt_{i}$ for each $\text{LR}_{i}$.

fMoW
We obtain the high-resolution images from [fmow], and the paired Sentinel-2 images from [satmae]. We first identify the area of interest on the HR images as given by [fmow], and then crop out other areas. Then we crop the corresponding LR images following the pre-processing steps given by [satmae]. We crop the HR images into patches of $512\times 512$, and align LR images into patches based on each HR image patch. Then we resize each LR image into the size of $512\times 512$ in accordance with the HR image. We consider 6 categories: airport, amusement parks, car dealership, crop field, educational institution, electric substation for training and testing. For training, we filter out HR images that do not have a corresponding LR, and those do not have three channels. We take the same training and validation split from [fmow]. When training, we consider all images from [fmow], and when testing, we pick the first 100 images from each selected category of the validation set of [fmow]. We also extract $dt_{i}$ from the metadata of fmow provided by [fmow], and the metadata of Sentinel-2 data provided by [satmae].

#### Model Training

We take the pretrained checkpoint (SD1.2) provided by [rombach2022highresolution], and then fine tune on the processed [fmow] and [WorldStrat] datasets. We rescale every LR image and HR image to the scale of [0,1]. Then, we use a learning rate $1e^{-5}$, and a batch size of 4 for both datasets. We stop training when the FID of sampled images stops improving. For [WorldStrat] dataset, we only train 800 iterations (partly due to the small dataset size). For [fmow] dataset, we train 4400 iterations. We then take the model for downstream inference tasks. For WorldStrat dataset, we use the prompt "Satellite images" for training. For fMoW dataset, we use the prompt "Satellite image of selected class from a direct overhead view", where selected class is the category name.

#### Model Inferencing

We use 50 DDIM steps with $\eta=0$ for inferencing. We perform optimization every 5 steps for computational efficiency, otherwise, we just perform conditional sampling. We set $\lambda=0.1$ and $\alpha=0.2$ for both datasets.

### A.2 More Ablation Studies

#### Effect of LPIPS weight α 𝛼 \alpha italic_α and optimization weight λ 𝜆 \lambda italic_λ

There are two hyper-parameters in our inference-time algorithm, and that is the weight for LPIPS distance $\alpha$ v.s. L2 distance, and the weight $\lambda$ for balancing the original predicted clean image component and the one after optimization.
We expect the perceptual quality of reconstructed images to improve when we increase $\alpha$ from 0. We also expect the reconstruction quality to improve when $\lambda$ increases from 0 since the weight of fusion increases. In Fig.5, we present the LPIPS score on 100 samples on the WorldStrat dataset with varying $\alpha$ and $\lambda$. We find that LPIPS score improves when both $\alpha$ and $\lambda$ increase from 0. Then, performance converges as $\alpha$ keeps increasing, and marginally degrades as $\lambda$ continues to increase. We observe that generally, the performance of our algorithm is insensitive to hyperparameter change.

Figure: Figure 5: (Left) Ablation study on LPIPS weight $\alpha$. (Right) Ablation study on optimization weight $\lambda$.
Refer to caption: extracted/5666614/figures/ablation_lambda.png

### A.3 Implementation Details of Baselines

#### WorldStrat

We follow the original codebase of [WorldStrat], where we train the HighResNet model on both [fmow] and [WorldStrat] datasets. We tune the hyperparameters of the loss function based on our validation set. We stop training when the validation performance converges. On [WorldStrat] we train 125000 iterations with a batch size of 32, and on [fmow] we take 100000 iterations with a batch size of 32.

#### MSRResNet

We follow the original codebase of [MSRResNet], where we train the MSRResNet model on both [fmow] and [WorldStrat] datasets. During training, we randomly pick a LR image and its paired HR image. We tune the hyperparameters of the loss function based on validation set performance. We train for 160000 iterations for both [WorldStrat] and [fmow] datasets with a batch size of 16.

#### DBPN

We follow the original codebase of [DBPN], where we train the MSRResNet model on both [fmow] and [WorldStrat] datasets. During training, we randomly pick a LR image and its paired HR image. We tune the hyperparameters of the loss function based on validation set performance. We train for 100 epochs for both [WorldStrat] and [fmow] datasets with a batch size of 16.

#### Pix2Pix

We follow the original codebase of [Pix2pix], where we train the MSRResNet model on both [fmow] and [WorldStrat] datasets. During training, we randomly pick a LR image and its paired HR image. We tune the hyperparameters of the loss function based on validation set performance. We train for 30 epochs for [fmow] and 100 epochs for [WorldStrat] with a batch size of 16.

#### ControlNet

We follow the original codebase of [ControlNet]. During training, we randomly pick a LR image and its paired HR image. We tune the hyperparameters of the loss function based on validation set performance. We train for 12500 iterations with a batch size of 16 for [WorldStrat], and 21500 iterations with a batch size of 16 for [fmow].

#### DiffusionSat

We implemented the 3D ControlNet architecture as mentioned in [DiffusionSat]. Then, we take the RGB band and the SWIR, NIR band from LR image for training 3D ControlNet. We tune the hyperparameters of the loss function based on validation set performance. We train for 12500 iterations with a batch size of 16 for [WorldStrat], and 20000 iterations with a batch size of 16 for [fmow]. We observe that further training worsened FID scores on both datasets.

### A.4 More Results

We report additional results on unconditional generation and conditioning on $dt_{i}$ as demonstrated in Fig.6, and Fig.7. We observe that we can generate realistic satellite images. Conditioning on $dt_{i}$ makes semantic changes in the image and can be applied to tasks such as cloud removal. We also report the PSNR and SSIM metrics in Table 6. The error bars are presented in Table 7.

**Table 6: Comparison of PSNR and SSIM metrics for super-resolution on WorldStrat dataset and fMoW. Best results are in bold. Second best results are underlined.**
| Method | WorldStrat | fMoW |  |  |
| --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ |  |
| WorldStrat [WorldStrat] | 17.98 | 0.396 | 13.42 | 0.443 |
| MSRResNet [MSRResNet] | 19.81 | 0.512 | 13.01 | 0.290 |
| DBPN [DBPN] | 19.17 | 0.471 | 11.90 | 0.268 |
| Pix2Pix [Pix2pix] | 19.76 | 0.448 | 12.21 | 0.180 |
| ControlNet [ControlNet] | 11.89 | 0.113 | 10.82 | 0.117 |
| DiffusionSat [DiffusionSat] | 12.34 | 0.133 | 10.63 | 0.109 |
| SatDiffMoE (Ours) | 17.40 | 0.396 | 11.96 | 0.172 |

Figure: Figure 6: Generated high resolution samples from our unconditional model.
Refer to caption: extracted/5666614/figures/generation.png

Figure: Figure 7: Here we consider the same LR image, and vary the relative time difference to generate different conditional samples.
Refer to caption: extracted/5666614/figures/vary_dt.png

**Table 7: Standard deviation of the quantitative metric LPIPS presented in Table 2 for super-resolution on fMoW and WorldStrat dataset.**
| Method | WorldStrat | MSRResNet | DBPN | Pix2Pix | ControlNet | DiffusionSat | SatDiffMoE(Ours) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| WorldStrat | 0.081 | 0.077 | 0.079 | 0.069 | 0.079 | 0.091 | 0.076 |
| fMoW | 0.092 | 0.081 | 0.052 | 0.045 | 0.034 | 0.034 | 0.044 |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Why the proposed model was designed for satellite image , why not for real image datasets like Urban100
2. Complexity of the season still huge, if the comparison will be made with weight methods

2. Address the weakness motioned in section.
soundness: 2
strengths: 1. This method is highly flexible, capable of handling sequences with varying numbers of low-resolution images for fusion
2. Its adaptability stems from a specific fusion strategy for sampling trajectories, enhanced by the novel use of timestamp differences as an embedding prompt to guide the fusion process effectively.
3. This strategy harmonizes intermediate outputs across varying low-resolution inputs, enabling a flexible and efficient fusion process independent of the image count.
summary: The paper llooks sound based on the idea "Super-resolution with Latent Diffusion Models for Satellite Image" This method introduces SatDiffMoE, a novel diffusion-based algorithm that fuses sequential low-resolution images into high-resolution outputs using complementary time-point data. The method demonstrates flexibility, superior performance, and computational efficiency on super-resolution tasks across various datasets.
weaknesses: 1. Limited comparison with SOTA methods
2 Literature review is not strong, problem statement
3. The use of diffusion models introduces a dependency on the quality of the generative priors
4. In real-world conditions (e.g., different climates or satellite systems), the generalizability of the results might be a concern.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: (1) How does the operator $F$ function in Eq. (2), and why is it necessary to use $F$ to mask out the HR component from the score? Under certain conditions, the score function $s_\theta(\cdot, t)$ can be interpreted as modeling the conditional probability $p(HR_t | HR_{t-1}, LR)$. Thus, once the available score function is obtained from the training dataset, it can gradually sample the HR from Gaussian noise at inference stage.
(2) Referring to the background on multi-image super-resolution, the rank 1 method [1] on the PROBA-V dataset (the mentioned HighRes-net ranked 6th) can take an arbitrary number of low-resolution inputs to generate a high-resolution satellite image. However, the authors did not address this point, which is closely related to the contributions of this paper, specifically regarding the concept of "arbitrary number."
(3) Referring to Line 133, "However, few works apply LDMs for image fusion yet", actually, there are lots of DM-based image fusion mothed have been developed last 3 years. 
(4) As a method to obtain accurate HR images rather than synthetic HR images, I am more focused on image content restoration quality, measured by PSNR and SSIM, rather than the quality of image generation and perceptual quality. If the goal is to generate high-quality satellite data, why not directly train a SAT Stable Diffusion model instead of using a fusion model? Compared to the regression-based model MSRResNet (Table 7), the restoration quality of the proposed method is not particularly outstanding. Additionally, some of the comparative methods are not specifically designed for multi-image super-resolution; rather, they are either single-image super-resolution methods or intended for synthetic data.

[1]An, Tai, et al. "TR-MISR: Multiimage super-resolution based on feature fusion with transformers." _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ 15 (2022): 1373-1388.
soundness: 2
strengths: (1) It is novel that using the timestamp differences as embedding prompts.
(2) The sampling strategy effectively combines the intermediate sampling results from different low-resolution conditional images, enabling the fusion of an arbitrary number of low-resolution images.
(3) SatDiffMoE can synthesize high-quality images.
summary: The authors propose a novel diffusion-based fusion algorithm called SatDiffMoE, which accepts an arbitrary number of sequential low-resolution satellite images to generate a high-resolution reconstructed image. During sampling, SatDiffMoE first obtains the center of the intermediate sampling results from different low-resolution conditional images and then weights this center along with the intermediate results to produce the next sampling. Experimental results demonstrate that SatDiffMoE can synthesize high-quality images.
weaknesses: (1) The mechanism of DiffusionDrop is unclear and needs further clarification.
(2) The content restoration of high-resolution images generated by SatDiffMoE lacks competitiveness.
(3) The literature review is insufficient.
(4) The comparison methods are either single-image super-resolution approaches or designed for synthetic data, lacking comparisons with multi-image super-resolution methods, especially those tailored for remote sensing imagery.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. What's the $F$ in Eq. (2), I did not find its implementaion in the rest of paper. Only description like ''mask out the HR component'' is too vague.

2. Please provide 1) the details of used datasets, including train/validation/test ratios, total number of images etc; 2) the computation resources to finetuning the SD1.2 model.
soundness: 2
strengths: 1. Propose DiffusionDrop and use previous $z_0$ optimization to fuse LRs to HR.

2. Good results on different satellite image dataset.
summary: This paper introduces a super-resolution latent diffusion model based on finetuning SD1.2 model, to fuse and SR and remote sensing images. To adapt fusing different LRs at different times, the authors proposed DiffusionDrop and optimized-based diffusion to fuse diffusion trajectories. Results on some remote sensing image SR datasets show its effectiveness.
weaknesses: Some typos:

1. At line 169 in Algo. 1, the comment ''Compute the score'' actually should be ''noise'', since you're using SD1.2 model, which is trained on noise-prediction objective.
2. At line 103, there should exist an semicolon.
3. At line 166, there should exist the encoder to encode the LR images.

Weakness:

1. Some key components like $F$ in Eq. (2), finding the center ($\bar z$) is too vague, hindering reader's understanding.

2. In inference, the authors claim that finding the center ($\bar z_0$) of $z_i$-s are based on Chung et al (2024), but actually, in this refered paper, when optimzing the $z_t$ (or says diffusion trajectory), CG steps should be taken to optimize with the help of the degraded operator ($\mathbf A$). It seems that you did not have the operator $\mathbf A$ and did not include the description of how to take the $\arg \min$.

3. According to your description in inference stage, finding the center of predicted $\hat z_0$ should include all LRs which is $LR_i, i\in \{0,\dots, N-1\}$ (if you have $N$ LRs when inference). This means in Algo. 1, there should be **two** for-loops, one for diffusion timesteps and one for $N$ LR samples. Did I misunderstand?

4. In the experiments, the authors compared with ControlNet, but only training it with only one LR and HR pair. This comparison is unfair, since the proposed method uses multiple LRs.

5. As shown in your Eq. (2), inputs should be encoded LRs and HR, but in Fig. 2, there is only LR and do not have $F$.

---

This paper propose some techniques to finetune the pretrained SD model to adapt on satellite SR task with various number of LR images, but the presentation is too poor to understand its contributions. 

I believe the proposed method's SR performance can beat previous methods, such as previous regressive models and diffusion models, but the presentation and writing is still an important part. The writting makes this paper can not be accepted by ICLR.

Now, I rate this paper to 5 (marginally below the acceptance threshold).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: + This method is very flexible and can accept sequences with different numbers of low-resolution images for fusion. This advantage is attributed to its specific fusion strategy for sampling trajectories.
+ The definition and conditional input of the relative time difference is also interesting, which can help the network capture the time-aware mapping distribution of low-resolution to high-resolution images.
+ Experimental results show that the reconstruction results are relatively better than competitors in terms of rationality and naturalness.
summary: This paper proposes a new fusion super-resolution method based on the diffusion model. Its feature is that it allows any number of low-resolution images to be used as input to reconstruct a high-resolution image. To enhance the mapping difference between the low-resolution images at different timestamps and the corresponding high-resolution images, it constructs the relative time difference and uses it as an additional condition input. Another contribution is that during the inference phase, the trajectories corresponding to each low-resolution image can be randomly selected for fusion and finally output to a single high-resolution image. Experiments show the advantages of the proposed method.
weaknesses: - A key operation is F, which masks out the HR component from the score function. However, this paper does not explain how F was designed. This makes the description of the relevant methodology unclear.
- There is also some disagreement about the definition of the relative time difference. In Figure 1, time t4 corresponds to the high-resolution image, and dti represents the difference between the low-resolution image at time ti and the high-resolution image. However, the definition of dt4 is obviously different from the others, which is not rigorous in expression.
- I am concerned about the significance of this technology. On the one hand, images captured at different times may show changes in surface cover, so is it appropriate to use a high-resolution image at one of these moments as a reference? On the other hand, the experiments also confirmed my concern that the reconstructed results of the proposed method, although better than those of other methods, still show considerable differences with GT. This difference is not only reflected in the intensity distribution, but also includes changes in some ground objects. This makes me concerned about whether the reconstructed high-resolution results are credible and applicable.
- Sequence fusion with multiple low-resolution images seems to only appear in ablation studies. Is the comparative experiment just a single image super-resolution?
- Could this method be extended to consider the possibility of parallax between low-resolution images? In reality, images captured at different times theoretically have geometric differences
- There are some typos. For example, the last line of page 4 is missing a period.