Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis
Authors: Yutong He
Dingjie Wang
Nicholas Lai
William Zhang
Chenlin Meng
Marshall Burke
David B. Lobell
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: Remote Sensing
Super-Resolution
Generative Models

Abstract: High-resolution satellite imagery has proven useful for a broad range of tasks, including measurement of global human population, local economic livelihoods, and biodiversity, among many others. Unfortunately, high-resolution imagery is both infrequently collected and expensive to purchase, making it hard to efficiently and effectively scale these downstream tasks over both time and space. We propose a new conditional pixel synthesis model that uses abundant, low-cost, low-resolution imagery to generate accurate high-resolution imagery at locations and times in which it is unavailable. We show that our model attains photo-realistic sample quality and outperforms competing baselines on a key downstream task – object counting – particularly in geographic locations where conditions on the ground are changing rapidly.

Spatial-Temporaa Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis
Yutong He
Dingjie Wang Marshall Burke
Nicholas Lai David B. Lobell Stanford University
William Zhang Stefano Ermon
Chenlin Meng
(kellyyhe, daviddw, nicklai, wxyz, chenlin, ermon)&cs. stanford.edi (mburke, dlobell)@stanford.
Abstract
High-resolution satellite imagery has proven useful for broad range tasks, in- cluding measurement of global human population, local economic livelihoods, and biodiversity, among many others. Unfortunately, high-resolution imagery is both infrequently collected and expensive to purchase, making it hard to efficiently and effectively scale these downstream tasks over both time and space. We propose a conditional pixel synthesis model that uses abundant, low-cost, low-resolution imagery to generate accurate high-resolution imagery at locations and times which it is unavailable. We show that model attains photo-realistic sample quality and outperforms competing baselines on key downstream task object counting particularly in geographic locations where conditions on the ground are changing rapidly.
Introduction
Recent advancements in satellite technology have enabled granular insight into the evolution of human activity the planet's surface. Multiple satellite sensors now collect imagery with spatial resolution less than 1m, and this high-resolution (HR) imagery can provide sufficient information for various fine-grained tasks such as post-disaster building damage estimation, poverty prediction, and crop phenotyping [15 B 4T. Unfortunately, HR imagery is captured infrequently over much of the planet's 's surface (once year or less), especially in developing countries where it is arguably most needed, and was historically captured even more rarely (once or twice a decade) 77]. Even when available, HR imagery is prohibitively expensive to purchase in large quantities. These limitations often result in an inability to scale promising HR algorithms and apply them to questions of broad social importance. Meanwhile, multiple sources of mublicly-availabl satellite imagery now provide sub-weekly coverage at global scale, albeit at lower spatial resolution (e.g. 10m resolution for Sentinel-2). Unfortunately, such buildings, swimming pools, and cars unrecognizable.
spatial resolution renders small objects like residential
LR 2016
HR 2018
Generated 2016
Ground Truth 2016
+
Figure 1: Given 10m low resolution (LR) image from 2016 and Im high resolution (HR) image from 2018, we generate photo-realistic and accurate HR image for 2016.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In the last few years, thanks to advances in deep learning and generative models, we have seen great progress in image processing tasks such as image colorization [3), denoising 6 35 inpainting
35 27), and super-resolution [1 21 16] Furthermore, pixel synthesis models such as neural radiance field (NeRF) [26] have demonstrated great potential for generating realistic and accurate scenes from different viewpoints. Motivated by these successes and the need for high-resolution images, we ask whether it is possible to synthesize high-resolution satellite images using deep generative models. For given time and location, can we generate a high-resolution image by interpolating the available low-resolution and high-resolution images collected over time?
To address this question, we propose conditional pixel synthesis model that leverages the fine- grained spatial information in HR images and the abundant temporal availability of LR images to create the desired synthetic HR images of the target location and time. Inspired by the recent
development of pixel synthesis models pioneered by the NeRF model [26 40 2], each pixel in
the output images is generated conditionally independently by perceptron- bassd generator given the encoded input image features associated with the pixel, the positional embedding of its spatial- temporal coordinates, and random vector. Instead of learning to adapt to different viewing directions in a single 3D scene [26), our model learns to interpolate across the time dimension for different geo-locations with the two multi-resolution satellite image time series.
To demonstrate the effectiveness of our model, we collect large-scale paired satellite image dataset of residential neighborhoods in Texas using nigh-resolution NAIP (National Agriculture Imagery Program, Im GSD) and low-resolution Sentinel-2 (10m GSD) imagery. This dataset consists of scenes in which housing construction occurred between 2014 and 2017 in major metropolitan areas of Texas, with construction verified using CoreLogic tax and deed data. These scenes thus provide a rapidly changing environment on which to assess model performance. As separate test, we also pair HR images (0.3m to Im GSD) from the Functional Map of the World (IMoW) dataset [9] crop field category with images from Sentinel-2.
To evaluate our model's performance, compare to state-of-the-art methods, including super- resolution models. Our model outperforms all competing models in sample quality on both datasets measured by both standard image quality assessment metrics and human perception (see example in Figure I]. Our model also achieves 0.92 and 0.62 Pearson's 's r² in reconstructing the correct numbers of buildings and swimming pools respectively in the images, outperforming other models these tasks. Results suggest our model's potential to scale to downstream tasks that use these object counts as input, including pocietally-importan tasks such as population measurement, poverty prediction, and humanitarian assessment 3].
2 Related Work
Image Super-resolution SRCNN 11 is the first paper to introduce convolutional layers into a SR context and demonstrate significant improvement over traditional SR models. SRGAN [21] improves on SRCNN with adversarial loss and is widely compared among many GAN-based SR models for remote sensing imagery [36 29], DBPN [6] is state-of-the-art SR solution that uses an iterative algorithm provide an error feedback system, and it is one of the most effective SR models for satellite imagery (28). However, [31] shows that SR is less beneficial at coarser resolution, especially when applied to downstream object detection on satellite imagery. In addition, most SR models test on henchmarks where LR images are artificially created, instead of collected from actual LR devices 17.. SR models also generally perform worse at larger scale factors, which is closer to settings for satellite imagery SR in real life.
39
super-resolution through neural texture transfer to mitigate information loss in LR images by leveraging texture details from HR reference images. While SRNTT also uses HR reference image, it does not learn the additional time dimension to leverage the HR image of the same object at a different time. In addition, our model uses a perceptron based generator while SRNTT uses CNN based generator.
SRNTT 445 applies reference-
Fusion Models for Satellite Imagery 12 first proposes STARFM to blend data from two remote sensing devices, MODIS 4 and Landsat 20 for spatial-temporal super resolution to predict land reflectance. [46] introduces an enhanced algorithm for the same task and [00 combines linear pixel unmixing and STARFM to improve spatial details in the generated images. cGAN Fusion

---

5 incorporates GAN-based models in the solution, using an architecture similar to Pix2Pix [18. In contrast to previous work, we are particularly interested in synthesizing images with very high resolution (s 1m GSD), enabling downstream applications such as poverty level estimation.
NeRF and Pixel Synthesis Models Recent developments in deep generative models, especially advances in perceptron basod generators, have yet to be explored in remote sensing applications.
Introduced by [26]. neural radiance fields (NeRF) demonstrates great success in constructing 3D
static scenes. 23 38 extends the notion of NeRF and incorporates time-variant representations
of the 3D scenes. 30 embeds NeRF generation into 3D aware image generator. These works,
however, are limited to generating individual scenes, in contrast with our model which can generalize different locations in the dataset. (40) proposes framework that predicts NeRF conditioning on spatial features from input images; however, it requires constructing the 3D scenes, which is less applicable to satellite imagery. 2 proposes a style-based 2D image generative model using an only perceptron based architecture; however, unlike our method, it doesn't It consider the task of conditional 2D image generation nor does it incorporate other variables such as time. In contrast, we propose a pixel synthesis model that learns a conditional 2D spatial coordinate grid along with continuous time dimension, which is tailored for remote sensing, where the same location can be captured by different devices (e.g. NAIP or Sentinel-2) at different times (e.g. year 2016 or year 2018).
Problem Setup
The goal of this work is to develop a method to synthesize high-resolution satellite images for locations and times for which these images are not available. As input are given two time-series of high-resolution (HR) and low-resolution (LR) images for the same location. Intuitively, we wish to leverage the rich information in HR images and the high temporal frequency of LR images to achieve the best of both worlds.
let 1/ E CXH be sequence of random variables representing high-resolution views of a location at various time steps t E T. Similarly, let T(()) E CxC.H, Wir denote low-resolution views of the same location over time. Our goal is develop a method estimate (()) given K ( for the same location. Note the available observations could be taken either before or after the target time t. Our task can be viewed as a special case of multivariate time-series imputation, where two concurrent but incomplete series of satellite images of the same location in different resolutions are given, and the model should predict the most likely pixel values an unseen time step of one image series.
Formally,
high resolution observations ( ..... T ((fk) and J. low-resolution ones {T((
In this paper, we consider a special case where the goal is to estimate T(!)) hrr given a single high-
resolution image 1!(') and a single low-resolution image I also from time We focus on this special case because while typically L >> K, it is reasonable to assumee I/0 IL r((") 7/() for t' + t, ie., given a LR image at the target time i, other LR views from different time steps provide little or no additional information. Given the abundant availability of LR imagery, it is often safe to assume access to ((() at target time t. Figure provides a visualization of this task.
For training, assume access to paired triplets {T I(00 [((()) collected across a geographic
region of interest where L / t. At inference time, we assume availability for ((() and 1 and the model needs to generalize to previously unseen locations. Note that at inference time, the target time I and reference time l' may not have been seen in the training set either.
4 Method
Given 1162 and r((') of the target location and target time t, our method generates 100 CxHW with a mod-le conditional pixel synthesis model. Figure is an illustration of our framework.
The generator G of our model consists of three parts: image feature mapper F R CxH Rea x positional encoder F,, and the pixel synthesizer Gp- For each spatial coordinate (æ, y) of the target HR image, the image feature mapper extracts the neighborhood information around
3

---

Gp
RGB
RGB
RGB
Mod
Mod
Mod
(0,0.))
F
Aur
Biz, U, 1)
FB
FD
(0)
/ea (z, v)
Med
FC
2X Mod
Mod
Mod
2X
f (HF,W
FA
2
Figure 2: An illustration of our proposed framework (discriminator omitted). The input images are processed by the image feature mapper F to obtain 1/20 Then with its spatial-tempon coordinate (a, v. encoded by E, each pixel is synthesized conditionally independently given the image feature associated with its spatial coordinate r(()
fee (Ir, y) and a random vector 2.
v2 {0, H) 10. W) from 1/ and 1 as well the global information associated with the coordinate in the two input images. The positional encoder learns a representation of the spatial-temporal coordinate (Ir, y, f), where is the temporal coordinate of the target image. The pixel synthesizer then uses the information obtained from the image feature mapper and the positional encoding to predict the pixel value at each coordinate. Finally, we incorporate an adversarial loss in our training, and thus include a discriminator D as the final component of our model.
Image Feature Mapper Before extracting features, first perform nearest neighbor resampling to the LR image 1 to match the dimensionality of the HR image and concatenate r/0 and // along the spectral bands to form the input r(0 = concat (1) n(t')
for E R22xHxW Then the mapper processes ( with a neighborhood encoder FE R2 FA: Re X''' xw' + R xH' xw' and a neighborhood decoder FD : RC x xw" ReCjeaXXXV The neighborhood encoder and decoder learn the fine structural features of the images, and the global encoder learns the overall inter-pixel relationships as it observes the entire image.
RC
global encoder
Fg uses sliding window filters to map small neighborhood of each coordinate into a value stored in the neighborhood feature map Fl2 E R. xw' and En uses another of filters trans- form the global feature map 1(t) E RC xW" back to the original coordinate grid. FA is a xexxHH'XX xW'
self-attention module that takes r(()) as the input and learns functions Q, K R Rea
R /8x V R you HB'x xw' CC X w' and scalar parameter y to map I r(()
The image feature mapper F = FF FA FD and we denote r/22 = F (())) and the image feature associated with coordinate (x., as T/t (x, v) Rra, Details are available in Appendix A.
F(r(f))
Positional Encoder Following 2 we also include both the Fourier feature and the spatial coor- dinate embedding in the positional encoder F.. The Fourier feature calculated as e,, (x., y, 4) = (Bn 2r 1, 2y 4)) where B,, E 3x is a learnable matrix and u is the time -
unit. This encoding of allows our model to handle time-series with various lengths and to ex- trapolate to time steps that are not seen at training time. E also learns a Cfea x H x W matrix ezo and the spatial coordinate embedding for (x. y, extracted from the vector at (ar, y) in c The positional encoding of (z, y, t) is the channel concatenation of e (z, y, t) and (2, y.,y,t). t), E(x, t) = concat(e,, (x, y, t), eco (2, y, t)] p0
Pixel Synthesizer Pixel Synthesizer Gp can be viewed as an analogy of simulating a conditional 2 + 1D neural radiance field with fixed viewing direction and camera ray using perceptron based

---

model. Instead of learning the breadth representation of the location, Gp learns to scale in the time dimension in a fixed spatial coordinate grid. Each pixel is synthesized conditionally independently
given ((() Exr, y, t), and a random vector 2 E R Go first learns function Is to map E(r, y, t) to R RC then obtains the input to the fully-connected layers e(z, v, t) 9= (E(x, y. t)) +))+ (r, v). Following 2 191, we use m-layer perceptron based mapping network M to map the noise vector into a style vector w, and use modulated fully-connected layers (ModFC) to inject the style vector into the generation to maintain style consistency among different pixels of the same image. We map the intermediate features to the output space for every two layers and accumulate the output values as the final pixel output.
With all components combined, the generated pixel value at (az, y, t) can be calculated
10 v) G(r,,,t,2I,, ) C,,,,,, F((,)))
Loss Function The generator is trained with the combination of the conditional GAN loss and T. loss. The objective function is
G = arg min max L.GAN ACI, (G)
LeGAN = Ellog 0 x, 16 1 Ei1 - log D(G -1 x, 19 (()
where X is the temporal-spatial coordinate grid {(,, 0) S x HH0 S y S W} for 1 hr-
LL, (G) = E1111/0 ((XX211K [/)))))
5
Experiments
5.1 Datasets
Texas Housing Dataset We collect dataset consisting of 286717 houses and their surrounding neighborhoods from CoreLogic tax and deed database that have an effective year built between 2014 and 2017 in Texas, US. We reserve 14101 houses from 20 randomly selected zip codes as the testing set and use the remaining 272616 houses from the other 759 zip codes as the training set. For each house in the dataset, we obtain two LR-HR image pairs, one from 2016 and another from 2018. In total, there are 1468668 multi-resolution images collected from different sensors for our experiments. We source high resolution images from NAIP (1m GSD) and low resolution images from Sentinel-2 (10m GSD) and only extract RGB bands from Google Earth Engine [14). More details can be found in Appendix C.
FMoW-Sentine22 Crop Field Dataset We derive this dataset from the crop field category in Functional Map of the World (fMoW) dataset [9] for the task of generating images over a greater number of time steps. We pair each fMoW image with lower resolution Sentinel-2 RGB image captured at the same location and a similar time. We prune locations with fewer than timestamps, yielding 1752 locations and total of 4898 MoW-SSentinel2 pairs. Each location contains between 2-15 timestamps spanning from 2015 to 2017. We reserve 237 locations as the testing set and the remaining 1515 locations the training set. More details can be found in Appendix C.
5.2 Implementation Details
Model Details We choose H = W = 256, C' = 3 (the concatenated RGB bands of the input images), Crea- 256, m 3,
14 and X = 100 for all of our experiments. We use non- saturating conditional GAN loss for G and R penalty for D, which has the same network structure as the discriminator in 19 2 We train all models using Adam optimizer with learning rate 2 x 10 3 Bu = 0, 31 = 0.99, c = 10-8 We train each model to convergence, which takes around 4-5 days 1 NVIDIA Titan XP GPU. Further details can be found in Appendix A and B.
We provide two versions of the image feature mapper. In version "EAD", we use convolutional layers and transpose convolutional layers with stride > in Fr and Fp. In version "EA", we use convolutional layers with stride in FF and an identity function in Fp. For version "EA", we use patch-based training and inference method with patch size of 64 because of memory constraints,
5

---

and denote it as "EA64". The motivation for including both "EAD" and "EA" is to examine the capabilities of F with and without spatial downsampling or upsampling. "EAD" can sample 1500 images in around 2.5 minutes (10 images/s) and "EA64" can sample 1500 images in around 19 minutes (1.3 image/s). More details can be found in Appendix and
Baselines We compare our method with two groups of baseline methods: image fusion models and super-resolution (SR) models. We use cGAN Fusion [3]. which leverages the network structure of the leading image-to-image translation model Pix2Pix [18 to combine different imagery products for surface reflectance prediction. We also compare our model with the original Pix2Pix framework. For SR baselines, choose SRGAN 21], which is widely compared among other GAN based SR
models for satellite imagery 36 25 29
We also compare our method with DBPN 16
which is
state-of-the-art SR model for satellite imagery
28
5.3 Image Generation Quality
Input LR 2016
Ours(EAD)
Pix2Pix
DBPN
Ground Truth HR 2016
Input HR 2018
Ours(EA64)
cGAN Fusion
SRGAN
Figure Samples from all models the Texas housing dataset with setting 8' Our models show advantages in both sample quality and structural detail consistency with the ground truth, especially in areas with house pool construction (zoomed in with colored boxes).
We examine generated image quality using both Texas housing dataset and our fMoW-Sentinel2 crop field dataset. Figures and presentt qualitative results from our approach and from baselines. Table I shows quantitative results on the Texas housing dataset and Table shows quantitative results on the [MoW-Sentinel2 crop field dataset. Overall, our models outperform all baseline approaches in all evaluation metrics.
Evaluation Metrics To assess image generation quality, we report standard sample quality metrics SSIM FSIM [42] and PSNR to quantify the visual similarity and pixel value accuracy of the generated images. We also include LPIPS [4 using VGG (2) features, which is deep perceptual metric used in previous works evaluate generation quality in satellite imagery [3]. LPIPS leverages visual features learned in deep convolutional neural networks that better reflect human perception. report the average score of each metrie given every possible pair of 1, where + 1'.
Texas Housing Dataset The dataset focuses regions with residential building construction between 2014 and 2017, so we separately analyze the task to predict j(t) when > / and when < on the Texas housing dataset. l' > / represents the task to "rewind" the construction of the neighborhood and < represents the task to "predict" construction. Our models achieve more photo-realistic sample quality (measured by LPIPS), maintain better structural similarity (measured

---

Table I: Image sample quality quantitative results Texas housing data. t' t generating an image in the past given a future HR image, and < denotes the task for generating an image in the future given past HR image.
denotes the task for
t' >>>
t' <t
Model
SSIM
PSNRT
FSIMT
LPIPS.
SSIM
PSNR
FSIM
LPIPS.
Pix2Pix
0.5432
20.8420
0.7522
0,4243
0.3909
17.9528
0.6802
0.4909
cGAN Fusion
0.5976
21.5226
0.7713
0.3936
0.4220
17.8763
0.6897
0.4726
DBPN
0.5781
21.4716
0.7102
0,5101
0.4572
18.9330
0.6384
0.5910
SRGAN
0.5361
21.1968
0.6999
0,5261
0.4221
18.9772
0.6387
0.5694
Ours (EAD)
0.6470
22.4906
0.7904
0.3695
0.5225
19.7675
0.7280
0.4275
Ours (EA64)
0.6570
22.5552
0.7902
0.3764
0.5338
19.8547
0.7269
0.4342
by SSIM and FSIM), and obtain higher pixel accuracy (measured by PSNR) in both tasks compared to other approaches. With the more challenging task t' < t, where input HR images contain fewer constructed buildings than the ground truth HR images, our method exceeds the performance of other models by greater margin.
Our results in Figure confirm these findings with qualitative examples. Patches selected and zoomed with colored bounding boxes show regions with significant change between f. and f'. Our model generates images with greater realism and higher structural information accuracy compared to baselines. While Pix2Pix and cGAN Fusion are also capable of synthesizing convincing images, they generate inconsistent building shapes, visual artifacts, and imaginary details like the swimming pool in the red bounding boxes. DBPN and SRGAN are faithful to information provided by the LR input but produce blurry images that are far from the ground truth.
Ground
Input
Input
(EAD)
(EA64)
SRGAN
2016-07-25
2016-10-09 10-09
2016-11-10
Figure 4: Samples from all models the fMoW-Sentines crop field dataset. Each row represents the results on the same location at different timestamp given the same HR input from 2016-09-06.
TMWW-Sentine22 Crop Field Dataset We conduct experiments on the dataset to compare model performance settings with less data, fewer structural changes, and longer time series with unseen timestamps test time. Our model outperforms baselines in all metrics, see Table 2 Figure Aegureessoww shows the image samples from different models on the MoW--entine12 crop field dataset. While image-to-image translation models fail to maintain structural similarity and SR models fail to attain realistic details, our model generates precise and realistic images.
crop field
Discussion is not surprising that our models generate high resolution details because they leverage rich prior of what HR images look like, acquired via the cGAN loss, and GANs are capable of learning to generate high frequency details [19, ( Despite considerable information loss, inputs from LR devices still provide sufficient signal for HR image generation (e.g. swimming pools may change LR pixel values in a way that is detectable by our models but not by human perception).
Experiments FigureBJan and Section show that these signals are enough for our model to reliably generate HR images that are high quality and applicable to downstream tasks.
7

---

Table 2: Image sample quality quantitative results IMoW- Sentine12 crop field dataset.
0.54
UPIPS
Model
SSIM
PSNR1
FSIM
LPIPSL
Pix2Pix
0.2144
14.0276
0.6418
0.5847
w/t
0.50
w/o
cGAN Fusion
0.2057
14.1353
0.6409
0,5912
DBPN
0.3621
15.7878
0.6323
0.6428
[t--]] (Day)
SRGAN
0.3479
15,3502
0.6323
0,6301
Ours (EAD)
0.3526
16.5769
0.6887
0.5629
Figure 5: Ablation study on learning the temporal embeddings model using MMWW-Sentinl22 crop field dataset.
Ours (EA64)
0.3905
16.8879
0.6827
0.5197
In more extreme scenarios (e.g. LR captured by MODIS with 250m GSD v.s. HR captured by NAIP with Im GSD), LR provides very limited information and therefore yields excessive uncertainty in generation. In this case, the high resolution details generated by our model are more likely to deviate from the ground truth.
is worth noting that our LR images are captured by remote sensing devices (e.g. Sentinel-2), as opposed to synthetic LR images created by downsampling used in many SR benchmarks. As shown our experiments, leading SR models such as DBPN and SRGAN do not perform well this setting.
5.4 Ablation Study
We perform an ablation study on different components of our framework. We consider the following configurations for comparison: "No Gp" setting removes the pixel synthesizer to examine the effects of Gp; "Linear F" and "E only" use single fully connecced layer and a single 3 convolutional layer with stride 1 respectively to verify the influence of deep multi-layer image feature mapper F. "ED Only" removes the global encoder FA and "A Only" removes the neighborhood encoder FE and decoder Fp. Note that because [2] has conducted thorough analysis various settings of the positional encoder, we omit the configurations to assess the effects of spatial encoding in E.
As shown in Table 3 each component contributes significantly to performance in all evaluation aspects. While "EA64" outperforms "EAD" SSIM and PSNR with a small margin, we observe slight checkerboard artifacts in the images generated by "EA64" (details in Appendix G). Overall, "EAD" the most realistic to the human eye, which is consistent with the LPIPS results. However, "EA64" has stronger performance in more data- constraindd setting shown in the fMoW-Sentinel2 crop field experiment. Samples generated by different configurations can be found Appendix F.
We also demonstrate the effects of learning the time dimension in our model. Parameterizing our model with continuous time dimension enables be applicable time series of varying lengths with non-uniform time intervals (e.g. IMoW-Sentines parameterization also improves model performance. We train a modified version of "EA64" to exclude the time component in E and compare the LPIPS values of the generated images to ones from the original "EA64" in Figure 5 In conjunction with the additional analysis in Appendix we show that the time-embedded model outperforms the same model without temporal encoding. Therefore, the time dimension is crucial to our model's performance, especially in areas with sparse HR satellite images over long periods of time.
Crop Field dataset). Moreover, this
We also include further details and analysis our ablation study Appendix F, including additional results for the effectiveness of the temporal dimension in F, comparison of different patch sizes for "EA" and training with different input choices.
5.5 Human Evaluation for Downstream Applications
Because our goal is to generate realistic and meaningful HR images that can benefit downstream tasks, we also conduct human evaluations to examine the potential of using our models for downstream applications. We deploy three human evaluation experiments on Amazon Mechanical Turk to measure

---

Table 3: Ablation study the effects of different components of model Texas housing dataset. "+" represents adding certain components, "-" represents removing the components, and "*" represents different configurations from the original setting. See Section 5.4 for more details.
Model
FE
FA
FD
Gp
SSIM
PSNR
FSIM
LPIPS_
"No Gp"
+
+
+
0.5338
20.2712
0.7399
0,4482
"Linear F"
8
0.4585
18.8164
0.7006
0.4845
"E Only"
0.4761
19.0881
0.7146
0,4604
"ED Only"
0.5414
20.2488
0.7392
0.4340
"A Only"
+
0.5280
20.03
0.7196
0,4418
"EA64"
+
+
+
0.5954
21.2050
0.7586
0.4053
"EAD"
+
0.5848
21.1291
0.7592
0.3985
Table 4: Human evaluation results on Texas housing dataset.
22 with mean count
r² with median count
% times selected
Images
Buildings
Pools
Buildings
Pools
Similarity
Realism
HR L' DBPN
0.1475
0.1009
0.1595
0.1997
0.8785
0.0227
0.8823
-0.0640
1,75%
1.25%
cGAN Fusion
0.8793
-0.0707
0.9093
-0.0367
45.00%
49.00%
Ours (EAD)
0.9174
0.6158
0.9298
0.5953
53.25 %
49.75%
the object reconstruction performance, similarity to ground truth HR images, and perceived realism of images generated by different models.
Building and Swimming Pool Count Object counting in HR satellite imagery has numerous applications, including environmental regulation [22). aid targeting (33], and local-level poverty mapping [3]. Therefore, choose object counting the primary downstream task for human evaluation. We randomly sample 200 locations in the test set of our Texas housing dataset, and assess the image quality generated under the setting t' 2018 > f 2016. Each image is evaluated by 3 workers, and each worker is asked to count the number of buildings as well as the number of swimming pools in the image. each location, we select images generated from our model (EAD),
cGAN Fusion, and DBPN, as well as the corresponding ground truth HR image r(() from 2016 and
HR image ((() from 2018 (denoted as HR t'). We choose buildings and swimming pools as our target objects since both can serve indicators of regional wealth [ 8 and both occur with high frequency in the areas of interest. Swimming pools are particularly challenging to reconstruct due to their small size and high shape variation, making them ideal candidate for measuring small-scale object reconstruction performance.
We measure the performance of each setting using the square of Pearson's correlation coefficient (r-2) between true and estimated counts, as in previous research 3. As human-level object detection still an open problem especially for satellite imagery [24] 34), human evaluation on this task serves as an upper bound on the performance of automatic methods on this task.
As shown in Table our model outperforms baselines on both tasks, with the most significant perfor- mance advantage in the swimming pool counting task. Note that in rapidly changing environments like our Texas housing dataset, using the HR image of a nearby timestamp t' cannot provide an accurate prediction of time f, which indicates the importance of obtaining higher temporal resolution in HR satellite imagery. Our model maintains the best object reconstruction performance among all models experimented, especially for small scale objects.
Similarity Ground Truth and Image Realism Aside from object counting, we also conduct human evaluation on the image sample quality. With 400 randomly selected testing locations in our Texas housing test set, each worker asked to either select the generated image that best matches a given ground truth HR image, or select the most realistic image from 3 generated images shown random order. All images are generated under the setting 2018 the same models as the ones in the object counting experiment. Human evaluation results on image
2016, and we choose

---

sample quality align with our quantitative metric results. Our model produces the most realistic and accurate images among the compared models. Note that although cGAN Fusion generates realistic images, it fails to maintain structural information accuracy, resulting in lower performance in the similarity to ground truth task.
5.6 Temporal Extrapolation
Ground Truth 2017
LR 2017
Generated 2017
HR 2018
Generated 2019
2019
Ground Truth 2019
Figure 6: Temporal extrapolation application of model. HR 2018 is the input NAIP image both generated images shown in the figure. LR 2017 and LR 2019 are Sentinel-2 images of the same region in 2017 and 2019. Since NAIP imagery is not available in Texas for 2017 and 2019, the ground truth is obtained via Google Earth Pro. Note that the capture dates of the ground truth and the LR images are not perfectly aligned due a lack of image availability.
Given HR image any time and LR images the desired timestamps, provide some evidence that our model is able to generate HR images at timestamps unavailable in the training dataset. Figure 6 demonstratts an example of such an application of our model. With the LR images from Sentinel-2, we generate HR images in 2017 and 2019, two years that do not have NAIP coverage in Texas. We compare the generated images with ground truth acquired from Google Earth Pro since the corresponding NAIP images are not available. Although the timestamps of the ground truth and LR images are not perfectly aligned, our generated images still show potential in reconstructing structural information reflected in the ground truth. More rigorous assessment of temporal extrapolation performance requires more extensive dataset, which we leave to future work.
6 Conclusion and Statement of Broader Impact
We propose conditional pixel synthesis model that uses the fine-grained spatial information in HR images and the abundant temporal availability of LR images to create the desired synthetic HR images the target location and time. We show that our model achieves photorealistic sample quality and outperforms competing baselines on a crucial downstream task, object counting.
We hope that the ability extend access HR satellite imagery in areas with temporally sparse HR imagery availability will help narrow the data gap between regions of varying economic development and aid in decision making. Our method can also reduce the costs of acquiring HR imagery, making cheaper to conduct social and economic studies over larger geographies and longer time scales.
That being said, our method does rely on trustworthy satellite imagery provided by reliable orga- nizations. Just like most SR models, our model is vulnerable to misinformation (e.g. the failure cases presented in Appendix G due to unreliable LR input). Therefore, caution against using generated images to inform individual policy (c.g. retroactively applying swimming pool permit fees) or military decisions. Exploration of performance robustness to ad verrsarial examples is left to future study. Furthermore, we acknowledge that increasing the temporal availability of HR satellite imagery has potential applications in surveillance. Finally, note that object counting performance measured through human evaluation due to dataset limitations, and we leave measurement of automated object counting performance to future work.
Acknowledgement
This research was supported in part by NSF (#1651565, #1522054, #1733686), ONR (V00014-99-1 2145), AFOSR (FA9550- 19-1-0249 ARO (W911NF-21-1-01255, Sloan Fellowship, HAI, TARPA, and Stanford DDI.
10

---

References
[1]
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
[2] Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image generators with conditionally-indepee pixel synthesis. arXiv preprint arXiv:2011.13775 2020.
[3] Kumar Ayush, Burak Uzkent, Marshall Burke, David Lobell, and Stefano Ermon. Gener- ating interpretable poverty maps using object detection in satellite images. arXiv preprint arXiv:2002.0166 016112 2020.
[4] W.I. Barnes, T.s. Pagano, and V.v. Salomonson. Prelaunch characteristics of the moderate resolution imaging spectroradiometer (modis) on eos-aml. IEEE Transactions on Geoscience and Remote Sensing, 3684):1088-1100, 1998.
[5] Shahine Bouabid, Maxim Chereetskiy, Maxime Rischard, and Jevgenij Gamper. Predicting landsat reflectance with deep generative fusion, 2020.
[6] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron, Unprocessing images for learned raw denoising. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[7] Marshall Burke, Anne Driscoll, David B Lobell, and Stefano Ermon. Using satellite imagery to understand and promote sustainable development. Science, 37136535), 2021.
[8] Margherita Carlucci, Sabato Vinci, Giuseppe Ricciardo Lamonica, and Luca Salvati. Socio- spatial disparities and the crisis: Swimming pools as a proxy of class segregation in athens. Social Indicators Research, Jul 2020.
[9] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world, 2018.
[10] Jintian Cui, Xin Zhang, and Muying Luo. Combining linear pixel unmixing and starfm for spatiotemporal fusion of gaofen-1 wide field of view imagery and modis imagery. Remote Sensing, 10(7), 2018.
[11] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks, 2015.
[12] Feng Gao, J. Masck, M. Schwaller, and F. Hall. On the blending of the landsat and modis surface reflectance: predicting daily landsat surface reflectance. IEEE Transactions on Geoscience and Remote Sensing, (217)))22222111 2006.
[13] Yuanfu Gong, Puyun Liao, Xiaodong Zhang, Lifei Zhang, Guanzhou Chen, Kun Zhu, Xiaoliang Tan. and Zhiyong Lv. Enlighten-gat for super resolution reconstruction in mid-resolution remote sensing images. Remote Sensing, 13(6), 2021.
[14] Noel Gorelick, Matt Hancher, Mike Dixon, Simon Ilyshccenkoo, David Thau, and Rebecca Moore. Google earth engine: Planetary-scale geospatial analysis for everyone. Remote Sensing of Environment, 2017.
[15] Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston. xbd: A dataset for assessing building damage from satellite imagery, 2019.
[16) Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. Deep back-projection net- works for super-resolution. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1664-1673, 2018.
[17] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5197-5206, 2015.
11

---

[18] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros, Image-to-image translation with conditional adversarial networks. CVPR, 2017.
[19] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
[20] Edward J. Knight and Geir Kvaran. Landsat-8 operational land imager design, characterization and performance. Remote Sensing, 6(11):1028-110055 2014.
[21] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang. and Wenzhe Shi. Photo- realistic single image super-resolution using generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 105-114, 2017.
[22] Jihyeon Lee, Nina R Brooks, Fahim Tajwar, Marshall Burke, Stefano Ermon, David B Lobell, Debashish Biswas, and Stephen P Luby. Scalable deep learning identify brick kilns and aid regulatory capacity. Proceedings of the National Academy of Sciences, 118(17), 2021.
[23] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. http:://ariivoorr/ass//00 13084,, 2020.
[24] Li Liu, Wanli Ouyang, Xiaogang Wang. Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pictikäinen. Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2))261-318, Feb 2020.
[25] Wen Ma, Zongxu Pan, Feng Yuan, and Bin Lei. Super-resolution of remote sensing images via a dense residual generative adversarial network. Remote Sensing, 11(21), 2019.
[26] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[27] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Generative image inpainting with adversarial edge learning. 2019.
[28] Shreya Roy and Anirban Chakraborty. Single image super-resolution with a switch guided hybrid network for satellite images, 2020.
[29] Luis Salgueiro Romero, Javier Marcello, and Verónica Vilaplana. Super-resolution of sentinel-2 imagery using generative adversarial networks. Remote Sensing, 12(15), 2020.
[30] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[31] Jacob Shermeyer and Adam Van Etten. The effects of super-resolution on object detection performance satellite imagery, 2019.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
[33] Kristin Spröhnle, Eva-Maria Fuchs, and Patrick Aravena Pelizari. Object-based analysis and fusion of optical and sar satellite data for dwelling detection in refugee camps. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(5):1780-177 2017.
[34] Hilal Tayara and Kil To Chong. Object detection in very high-resolution aerial images using one- stage densely connected feature pyramid network. Sensors (Basel, Swirzerland), 18(10):3311, Oct 2018.
[35] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. 1rXiv:1711.1092. 2017."
[36] Zhongyuan Wang, Kui Jiang, Peng Yi, Zhen Han, and Zheng He. Ultra-dense gan for satellite imagery super-resolution. Neurocomputing, 398:328-337,, 2020.
12

---

[37] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on Image Processing, 13(4):600-612 2004.
[38] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. http::/arai..rr//b22111...10 2020.
[39] Chih- Yuan Yang, Chao Ma, and Ming-Hsuan Yang. Single-image super-resolution: A bench- mark. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014, pages 372-386, Cham, 2014. Springer International Publishing.
[40] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[41] Chongyuan Zhang, Afef Marzougui, and Sindhuja Sankaran. High-resolution satellite imagery applications in crop phenotyping: An overview. Computers and Electronics in Agriculture, 175:105584, 2020.
[42] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: feature similarity index for image quality assessment. IEEE Transactions Image Processing, 20(8):2378-2333 2011.
[43] Richard Zhang. Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[44] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[45] Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. Image super-resolution by neural texture transfer, 2019.
[46] Xiaolin Zhu, Chen, Feng Gao, Xuehong Chen, and Jeffrey G. Masek. An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions. Remote Sensing of Environment, (44(1))2610-2622 2010.
13


Reviews:
[{"review_id": "aXnCNZ8xvr", "reviewer": "Reviewer_3V3Y", "summary": "This paper presents a method to generate high-resolution satellite imagery from a pair of input low resolution image and a reference high resolution image of the same location but taken at a different time instant. The main problem the author(s) are addressing is the lack of high quality satellite imagery(specially temporally), and that can be resolved using multiple low resolution and a reference HR image. The authors claim that such a high resolution image will be useful for object counting tasks, specially over time when the intermediate high-res images are not available. Their model comprises of a feature mapper (for fusion of input low-res and reference high-res image),  feature encoder (for both the Fourier and spatial feature) and finally the synthesiser (inspired by nerf operating in the temporal domain keeping the spatial domain fixed) they curate a dataset from the publicly available datasets or services like google earth and perform their experiments on them. The authors also conduct a Human study for the verification of their approach along with the usual metrics for image similarity. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a interesting application of the already well researched problem if image super resolution. The super resolution is applied on the satellite images, where input to the system is a low resolution image and a reference high resolution image, but taken in a different time instant. The reference HR image provides as a guidance to for high quality realistic texture in the final synthesized image. Similar attempts have been done in general Single image SR ( e.g. SRNTT: Image Super-Resolution by Neural Texture Transfer, CVPR19). \n\nIn the paper the LR image corresponds to 10m/pixel (assuming in each spatial dimension) and the HR image is 1m/pixel. The method is trained using a single LR, reference and the target image. It is claimed that having multiple LR image as input does not add to the final reconstruction as the inputs low resolution images across time prove little additional information. However it seems counter-intuitive as you would expect the system to be able to bridge the gap better between the reference HR time t^{prime} and t. Have the authors tried it or there is lack of granular data to do the verification.\n\nCan the authors elaborate more on the image registration between the input LR and input reference HR.  The reference HR image at 1m/pix resolution might contain objects like vehicles which might not be present in the LR time or not visible in the lower resolution. Does that have an impact on the overall performance of the system.\n\nCan the authors provide some information on the ratio of the rectangles (bounded by the latitude/longitude, Appendix section C) of the are of interest  for the LR and the HR image. That give an indication of how much SR (e.g. 4x, 8x,..) is being done, and put this method in the perspective of the general task of Single image SR.\n\nCan the authors please provide some additional experimental details like input image resolutions, output image resolutions (in spatial coordinates)\n\nThe paper is clear and easy to understand. Additional materials in Appendix are useful for understanding the experiments and the dataset. \n\nRegarding the human evaluation of object counting task, the HR (at t^{prime}) is expected to be different at different time stamp (t). However for the building counting task, the reference HR(t^{prime}) image is significantly different than the target HR(t). This might make the generation task harder.  Can the author please comment on temporal granularity of reference and input images.\n\nCan the authors please comment/discuss on the potential drawbacks of their systems. Some failure examples will be a good to demonstrate the limits of the system, and help further research. Further if they can please comment on how the proposed method will cope with large moving urban artifacts (often captured in satellite images ) like large vehicles, etc.\n\nCan the authors please mention running performance (training times, inference times) for their method.  \n\nIn fig. 4, there seems to be a slight colour distortion from the Ground Truth and methods EAD/EA64. However  the baseline methods are closer to ground truth in terms of colour temporature. Can the authors comment if this kind of distortion an atrifact of the system which is guided by reference HR image?\n\nTheir model has three part structure, first a feature mapper for combining the LR and ref HR images. Then their encoder combine Fourier the features (as proposed initially by [2]) to allow  their model to learn about temporal variation of the region. The authors take inspiration from the Neural radiance field to generate a temporally novel viewpoint while keeping the spatial dimension fixed. Thus the authors propose a system comprising of well known modules that work and have applied for satellite imagery, which is lacking in novelty. Finally, the resulting image is used for downstream object counting task, where it shows that human subjects perform better with the proposer sure-resolution method. However, this might not be the same for automated object counting. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nCan the authors comment on hoe their system will cope with dense urban areas specially areas which have high population where objects in the contention (like buildings) are smaller than dataset. Also how well will this method generalise to different scene with differing urban landscape.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "NVBZ2BqmzyD", "reviewer": "Reviewer_AEn7", "summary": "This paper proposes using a NeRF-like approach for generating high-resolution imagery at some time t based off of high-resolution imagery at time t' and low-resolution imagery at t. This is a practically useful application for different problems in conservation and sustainability as the authors note. They compare the proposed method with SR based methods and image fusion models (e.g. cGAN and Pix2Pix) on two datasets.\n\nThe approach consists of:\n1.) An image encoder that takes the conactenated HR and LR images as input\n2.) A decoder that produces pixel level features\n3.) A learned positional embedding \n4.) A pixel-wise decoder that produces the desired high-resolution imagery from the pixel level features and position embedding\n5.) A discriminator network\n\nThe entire process is trained using a cGAN loss and L1 reconstruction loss.\n\nEvaluation is performed with SSIM, FSIM, and LPIPs.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well written paper with very clear experiments and contributions with immediate practical application in remote sensing.\n\nQuestions for the authors:\n- Is LPIPs with VGG features acceptable in remote sensing contexts? What is the reason that VGG features from ImageNet should be able to measure visual perception of high-resolution imagery?\n- Did you consider using the NIR bands from Sentinel / NAIP? These spectral values may better constrain the output of the model \n- What is the motivation for EA64 vs. EAD?\n- Is it possible to evaluate counting based downstream tasks on fMoW?\n\n## Minor Comments\n\n- Line 126-127, should one of the three F_E be F_A and another be F_D (matching Figure 2)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address the limitations and potential negative societal impacts. While they acknowledge that their method has potential applications in surveillance, high-resolution on demand satellite imagery is both commercially available and available to government level entities, therefore the additional negative impacts of their method are minimal.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "G9X9pG9dkGY", "reviewer": "Reviewer_Gndk", "summary": "The submission deals with high resolution satellite images, given a low resolution image. The method is trained by learning to predict a high resolution image given a low resolution one, adversiarially, knowing the true high resolution. The point is to use LR imagery to predict HR at points in time where actual HR images are not available. \n\nExperiments aim at quantifying the quality of generated images, both from a structural and reconstruction quality standpoint, as well as using the generated images in a potential downstream task of object counting, specifically, buildings and swimming pools. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper deals with an interesting problem and surely deserves some attention. I think the method presented is robust and results encouraging, but I also have some concerns about practical scenarios and settings which maybe should be better presented and motivated in the manuscript.  \n\nIn more detail: \n- It is clear that the method aims at good reconstruction, but from the paper is somewhat unclear whether this can be extended from plain RGB to more wavelenghts. For many applications, having near infrared information is crucial, specifically crop monitoring and precision agriculture. In the paper, dataset only aim at reconstructing RGB images, which is a very fair target, but would it perform better if the LR is characterised also by near-infrared information, or if the target HR also contains NIR? I understand only using RGB makes more data available, but in the specific experiments, using sentinel data, other channels would be at hand. \n- I would like to see a better distinction between learninng methods for superresolution. At the end of the day, one can train a SR model on LR-HR pairs, and use it to predict HR from a given LR. I see that standard SR methods do not make use of time, but from the plot in Fig. 5, it does not seem to _that_ relevant. \n- Fig. 5. Is, in facts, statistically significant the difference of using or not using temporal information? I think that the model can struggle not only with ground cover changes (e.g. LL 210-212) but also with seasonal effects, atmospheric effects, sun-angle, etc. These aspects are not well considered in the manuscript and would be very beneficial to discuss these limitations.  The simples example would be the presence of occluding clouds or opaque clouds attenuating signals. \n- In the same figure, I was expecting to see a spike in accuracy after roughly 1y, where the vegetation cycle is roughly at the same point. but I do see a spike at ~510 days, which is weird. What is the cause of that? only a noisy \"by chance\" spike, or something more? What about crop rotation for this dataset, could it influence the results? This spike is the only place where the time and not time models differ (but also larger error bars -- if the shaded area are std of experiment replica -- it is not explained). \n- Another important point on the same line, is the argument about spatial resolution and the GSD. What is the maximum difference in GSD the model can handle? I am curious since for the swimming pools, LR image resolution (RGB Sentinel 2) often is larger than most small private swimming pools, where a given swimming pool is either contained in a pixel or diluted over a few. How can the model reconstruct such things, if those elements are not visible in the LR images? So I understand one should find best images for the downstream task, and it is obvious that one cannot reconstruct 10cm imagery from 250m GSD MODIS imagery, but some discussion around the issue would be welcome. I think that linking these to the amount of spectral information, could give hints about how to extend the work, since using infrared, for instance, could provide information about land covers not clearly distinguishable in RGB. \n- It is mentioned that land cover changes affect results. This, coupled to the GSD problem of the point above, would suggest that there is a setting in which changes are modes of failures. Can this be a problem also for downstream tasks, and how can one prevent downstream task failures? For instance, not seeing swimming pools in LR data prevents them from being counted, which is a clear limitation.  Also, I can imagine that generating HR images far in the future, would greatly reduce confidence of the objects contained in the HR image. All in all, I wonder (since LR/HR pairs are rarely at uniform time intervals) how this can be detected at inference. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "limitations are not very clearly discussed, while i do not see any potential adverse societal impact.  ", "review_text": "", "extra_scores": {"ethical_concerns": "None ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "3Q1zI0G-5B", "reviewer": "Reviewer_F5jU", "summary": "The authors propose a conditional pixel synthesis approach which combines frequent low-resolution aerial imagery with less widely available high-resolution imagery to generate photo-realistic and accurate high-resolution imagery.  The approach is largely based on NeRF with modifications for the remote sensing domain.  Results are compared to other techniques such as image fusion models, Pix2Pix, SuperResolution GAN, and others.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address an important issue in remote sensing which is the availability of high-resolution imagery.  Frequent, low-resolution imagery is abundant, but high-resolution imagery is much more difficult to obtain.  They use the combination of LR and HR data to produce a photorealistic HR data at the point of interest.  They demonstrate this is both photo-realistic as well as useful for downstream tasks such as counting.\n\nThe approach is largely based on NeRF although addresses challenges specific to the remote sensing domain, such as the construction of the 3D scene.  The theoretical derivation is clear, appears correct, and is easy to follow.  However, it would be useful to highlight where the current approach differs from some of the cited related works as NeRF and its derivatives are a relatively new approach.  It is hard to tell which elements are completely new, adapted, or directly used from these other works. \n The authors reference how their approach differs in relation to SR and Fusion RM approaches, but doesn't draw a direct comparison to what modifications are made relative to the works cited around NeRF and Pixel Synthesis.  Within the derivation, it would be useful to include remarks such as \"following the approach of\" or \"unlike X which did Y\", to highlight the originality of the various components in their approach.\n\nExperiments are performed on two datasets with different domains- Texas Housing Dataset which is more \"industrial\" focused, and Sentinel2 Crop Field Dataset, which has a more \"agricultural focus\".  This is a very nice addition as it shows the approach is applicable over two domains with likely different statistics (THD likely containing more geometric features than Sentinel2).  Performing a downstream counting task is also powerful in addition to providing a human-perception evaluation.  The only piece I would be interested in seeing is a similar task on the agriculture dataset- there are agricultural metrics and tasks which are often very color-focused as opposed to shape focused.  Even if the images look photo-realistic, slight color variations could be problematic on downstream tasks such as determining crop type, land usage, growth stage, stress/damage, etc.  Those tasks may rely on subtle qualities of the image which the human does not perceive easily.  Performing a similar downstream task on this dataset would be very useful.\n\nAblation studies are performed to determine the impact on the different components of the framework.  \n\nRegarding both the experiments and ablation studies, while the improvements are so large that it is reasonable to believe they are significantly better there is an increasing desire to see experiments run multiple times so that errorbars around these measurements can be provided and the degree of significance can be determined.\n\nThere is always a worry around adversarial examples, especially in a generative setting, especially when there may be surveillance use cases.  It would be interesting to see how small pixel shifts, particularly in the low-resolution imagery, impacts the final generated results.  \n\nThe paper is overall clear and well written.  Implementation details are given in the appendix as opposed to the main text, which is suboptimal.  Because all elements enabling reproducibility should be included in the main text, I believe it's important to move as much of that as possible to the body of the text.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In the Implementation Details I see the hardware used, but do not see any indication as to the time trained (either in epochs/steps, or wall time).  Similarly, there is no mention to inference speed or limitations.  Because this paper is focused on a domain involving huge amounts of data, I believe some discussion around this is warranted.  For example, the authors cite an important usecase of the HR data as exploring poverty metrics.  However, if the cost to run such a model is massive, perhaps there is some drawback to this approach.  \n\nSocietal Impacts, particularly in reference to security/surveillance are mentioned, but not discussed at length.  There are some fairly significant implications here both in terms of individual privacy as well as defense/military applications.  Especially regarding the later, generating high-resolution images with military implications could be problematic if the model is incorrect (both in terms of false positives as well as negatives).  Photorealistic images could be generated and shared with the public to bolster support for political agendas- I believe speaking to this even briefly is an important consideration.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three of the four reviewers recommended accepting the paper , and one of them increased the score following the rebuttal. I am happy to accept it, I encourage the authors to include the additional material that they discussed in the rebuttal in the final version.
Decision: Accept (Poster)