Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis

1 Introduction . Recent advancements in satellite technology have enabled granular insight into the evolution of human activity on the planet ’ s surface . Multiple satellite sensors now collect imagery with spatial resolution less than 1m , and this high-resolution ( HR ) imagery can provide sufficient information for various fine-grained tasks such as post-disaster building damage estimation , poverty prediction , and crop phenotyping [ 15 , 3 , 41 ] . Unfortunately , HR imagery is captured infrequently over much of the planet ’ s surface ( once a year or less ) , especially in developing countries where it is arguably most needed , and was historically captured even more rarely ( once or twice a decade ) [ 7 ] . Even when available , HR imagery is prohibitively expensive to purchase in large quantities . These limitations often result in an inability to scale promising HR algorithms and apply them to questions of broad social importance . Meanwhile , multiple sources of publicly-available satellite imagery now provide sub-weekly coverage at global scale , albeit at lower spatial resolution ( e.g . 10m resolution for Sentinel-2 ) . Unfortunately , such coarse spatial resolution renders small objects like residential buildings , swimming pools , and cars unrecognizable . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In the last few years , thanks to advances in deep learning and generative models , we have seen great progress in image processing tasks such as image colorization [ 43 ] , denoising [ 6 , 35 ] , inpainting [ 35 , 27 ] , and super-resolution [ 11 , 21 , 16 ] . Furthermore , pixel synthesis models such as neural radiance field ( NeRF ) [ 26 ] have demonstrated great potential for generating realistic and accurate scenes from different viewpoints . Motivated by these successes and the need for high-resolution images , we ask whether it is possible to synthesize high-resolution satellite images using deep generative models . For a given time and location , can we generate a high-resolution image by interpolating the available low-resolution and high-resolution images collected over time ? To address this question , we propose a conditional pixel synthesis model that leverages the finegrained spatial information in HR images and the abundant temporal availability of LR images to create the desired synthetic HR images of the target location and time . Inspired by the recent development of pixel synthesis models pioneered by the NeRF model [ 26 , 40 , 2 ] , each pixel in the output images is generated conditionally independently by a perceptron-based generator given the encoded input image features associated with the pixel , the positional embedding of its spatialtemporal coordinates , and a random vector . Instead of learning to adapt to different viewing directions in a single 3D scene [ 26 ] , our model learns to interpolate across the time dimension for different geo-locations with the two multi-resolution satellite image time series . To demonstrate the effectiveness of our model , we collect a large-scale paired satellite image dataset of residential neighborhoods in Texas using high-resolution NAIP ( National Agriculture Imagery Program , 1m GSD ) and low-resolution Sentinel-2 ( 10m GSD ) imagery . This dataset consists of scenes in which housing construction occurred between 2014 and 2017 in major metropolitan areas of Texas , with construction verified using CoreLogic tax and deed data . These scenes thus provide a rapidly changing environment on which to assess model performance . As a separate test , we also pair HR images ( 0.3m to 1m GSD ) from the Functional Map of the World ( fMoW ) dataset [ 9 ] crop field category with images from Sentinel-2 . To evaluate our model ’ s performance , we compare to state-of-the-art methods , including superresolution models . Our model outperforms all competing models in sample quality on both datasets measured by both standard image quality assessment metrics and human perception ( see example in Figure 1 ) . Our model also achieves 0.92 and 0.62 Pearson ’ s r2 in reconstructing the correct numbers of buildings and swimming pools respectively in the images , outperforming other models in these tasks . Results suggest our model ’ s potential to scale to downstream tasks that use these object counts as input , including societally-important tasks such as population measurement , poverty prediction , and humanitarian assessment [ 7 , 3 ] . 2 Related Work . Image Super-resolution SRCNN [ 11 ] is the first paper to introduce convolutional layers into a SR context and demonstrate significant improvement over traditional SR models . SRGAN [ 21 ] improves on SRCNN with adversarial loss and is widely compared among many GAN-based SR models for remote sensing imagery [ 36 , 25 , 29 ] . DBPN [ 16 ] is a state-of-the-art SR solution that uses an iterative algorithm to provide an error feedback system , and it is one of the most effective SR models for satellite imagery [ 28 ] . However , [ 31 ] shows that SR is less beneficial at coarser resolution , especially when applied to downstream object detection on satellite imagery . In addition , most SR models test on benchmarks where LR images are artificially created , instead of collected from actual LR devices [ 1 , 39 , 17 ] . SR models also generally perform worse at larger scale factors , which is closer to settings for satellite imagery SR in real life . SRNTT [ 45 ] applies reference-based super-resolution through neural texture transfer to mitigate information loss in LR images by leveraging texture details from HR reference images . While SRNTT also uses a HR reference image , it does not learn the additional time dimension to leverage the HR image of the same object at a different time . In addition , our model uses a perceptron based generator while SRNTT uses a CNN based generator . Fusion Models for Satellite Imagery [ 12 ] first proposes STARFM to blend data from two remote sensing devices , MODIS [ 4 ] and Landsat [ 20 ] , for spatial-temporal super resolution to predict land reflectance . [ 46 ] introduces an enhanced algorithm for the same task and [ 10 ] combines linear pixel unmixing and STARFM to improve spatial details in the generated images . cGAN Fusion [ 5 ] incorporates GAN-based models in the solution , using an architecture similar to Pix2Pix [ 18 ] . In contrast to previous work , we are particularly interested in synthesizing images with very high resolution ( ≤ 1m GSD ) , enabling downstream applications such as poverty level estimation . NeRF and Pixel Synthesis Models Recent developments in deep generative models , especially advances in perceptron-based generators , have yet to be explored in remote sensing applications . Introduced by [ 26 ] , neural radiance fields ( NeRF ) demonstrates great success in constructing 3D static scenes . [ 23 , 38 ] extends the notion of NeRF and incorporates time-variant representations of the 3D scenes . [ 30 ] embeds NeRF generation into a 3D aware image generator . These works , however , are limited to generating individual scenes , in contrast with our model which can generalize to different locations in the dataset . [ 40 ] proposes a framework that predicts NeRF conditioning on spatial features from input images ; however , it requires constructing the 3D scenes , which is less applicable to satellite imagery . [ 2 ] proposes a style-based 2D image generative model using an only perceptron-based architecture ; however , unlike our method , it doesn ’ t consider the task of conditional 2D image generation nor does it incorporate other variables such as time . In contrast , we propose a pixel synthesis model that learns a conditional 2D spatial coordinate grid along with a continuous time dimension , which is tailored for remote sensing , where the same location can be captured by different devices ( e.g . NAIP or Sentinel-2 ) at different times ( e.g . year 2016 or year 2018 ) . 3 Problem Setup . The goal of this work is to develop a method to synthesize high-resolution satellite images for locations and times for which these images are not available . As input we are given two time-series of high-resolution ( HR ) and low-resolution ( LR ) images for the same location . Intuitively , we wish to leverage the rich information in HR images and the high temporal frequency of LR images to achieve the best of both worlds . Formally , let I ( t ) hr ∈ RC×H×W be a sequence of random variables representing high-resolution views of a location at various time steps t ∈ T . Similarly , let I ( t ) lr ∈ RC×Hlr×Wlr denote low-resolution views of the same location over time . Our goal is to develop a method to estimate I ( t ) hr , given K high resolution observations { I ( t ′ 1 ) hr , · · · , I ( t′K ) hr } and L low-resolution ones { I ( t′′1 ) lr , · · · , I ( t′′L ) lr } for the same location . Note the available observations could be taken either before or after the target time t. Our task can be viewed as a special case of multivariate time-series imputation , where two concurrent but incomplete series of satellite images of the same location in different resolutions are given , and the model should predict the most likely pixel values at an unseen time step of one image series . In this paper , we consider a special case where the goal is to estimate I ( t ) hr given a single highresolution image I ( t ′ ) hr and a single low-resolution image I ( t ) lr also from time t. We focus on this special case because while typically L K , it is reasonable to assume I ( t ) hr ⊥ I ( t′ ) lr | I ( t ) lr for t ′ 6= t , i.e. , given a LR image at the target time t , other LR views from different time steps provide little or no additional information . Given the abundant availability of LR imagery , it is often safe to assume access to I ( t ) lr at target time t. Figure 1 provides a visualization of this task . For training , we assume access to paired triplets { I ( t ) hr , I ( t ) lr , I ( t′ ) hr } collected across a geographic region of interest where t′ 6= t. At inference time , we assume availability for I ( t ) lr and I ( t′ ) hr and the model needs to generalize to previously unseen locations . Note that at inference time , the target time t and reference time t′ may not have been seen in the training set either . 4 Method . Given I ( t ) lr and I ( t′ ) hr of the target location and target time t , our method generates Î ( t ) hr ∈ RC×H×W with a four-module conditional pixel synthesis model . Figure 2 is an illustration of our framework . The generator G of our model consists of three parts : image feature mapper F : RC×H×W → RCfea×H×W , positional encoder E , and the pixel synthesizer Gp . For each spatial coordinate ( x , y ) of the target HR image , the image feature mapper extracts the neighborhood information around ( x , y ) ∈ { 0 , 1 , ... , H } × { 0 , 1 , ... , W } from I ( t ) lr and I ( t′ ) hr , as well as the global information associated with the coordinate in the two input images . The positional encoder learns a representation of the spatial-temporal coordinate ( x , y , t ) , where t is the temporal coordinate of the target image . The pixel synthesizer then uses the information obtained from the image feature mapper and the positional encoding to predict the pixel value at each coordinate . Finally , we incorporate an adversarial loss in our training , and thus include a discriminator D as the final component of our model . Image Feature Mapper Before extracting features , we first perform nearest neighbor resampling to the LR image I ( t ) lr to match the dimensionality of the HR image and concatenate I ( t ) lr and I ( t′ ) hr along the spectral bands to form the input I ( t ) cat = concat [ I ( t ) lr , I ( t′ ) hr ] ∈ R2C×H×W . Then the mapper processes I ( t ) cat with a neighborhood encoder FE : R 2C×H×W → RCfea×H′×W ′ , a global encoder FA : R Cfea×H′×W ′ → RCfea×H′×W ′ and a neighborhood decoder FD : RCfea×H ′×W ′ → RCfea×H×W . The neighborhood encoder and decoder learn the fine structural features of the images , and the global encoder learns the overall inter-pixel relationships as it observes the entire image . FE uses sliding window filters to map a small neighborhood of each coordinate into a value stored in the neighborhood feature map I ( t ) ne ∈ RCfea×H ′×W ′ and FD uses another set of filters to transform the global feature map I ( t ) gl ∈ RCfea×H ′×W ′ back to the original coordinate grid . FA is a self-attention module that takes I ( t ) ne as the input and learns functions Q , K : RCfea×H ′×W ′ → RCfea/8×H ′W ′ , V : RCfea×H ′×W ′ → RCfea×H′W ′ and a scalar parameter γ to map I ( t ) ne to I ( t ) gl . The image feature mapper F = FE ◦ FA ◦ FD and we denote I ( t ) fea = F ( I ( t ) cat ) and the image feature associated with coordinate ( x , y ) as I ( t ) fea ( x , y ) ∈ RCfea . Details are available in Appendix A. Positional Encoder Following [ 2 ] , we also include both the Fourier feature and the spatial coordinate embedding in the positional encoder E. The Fourier feature is calculated as efo ( x , y , t ) = sin ( Bfo ( 2x H−1 − 1 , 2y H−1 − 1 , t u ) ) where Bfo ∈ R 3×Cfea is a learnable matrix and u is the time unit . This encoding of t allows our model to handle time-series with various lengths and to extrapolate to time steps that are not seen at training time . E also learns a Cfea × H ×W matrix eco and the spatial coordinate embedding for ( x , y , t ) is extracted from the vector at ( x , y ) in eco . The positional encoding of ( x , y , t ) is the channel concatenation of efo ( x , y , t ) and eco ( x , y , t ) , E ( x , y , t ) = concat [ efo ( x , y , t ) , eco ( x , y , t ) ] ∈ R2Cfea . Pixel Synthesizer Pixel Synthesizer Gp can be viewed as an analogy of simulating a conditional 2 + 1D neural radiance field with fixed viewing direction and camera ray using a perceptron based model . Instead of learning the breadth representation of the location , Gp learns to scale in the time dimension in a fixed spatial coordinate grid . Each pixel is synthesized conditionally independently given I ( t ) fea , E ( x , y , t ) , and a random vector z ∈ RZ . Gp first learns a function gz to map E ( x , y , t ) to RCfea , then obtains the input to the fully-connected layers e ( x , y , t ) = gz ( E ( x , y , t ) ) +I ( t ) fea ( x , y ) . Following [ 2 , 19 ] , we use a m-layer perceptron based mapping network M to map the noise vector z into a style vector w , and use n modulated fully-connected layers ( ModFC ) to inject the style vector into the generation to maintain style consistency among different pixels of the same image . We map the intermediate features to the output space for every two layers and accumulate the output values as the final pixel output . With all components combined , the generated pixel value at ( x , y , t ) can be calculated as Î ( t ) hr ( x , y ) = G ( x , y , t , z|I ( t ) lr , I ( t′ ) hr ) = Gp ( E ( x , y , t ) , F ( I ( t ) cat ) , z ) Loss Function The generator is trained with the combination of the conditional GAN loss and L1 loss . The objective function is G∗ = argmin G max D LcGAN ( G , D ) + λLL1 ( G ) LcGAN ( G , D ) = E [ logD ( I ( t ) hr , X , I ( t ) lr , I ( t′ ) hr ) ] + E [ 1 − logD ( G ( X , z|I ( t ) lr , I ( t′ ) hr ) , X , I ( t ) lr , I ( t′ ) hr ) ] where X is the temporal-spatial coordinate grid { ( x , y , t ) |0 ≤ x ≤ H , 0 ≤ y ≤ W } for I ( t ) hr . LL1 ( G ) = E [ ||I ( t ) hr −G ( X , z|I ( t ) lr , I ( t′ ) hr ) ||1 ] .

Summary: This paper proposes using a NeRF-like approach for generating high-resolution imagery at some time t based off of high-resolution imagery at time t' and low-resolution imagery at t. This is a practically useful application for different problems in conservation and sustainability as the authors note. They compare the proposed method with SR based methods and image fusion models (e.g. cGAN and Pix2Pix) on two datasets.  The approach consists of: 1.) An image encoder that takes the conactenated HR and LR images as input 2.) A decoder that produces pixel level features 3.) A learned positional embedding  4.) A pixel-wise decoder that produces the desired high-resolution imagery from the pixel level features and position embedding 5.) A discriminator network  The entire process is trained using a cGAN loss and L1 reconstruction loss.  Evaluation is performed with SSIM, FSIM, and LPIPs.