Spectral Synthesis for Satellite-to-Satellite Translation

1 INTRODUCTION . Climate change and related environmental issues - including the loss of biodiversity and extreme weather - are listed by the World Economic Forum as the most important risks to our planet ( 7 ) . Monitoring the Earth is critical to mitigating these risks , understanding the effects , and making future predictions ( 38 ) . Multi- and hyper-spectral satellite-based remote sensing enables global observation of the Earth , allowing scientists to study large-scale system dynamics and inform general circulation models ( 26 ) . In weather forecasts satellite data initializes the atmospheric state for future predictions . On longer time scales , these data are used to measure the effects of climate change such as land-cover variations , temperature trends , solar radiation levels , and the rate of snow/ice melt . In the coming decades , increased investments from the public and private sectors in satellite-based observations will continue to improve global monitoring , as highlighted in NASA ’ s decadal survey ( 25 ) . Satellites are designed based on specifications for a given set of applications with fiscal , technological , and physical constraints which limit their temporal , spatial , and spectral resolutions . Geostationary ( GEO ) satellites rotate with the Earth to stay over a constant position above the equator at a high elevation of 35,786km . This position enables GEO satellites with on-board multi-spectral imagers to take continuous and high-temporal snapshots over large spatial regions and are ideal for monitoring diurnal and fast moving events . Spectral bands measure brightness and radiance intensities of the electromagnetic spectrum at a specified center wavelength and bandwidth . Bands are selected to satisfy defined variables of interest constrained by technological cost and accuracy . Applications of GEO sensors include atmospheric winds measurement ( 35 ) , tropical cyclone tracking ( 36 ) , wildfire monitoring ( 41 ) , and short-term forecasting ( 24 ) . Multiple GEO satellites are needed to generate global high-temporal resolution datasets to better monitor these events around the world . However , variations in resolutions , sensor uncertainties , and temporal life spans leads to a set of separate datasets which are not consistent , making this process very challenging ( 26 ) . Developing consistent and homogeneous global datasets would relieve many of these challenges . Supplementary Material : https : //github.com/anonymous-ai-for-earth/ satellite-to-satellite-translation The current generation of GEO satellites are no exception . The GOES-16/17 satellites operated by NASA/NOAA ( cost : $ 11 billion ) have a set of 16 imaging bands covering the visible , near- , and thermal-infrared spectral range ( 29 ) . The Himawari-8 satellite operated by the Japanese Space Agency ( cost : $ 800 million ) similarly has 16 bands but swaps a NIR ( 1.38µm ) band for a green channel ( 0.51µm ) , enabling the construction of true color images ( 3 ) . The 1.38µm band is ideal for measuring Cirrus clouds , composed of ice particles in the upper troposphere , a major contributor to regulating the Earth ’ s climate that is not yet well understood ( 21 ; 10 ) . Without capturing this band , directly observing Cirrus clouds over Japan , East Asia , and Western Pacific region from Himawari-8 is not possible . Synthetic observations via virtual spectral sensors could be a low-cost solution to improving coverage availability and consistency with current satellites . We present an approach to generate synthetic spectral channels from a multi-domain unpaired satellite dataset . We treat satellites with either dissimilar spectral coverage or varying vantage points as separate spectral sets . In this way , the problem closely resembles that of colorization ( 40 ) and image-to-image translation tasks ( 22 ; 42 ; 9 ) in the case where paired images are not available but with the added complexity of a large number of spectral bands . We use a combination of variational autoencoder ( VAE ) and generative adverserial network ( GAN ) ( 8 ) architectures adapted to our problem to model a shared latent space , as in unsupervised image-to-image translation ( 22 ) . Generating synthetic bands is an under-constrained problem that paired with an adverserial loss in high dimensions promotes overfitting . Our approach mitigates these challenges by leveraging a weak supervision signal based on partial overlap in spectral bands between domains . By including a reconstruction loss on overlapping spectral bands between domain pairs we can substantially improve spectral band synthesis . To summarize our contributions , we 1 ) introduce a shared spectral reconstruction loss to a VAE-GAN architecture for synthetic band generation ; 2 ) test our methodology on real-world scenarios ; 3 ) present and release a test dataset of four hemispheric snapshots from three publicly available geostationary satellites for future research . In the following sections , we will introduce related work in remote sensing and image-to-image translation , describe the architecture , and review experiments . Lastly , we will discuss the implications on this work and conclude with future directions . 2 BACKGROUND . Remote Sensing . Current generation GEO satellites observe 16 spectral bands over large regions every 10-15 minutes at a 0.5-2km resolution . At a sub-optimal 2km , this produces full-disk images of size 5,424×5,424×16 which causes storage constraints while being computationally expensive to process . Physical and statistical models are used to convert these images into more easily interpreted variables such as precipitation , cloud cover , and surface temperature ( 30 ) . Multiple GEO satellites , currently in orbit , extend the spatial ranges to actively monitoring larger regions . However , differences in spectral bands and sensor uncertainties/biases present challenges to commonly used sensor specific models , especially existing downstream models do not generalize well to missing spectral information . Neural models have long been applied to process remote sensing data and generate downstream products . Hsu et al . ( 12 ) presented some of the first work that showed neural networks ( NNs ) could generate accurate and high-resolution precipitation products from satellite observations . In recent years , convolutional neural networks ( CNNs ) have been found to further improve this task ( 27 ) . Similarly , CNNs have successfully been applied to poverty mapping ( 15 ) , super-resolution ( 18 ) , subpixel classification ( 20 ) , model emulation ( 6 ) , and land-cover classification ( 2 ) , all from low-level satellite products . In terms of spectral synthesis , few studies have explored reconstruction of hyperspectral bands from RGB bands with supervised approaches ( 32 ; 1 ) . While many of these problems are within the class of image-to-image translation , they generally assume labels are widely available and focus on individual sensors . To the best of our knowledge , no studies have developed approaches to synthesize spectral information by learning across satellites in the unsupervised setting . Image-to-Image Translation . Many problems can be defined as an image-to-image translation task including super-resolution , style transfer , and colorization . Approaches to image-to-image translation have been developed for both supervised and unsupervised settings to map images from one domain to another . In the supervised setting , image pairs are available to learn a direct mapping from one to the other . Generative adverserial networks have been shown to be highly successful at this task ( 14 ; 33 ) . Numerous unsupervised learning methods have been developed for the common case of large unpaired datasets ( 22 ; 42 ; 39 ; 23 ) . CycleGAN , for instance , proposed an approach to directly map from one domain to another and back by incorporating a cycle consistency loss with a GAN ( 42 ) . UNIT ( 22 ) proposed a probabilistic approach that uses an intermediate latent space between domains with a Variational Autoencoder ( VAE ) ( 16 ) and GANs ( 8 ) . In contrast to prior work on image-to-image translation our scenario specifically requires spectral translation and across multiple domains . Rather than translating between relatively low-dimensional RGB images and segmentation maps , as is found in traditional multimodal image-to-image translation ( 43 ; 4 ; 13 ) , satellite imagery contains tens to hundreds of spectral bands . Domain adaptation is another area of active research which also considers the case of effectiveness in unseen environments with cycle consistency and domain invariant ( 11 ; 5 ) . ( 17 ) using a shared content loss to translate between RGB image styles . ( 28 ) presented an application of image-to-image translation for 4-band Sentinel-4 images between different times of day . Our approach is based on the proven fundamental techniques of learning a shared latent space using cycle consistency and adverserial losses extended in the spectral dimension . We also use the prior understanding of spatial consistency between domains to implement a partial skip connection . 3 APPROACH . VAEs and GANs are effective for image-to-image translation where pairs of images are not available ( 22 ) . This is the case with for satellites with no space-time overlap . However , as in ( 22 ) , a shared latent variable z can be used to approximate the joint distribution from marginals . An adverserial loss applied to cross reconstructions satisfies the shared latent space assumption but is under-constrained for high-dimensional , multi-spectral images . We shall observe that this leads to large errors in our task . To address this , we introduce a shared spectral reconstruction loss and skip connection to effectively generate synthetic spectral bands ( see Fig . 1 ) , the result is a 50-80 % reduction in mean absolute error . In the spectral domain , we consider the case of K satellites , S = { S1 , S2 , ... , SK } , such that Sk ∈ RH×W×Bk is a set of Bk spectral bands with height H and width W , illustrated as a Venn diagram in Fig . 1 . The union of all sets , ∪Ki=1Sk , represents the complete set of spectral channels in the data . We denote the intersection of two spectral sets as overlapping bands . Our goal is to generate synthetic bands will where Si ∩ Scj 6= ∅ for ∀ ( i , j ) where c denotes the complement . A shared latent variable z is modeled with a Gaussian prior to learn a general representation for mapping between sets such that the assumptions of shared spectral reconstruction , weight sharing , cycle consistency , and cross-domain adverserial losses are satisfied . VAE-GAN . For a given spectral set k , we define encoder-generator pairs { Ek , Gk } such that q ( zk|sk ) = N ( Ek ( sk ) , I ) and ŝk→kk = Gk ( zk ∼ qk ( zk|sk ) ) for a sk ∈ Sk . For any set j , ŝ k→j k corresponds to reconstruction from set k to j . The set of encoders { E1 , E2 , ... , Ek } share their last layer of weights to constrain the latent space to high-level representations . Using prior pη ( z ) ∼ N ( 0 , I ) , the VAE likelihood is defined as : LV AEk ( E , G ) = λ1KL ( qk ( zk|sk ) ||pη ( z ) ) − λ2Ezk∼qk ( zk|sk ) [ logpGk ( sk|zk ) ] . ( 1 ) Distributions pGk are modeled as Laplacian distributions and a Gaussian latent space with prior z ∼ N ( 0 , I ) . GANs are used to enforce realistic spatial/spectral distributions of reconstructed images from the latent space . Discriminator networks D1 to Dk compare observations with cross reconstructions from the latent space . LGANk = λ3Esk∼PSk [ log Dk ( sk ) ] + λ3 ∑ j 6=k Ezj∼PSk [ log ( 1−Dk ( Gk ( zj ) ) ) ] . ( 2 ) Cycle Consistency . VAE and GAN losses are under-constrained and do not satisfy the shared latent space constraint alone . As in ( 22 ) , a cycle consistent loss is used such that sk = Fj→k ( Fk→j ( sk ) ) for all satellite pairs ( j , k ) and where Fk→j ( sk ) = Gj ( Ek ( sk ) ) . The loss between sk and cycled reconstruction ŝk→j→kk is written as : LCCk→j ( Ek , Ej , Gk , Gj ) =λ4KL ( qk ( zk|sk ) ||pη ( z ) ) + λ4KL ( qj ( zj |s k→j k ) ||pη ( z ) ) − λ5Ezj∼qj ( zj |sk→jk ) [ logp ( Gk ( sk|zj ) ) ] ( 3 ) With multiple domains , each domain should cycle through every other domain . The cycle-consistency loss for each permutation results in a complete cyclical graph . This loss is written as : LCCk = ∑ k 6=j LCCk→j ( Ek , Ej , Gk , Gj ) ( 4 ) Shared Spectral Reconstruction Loss . Adverserial losses can be easily fooled with increased dimensions . To help avoid this we introduce an additional loss , LSSRk . In this problem , if the intersection of spectral channels Sk , j = Sk ∩ Sj between domains is not empty then the difference between p ( sk→kk |zk ) and p ( sk→kk |zk ) can be minimized with KL divergence : LSSRk = λ6 ∑ j 6=k KL ( p ( s̃k→kk |zk ) ||p ( s̃ k→j k |zk ) ) ( 5 ) where s̃k ∈ Sk , j . The SSR loss encourages decoders to reconstruct identical spectral wavelengths with similar distributions while still synthesizing dissimilar bands . In this scenario , partial constraints are placed between domains and allows sampling of unobserved spectra from the shared latent space . By decreasing λ6 the bias between bands will be relaxed which may reduce the effect of more uncertain domains . Total Loss The likelihood is maximized by optimizing the GAN mini-max problem such that the generator aims to fool the discriminator , alternating updates between ( E , G ) and ( G , D ) . L = min E , G max D K∑ k=1 [ LV AEk + LCCk + LGANk + LSSRk ] ( 6 ) The hyper-parameters used correspond to those in ( 22 ) and set as λ1 = 1 , λ2 = 0.01 , λ3 = 1 , λ4 = 1 , λ5 = 0.01 , and λ6 = 0.1 . Adam optimization is used to train the networks for 200,000 steps with a batch size of 8 with parameters β1 = 0.5 , β2 = 0.999 and learning rate 1e− 5 . The reader can find detailed information in the supplementary material . Below we show the steps for generating a new band . Algorithm 1 : Generate a synthetic band by translating from one satellite to another Result : Synthetic spectral band Image sk from satellite k ; Encode to latent space z = Ek ( sk ) ; Decode to other satellite s̃j = Gj ( z ) ; Select synthetic band from s̃j ; Data . Three geostationary satellite imagery datasets , GOES-16 ( G16 ) , GOES-17 ( G17 ) , and Himawari-8 ( H8 ) are used in our experiments . Each satellite captures hemispheric ( full-disk ) snapshots from a constant vantage point over time but of different regions . Examples shown in supplement . Images contain 16 bands ( channels ) in the visible , near-infrared , and thermal spectrum at 0.5-2km spatial resolution . G16 and G17 have identical specifications viewing the east and west regions of North America and include two visible ( blue , red ) , four near-infrared ( including cirrus ) , and ten thermal infrared bands . H8 has 15 overlapping bands with G16/G17 viewing the Pacific Ocean and East Asia , this ensures similar information content . H8 captures three visible ( blue , green , red ) , three near-infrared ( missing cirrus ) , and the same ten thermal infrared bands as G16/G17 . Thus , the G16 and G17 bands all overlap , cirrus ( 1.37µm ) exists in G16 and G17 but is not in H8 and green ( 0.51µm ) exists in H8 but not in G16 or G17 . These differences cause difficulties when applying models relying on green or cirrus bands across satellite sets . G16 observes the North , Central and South Americas , capturing a good distribution of land and ocean . G17 observes the Pacific Ocean as well as most of North and Central America . However , G17 has known problems with its thermal cooling system causing the near-infrared and thermalinfrared channels to be unusable during periods of high heat and biased throughout ( 31 ) . This further highlights the gain in replacing low quality bands of G17 with a virtual sensor . Periods of high heat are filtered out of our training and test sets with quality control checks to eliminate temporal periods of known uncertainty . After quality control , considerable space-time overlap between G16 and G17 can be used for testing . H8 observes East Asia , Australia , and the Western Pacific , partially overlapping with G17 . Discrepancies are expected between sensors caused by different solar and sensor viewing angles but we are not aware of a more appropriate dataset for evaluation . The data generated by ( 37 ) is used which normalized G16 , G17 , and H8 to a common geo-referenced gridding system in order to facilitate intercomparisons and processed with the Bidirectional Reflectance Distribution Function ( BRDF ) . Bands have resolutions varying from 500m to 2km which we interpolate to a common sub-optimal resolution of 2km . Full-disk images are on a common grid with tiles of size 300× 300×16 . Training data is generated from the multi-petabyte datasets . We randomly sample images to build a well distributed and large training dataset from years 2018 ( G16 , H8 ) and 2019 ( G17 ) which totaled 359GB of data . Each tile is split into 64×64×16 non-overlapping patches for training , generating millions of samples . A test set including 500 random tiles from 25 days in February 2019 from overlapping G16 and G17 observations . The random set of tiles assures a range of solar angles , system patterns , and land cover types . Similarly , four tiles of data from G17 and H8 on January 2 , 2019 at 04:00UTC are selected to evaluate synthetically generated green and cirrus bands ( spatial overlap of G17/H8 is mostly ocean ) . This dataset will be made publicly available consisting of tiles from each satellite .

Summary: This paper proposes a new method for image-to-image translation on multi-spectral imagery. The proposed method uses variational auto-encoders and generative adversarial networks to generate synthetic bands in satellite imagery.  Novelty of the proposed method is that the authors introduce a shared spectral reconstruction loss and skip connection to generate synthetic spectral bands. This allows to generate synthetic bands with higher accuracy that the original image-to image translation method.