Spectral Synthesis for Satellite-to-Satellite Translation
Authors: Thomas Vandal
Daniel McDuff
Weile Wang
Andrew Michaelis
Ramakrishna Nemani
Venue:  (2021)

Abstract: Earth observing satellites carrying multi-spectral sensors are widely used to monitor the physical and biological states of the atmosphere, land, and oceans. These satellites have different vantage points above the earth and different spectral imaging bands resulting in inconsistent imagery from one to another. This presents challenges in building downstream applications. What if we could generate synthetic bands for existing satellites from the union of all domains? We tackle the problem of generating synthetic spectral imagery for multispectral sensors as an unsupervised image-to-image translation problem with partial labels and introduce a novel shared spectral reconstruction loss. Simulated experiments performed by dropping one or more spectral bands show that cross-domain reconstruction outperforms measurements obtained from a second vantage point. On a downstream cloud detection task, we show that generating synthetic bands with our model improves segmentation performance beyond our baseline. Our proposed approach enables synchronization of multispectral data and provides a basis for more homogeneous remote sensing datasets.

Under review as a conference paper at ICLR 2021
SPECTRAL SYNTHESIS FOR SATELLITE-TO-SATELLIYT TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Earth observing satellites carrying multi-spectral sensors are widely used to monitor the physical and biological states of the atmosphere, land, and oceans. These satel- lites have different vantage points above the Earth and different spectral imaging bands resulting in inconsistent imagery from one to another. This presents chal- lenges building downstream applications. What if we could generate synthetic bands for existing satellites from the union of all domains? We tackle the problem generating synthetic spectral imagery for multispectral sensors as an unsuper- vised imaee-to-image translation problem with partial labels and introduce novel shared spectral reconstruction loss. Simulated experiments performed by dropping or more spectral bands show that cross-domale reconstruction outperforms measurements obtained from a second vantage point. On a downstream cloud detection task, show that generating synthetic bands with our model improves segmentation performance beyond our baseline. Our proposed approach enables synchronization of multispectral data and provides basis for more homogeneous remote sensing datasets.
INTRODUCTION
Climate change and related environmental issues including the loss of biodiversity and extreme weather are listed by the World Economic Forum as the most important risks to our planet (7). Monitoring the Earth is critical mitigating these risks, understanding the effects, and making future predictions (38). Multi- and hyper-spectral satellite-based remote sensing enables global observation of the Earth, allowing scientists to study large-scale system dynamics and inform general circulation models (2)). In weather forecasts satellite data initializes the atmospheric state for future predictions] On longer time scales, these data are used to measure the effects of climate change such as land-cover variations, temperature trends, solar radiation levels, and the rate of snow/ice melt. In the coming decades, increased investments from the public and private sectors in satellite-based observations will continue improve global monitoring, as highlighted in NASA's decadal survey (55)
Satellites are designed based on specifications for a given set of applications with fiscal, technological, and physical constraints which limit their temporal, spatial, and spectral resolutions. Geostationary (GEO) satellites rotate with the Earth to stay over a constant position above the equator at a high elevation of 35,786km. This position enables GEO satellites with on- on- on-board multi-spectral imagers to take continuous and high-temporal snapshots over large spatial regions and are ideal for monitoring diurnal and fast moving events. Spectral bands measure brightness and radiance intensities of the electromagnetic spectrum at specified center wavelength and bandwidth. Bands are selected to satisfy defined variables of interest constrained by technological cost and accuracy. Applications of GEO sensors include atmospheric winds measurement (35), tropical cyclone tracking (36), wildfire monitoring (41,, and short-term forecasting (24). Multiple GEO satellites are needed to generate global high-temporal resolution datasets to better monitor these events around the world. However, variations in resolutions, sensor uncertainties, and temporal life spans leads to a set of separate datasets which are not consistent, making this process very challenging (26). Developing consistent and homogeneous global datasets would relieve many of these challenges.
Supplementary
Material:
ht http:://giitubb.cmmmmmmnoo -oor-ee-ar--------r----------
satell Teeeliteetto satel lieellitetttee

---

Under review as a conference paper at ICLR 2021
E
D
Si
LGAN
Lssn
E2
#2
34-12
LGAN
Lassi
Ex
Loc.
Gk
Dx
Sq
LGAN.
82
Shared WAIgHtS
LVAE.
Figure (Left) Network architecture for K satellites. Encoders (E&), decoders (Gx), and discriminators (Dx) networks with residual blocks. Losses terms highlighted in red. (Right) Venn diagram shows how spectral bands can overlap between pairs and multiple satellites.
The current generation of GEO satellites are exception. The GOES-16/17 satellites operated by NASA/NOAA (cost: $11 billion) have a set of 16 imaging bands covering the visible, near-, and thermal-infrared spectral range (29). The Himawari-8 satellite operated by the Japanese Space Agency (cost: $800 million) similarly has 16 bands but swaps a NIR (1.38um band for a green channel (0.51 enabling the construction of true color images (3). The 1.38pm band is ideal for measuring Cirrus clouds, composed of ice particles the upper troposphere, a major contributor to regulating the Earth's climate that is not yet well understood (1 IO). Without capturing this band, directly observing Cirrus clouds over Japan, East Asia, and Western Pacific region from Himawari-8 not possible. Synthetic observations via virtual spectral sensors could be low-cost solution to improving coverage availability and consistency with current satellites.
We present an approach to generate synthetic spectral channels from dataset. We treat satellites with either dissimilar spectral coverage or varying vantage points as separate spectral sets. In this way, the problem closely resembles that of colorization (40) and
unpaired satellite
image-to-image translation tasks (22 42 9 in the case where paired images are not available but with the added complexity of large number of spectral bands. We use a combination of variational autoencoder (VAE) and generative adverserial network (GAN) (8) architectures adapted to our problem to model a shared latent space, as in unsupervised image-to-image translation Generating synthetic bands is under-constrained problem that paired with adverserial loss in high dimensions promotes overfitting. Our approach mitigates these challenges by leveraging a weak supervision signal based partial overlap in spectral bands between domains. By including reconstruction loss on overlapping spectral bands between domain pairs we can substantially improve spectral band synthesis.
To summarize our contributions, we 1) introduce a shared spectral reconstruction loss to a VAE-GAN architecture for synthetic band generation; test our methodology real-world scenarios: 3) present and release test dataset of four hemispheric snapshots from three publicly available geostationary satellites for future research. In the following sections, we will introduce related work in remote sensing and image-to mmage-to-image tm-ge translation. describe the architecture, and review experiments. Lastly, we will discuss the implications on this work and conclude with future directions.
BACKGROUND
Remote Sensing. Current generation GEO satellites observe 16 spectral bands over large regions every 10-15 minutes at 0.5-2km resolution. At sub-optimal 2km, this produces full-disk images of size 5,422x5,4244 5,424x 16 which causes storage constraints while being computationally expensive to process. Physical and statistical models used to convert these images into variables such as precipitation, cloud cover, and surface temperature (30). Multiple GEO satellites, currently in orbit, extend the spatial ranges actively monitoring larger regions. However, differences
easily interpreted
2

---

Under review as a conference paper at ICLR 2021
in spectral bands and sensor uncertainnieesbiases present challenges to commonly used sensor specific models, especially existing downstream models do not generalize well to missing spectral information.
Neural models have long been applied to process remote sensing data and generate downstream products. Hsu et al. (12) presented some of the first work that showed neural networks (NNs) could generate accurate and high-resolution precipitation products from satellite observations. In recent years, convolutional neural networks (CNNs) have been found to further improve this task (27). Similarly, CNNs have successfully been applied to poverty mapping (15), super-resolution 18). subpixel classification (20), model emulation (6), and land-cover classification (2), all from low-level satellite products. In terms of spectral synthesis, few studies have explored reconstruction of hyperspectral bands from RGB bands with supervised approaches [.. While many of these problems are within the class of image-to-image translation, they generally assume labels are widely available and focus on individual sensors. To the best of our knowledge, no studies have developed approaches to synthesize spectral information by learning across satellites in the unsupervised setting.
Image-to- Image Translation. Many problems can be defined as an image-to-image translation task including super-resolution, style transfer, and colorization. Approaches to image-to-image translation have been developed for both supervised and unsupervised settings to map images from one domain to another. In the supervised setting, image pairs are available to learn a direct mapping from one to the other. Generative adverserial networks have been shown to be highly successful at this task (14 33). Numerous unsupervised learning methods have been developed for the common case of large unpaired datasets (22 42 39) 23). CycleGAN, for instance, proposed an approach directly
map from one domain to another and back by incorporating a cycle consistency loss with a GAN (42). UNIT (22) proposed a probabilistic approach that uses an intermediate latent space between domains with a Variational Autoencoder (VAE) (16) and GANs (8. In contrast to prior work on image-to-image translation our scenario specifically requires spectral translation and across multiple domains. Rather than translating between relatively low-dimensional RGB images and segmentation maps, as found in traditional multimodal image-to-image translation contains tens to hundreds of spectral bands. Domain adaptation is another area of active research which also considers the case of effectiveness in unseen environments with cycle consistency and domain invariant (1 5). (17 using shared content loss to translate between RGB image styles. (28) presented an application of mage-tt--image translation for 4-band Sentinel-4 images between different times of day. Our approach is based the proven fundamental techniques of learning shared latent space using cycle consistency and adverserial losses extended in the spectral dimension. We also use the prior understanding of spatial consistency between domains to implement partial skip connection.
13, satellite imagery
3 APPROACH
VAFs and GANs are effective for image-to- image translation where pairs of images are not available (22). This the case with for satellites with no space-time overlap. However, as in (22). a shared latent variable = can be used to approximate the joint distribution from marginals. An adverserial loss applied to cross reconstructions satisfies the shared latent space assumption but under-constrained for high-dimensional, multi-spectral images. We shall observe that this leads to large errors in our task. To address this, we introduce a shared spectral reconstruction loss and skip connection to effectively generate synthetic spectral bands (see Fig. D. the result is a 50-80% reduction in mean absolute error.
In the spectral domain, we consider the case of K satellites, S S2,, (k)...,S),, such that Se E RHxWxBB is a set of B& spectral bands with height H and width W, illustrated as a Venn diagram in Fig. I The union of all sets, S represents the complete set of spectral channels in the data. We denote the intersection of two spectral sets as overlapping bands. Our goal is to generate synthetic bands will where S, S # for (i. where e ddenotes the complement. A shared latent variable is modeled with Gaussian prior to learn general representation for mapping between sets such that the assumptions of shared spectral reconstruction, weight sharing, cycle consistency, and cross-domain adverserial losses are satisfied.
VAE-GAN. For given spectral set k. we define encoder-generator pairs {Ex, G&) such that q(2 (st) = (N (sx), I) and s% gh>> = GG (2x ~ (x<x)ss)) for a 8y E S₁. For any set j. st gk-s cor- responds to reconstruction from set k j. The set of encoders (F,, Fey..., :.. F&) share their last layer
3

---

Under review as a conference paper at ICLR 2021
of weights to constrain the latent space to high-level representations. Using prior p., NN the VAE likelihood is defined as:
AV G)
AgE (sw) (ss)) (sx [x)]
(1)
Distributions pG, are modeled as Laplacian distributions and a Gaussian latent space with prior (O,, I). GANs are used to enforee realistic spatial/spectral distributions of reconstructed images from the latent space. Discriminator networks D1 to Dj compare observations with cross reconstructions from the latent space.
LGAN, Xz [log DR(sx)] + Xa
Σ Ps log (1 De(GG( jæh
Cycle Consistency. VAE and GAN losses are under-constrained and do not satisfy the shared latent space constraint alone. As in (22), cycle consistent loss is used such that 8&: (Fk for all satellite pairs (j, k) and where FR. FR- (8x) = G5(Fx((A)) The loss between Sy and cycled
reconstruction $% k is written as:
Loou vs (ER, Ej, Gk, G2) =N, 8x) (z)) + (4 (1p,)))
(3)
As (2,la (bx
With multiple domains, each domain should cycle through every other domain. The cycle-consistency loss for each permutation results in complete cyclical graph. This loss is written as:
Loo, Σ Lock., (Ex, Ej, G3,
(4)
knj
Shared Spectral Reconstruction Loss. Adverserial losses can be easily fooled with increased dimensions. To help avoid this we introduce an additional loss, LSSR,- In this problem, if the intersection of spectral channels Sk.j = Ss n nSy between domains is not empty then the difference
between p and p(s k-tk (zn) can be minimized with KI. divergence:
LSSRL A
1= ((x))
(5)
jØk
where $ E Sk.j. The SSR loss encourages decoders to reconstruct identical spectral wavelengths with similar distributions while still synthesizing dissimilar bands. In this scenario, partial constraints are placed between domains and allows sampling of unobserved spectra from the shared latent space. By decreasing A the bias between bands will be relaxed which may reduce the effect of more uncertain domains.
Total Loss The likelihood is maximized by optimizing the GAN mini-max problem such that the generator aims to fool the discriminator, alternating updates between (E, G) and (G, D).
K
== = max F.G D
Σ [CV + Loc, CGAN, + LssRg] A=1
(6)
The hyper-parameters used correspond to those in (22 and set Å₁ = 1, A2 03 11
1, As = 0.01, and Åg = 0.1. Adam optimization used to train the networks for 200,000 steps with batch size of 8 with parameters 81 0.5, B2 0.999 and learning rate le 5. The reader find
detailed information in the supplementary material. Below we show the steps for generating new band.
Algorithm 1: Generate a synthetic band by translating from one satellite to another
Result: Synthetic spectral band
Image 85 from satellite k; Encode to latent space Eg (sk);
Decode to other satellite &, = G,(z);
Select synthetic band from šy;
Data. Three geostationary satellite imagery datasets, GOES-16 (G16), GOES-17 (G17), and Himawari-8 (H8) are used in experiments. Each satellite captures hemispheric (full-disk)
4

---

Under review as a conference paper at ICLR 2021
A1
83
Scree
30-
09.4
1964
(.00
0..00
11.0
(2.0)
0..0 09:33
15090
0.30
(5.4)
13.50
D6.0
01.00
10.00
D53)
Band Cooner Meveinnghh jm]
Sand (Cartor Wisollnghh uor)
Figure 2: Mean absolute error (MAB) of a substitute sensor (GT observations from a separate satellite), synthetic sensor without SSR, synthetic sensor without skip, synthetic sensor, and reconstructed sensor (reconstructed images from a full model with access to all bands) for each band. The wavelength of each band is shown below the band number. A) include visible and near infra-red (no physical units) and B) includes thermal infrared measured in Kelvin.
snapshots from constant vantage point over time but of different regions. Examples shown in supplement. Images contain 16 bands (channels) in the visible, near-infrared, and thermal spectrum at 0.5-2km spatial resolution. G16 and G17 have identical specifications viewing the east and west regions of North America and include two visible (blue, red), four near-infrared (including cirrus), and ten thermal infrared bands. H8 has 15 overlapping bands with G16/G17 viewing the Pacific Ocean and East Asia, this ensures similar information content. H8 captures three visible (blue, green, red), three near-infrared (missing cirrus), and the same ten thermal infrared bands as G16/G17. Thus, the G16 and G17 bands all overlap, cirrus (1.37um) exists in G16 and G17 but not in H8 and green (0.51 exists in H8 but not in G16 or G17. These differences cause difficulties when applying models relying on green or cirrus bands across satellite sets.
G16 observes the North, Central and South Americas, capturing a good distribution of land and ocean. G17 observes the Pacific Ocean as well as most of North and Central America. However, G17 has known problems with its thermal cooling system causing the near-infrared and thermal- infrared channels to be unusable during periods of high heat and biased throughout (31). This further highlights the gain in replacing low quality bands of G17 with a virtual sensor. Periods of high heat are filtered out of our training and test sets with quality control checks to eliminate temporal periods of known uncertainty. After quality control, considerable space-time overlap between G16 and G17 can be used for testing. H8 observes East Asia, Australia, and the Western Pacific, partially overlapping with G17. Discrepancies are expected between sensors caused by different solar and sensor viewing angles but we are not aware of a more appropriate dataset for evaluation. The data generated by (37) is used which normalized G16, G17, and H8 to a common geo-referenced gridding system in order to facilitate intercomparisons and processed with the Bidirectional Reflectance Distribution Function (BRDF). Bands have resolutions varying from 500m to 2km which we interpolate to a common sub-optimal resolution of 2km. Full-disk images 300 x 300 16. Training data is generated from the multi-petabyte datasets. We randomly sample images to build well distributed and large training dataset from years 2018 (G16.H8) and 2019 (G17) which totaled 359GB of data. Each tile is split into 64x64x 16 non-overlapping patches for training. generating millions of samples.
common grid with riles of size
A test set including 500 random tiles from 25 days in February 2019 from overlapping G16 and G17 observations. The random set of tiles assures a range of solar angles, system patterns, and land cover types. Similarly, four tiles of data from G17 and H8 on January 2, 2019 at 04:00UTC are selected to evaluate synthetically generated green and cirrus bands (spatial overlap of G17/H8 is mostly ocean). This dataset will be made publicly available consisting of tiles from each satellite.
4 EXPERIMENTS AND DISCUSSION
In this section present set of experiments to explore the properties of our approach by testing which bands can be robustly synthesized, how many bands can be generated, how effectively the proposed loss performs, and the ability perform downstream tasks.
5

---

Under review as a conference paper at ICLR 2021
Cross Satellite Band Synthesis. Our experiments start with testing how well each spectral band can be synthesized. To do this we remove individual bands from one satellite (G16) during training, synthesize these bands and compare with the ground-truth observations. We use the full set of bands from the other two satellites during training. This approach is applied on G16 such that each model takes 15 bands of G16 and 16 of G17 and H8.
Three comparisons are used to help put the accuracy of synthesized bands for G16 into context. Ours refers to bands generated using our proposed approach with both SSR loss and skip connection. Ours w/o SSR refers to bands generated with our proposed approach without the shared spectral reconstruction loss. Ours w/o Skip refers to bands generated with our proposed approach without the skip connection between the input and generator. UNIT refers to the unsupervised image-to- image translation baseline as presented connection. Sensor refers to the performance if we simply use overlapping observations from another satellite (G17), this acts as our lower bound in performance and is actually the status-quo (essentially substituting the missing band with images from the same band but from another satellite, which as we shall see is a sub-optimal solution). Reconstruction refers to the images reconstructed from a full model trained on all satellites with no missing bands, this acts our upper bound in performance. Each of these signals are computed on our test set of 500 overlapping overlapping tiles from G16/117. Fig. shows the mean absolute error (MAE) for each condition. Table I shows the average MAE for VIS/NIR and TIR of each method.
(22) and equivalent to ours without SSR and skip
The MAE in the sensor condition is substantial and largely caused by clouds/aerosols in the vertical direction (see gif in supplementary material). On the other side, synthetically generating bands using our approach substantially reduces error by over 60% compared to both this baseline and UNIT(see Table I. Similarly, synthesized bands also improve upon the view from G17 even though during training they did not see examples of the corresponding band from G16. Overall, the reconstructed and synthesized images have similar Eegnaltt-noiss ratios. Ablation experiments removing the shared spectral reconstruction loss and the skip connection show their ef- fectiveness. SSR is critical to learning a robust latent space and the skip connection improves both VIS/NIR and TIR predictions. We observe that without introducing the SSR loss, performance is even worse than the sensor baseline. From this we learn that applying an existing image-to-image translation model (22) to our task, without adaptation, performs poorly. We find that band the shortwave infrared band (3.9um)), is particularly difficult to synthesize with MAE significantly above that of the full reconstruction. This result suggests that the shortwave infrared band captures information which cannot be inferred from the others. Notice how the wavelength gap between bands 7 and 8 is relatively large (2.3um), this may explain why the performance is poor. In the future, a similar analysis could be used to inform future satellite design configurations.
Table
The MAE for VIS/NIR and TIR bands by method.
Method
VIS/NIR
TIR
Cross-Sensor
.186
4.45
UNIT
.183
12.64
Ours w/o SSR
.101
7.84
Ours w/o Skip
.049
1.68
Ours
.048
1.48
Reconstruct
.036
1.14
We show qualitative examples of generating synthetic bands in Figs. 3b andand 4 Two images are shown in Fig. (3b including a false color image, composed of hear-infrared, red, and blue bands, and a true color image from synthetically generated green band and real red and blue bands. This process is applied to Himawari-8 to generate a cirrus band (shown in Fig. 4 While there may be challenges in synthetically generating all bands, most can be reconstructed with high signal-to-non
ratio and this suggests that our approach could be used to make software updates to current satellite datasets.
Synthesizing Multiple Bands. Generating synthetic channels from satellites with limited number of spectral bands could be of significant value for long-term analysis. For example, older generation satellites often have fewer channels and could provide greater utility in downstream tasks if it was possible to generate images in additional frequency bands. Therefore, we set up an experiment to test how many additional bands can be synthesized reliably and how many initial bands are required. A set of synthesis models were trained on G16, removing bands one by one until just one band was left and while keeping all 16 G17 and H8 bands. For simplicity, and reduce computation, we dropped bands in a fixed order: 2,4,13,2,15,12,66, 10.8, 14,5,1 1,7,16. In the most extreme case we use visible band and attempt to synthesize the remaining 15. As above, results are computed on the test set of 59 overlapping G16 and G17 tiles. The results presented in Fig. Ba (left) show how the number of available input bands effects the MAE for VIS/NIR (bands 1-6) and TIR (bands 7-16). As expected, MAE falls more or less monotonically as more bands are given inputs. When just two bands, 1
6

---

Under review as a conference paper at ICLR 2021
0.30
0.20
(iii
0.15
VIS/NIR
0.15
VIS/NIR
MAE (Rell)
0.10
0.10
0.05
0.05
0.00
0.00
10
10
(iv)
8-
TIR
8-
TIR
MAE (K)
44
2
4 6
10
12
14
14 Number of Shared Bands
4
6
10 12
Number Input Bands
(a)
(b)
Figure 3: (a) Horizontal lines correspond to the sensor signal and eтrоr bars represent 95% confidence intervals of tile MAE. (i-ii) show results of an experiment synthetically generating more and more bands. (iii-iv) show MAE function of number of bands shared between spectral sets. (i and iii) correspond to visible/near-infrraed and (ii and iv) to thermal-infrared. (b) False (top) and true (bottom) color images generated by GOES-17 on January 2019.
(blue) and 16 (TIR), are used as inputs, the synthetic TIR reconstruction of G16 still has lower error than the observed sensor difference between G16 and G17. These results show that few bands are needed to synthesize images that improve upon the status quo. In the TIR range, we find that MAE plateaus after 3-4 bands are used as inputs. These results suggest that the information content in a subset of bands may be sufficient for many applications. However, should be prepared that some bands may contain specific information useful for monitoring rare events. Overall, these results show that good proportion of bands can be synthesized remarkably well.
Sharing Spectral Losses. The effectiveness of the shared spectral reconstruc- tion loss tested by gradually increasing the number of shared bands included in the loss one by one. Mathematically. this corresponds to the number of bands included in the set Sk.j. In all runs, 16 bands of G16, G17, and H8 are used even if ignored by the SSR loss. Fig. [3 (afiii,iv) shows the effect of adding shared bands during training leads to dramatic decrease in MAE. Cor- responding cross-sensor signals are shown as horizontal lines. In this setting, we find using the SSR loss critical to learning this model. Sharing two spectral bands in the loss function improves the signal and is almost all that is needed for accurate reconstruction. This further reinforces our insight above that large amount of the information is captured in just few spectral bands. In Table 2 we further explore the SSR loss by testing a range of values for A
Table 2: The MAE for VIS/NIR and TIR bands by As-
AS
VIS/NIR TIR
0.01
.068
5.03
0.10
.048
1.48
1.0
.042
1.34
10.0
.049
1.28
from 0.01 to 10. Our results suggest that increasing SSR weighting factor improves performance on the test set.
Synthesizing Cirrus for Himawari-8, As discussed above, the cirrus band (1.38 µu) monitors ice particles in the upper troposphere which regulate the climate, and H8 is missing this band. These ice particles are often seen as thin clouds high in the atmosphere which may be viewed in the visible range, along with other clouds. To generate a synthetic cirrus band, an H8 observation is translated to G17. In Fig. 4 we show five images where G17 and H8 have space time overlap. G17 observations of false color and cirrus bands present baseline. The observed H8 true color depicts the same scene and corresponding synthetic cirrus band. This scene consists of clouds of multiple types and atmospheric heights on January 2, 04:00UTC. Cirrus clouds are found high in the atmosphere and are seen as thin or wispy (see lower right portion of the images). Comparing images 4b and 4d shows the similarity between synthetic bands and observations. Lower-level clouds, which can be seen on the lower right of the images, are ignored by both the observed and synthetic cirrus bands. Fig. 44 also shows the corresponding latent space from H8 where highlight the feature maps corresponding
7

---

Under review as a conference paper at ICLR 2021
(a) G17 False Color
(b) G17 Cirrus
(c) H8 True Color
(d) H8 Synthetic
(e) Latent features
Figure 4: Synthetically sensing cirrus from Himawari-8 using GOES-17 17. Images are taken from the test set (January 2, 2019 at 4:00UTC), (a) G17 false color image (NIR.R.B) (b) observed cirrus, (c) corresponding true color H8 image and (d) synthetically generated cirrus. Latent features are shown in (e) with red boxes highlighting feature maps capturing cirrus clouds.
the cirrus band. These suggest that our approach has learned to distinguish that clouds of different types are visible from different bands, a particularly important result.
Cloud Detection. The goal of generating synthetic satellite datasets is enable us to run downstream applications and models. Cloud/aerosol segmentation is one important task. To demonstrate this we take learned cloud detection model that is dependent on H8 bands 1-4, including the green band. However, the green band is not available from G16/G17 and therefore cannot directly use the cloud detection model with these data, we need to syn- thesize the green band first. For background, the cloud detection model was trained using labels for clouds and aerosols from dataset gener- ated with a physically- -based land surface model tuned specifically for H8 (I9). With this training dataset, a model can be trained to perform cloud detection (19), specifically we use Bayesian convolutional neural network as introduced in 34 6 for the model.
Examplé
Example
Example
Example
Observation
Falsse
Synthetic
Cloud Reconstructio
Now we can compare the cloud detection ac- curacy with and without H8's green band and with a synthetic green band. We take version of our synthesis model trained to synthesize a green band. Our model is able translate be- tween H8 views with and without green bands. Quantitatively, we compute the area under the receiver operating characteristic curve (AUC) when comparing labels to segmented probabil- ities over 100 random samples. On the upper limit, observed and reconstructed AUCs are 0.93 and 0.89, respectively. The AUC falls to 0.73 using a false green band versus 0.88 for the syn- thetically generated green.
Figure 5: Cloud segmentation of four Himawari-8 observations. From top to bottom, rows show ob- servation and cloud mask labels followed by false color, synthesized, reconstructed, and observed segmentation. Yellow denotes cloud pixels and blue non-cloudy pixels.
In Fig. 5 we show qualitative examples of em- ulated cloud detection results. The first row shows the inputs as true color observation with visible clouds and the second shows the physically generated cloud/aerosol labels. As baseline, false green is generated as the average of red and blue bands (R, R+ 2 BB) and used for cloud detection. The fourth row shows the results using our synthetic green band translating between 15-band H8 and 16-band H8. Similarly, the last two rows show results when using reconstructed and observed bands for segmentation, respectively. Segmentation from the false color images often produces unclear results and overestimates cloud cover. In contrast, the
8

---

Under review as a conference paper at ICLR 2021
synthetically generated bands produce segmentation maps that look nearly identical to reconstructed and observed examples. In the last column, we show an example of our approach acting as a denoiser to improve cloud detection even beyond the observed cloud segmentation. In sum, results suggest that synthetic data generated from our approach is applicable and may even have the potential to improve downstream tasks.
Limitations. While the VAE-GAN architecture performs well overall, it does present some limita- tions. VAEs aim to explicitly model the data as multivariate Gaussian and often produces blurry outputs. The GAN counteracts this effect by discriminating between real and generated images. However, there concem that this reduces data precision and fails detect rare and anomalous events which may effect scientific applications. Extending our work to use normalizing flows, as in (9, may reduce this limitation.
CONCLUSION
We have presented an unsupervised learning approach for used to synthesize unobserved spectral bands. A novel shared spectral reconstruction loss is presented to further constrain learning and conserve spectral information and partial skip connection maintains spatial consistency. Experiments with sensors the GOES-16/1 and Himawari-8 satellites show that synthetic spectral bands can be generated through reconstruction from a shared latent space. For the first time, we able to generate true color images from GOES-16/T and the eirrus band from Himawari-8, generating further value from these satellites. Further, cloud detection model is used show the applicability synthetically generated bands for downstream tasks, Future work may consider conditioning the shared latent space with known physical properties and extending to additional tasks.
translation that can be
REFERENCES
[1] Naveed Akhtar and Ajmal Mian. Hyperspectral recovery from rgb images using gaussian processes. IEEE transactions on pattern analysis and machine intelligence, 42(1):100-113, 2018.
[2] Saikat Basu, Sangram Ganguly, Supratik Mukhopadhyay, Robert DiBiano, Manohar Karki, and Ramakrishna Nemani. Deepsat: a learning framework for satellite imagery. In Proceedings of the 23rd SIGSPATIAL international conference advances in geographic information systems, pages 1-10, 2015.
[3] Kotaro Bessho, Kenji Date, Masahiro Hayashi, Akio Ikeda, Takahito Imai, Hidekazu Inoue, Yukihiro Kumagai, Takuya Miyakawa, Hidehiko Murata, Tomoo Ohno, et al. An introduction to 1armawari-/9-jaapae 's new-generation geostationary meteorological satellites. Journal of the Meteorological Society of Japan. Ser. II, 94(2):151-183, 2016.
[4] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wor Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-imade translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8789-8799 879998877,200 2018
[5] Weijian Deng, Liang Zheng, Qixiang Ye, Guoliang Kang, Yi Yang, and Jianbin Jiao. Image- image domain adaptation with preserved self-similarity and re-identification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 994-1003, 2018
for person
[6] Kate Duffy, Thomas J Vandal, Shuang Li, Ramakrishna R Nemani, and Auroop Ganguly. Deep learning emulation of atmospheric correction for geostationary sensors. 2019.
[7] World Economic Forum. Global risks report 2020. 2020.
[8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.

---

Under review as a conference paper at ICLR 2021
[9] Aditya Grover, Christopher Chute, Rui Shu, Zhangjie Cao, and Stefano Ermon. Align- flow: Cycle consistent learning from multiple domains via normalizing flows. arXiv preprint 190.::9900.//892,, 2019.
[10] Andrew J Heymsfield, Martina Krämer, Anna Luebke, Phil Brown, Daniel Cziczo, Charmaine Franklin, Paul Lawson, Ulrike Lohmann, Greg McFarquhar, Zbigniew Ulanowski, et al. Cirrus clouds. Meteorological Monographs, 58:2-1, 2017.
[11] Judy Hoffman, Eric Tzeng, Taesung Park, Jan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference machine learning, pages 1989-1998, 2018.
[12] Kou-lin Hsu, Xiaogang Gao, Soroosh Sorooshian, and Hoshin V Gupta. Precipitation estima- tion from remotely sensed information using artificial neural networks. Journal of Applied Meteorology, 36(9):1 177-11990 1997.
[13] Xun Huang. Ming- Yu Liu. Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 172-189, 2018.
[14] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1125-1134 1134, 2017.
[15] Neal Jean, Marshall Burke, Michael Xie, W Matthew Davis, David B Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty. Science, 353(630 794, 2016.
[16] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13 14, 2013.
[17] Hsin- Ying Lee, Yung-YY Tseng, Jia-Bin Huang, Maneesh Singh, and Ming- Yang. Diverse image-to- mmage-to-image image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pages 35 51, 2018.
[18] Sen Lei, Zhenwei Shi, and Zhengxia Zou. Super-resolution for remote sensing images via local- global combined network. IEEE Geoscience and Remote Sensing Letters, (188):243-1244 2017.
[19] Shuang Li, Weile Wang, Hirofumi Hashimoto, Jun Xiong, Thomas Vandal, Jing Yao, Lexiang Qian, Kazuhito Ichii, Alexei Lyapustin, Yujie Wang, et al. First provisional land surface reflectance product from geostationary satellite himawari-8 ahi. Remote Sensing, 11(24):2990, 2019.
[20] Feng Ling, Doreen Boyd, Yong Ge, Giles M Foody, Xiaodong Li, Lihui Wang, Yihang Zhang, Lingfei Shi, Cheng Shang, Xinyan Li, et al. Measuring river wetted width from remotely sensed imagery at the subpixel scale with a deep convolutional neural network. Water Resources Research, 55(7):5631-5649) 2019.
[21] Kuo-Nan Liou. Influence of cirrus clouds on weather and climate processes: A global perspec- tive. Monthly Weather Review, 1 114(6): 167-1199, 1986.
[22] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image to--tage translation networks. In Advances in neural information processing systems, pages 700-708, 2017.
[23] Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Fho- unsupervised image-to- -onggeet--maeeee translation. In Proceedings of the IEEE Interna- lional Conference on Computer Vision, pages 10551-10560, 2019.
[24] W Paul Menzel, Frances C Holt, Timothy J Schmit, Robert M Aune, Anthony J Schreiner, Gary S Wade, and Donald G Gray. Application of goes-8/9 soundings to weather forecasting and nowcasting. Bulletin of the American Meteorological Society, 79(10)2059-2277 1998.
10

---

Under review as a conference paper at ICLR 2021
[25] National Academies of Sciences, Engineering, and Medicine. Thriving Our Changing Planet: Decadal Strategy for Earth Observation from Space. The National Academies Press, Washington, DC, 2018.
[26] Jonathan T Overpeck, Gerald A Meehl, Sandrine Bony, and David R Easterling. Climate data challenges in the 21st century. science, (10166600000700000 2011.
[27] Mojtaba Sadeghi, Ata Akbari Asanjan, Mohammad Faridzad, Phu Nguyen, Kuolin Hsu, Soroosh Sorooshian, and Dan Braithwaite. Persiamn-enn Precipitation estimation from remotely sensed information using artificial neural eteorology, 20 20(12))2273-2283 2019.
neural networks. Journal of Hydrom-
[28] Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled repre- sentations via mutual information estimation. arXiv preprint arXiv: drXiv:192..33915. 2019.
[29] Timothy J Schmit, Paul Griffith, Mathew M Gunshor, Jaime M Daniels, Steven Goodman, and William J Lebair. A closer look at the abi on the goes-r series. Bulletin of the American Meteorological Society, 98(4))681-66) 2017.
[30] Timothy J Schmit, SCOTT S Lindstrom, JORDAN J Gerth, and MATHEW M Gunshor. Appli- cations of the 16 spectral bands on the advanced baseline imager (abi). 2018.
[31] GOES-R Series. Goes-17 abi performance. https//www..eesrrgo/iiee//ssos performance.hun (accessed may 30, 2020).
[32] Zhan Shi, Chang Chen, Zhiwei Xiong, Dong Liu, and Feng Wu. Hsenn+: Advanced cnn-based hyperspectral recovery from rgb images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 939-947, 2018.
[33] Hao Tang, Dan Xu, Nicu Sehe, Yanzhi Wang, Jason J Corso, and Yan Yan. Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2417-2426, 2019.
[34] Thomas Vandal, Evan Kodra, Jennifer Dy, Sangram Ganguly, Ramakrishna Nemani, and Auroop R Ganguly. Quantifying uncertainty in discrete-coniinouus and skewed data with bayesian deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2377-2335 2018.
[35] Christopher S Velden, Christopher M Hayden, Steven W Nieman, W Paul Menzel, Steven Wanzong, and James S Goerss. Spper-tropospherr winds derived from geostationary satellite water vapor observations. Bulletin of the American Meteorological Society, 78(2):173-196, 1997.
[36] Christopher S Velden, Timothy L Olander, and Raymond M Zehr. Development of an objective scheme to estimate tropical cyclone intensity from digital geostationary satellite infrared imagery. Weather and Forecasting, 1721)1172-186, 1998.
[37] Weile Wang, Shuang Li, Hirofumi Hashimoto, Hideaki Takenaka, Atsushi Higuchi, Satya Kalluri, and Ramakrishna Nemani. An introduction to the geostationary-nase earth exchange (geonex) products: top-of-atmosphere reflectance and brightness temperature. Remote Sensing, 12(8):1267) 2020.
[38] Michael A Wulder and Nicholas C Coops. Satellites: Make earth observations open access. Nature News, 513(7516):30, 2014.
[39] Zili Yi, Hao Zhang. Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision, pages 2849-2857, 2017.
[40] Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016.
11

---

Under review as a conference paper at ICLR 2021
[41] Xiaoyang Zhang, Shobha Kondragunta, Jessica Ram, Christopher Schmidt, and Ho-Chun Huang. Near-real-time global biomass burning emissions product from geostationary satellite constellation. Journal of Geophysical Research: Ammoshhrre, 117(D14), 2012.
[42] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223 2223-2232, 2017.
[43] Jun- Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei Efros, Oliver Wang. and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pages 465-476, 2017.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 TEST IMAGES
Figure 6: Full Disk Image of GOES-16 at 20:00 UTC on January 1 2019 False Color Red,, You may include other additional sections here.
13

---

Under review as a conference paper at ICLR 2021
Figure 7: G7ES-17 Coverage overlapping GOES-16 16 at 20:00 UTC on January 1 2019 False Color
Bluu
14

---

Under review as a conference paper at ICLR 2021
Figure 8: Full Disk Image of Himawari-8 at 04:00 UTC on January 2 2019 True Color (lee,
15

---

Under review as a conference paper at ICLR 2021
Figure 9: G0ES-17 Coverage overlapping Himawari-8 04:00 UTC on January 2020 False Color Bed,
16


Reviews:
[{"review_id": "uMdZM4XjYIQ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applied the unsupervised VAE-GAN model for satellite-to-satellite translation to generate synthetic spectral bands. My general evaluations of this paper are:\npros: The application is meaningful. This paper exploits the possible application of VAE-GAN based image-to-image translation model to the generation of synthetic spectral bands. Of course, it will bring new solutions for the subsequent application of satellite images. From the method perspective, the proposed Shared Spectral Reconstruction Loss (SSRS) has been testified to significantly improve the final spectral band simulation results.\ncons: From the model perspective, the only shining point compared to the previous model is the introduction of SSRS loss.  The notations are not clear. In the experiments, the comparison methods are missing. It is hard to distinguish the performance of the proposed model.\n\nMore general comments are listed as the following.\nThere are so many notations in the proposed method introduction,  however, the meaning of notations to the satellite images are not clear. I think most notations are from ref[23], but the most contribution of this paper is the application to satellite images. It may be more meaningful to introduce the notations from the satellite images perspective.\n\nThe experimental dataset is complex, but the introduction is insufficient.  It is difficult to make the readers understand the task of the experiment. I suggest to enhance the introduction of the problem and formulate the mathematical model from satellite-to-satellite translation clearly.\n\nIn the experiments, it is hard to rate the performance of the proposed model. There are no comparisons to previous state-of-the-art, and I'm not so familiar with this dataset. The authors should pay more attention to make their datasets understood to the readers, including the performance judgment and possible further applications.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cUSvYJPUn9T", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary ====\nThis paper proposes a new method for image-to-image translation on multi-spectral imagery. The proposed method uses variational auto-encoders and generative adversarial networks to generate synthetic bands in satellite imagery.  Novelty of the proposed method is that the authors introduce a shared spectral reconstruction loss and skip connection to generate synthetic spectral bands. This allows to generate synthetic bands with higher accuracy that the original image-to image translation method.\n\n=== Pros ==== \nObtaining additional data for satellite imagery is notoriously difficult. Moreover, attempts to combine different sources of satellite imagery often fail due to the difference in resolution, band spectrum and the time, when the imagery was obtained. Therefore, all the work that aims to help solving these problems is extremely important. Authors here do not address the problem of time when the imagery was obtained (since they work with geostationary satellites), but they do address the other two: lack of data and band incompatibility. Therefore, the approach, presented in this paper, is very interesting and important application. \nTo our knowledge, VAE-GAN approach to satellite imagery was used before for Sentinel imagery [1], however the method, proposed here, improves this approach by introducing a new type of loss. \nTo summarise, the proposed method is technically correct and the application is interesting and important.\n\n=== Cons === \nThe authors claim three main contributions in the paper: “1) introduce a shared spectral reconstruction loss to a VAE-GAN architecture for synthetic band generation; 2) test our methodology on real-world scenarios; 3) present and release a test dataset of four hemispheric snapshots from three publicly available geostationary satellites for future research.”\n1) The novelty of the method seems to be relatively limited and domain specific. Specifically, the original method was introduced before [2] and [3] in other applications. \nAuthors introduce a new type of loss to improve the performance of the model on a particular type of the input data (multispectral). This new type of loss improves the performance of the model from [3] for this particular type of input. However, the other properties of shared spectrum reconstruction loss are unclear: is it domain-specific or it can be extended to other problems?\n3) Sorry, I have not found the link to the dataset, which is mentioned as the third contribution of the paper. Could you please explicitly state where you present it or remove this contribution?\n\n=== Basis for recommendation ===\nI’m borderline on this paper since the paper is very interesting and the application is important, but technical novelty is slightly limited. \nQuality. The quality of the paper is very high and satisfy the standards of the ICLR publication.\nClarity. The material is presented clearly and correctly.\nOriginality. The paper introduces an original contribution, which was not published before (to my knowledge).  \nSignificance. The results are important to the domain of climate-change related applications, satellite imagery processing and EO/GIS . \n\n=== Questions to address in the rebuttal ===\n1.\tPlease specify the novelty and importance of the proposed method. Is it possible to apply it elsewhere?\n2.\tDoes the proposed method work with other types of imagery (other satellites/other types of sensors?) If yes, what other types of satellites/sensors can it be applied to?\n3.\tWhat are the limitations of the proposed approach?\n4.\tCould you please explicitly state where you present the dataset or remove this contribution?\n5.\tCould you explicitly state the steps for generation of new bands? \n6.\tFor the readers, unfamiliar with satellite imagery processing, it might be interesting to understand why generation of synthetic bands is of particular importance.\n\n=== Minor comments and additional feedback (not necessary to address in rebuttal)===\n1.\tIt’s not very clear how authors address the problem of different band resolution.\n2.\tHow would performance change if the tiles (or parts of tiles)  with clouds are removed from original data and there is less noise in the Sensor condition?\n3.\t“Shortwave infrared band cannot be inferred from others”: this is an interesting observation. It would be interesting to see in the future the accuracy with which the missing bands can be synthesised from a (random) set of bands and how accuracy increases based on the initial number of bands that were used. \n4.\tCan this method be applied to radar data?\nReferences\n1.\tE. H. Sanchez, M. Serrurier, and M. Ortner, “Learning disentangled representations via mutual information estimation,” 2019. https://arxiv.org/abs/1912.03915 \n2.\tA. B. L. Larsen, S. K. Sønderby, and O. Winther, “Autoencoding beyond pixels using a learned similarity metric,”  (2015) http://arxiv.org/abs/1512.09300\n3.\tM.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in Advances in Neural Information Processing Systems 30, (2017) http://papers.nips.cc/paper/6672-unsupervised-image-to-image-translation-networks.pdf\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UgRNT83zTkE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to generate synthetic unobserved spectral imagery from a set of existing spectral channels. This is an image-to-image translation task, for which VAEs and GANs are effective to address. For this specific satellite band-to-band translation task, authors adopt the VAE-GAN framework (adding a skip connection between the input and generator), with a new added spectral reconstruction loss.  Experiments show the effectiveness of the proposed method.\n\nPros:\n\n1. As stated by the authors, this paper introduces a shared spectral reconstruction loss to a VAE-GAN architecture for synthetic band generation.\n2. Simulated experiments show that the generated banded is possible and effective.\n3. Using the generated band, the proposed model improves segmentation performance beyond baselines.\n4. A test dataset of four hemispheric snapshots from three publicly available geostationary satellites is thankfully received.\n\n\nCons:\n\nI would say this paper has limited contribution, in terms of the novelty of the method. For me, this paper is an application paper. Specifically:\n\n1. I would say among the three contributions (listed before section 2. background), only the first contribution of proposing a new loss makes sense for me as we always need to validate methods on real datasets. For the proposed shared spectral reconstruction loss, it's a simple KL divergence loss (Equation 5) imposed on overlapping channels, specifically tailored for this band-to-band translation task under the VAE-GAN framework. In one word, the contribution is limited.\n\n2. Throughout the experiments section, I cannot find a baseline method from a third-party. The reason could be that no one did this task before, which is good. However, including an off-the-shelf image-to-image translation method as a baseline is necessary. It will help readers to understand the difficulty of this band-to-band translation task.\n\n3. To make the proposed shared spectral reconstruction loss stronger, I would like to see an ablation study concerning how it interacts with other losses in equation 6. Please use a different number of $\\lambda_6$ to illustrate.\n\nIn summary, I would say this paper demonstrates the success of VAE-GAN on the band-to-band reconstruction task, which is good, but with limited technical novelty.\n\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an interesting application of VAE-GAN to the problem of Spectral Synthesis across satellite observations with some additional domain specific changes (new loss, ...). The introduction of a new dataset is also very interesting and can open the door for more methodological development in the community. 

While the application is original, the methodological contributions have been judged limited and domain specific by most of the reviewers. The responses from the authors was appreciated by the reviewers (especially the comparison to the UNIT baseline) but did not change their opinion about the limited methodological novelty of the approach. The AC recommends a reject but encourages the authors to resubmit it to a more applied remote sensing venue.

Decision: Reject