Single Image Super-Resolution with Uncertainty
Estimation for Lunar Satellite Images

Jose I. Delgado-Centeno∗
University of Luxembourg
jose.delgado@uni.lu
Ben Moseley
University of Oxford
bmoseley@robots.ox.ac.uk

Paula Harder∗
Fraunhofer Institutepaula.harder@itwm.fraunhofer.de
Valentin T. Bickel
MPS Goettingen
bickel@mps.mpg.de

Freddie Kalaitzis
University of Oxford
freddie.kalaitzis@cs.ox.ac.uk

Siddha Ganju
NVIDIA, SpaceML
sganju@nvidia.com

Miguel A. Olivares-Mendez
University of Luxembourg
miguel.olivaresmendez@uni.lu

Abstract
Recently, there has been a renewed interest in returning to the Moon, with manyplanned missions targeting the south pole. This region is of high scientific andcommercial interest, mostly due to the presence of water-ice and other volatiles
which could enable our sustainable presence on the Moon and beyond. In orderto plan safe and effective crewed and robotic missions, access to high-resolution(<0.5 m) surface imagery is critical. However, the overwhelming majority (99.7%)of existing images over the south pole have spatial resolutions >1 m. In order toobtain better images, the only currently available way is to launch a new satellite
mission to the Moon with better equipment to gather more precise data. In this work,
we develop an alternative that can be used directly on previously gathered data andtherefore saving a lot of resources. It consists of a single image super-resolution(SR) approach based on generative adversarial networks that is able to super-resolve
existing images from 1 m to 0.5 m resolution, unlocking a large catalogue of images(∼50,000) for a more accurate mission planning in the region of interest for theupcoming missions. We show that our enhanced images reveal previously unseen
hazards such as small craters and boulders, allowing safer traverse planning. Our
approach also includes uncertainty estimation, which allows mission planners tounderstand the reliability of the super-resolved images.

1Introduction

The Moon has been receiving increasing interest in recent years. Many upcoming missions have beenplanned, for example in the frame of NASA’s Artemis program which aims to put humans back on
the Moon within this decade (Smith et al., 2020). A major goal is to achieve a permanent presence on
the Moon, and for many of these missions the south pole is the main target. This region is of highscientific and commercial interest because it is expected to host water-ice and other resources which
could enable our sustainable presence.
High-resolution (HR) satellite imagery is essential for planning such missions. For example, whenplanning human and rover traverses, mission planners need imagery with a resolution of ∼0.5m/px or∗

equal contribution

Preprint. Under review.

Figure 1: Using ANUBIS to super-resolve real LRO NAC imagery (Lat: -64.51, Long: 299.6). (a) A
real 1 m/px summed mode image, (b) after LSRRes (our work), (c) after LSRGAN (our work) , (d) acoinciding real 0.5 m/px regular mode image (ground truth). The uncertainty map displayed next
to the enhanced image shows our confidence in the super-resolved image (yellow: high uncertainty,purple: low uncertainty).
better to detect relevant features at the rover-/astronaut-scale, such as craters and boulders (Robinsonet al., 2010). This allows small hazards to be identified in advance, ensuring safe and efficient traverses.
Over the past 12 years NASA’s Lunar Reconnaissance Orbiter (LRO) mission has been capturing
state-of-the-art images of the lunar surface using its Narrow Angle Camera (NAC) (Robinson et al.,2010), and their spatial resolutions typically vary in the range 0.5-2 m. However, at the south pole
the overwhelming majority of these images have resolutions >1m. This is due to the inherent low
lighting conditions. Over the south pole, most images (99.7%) are captured using what is known
as “summed mode” operation of the NAC, which sums two adjacent pixels to improve SNR at theexpense of reducing resolution by a factor of 2 (Humm et al., 2015). Unfortunately, this limits theirusefulness for mission planning.
In this work we introduce ANUBIS (Adversarial Network for Uncertainty-Based Image Superresolution). ANUBIS uses a deep ensemble of GANs to super-resolve these images, increasing
their resolution by a factor of 2 (from 1 m to 0.5 m), providing enhanced data products for mission
planners. We show that these images enable safer traverse planning, and by using an ensemble we arealso able to output uncertainty estimates, which is essential for their safety-critical application.

2Methodology

The main goal of this work is to learn to super-resolve existing summed-mode NAC images over thesouth pole from ∼1m/px to ∼ 0.5m/px. We note that a small number of “regular mode” images overthe south pole are taken without the summing operation which have ∼0.5m/px resolution. These
images are not common (∼ 0.0028%), but there are enough examples with high SNR to construct alabelled training dataset. An example regular mode image is shown in Figure 1 (d).
Specifically, we generate a training, validation, and testing data starting from 121 (52,224×5,064 px)
regular mode NAC images of the south pole. The selected images have a resolution of 0.5 ± 0.05
m/px and similar illumination conditions to the summed mode polar images. We divide the images
into 128 × 128 patches, resulting in 220,000 HR patches. To approximate the NAC summed mode,
we apply a local 2 × 2 mean and round to the next integer value to obtain their low-resolution(LR) ∼1m/px counterparts. This operation represents a best effort approach to approximate the realsumming performed on the spacecraft. We split this dataset into a training, validation and test dataset
using a 80:10:10 split.
Two separate single image SR approaches are designed. The first uses a residual network (LSRRes)
which takes a LR image and outputs a HR image, trained with an L2 loss function. The second builds
on the first, training LSRRes with an additional adversarial loss (LSRGAN), and is shown in Figure 2.
The goal of this approach is to ensure the output image is as realistic as possible via the interaction of
the generator (LSRRes) and discriminator network.
2Figure 2: The ANUBIS LSRGAN architecture, consisting of a residual generator and a fullyconvolutional discriminator. Colors indicate different block types.
Method

MAE↓

PSNR↑

SSIM↑

LR
BilinearBicubic
LSRGAN (ours)
LSRRes (ours)

0.00420.00370.00320.00340.0027

44.81
45.89
47.26
47.23
49.13

0.9745
0.979
0.984
0.983
0.989

Table 1: Performance of SR methods on our test set. Best scores in bold. The LSRRes/LSRGAN
metrics are computed using only a single randomly selected ensemble member.

A major concern when using single image SR is the potential for generating artifacts (or “hallucinations”), which are generally related to the ill-posedness of the inverse problem. This is of key concern
in our application, as mission planners will use our images in safety-critical applications. Thus toincrease the reliability of our images we also explore uncertainty estimation. Specifically, we train
an ensemble of 24 LSRGANs with different weight initialisations, allowing us to output multiple
realisations of the same super-resolved image. Each realisation differs slightly due to ill-posedness ofthe problem. The pixel-wise standard deviation of these images is used to generate an uncertainty
map, which provides an estimate of the reliability of our SR process. The same uncertainty workflow
is carried out for the LSRRes-only approach too. All training details are described in Appendix A.

3

Results and discussion

An example real 1 m/px summed mode image over the south pole is shown in Figure 1 (a), and theresult after applying LSRRes and LSRGAN (using a single randomly selected ensemble member) is
shown in (b) and (c). A real 0.5 m/px regular mode image is also available over this region, and isshown in (d). We find that both LSRRes and LSRGAN are able to significantly enhance the resolutionof the LR image, revealing small craters and other surface features which are present in the HR imageand difficult to see in the LR image. When evaluating more images, we find that LSRGAN tends
to output images with higher perceptual quality than LSRRes. Specifically, LSRGAN adds moresmall scale features such as craters and boulders to the image compared to LSRRes, resulting in afrequency spectrum which is more comparable to ground truth HR images.
We evaluate the quantitative performance of our methods on our test set of images, shown in Table 1.
As metrics, we consider the mean absolute error (MAE), peak signal to noise ratio (PSNR), andstructural similarity index measurement (SSIM). We also compare our methods to two baseline
approaches (bilinear and bicubic interpolation). Under these metrics, we find that LSRRes gives
the highest performance. However, given the improved perceptual quality of the LSRGAN images
observed above, these metrics may not be the best suited for this task.
3Figure 3: Validating ANUBIS with traverse planning. (a) Low-resolution (input), (b) Super-resolution+ Uncertainty map (our work), (c) High-resolution (Ground truth).

We also show the uncertainty maps generated by our ensembles of LSRRes/LSRGANs for the superresolved images in Figure 1. These maps are most uncertain around crater rims, which is believableas they represent high-frequency features in the image which are likely difficult to reconstruct.

4Validation with traverse planning

Finally, we show the value of our super-resolved images on a downstream task, here: traverseplanning. We set up a hypothetical problem, where a rover must plan a safe traverse from its starting
position to a goal, given an image and the locations shown in Figure 3. Given an image, we detectobstacles based on the image gradients and then use an A* path planning algorithm to plan a safe
traverse around them. During path planning using our super-resolved images, we also add obstacles
derived from our uncertainty map (using the same obstacle detection process, i.e., high uncertainty
pixels are considered as obstacles). Figure 3 shows the resulting traverses when using a LR summed
mode image, its super-resolved version using LSRGAN, and a ground truth regular mode HR image.
We find that for this location the LSRGAN paths more closely match the HR image paths than theLR image paths, i.e. allowing for a safer and more efficient traverse.

5Limitations and future work

There are multiple limitations and future directions of our work. Firstly, we restricted ourselves toonly learning a mapping from 1 m/px to 0.5 m/px, whereas in reality NAC images of the south polespan a range of resolutions >1 m/px. In future work, we aim to develop a workflow which is able tolearn a mapping from any input resolution to any output resolution, within reason, allowing more
NAC images to be super-resolved. Secondly, there are many other uncertainty estimation techniques
which could be tested. For example, GANs with random input vectors (Abid et al., 2021), normalisingflows (Lugmayr et al., 2020), dropout or other Bayesian inference approaches could be more effective.
Finally, we assume that our downsampling operator can be approximated by a simple summation androunding. However, in reality NAC images are subject to a lossy compression before downlink toEarth (Robinson et al., 2010), and therefore this degradation process is not modelled; we intend toinclude this in future work.

6Conclusions and broader impact

We have investigated methods for super-resolving lunar imagery over the south pole by a factorof 2 (from 1 m/px to 0.5 m/px) and shown that they can improve image quality and help achieve
downstream tasks such as traverse planning. Furthermore we provided uncertainty estimates foreach super-resolved image, which is essential for safety-critical applications. Our approach provides
higher resolution images over the south pole of the Moon, where they are not currently available,
allowing mission planners to plan safer and more effective missions.
4This work could have a positive impact on the planning and execution of future lunar exploration
missions, specifically by reducing their risk and maximizing their efficacy. The uncertainty mapsthat are delivered along with the improved images greatly increase the reliability of ANUBIS forsafety-critical applications. Upcoming robotic and crewed missions, such as NASA’s VIPER or other
Artemis missions, are a few of many possible beneficiaries of our work.

References
Abid, M. A., Hedhli, I., and Gagné, C. (2021). A generative model for hallucinating diverse versionsof super resolution images. arXiv preprint arXiv:2102.06624.
Humm, D., Tschimmel, M., Brylow, S., Mahanti, P., Tran, T., Braden, S., Wiseman, S., Danton, J.,
Eliason, E., and Robinson, M. (2015). Flight calibration of the lroc narrow angle camera. SpaceScience Reviews, 200.
Lugmayr, A., Danelljan, M., Van Gool, L., and Timofte, R. (2020). Srflow: Learning the superresolution space with normalizing flow. In European Conference on Computer Vision, pages
715–732. Springer.
Robinson, M., Brylow, S., Tschimmel, M., Humm, D., Lawrence, S., Thomas, P., Denevi, B.,
Bowman-Cisneros, E., Zerr, J., Ravine, M., et al. (2010). Lunar reconnaissance orbiter camera(lroc) instrument overview. Space science reviews, 150(1-4):81–124.
Smith, M., Craig, D., Herrmann, N., Mahoney, E., Krezel, J., McIntyre, N., and Goodliff, K. (2020).
The artemis program: An overview of nasa’s activities to return humans to the moon. In 2020
IEEE Aerospace Conference, pages 1–10. IEEE.

ATraining details

Our LSRRes network uses a convolutional residual architecture with 6 layers, and ReLU activation
functions (Generator section in Fig. 2). The discriminator in our LSRGAN uses a convolutionalarchitecture with 6 layers, and ReLU activation functions. All approaches were trained using theAdam optimizer, a learning rate of 0.0002 and a batch size of 256. The inputs and outputs of thenetworks are normalised before training. Training takes approximately 3 hours on a single NVIDIA
Tesla A100 GPU.
To create the deep ensemble of GANs used to generate the SR image distribution, the same architecture
was trained 24 different times using random weight initialization.

B

NeurIPS 2021 Paper Checklist
1. Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?: Yes
2. Have you read the ethics review guidelines and ensured that your paper conforms to them?:Yes
3. Did you discuss any potential negative societal impacts of your work?: Yes
4. Did you describe the limitations of your work?: Yes
5. Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?: The data used during this work is
publicly available at https://www.lroc.asu.edu/. The code will be released in a future
journal publication.
6. Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?: Yes
7. Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?: Yes, with the uncertainty estimation.
8. Did you include the amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?: Yes

5