GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution
Kelvin C.K. Chan1

Xintao Wang2 Xiangyu Xu1 Jinwei Gu3 Chen Change Loy1∗
Nanyang Technological University, Singapore
2Applied Research Center, Tencent PCG 3 SenseBrain
1{chan0899, xiangyu.xu, ccloy}@ntu.edu.sg(a) Low-Resolution(b) ESRGAN

xintao.wang@outlook.com(c) PULSE

gujinwei@sensebrain.ai(d) GLEAN (ours)(e) Ground-truth

Figure 1: Example of large-factor super-resolution (16×). (a) The low-resolution input (LR). (b) ESRGAN [35] trains the SR generator
from scratch, which often produces artifacts and unnatural textures. (c) PULSE [27] achieves more realistic results by GAN inversion,
which, however, cannot faithfully recover the structures of the ground-truth. (d) With the proposed generative latent bank, GLEAN is able
to generate output that not only is close to the ground-truth, but also possesses realistic textures. (e) The ground-truth (GT).

Abstract

methods fail to handle since most details and textures arelost during downsampling. Since the problem is severely
underspecified, informative priors become inevitable in thissetting, especially in restoring the textural details. Studying
large-factor image SR is meaningful as it can potentially
improve the state of the arts in SR, and more generally conditional generative models for images.
The notion of GAN has been extensively used in SR withthe aim to enrich texture details in an upscaled image. There
are two popular approaches to deploy GANs for this task.
The more common paradigm [21, 34, 35] trains a generator to handle the upscaling task, where adversarial training
is performed by using a discriminator to differentiate real
images from the upscaled images produced by the generator. Another possible way to exploit GAN for the task is
by GAN inversion [1, 11, 27, 28]. In this setting, one willneed to ‘invert’ the generation process of a pre-trained GAN
by mapping a corrupted image back to the latent space. A
restored image can then be reconstructed from the optimal
vector in the latent space.
While both methods are capable of generating more realistic results than approaches that solely rely on `2 loss, they
have some inherent shortcomings. The first paradigm typically trains the SR generator from scratch using a combined
objective function consisting of a fidelity term and an adversarial loss. In this setting, the generator is responsible forboth capturing the natural image characteristics and main-

We show that pre-trained Generative Adversarial Networks (GANs), e.g., StyleGAN, can be used as a latent bankto improve the restoration quality of large-factor image
super-resolution (SR). While most existing SR approaches
attempt to generate realistic textures through learning withadversarial loss, our method, Generative LatEnt bANk
(GLEAN), goes beyond existing practices by directly leveraging rich and diverse priors encapsulated in a pre-trained
GAN. But unlike prevalent GAN inversion methods that require expensive image-specific optimization at runtime, ourapproach only needs a single forward pass to generate
the upscaled image. GLEAN can be easily incorporated
in a simple encoder-bank-decoder architecture with multiresolution skip connections. Switching the bank allows themethod to deal with images from diverse categories, e.g.,cat, building, human face, and car. Images upscaled by
GLEAN show clear improvements in terms of fidelity andtexture faithfulness in comparison to existing methods asshown in Fig. 1.

1. Introduction
In this study, we explore a new way to employ GAN [9]for image super-resolution. We are interested in the regimeof high magnification factors (8× to 64×), which typical SR∗ Corresponding author1

taining the fidelity to the ground-truth. This inevitably limits the capability of approximating the natural image manifold. As a result, these methods often produce artifacts andunnatural textures. As shown in Fig. 1, while ESRGAN [35]faithfully recovers the structures (e.g. pose, ear shape) of thecat, it struggles to produce realistic textures.
The second paradigm resolves the aforementioned problem by making better use of the latent space of GAN
through optimization. However, as the low-dimensional latent codes and the constraints in the image space are insufficient to guide the restoration process, these methods often
generate images with low fidelity. As shown in Fig. 1, despite being realistic, the output of a representative method,
PULSE [27], fails to recover the structures of the groundtruth faithfully. In addition, as the optimization is usuallyconducted in an iterative manner for each image at runtime,these approaches are often time-consuming.
In our approach, we leverage pre-trained GANs such as
StyleGAN [16] to provide rich and diverse priors for thetask. Unlike most GAN inversion methods, which also
use pre-trained GANs, our method does not involve imagespecific optimization at runtime. Once trained, the model
only needs a single forward pass to upscale an image, which
is more practical for applications that demand fast response.
The idea is partially inspired by the classic notion of dictionary [40]. But unlike conventional approaches that construct a finite and imagery-derived dictionary, we exploit
GAN as a more effective way for storing priors.
Conditioning and retrieving from a GAN-based dictionary is a new and non-trivial question we need to addressin this work. We show that pre-trained GANs can be employed as a latent bank in a succinct encoder-bank-decoderarchitecture. This novel architecture allows us to lift theburden of learning both fidelity and texture generation simultaneously in a typical encoder-decoder network since
the latent bank already captures rich texture priors. In addition, we show that it is pivotal to condition the bank bypassing both the latent vectors and multi-resolution convolutional features from the encoder to achieve high-fidelity
results. Symmetrically, multi-resolution cues need to bepassed from the bank to the decoder. We show the effectiveness of the proposed method in handling images withchallenging poses and structures apart from the large magnification factor. We also demonstrate how the method canbe generalized to different categories, e.g., human faces,
cats, buildings, by switching different pre-trained GAN latent banks.

ods achieve remarkable results in terms of PSNR, training
solely with pixel-wise constraints often results in perceptually unconvincing outputs with severe over-smoothing artifacts [21, 27]. To alleviate the problem, GANs [21, 29, 35,37] are employed to approximate the natural image manifold, yielding more photo-realistic results. However, as
the generator needs to learn both fidelity and natural image
characteristics, unnatural artifacts could still be observedin the outputs, especially if one trains the generator fromscratch.
Recent interests have shifted to large-factor SR beyond
the typical upscaling factors (2× or 4×) [3, 13, 31, 46].
Dahl et al. [3] propose a fully probabilistic pixel recursive
network for upsampling extremely coarse images with resolution 8×8. RFB-ESRGAN [31] builds upon ESRGAN
and adopts multi-scale receptive fields blocks for 16× SR.
VarSR [13] achieves 8× SR by matching the latent distributions of LR and HR images to recover the missing details.
Zhang et al. [46] perform 16× reference-based SR on paintings with a non-local matching module and a wavelet texture loss. To handle even larger magnification factors, one
would need to rely on stronger priors. SR methods specialized on large magnification factors are typically dedicatedto the human face category as one could exploit the strong
structural prior of faces. Facial priors including facial attributes [22], facial landmarks [18, 26], and identity [10]have been studied. Our work goes beyond previous works
and pushes the limit to 64× and generalizes to more categories. Such a large magnification factor is challenging due
to its highly ill-posed nature.
GAN Inversion. Given a degraded image x, GAN
inversion-based methods [1, 11, 27, 28] in general produce a natural image best approximating x by optimizing z ∗ = argminz∈Z L (G(z), x), where Z is the latent
space and L(·, ·) denotes the task-specific objective function. For instance, PULSE [27] iteratively optimizes thelatent code of StyleGAN [16] with a pixel-wise constraint
between the input and output. mGANprior [11] optimizes
multiple latent codes to increase the expressiveness of themodel. DGP [28] further finetunes the generator together
with the latent code to reduce the gap between the distributions of the training and testing images. A common issue
with GAN inversion is that important spatial information
may not be faithfully kept due the low-dimensionality of
the latent code. Thus, these methods often generate undesirable results that do not resemble the ground-truth. Different
from GAN inversion, GLEAN conditions the pre-trained
generator with both the latent codes and multi-resolution
convolutional features, providing additional spatial guidance for restoration. In addition, GLEAN does not require
iterative optimization during inference.

2. Related Work
Image Super-Resolution. Many existing SR algorithms [4,6, 7, 8, 12, 36, 44, 48] directly learn a mapping fromthe low-resolution images to high-resolution images witha pixel-wise constraint (e.g. `2 loss). While these meth2

= Ei

32⇥32

<latexit sha1_base64="bHGvdUyNA2RVMJ51zI2oCSOUC8U=">AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoBchKILHCMYEkiXMTmaTIfNYZmaFsOQXvHhQxKs/5M2/cTbZgyYWNBRV3XR3RQlnxvr+t1daWV1b3yhvVra2d3b3qvsHj0almtAWUVzpToQN5UzSlmWW006iKRYRp+1ofJP77SeqDVPywU4SGgo8lCxmBNtcurrts3615tf9GdAyCQpSgwLNfvWrN1AkFVRawrEx3cBPbJhhbRnhdFrppYYmmIzxkHYdlVhQE2azW6foxCkDFCvtSlo0U39PZFgYMxGR6xTYjsyil4v/ed3UxpdhxmSSWirJfFGccmQVyh9HA6YpsXziCCaauVsRGWGNiXXxVFwIweLLy+TxrB749eD+vNa4LuIowxEcwykEcAENuIMmtIDACJ7hFd484b14797HvLXkFTOH8Afe5w+U8Y3s</latexit><latexit sha1_base64="SkG7ZIo5s5bVr0KcD70XgA+I5GY=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0laQY9FLx4r2A9oQtlst+3SzSbsToQS+je8eFDEq3/Gm//GbZuDtj4YeLw3w8y8MJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1LDpVC8hQIl7yaa0yiUvBNO7uZ+54lrI2L1iNOEBxEdKTEUjKKV/Hot81FE3MzqtX654lbdBcg68XJSgRzNfvnLH8QsjbhCJqkxPc9NMMioRsEkn5X81PCEsgkd8Z6lito9Qba4eUYurDIgw1jbUkgW6u+JjEbGTKPQdkYUx2bVm4v/eb0UhzdBJlSSIldsuWiYSoIxmQdABkJzhnJqCWVa2FsJG1NNGdqYSjYEb/XlddKuVT236j1cVRq3eRxFOINzuAQPrqEB99CEFjBI4Ble4c1JnRfn3flYthacfOYU/sD5/AFPQJEy</latexit>

16⇥16
64⇥64

<latexit sha1_base64="ADbftRqiR2c7Pjd+JZBL0Mwq/7M=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2A9sQ9lsN+3SzSbsToQS+i+8eFDEq//Gm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKREKRtFKj7WsjyLiZlYblCtu1V2ArBMvJxXI0RyUv/rDmKURV8gkNabnuQn6GdUomOSzUj81PKFsQke8Z6mido2fLS6ekQurDEkYa1sKyUL9PZHRyJhpFNjOiOLYrHpz8T+vl2J47WdCJSlyxZaLwlQSjMn8fTIUmjOUU0so08LeStiYasrQhlSyIXirL6+T9lXVc6vefa3SuMnjKMIZnMMleFCHBtxBE1rAQMEzvMKbY5wX5935WLYWnHzmFP7A+fwBasGQvA==</latexit><latexit sha1_base64="2aK1VPzq4w4JpYf12lURKj/nsp4=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkVI9FLx4r2A9oQtlst+3SzSbsToQS+je8eFDEq3/Gm//GbZuDtj4YeLw3w8y8MJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1LDpVC8hQIl7yaa0yiUvBNO7uZ+54lrI2L1iNOEBxEdKTEUjKKV/Hot81FE3MzqtX654lbdBcg68XJSgRzNfvnLH8QsjbhCJqkxPc9NMMioRsEkn5X81PCEsgkd8Z6lito9Qba4eUYurDIgw1jbUkgW6u+JjEbGTKPQdkYUx2bVm4v/eb0UhzdBJlSSIldsuWiYSoIxmQdABkJzhnJqCWVa2FsJG1NNGdqYSjYEb/XlddK+qnpu1XuoVRq3eRxFOINzuAQPrqEB99CEFjBI4Ble4c1JnRfn3flYthacfOYU/sD5/AFeoJE8</latexit>

f3<latexit sha1_base64="eRUltlz3W0AmByeXC+PBnjX7dNk=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0EPYv++WKW3XnIKvEy0kFcjT65a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi2VNELtZ/NTp+TMKgMSxsqWNGSu/p7IaKT1JApsZ0TNSC97M/E/r5ua8NrPuExSg5ItFoWpICYms7/JgCtkRkwsoUxxeythI6ooMzadkg3BW355lbQuqp5b9e5rlfpNHkcRTuAUzsGDK6jDHTSgCQyG8Ayv8OYI58V5dz4WrQUnnzmGP3A+fwDzPY2Q</latexit>

E2<latexit sha1_base64="HOcG5SYhygUQuJCCCey8ctdv9Hg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiCB4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCopeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPxzcxvP6HSPJaPZpKgH9Gh5CFn1Fjp4bZf65crbtWdg6wSLycVyNHol796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n81PnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwis/4zJJDUq2WBSmgpiYzP4mA66QGTGxhDLF7a2EjaiizNh0SjYEb/nlVdKqVT236t1fVOrXeRxFOIFTOAcPLqEOd9CAJjAYwjO8wpsjnBfn3flYtBacfOYY/sD5/AG/c41u</latexit>

E3<latexit sha1_base64="VxTYfOToKBazKW23ss0RsF6C88g=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GNRBI8V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4WV1bX1jeJmaWt7Z3evvH/Q1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8HoZuq3nlBpHstHM07Qj+hA8pAzaqz0cNs775UrbtWdgSwTLycVyFHvlb+6/ZilEUrDBNW647mJ8TOqDGcCJ6VuqjGhbEQH2LFU0gi1n81OnZATq/RJGCtb0pCZ+nsio5HW4yiwnRE1Q73oTcX/vE5qwis/4zJJDUo2XxSmgpiYTP8mfa6QGTG2hDLF7a2EDamizNh0SjYEb/HlZdI8q3pu1bu/qNSu8ziKcATHcAoeXEIN7qAODWAwgGd4hTdHOC/Ou/Mxby04+cwh/IHz+QPA941v</latexit>

E4<latexit sha1_base64="PE0p+lONVmlxORKCAGPgTNNL8AM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiCB4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCopeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPxzcxvP6HSPJaPZpKgH9Gh5CFn1Fjp4bZf65crbtWdg6wSLycVyNHol796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n81PnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwis/4zJJDUq2WBSmgpiYzP4mA66QGTGxhDLF7a2EjaiizNh0SjYEb/nlVdK6qHpu1buvVerXeRxFOIFTOAcPLqEOd9CAJjAYwjO8wpsjnBfn3flYtBacfOYY/sD5/AHCe41w</latexit>

C = (c0<latexit sha1_base64="Bi34J8SYWq1KLtBcT2QBNyIgAIM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiCB4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2Mb2Z++4lrI2L1iJOE+xEdKhEKRtFKD7d9r1+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophld+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t1fVurXeRxFOIFTOAcPalCHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gC9741t</latexit><latexit sha1_base64="MIao4A2XiM4eXuO0aIn84r77UjE=">AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoBchqAePEYwJJEuYncwmQ+axzMwKYckvePGgiFd/yJt/42yyB00saCiquunuihLOjPX9b6+0srq2vlHerGxt7+zuVfcPHo1KNaEtorjSnQgbypmkLcssp51EUywiTtvR+Cb3209UG6bkg50kNBR4KFnMCLa5dHXbZ/1qza/7M6BlEhSkBgWa/epXb6BIKqi0hGNjuoGf2DDD2jLC6bTSSw1NMBnjIe06KrGgJsxmt07RiVMGKFbalbRopv6eyLAwZiIi1ymwHZlFLxf/87qpjS/DjMkktVSS+aI45cgqlD+OBkxTYvnEEUw0c7ciMsIaE+viqbgQgsWXl8njWT3w68H9ea1xXcRRhiM4hlMI4AIacAdNaAGBETzDK7x5wnvx3r2PeWvJK2YO4Q+8zx+Ta43r</latexit><latexit sha1_base64="B4w5QLAbIi3D6iLuMg1ZyWOTNlY=">AAAB9XicbZBNSwMxEIZn61etX1WPXoJF8FR2i2CPRS8eK9hWaNeSTbNtaDa7JLNKWfo/vHhQxKv/xZv/xrTdg7a+EHh4Z4aZvEEihUHX/XYKa+sbm1vF7dLO7t7+QfnwqG3iVDPeYrGM9X1ADZdC8RYKlPw+0ZxGgeSdYHw9q3ceuTYiVnc4Sbgf0aESoWAUrfXg1epZD0XEzdRiv1xxq+5cZBW8HCqQq9kvf/UGMUsjrpBJakzXcxP0M6pRMMmnpV5qeELZmA5516KidpGfza+ekjPrDEgYa/sUkrn7eyKjkTGTKLCdEcWRWa7NzP9q3RTDup8JlaTIFVssClNJMCazCMhAaM5QTixQpoW9lbAR1ZShDapkQ/CWv7wK7VrVc6ve7UWlcZXHUYQTOIVz8OASGnADTWgBAw3P8ApvzpPz4rw7H4vWgpPPHMMfOZ8/Q+SRsg==</latexit>

4⇥4

E1

= Di

128⇥128<latexit sha1_base64="zKOW3Oe/zundkHQsQnGZ0yIn9oI=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEaI9FLx4r2A9sQ9lsN+3SzSbsToQS+i+8eFDEq//Gm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKREKRtFKj/WsjyLiZlYflCtu1V2ArBMvJxXI0RyUv/rDmKURV8gkNabnuQn6GdUomOSzUj81PKFsQke8Z6mido2fLS6ekQurDEkYa1sKyUL9PZHRyJhpFNjOiOLYrHpz8T+vl2JY9zOhkhS5YstFYSoJxmT+PhkKzRnKqSWUaWFvJWxMNWVoQyrZELzVl9dJ+6rquVXv/rrSuMnjKMIZnMMleFCDBtxBE1rAQMEzvMKbY5wX5935WLYWnHzmFP7A+fwBdwWQxA==</latexit><latexit sha1_base64="3HywaIiAuMF36typ7BbuBjMzoB8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiCB4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2Mb2Z++4lrI2L1iJOE+xEdKhEKRtFKD7d9t1+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophld+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t1fVurXeRxFOIFTOAcPalCHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gC8a41s</latexit><latexit sha1_base64="+RGdV9zgChrL9j1w3LsCjzg40g4=">AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoBch6MVjRGMCyRJmJ7PJkHksM7NCWPILXjwo4tUf8ubfOJvsQRMLGoqqbrq7ooQzY33/2yutrK6tb5Q3K1vbO7t71f2DR6NSTWiLKK50J8KGciZpyzLLaSfRFIuI03Y0vsn99hPVhin5YCcJDQUeShYzgm0uXd33Wb9a8+v+DGiZBAWpQYFmv/rVGyiSCiot4diYbuAnNsywtoxwOq30UkMTTMZ4SLuOSiyoCbPZrVN04pQBipV2JS2aqb8nMiyMmYjIdQpsR2bRy8X/vG5q48swYzJJLZVkvihOObIK5Y+jAdOUWD5xBBPN3K2IjLDGxLp4Ki6EYPHlZfJ4Vg/8enB3XmtcF3GU4QiO4RQCuIAG3EITWkBgBM/wCm+e8F68d+9j3lryiplD+APv8weqRY36</latexit><latexit sha1_base64="s0URI4v6Xa23UXcH+I4c/lLz1nM=">AAAB9XicbZDLSgMxFIbP1Futt6pLN8EiuCozxduy6MZlBXuBdiyZNG1DM5khOaOUoe/hxoUibn0Xd76NaTsLbf0h8PGfczgnfxBLYdB1v53cyura+kZ+s7C1vbO7V9w/aJgo0YzXWSQj3Qqo4VIoXkeBkrdizWkYSN4MRjfTevORayMidY/jmPshHSjRF4yitR4q5xdpB0XIzcRit1hyy+5MZBm8DEqQqdYtfnV6EUtCrpBJakzbc2P0U6pRMMknhU5ieEzZiA5426KidpGfzq6ekBPr9Eg/0vYpJDP390RKQ2PGYWA7Q4pDs1ibmv/V2gn2r/xUqDhBrth8UT+RBCMyjYD0hOYM5dgCZVrYWwkbUk0Z2qAKNgRv8cvL0KiUPbfs3Z2VqtdZHHk4gmM4BQ8uoQq3UIM6MNDwDK/w5jw5L8678zFvzTnZzCH8kfP5A0oUkbY=</latexit>

8⇥8

E0

= Si

256⇥256<latexit sha1_base64="pw1HccPcwMvyKxFZxkHFeQqFOns=">AAAB83icbVDLSgNBEOyNrxhfUY9eBoPgKeyIRI9BLx4jmAdklzA7mU2GzD6Y6RXCkt/w4kERr/6MN//GSbIHjRY0FFXddHcFqZIGXffLKa2tb2xulbcrO7t7+wfVw6OOSTLNRZsnKtG9gBmhZCzaKFGJXqoFiwIlusHkdu53H4U2MokfcJoKP2KjWIaSM7SSRxu5hzISZkYbg2rNrbsLkL+EFqQGBVqD6qc3THgWiRi5Ysb0qZuinzONkisxq3iZESnjEzYSfUtjZvf4+eLmGTmzypCEibYVI1moPydyFhkzjQLbGTEcm1VvLv7n9TMMr/1cxmmGIubLRWGmCCZkHgAZSi04qqkljGtpbyV8zDTjaGOq2BDo6st/SeeiTt06vb+sNW+KOMpwAqdwDhSuoAl30II2cEjhCV7g1cmcZ+fNeV+2lpxi5hh+wfn4BlVgkTY=</latexit>

S0

c5 )

c0

Encoder

f2

f1<latexit sha1_base64="EhgoIcdngRZ62TNDIgUrlHkxJGU=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKD+GgNihX3Kq7AFknXk4qkKM5KH/1hzFLI66QSWpMz3MT9DOqUTDJZ6V+anhC2YSOeM9SRSNu/Gxx6oxcWGVIwljbUkgW6u+JjEbGTKPAdkYUx2bVm4v/eb0Uw2s/EypJkSu2XBSmkmBM5n+TodCcoZxaQpkW9lbCxlRThjadkg3BW315nbRrVc+tevdXlcZNHkcRzuAcLsGDOjTgDprQAgYjeIZXeHOk8+K8Ox/L1oKTz5zCHzifP/G5jY8=</latexit><latexit sha1_base64="BZqwz6C361yJqVrSQmvMfMSwMxU=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9RH2/71a9mjcHWSV+QapQoNF3v3qDhGUxSsME1brre6kJcqoMZwKnlV6mMaVsTIfYtVTSGHWQz0+dkjOrDEiUKFvSkLn6eyKnsdaTOLSdMTUjvezNxP+8bmai6yDnMs0MSrZYFGWCmITM/iYDrpAZMbGEMsXtrYSNqKLM2HQqNgR/+eVV0rqo+V7Nv7+s1m+KOMpwAqdwDj5cQR3uoAFNYDCEZ3iFN0c4L86787FoLTnFzDH8gfP5A/A1jY4=</latexit>

S1<latexit sha1_base64="TF73DO+/a0qiR5HKGoe/eBOCNr8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF4+V2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJaPZpqgH9GR5CFn1Fip2Rx4g3LFrboLkHXi5aQCORqD8ld/GLM0QmmYoFr3PDcxfkaV4UzgrNRPNSaUTegIe5ZKGqH2s8WpM3JhlSEJY2VLGrJQf09kNNJ6GgW2M6JmrFe9ufif10tNeONnXCapQcmWi8JUEBOT+d9kyBUyI6aWUKa4vZWwMVWUGZtOyYbgrb68TtpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQAgYjeIZXeHOE8+K8Ox/L1oKTz5zCHzifP9NDjXs=</latexit>

c1

g3

f0

g4
<latexit sha1_base64="q8fFc5VyhmT1LszF3qzznMujotg=">AAAB63icbVBNSwMxEJ3Ur1q/qh69BIvgqexKQY9FLx4r2A9ol5JNs21okl2SrFCW/gUvHhTx6h/y5r8x2+5BWx8MPN6bYWZemAhurOd9o9LG5tb2Tnm3srd/cHhUPT7pmDjVlLVpLGLdC4lhgivWttwK1ks0IzIUrBtO73K/+8S04bF6tLOEBZKMFY84JTaXxsMGHlZrXt1bAK8TvyA1KNAaVr8Go5imkilLBTGm73uJDTKiLaeCzSuD1LCE0CkZs76jikhmgmxx6xxfOGWEo1i7UhYv1N8TGZHGzGToOiWxE7Pq5eJ/Xj+10U2QcZWklim6XBSlAtsY54/jEdeMWjFzhFDN3a2YTogm1Lp4Ki4Ef/XlddK5qvte3X9o1Jq3RRxlOINzuAQfrqEJ99CCNlCYwDO8whuS6AW9o49lawkVM6fwB+jzB0y4jbw=</latexit>

g5<latexit sha1_base64="Z0zSEqu0cMboNHWCQi22ApcpCx8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0WPRi8eK1hbaUDbbTbp0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmlldW19o7xZ2dre2d2r7h88miTTjLdYIhPdCajhUijeQoGSd1LNaRxI3g5GN1O//cS1EYl6wHHK/ZhGSoSCUbTSfdS/6Fdrbt2dgSwTryA1KNDsV796g4RlMVfIJDWm67kp+jnVKJjkk0ovMzylbEQj3rVU0ZgbP5+dOiEnVhmQMNG2FJKZ+nsip7Ex4ziwnTHFoVn0puJ/XjfD8MrPhUoz5IrNF4WZJJiQ6d9kIDRnKMeWUKaFvZWwIdWUoU2nYkPwFl9eJo9ndc+te3fntcZ1EUcZjuAYTsGDS2jALTShBQwieIZXeHOk8+K8Ox/z1pJTzBzCHzifP/fLjZM=</latexit><latexit sha1_base64="a26MgRSJqhxU/OloBGZ3gwsuh3g=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK1hbaUDbbTbp0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmlldW19o7xZ2dre2d2r7h88miTTjLdYIhPdCajhUijeQoGSd1LNaRxI3g5GN1O//cS1EYl6wHHK/ZhGSoSCUbTSfdQ/71drbt2dgSwTryA1KNDsV796g4RlMVfIJDWm67kp+jnVKJjkk0ovMzylbEQj3rVU0ZgbP5+dOiEnVhmQMNG2FJKZ+nsip7Ex4ziwnTHFoVn0puJ/XjfD8MrPhUoz5IrNF4WZJJiQ6d9kIDRnKMeWUKaFvZWwIdWUoU2nYkPwFl9eJo9ndc+te3cXtcZ1EUcZjuAYTsGDS2jALTShBQwieIZXeHOk8+K8Ox/z1pJTzBzCHzifP/TDjZE=</latexit><latexit sha1_base64="u9JS5HeWpCubI+DAfVYCqMXLt+0=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9RH2v71a9mjcHWSV+QapQoNF3v3qDhGUxSsME1brre6kJcqoMZwKnlV6mMaVsTIfYtVTSGHWQz0+dkjOrDEiUKFvSkLn6eyKnsdaTOLSdMTUjvezNxP+8bmai6yDnMs0MSrZYFGWCmITM/iYDrpAZMbGEMsXtrYSNqKLM2HQqNgR/+eVV0rqo+V7Nv7+s1m+KOMpwAqdwDj5cQR3uoAFNYDCEZ3iFN0c4L86787FoLTnFzDH8gfP5A+6xjY0=</latexit>

S2<latexit sha1_base64="80m6n5PDVYdXZSas7IYeWAgaQkU=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF4+V2g9oQ9lsJ+3SzSbs
sha1_base64="80m6n5PDVYdXZSas7IYeWAgaQkU=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF4+V2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJaPZpqgH9GR5CFn1Fip2RzUBuWKW3UXIOvEy0kFcjQG5a/+MGZphNIwQbXueW5i/Iwqw5nAWamfakwom9AR9iyVNELtZ4tTZ+TCKkMSxsqWNGSh/p7IaKT1NApsZ0TNWK96c/E/r5ea8MbPuExSg5ItF4WpICYm87/JkCtkRkwtoUxxeythY6ooMzadkg3BW315nbRrVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDUx418</latexit>

S3

c3

c2

S4

S5

c5

c4

Generative Latent Bank

D0

D1

D2

D3

f0
Decoder

32⇥32<latexit sha1_base64="SkG7ZIo5s5bVr0KcD70XgA+I5GY=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0laQY9FLx4r2A9oQtlst+3SzSbsToQS+je8eFDEq3/Gm//GbZuDtj4YeLw3w8y8MJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1LDpVC8hQIl7yaa0yiUvBNO7uZ+54lrI2L1iNOEBxEdKTEUjKKV/Hot81FE3MzqtX654lbdBcg68XJSgRzNfvnLH8QsjbhCJqkxPc9NMMioRsEkn5X81PCEsgkd8Z6lito9Qba4eUYurDIgw1jbUkgW6u+JjEbGTKPQdkYUx2bVm4v/eb0UhzdBJlSSIldsuWiYSoIxmQdABkJzhnJqCWVa2FsJG1NNGdqYSjYEb/XlddKuVT236j1cVRq3eRxFOINzuAQPrqEB99CEFjBI4Ble4c1JnRfn3flYthacfOYU/sD5/AFPQJEy</latexit>

F gu e 2 Overv ew o GLEAN n add on o he a en vec o s c he gene a oe he gene a ve a en bank s a so cond onedon he mu
eso u on ea u es f W h a p e a ned GAN cap u ng he na u a mage p o h s encode bank decode des gn s hebu den o ea n ng bo h fide y and na u a ness n he conven ona encode decode a ch ec u e E S and D deno e he encode b ocks
a en bank b ocks and decode b ocks espec ve y Th s examp e co esponds o an npu s ze o 32×32 and an ou pu s ze o 256×256

3 Methodology

where E i ∈ {1N } deno es a s ack of a s r de-2
convo u on and a s r de-1 convo u on F na y a convou on and a fu y-connec ed ayer are used o genera e hea en vec ors
C = EN +1 (fN )(2)

A GAN mode ha s ra ned on arge-sca e na ura mages cap ures r ch ex ure and shape pr ors Prev ous s udes [1 11 27 28] have shown ha such pr ors can be
harves ed hrough GAN nvers on o benefi var ous mageres ora on asks None he essrema ns underexp ored
how o exp o he pr ors w hou he expens ve op m zaon dur ng nvers on
In h s s udy we dev se GLEAN w h n a nove encoderbank-decoder arch ec ure wh ch a ows one o exp o hegenera ve pr ors by need ng us a s ng e forward pass An
overv ew of he arch ec ure s dep c ed n F g 2 G ven asevere y downsamp ed LR mage GLEAN app es an encoder o ex rac a en vec ors and mu -reso u on convou ona fea ures wh ch cap ure mpor an h gh- eve cues
as we as spa a s ruc ure of he LR mage Such cues areused o cond on he a en bank wh ch fur her produces
ano her se of mu -reso u on fea ures for he decoder F na y he decoder genera es he fina ou pu by n egra ng
he fea ures from bo h he encoder and he a en bank In
h s work we adop S y eGAN [16 17] as he genera ve
a en bank due o s excep ona performance The deaof a en bank can be ex ended o o her genera ors such as
B gGAN [2]where C s a ma r x whose co umns represen he a en vecors for he S y eGAN
The a en vec ors n C cap ure a compressed represena on of he mages prov d ng he genera ve a en bankw h h gh- eve nforma on To fur her cap ure he ocas ruc ures of he LR mage and o prov de add ona gu dance for s ruc ure res ora on we a so feed mu -reso u on
convo u ona fea ures {f } n o he a en bank

3 2 Generative Latent Bank
G ven he convo u ona fea ures {f } and he a en vecors C we everage a pre- ra ned genera or as a a en banko prov de pr ors for ex ure and de a genera on As S y eGAN s or g na y des gned for mage genera on asks
canno be d rec y n egra ed n o he proposed encoderbank-decoder framework In h s work we adap S y eGAN
o our SR ne work by mak ng hree mod fica ons
1 Ins ead of ak ng one s ng e a en vec or as he npu each b ock of he genera or akes a d fferen aen vec or o mprove express veness More spec fica y we have C=(c0ck−1 ) for k b ocks where
each c corresponds o one a en vec or We findha h s mod fica on eads o ou pu s w h fewer arfac s Th s mod fica on s a so seen n prev ous
works [11 38 49]

3 1 Encoder
To genera e he a en vec ors we firs use an RRDBNe [35] (deno ed as E0 ) o ex rac fea ures f0 from he
npu LR mage Then we gradua y reduce he reso u onof he fea ures by
f = E (f −1 )i ∈ {1N}(1)

2 To a ow cond on ng on he add ona fea ures from3

3.4. Training

the encoder, we use an additional convolution in eachstyle block for feature fusion:(
S0 (c0 , fN ),if i = 0,(3)
gi =Si (ci , gi−1 , fN −i ), otherwise,

Similar to existing works [21, 34, 35], we adopt the standard l2 loss, perceptual loss [14], and adversarial loss fortraining. More details on the loss function can be foundin the appendix. To exploit the generative prior, we keep
the weights of the latent bank fixed throughout training.
In our preliminary experiments, finetuning the latent bankwith the encoder and decoder demonstrates no noticeable
improvements. Moreover, it potentially harms the generalizability of the model as the latent bank may eventuallybias to the training distribution. It is worth emphasizing that
despite GLEAN is trained with similar objectives as in existing works (e.g. ESRGAN), the main difference to thesemethods is that GLEAN leverages a pre-trained generator
to directly incorporate the priors into the network, further
improving the output quality. We show that the improvement is not due to additional parameters in the generator
by comparing GLEAN with ESRGAN+ , a larger ESRGAN
that has similar FLOPs to GLEAN.

where Si denotes the augmented style block with anadditional convolution, and gi corresponds to the output feature of the i-th augmented style block.
3. Instead of directly generating outputs from the generator, we output the features {gi } and pass them to thedecoder to better fuse the features from the latent bankand encoder.
Advantages. The use of generative latent bank is reminiscent of the task of reference-based SR [23, 24, 39, 45, 47],
where external HR reference image(s) are employed as an
explicit imagery dictionary. While the external HR information leads to marked improvements, the performanceis sensitive to the similarity between the inputs and references. This sensitivity may eventually lead to degraded results when the reference images/components are not wellselected. Moreover, the size and diversity of those imagery
dictionaries are limited by the selected components, impeding the generalization to diverse scenes in practice. In addition, computationally-intensive global matching [47] orcomponent detection/selection [23] is often required to aggregate appropriate information from the references, hindering the applications to scenarios with tight computational constraints. Instead of constructing an imagery dictionary, GLEAN adopts a GAN-based dictionary conditioned on a pre-trained GAN. Our dictionary does not depend on any specific components or images. Instead, it
captures the distribution of the images and has potentiallyunlimited size and diversity. Furthermore, GLEAN is computationally efficient without requiring global matching andreference images/components selection.

4. Experiments
We adopt pre-trained StyleGAN1 [16] or StyleGAN22 [17] (depending on the availability of pre-trainedmodels) as our latent bank, and use the publicly available
codes of existing methods for the comparison in this section. To maintain fairness, we train our model and baselines
on the same datasets, including FFHQ [16] and LSUN [41],so that the difference in restoration quality is mainly causedby the algorithms instead of the training distribution. Test
set is strictly exclusive from the training. Detailed experimental settings are provided in the appendix.
Qualitative comparison. The qualitative comparison on
16× SR is shown in Fig. 3. Guided by low-dimensional
vectors and constraints in LR space, the outputs of GAN inversion methods are unable to maintain a good fidelity. In
particular, PULSE [27] and mGANprior [11] fail to restore
a face image with the same identity. In addition, artifacts areobserved in their outputs. Through finetuning the generator
during optimization, the result of DGP [28] demonstrates
significant improvements in both quality and fidelity. However, a slight difference between the identities of the output
and ground-truth is still observed. For example, the eyes
and lips show noticeable differences.
Methods trained with adversarial loss (SinGAN [30],
ESRGAN+3 [35]) can preserve the local structures, but failin synthesizing convincing textures and details. Specifically, SinGAN fails to capture the natural image style, producing a painting-like image. Although ESRGAN+ is capable of generating a realistic image, it struggles to synthesize

3.3. Decoder
GLEAN uses an additional decoder with progressive fusion to integrate the features from the encoder and latentbank to generate the output image. It takes the RRDBNet
features as inputs and progressively fuse the features withthe multi-resolution features from the latent bank:(
D0 (f0 )if i = 0,di =(4)
Di (di−1 , gN −2+i ) otherwise,
where Di and di denote a 3×3 convolution and its output, respectively. Each convolution is followed by a pixelshuffle [32] layer except the final output layer. With theskip-connection between the encoder and decoder, the information captured by the encoder can be reinforced andhence the latent bank could focus more on the texture anddetail generation.

1 GenForce: https://github.com/genforce/genforce
2 BasicSR: https://github.com/xinntao/BasicSR
3 A larger version of ESRGAN with similar FLOPs to GLEAN.

4

LR

PULSE

mGANprior

DGP

SinGANESRGAN+GLEAN (ours)

GT

ESRGAN+GLEAN (ours)

GT

Figure 3: Comparisons on 16× SR on CelebA-HQ [15]. Only GLEAN is able to maintain high fidelity while synthesizing realistic
textures and details: GAN inversion methods fail to preserve the identity, and adversarial loss methods struggle to synthesize fine details.
ESRGAN+ denotes a larger version with similar FLOPs to GLEAN. (Zoom-in for best view)LR

ESRGAN+GLEAN (ours)

GTLR

Output16⇥<latexit sha1_base64="rxRIt1AQ5D/shYQTPNM0J0Cl5ro=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFhPBKtylUMuAjWUE84HJEfY2e8mSvb1jd04IR/6FjYUitv4bO/+Nm+QKTXww8Hhvhpl5QSKFQdf9dgobm1vbO8Xd0t7+weFR+fikbeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqAacJ9yM6UiIUjKKVHr2rah9FxE11UK64NXcBsk68nFQgR3NQ/uoPY5ZGXCGT1Jie5yboZ1SjYJLPSv3U8ISyCR3xnqWK2i1+trh4Ri6sMiRhrG0pJAv190RGI2OmUWA7I4pjs+rNxf+8XorhjZ8JlaTIFVsuClNJMCbz98lQaM5QTi2hTAt7K2FjqilDG1LJhuCtvrxO2vWa59a8+3qlUc3jKMIZnMMleHANDbiDJrSAgYJneIU3xzgvzrvzsWwtOPnMKfyB8/kDUIaP6w==</latexit>

GT<latexit sha1_base64="4xx79QfBZSK1kvics7A9yt22cVk=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LLaCp5LUgx4LXjxWsK3QhLLZbtqlm03YnQgl9G948aCIV/+MN/+N2zYHbX0w8Hhvhpl5YSqFQdf9dkobm1vbO+Xdyt7+weFR9fika5JMM95hiUz0Y0gNl0LxDgqU/DHVnMah5L1wcjv3e09cG5GoB5ymPIjpSIlIMIpW8utXzdxHEXMzqw+qNbfhLkDWiVeQGhRoD6pf/jBhWcwVMkmN6XtuikFONQom+aziZ4anlE3oiPctVdSuCfLFzTNyYZUhiRJtSyFZqL8nchobM41D2xlTHJtVby7+5/UzjG6CXKg0Q67YclGUSYIJmQdAhkJzhnJqCWVa2FsJG1NNGdqYKjYEb/XlddJtNjy34d03a616EUcZzuAcLsGDa2jBHbShAwxSeIZXeHMy58V5dz6WrSWnmDmFP3A+fwAXJJD1</latexit>

32⇥<latexit sha1_base64="z5JffJQSFanlcryPKpfYahrU7Zs=">AAAB83icbVDLSgNBEOyNrxhfUY9eBhPBU9gNoh4DXjxGMA/ILmF2MpsMmZ1dZnqFsOQ3vHhQxKs/482/cfI4aGJBQ1HVTXdXmEph0HW/ncLG5tb2TnG3tLd/cHhUPj5pmyTTjLdYIhPdDanhUijeQoGSd1PNaRxK3gnHdzO/88S1EYl6xEnKg5gOlYgEo2glv3p9lfsoYm6m1X654tbcOcg68ZakAks0++Uvf5CwLOYKmaTG9Dw3xSCnGgWTfFryM8NTysZ0yHuWKmrXBPn85im5sMqARIm2pZDM1d8TOY2NmcSh7YwpjsyqNxP/83oZRrdBLlSaIVdssSjKJMGEzAIgA6E5QzmxhDIt7K2EjaimDG1MJRuCt/ryOmnXa55b8x7qlUZ1GUcRzuAcLsGDG2jAPTShBQxSeIZXeHMy58V5dz4WrQVnOXMKf+B8/gAe6JD6</latexit>

64⇥

Figure 5: Results on larger scale factors. GLEAN reconstructs
realistic images highly similar to the GT for up to 64× upscaling
factor. (Zoom-in for best view)

Figure 4: Results of 16× SR on other categories. GLEAN canbe applied to various categories by switching between StyleGANs
trained on different categories. (Zoom-in for best view)and details. We further extend our method to larger scale
factors in Fig. 5. GLEAN successfully generates perceptually convincing images resembling the ground-truth for up
to 64× upscaling.

fine details and introduces unnatural artifacts in detailed regions. It is worth emphasizing that although ESRGAN+
achieves competitive results on human faces, its performances on other categories such as cats and cars are lesspromising (see Fig. 1 and Fig. 4). With the latent bankproviding natural image priors, GLEAN succeeds in bothfidelity and naturalness. For example, when compared to
ESRGAN+ , GLEAN reconstructs eyes with better shape

Robustness to poses and contents. Another appealing
property of GLEAN is its robustness to the changes in posesand contents. As shown in Fig. 6, guided by the convolutional features, GLEAN is still able to construct realistic
5

GT

Only
latent vectors

GLEAN (ours)

Resolution{4, 8, 16, 32, 64}

Resolution{4, 8, 16}

PULSE

GT

Figure 6: Outputs with diverse poses and contents. Despite
GLEAN is trained with aligned human faces, it is able to reconstruct faithful images for non-aligned and non-human faces.
PULSE approximates the GT in low resolution (bottom left), but
its outputs are significantly different from the GT when viewed inhigh resolution.

Table 1: Cosine similarity of ArcFace Embeddings [5]. GLEAN
achieves a higher similarity than baselines. Bolded texts represent
the best performance.
Figure 7: Effects of the multi-resolution encoder features.
Without the convolutional features, the outputs can only resemble the global attributes (e.g. hair color, pose). When adding theencoder features progressively, the network can capture more local
structures, better approximating the GT.

PULSE [27] mGANprior [11] DGP [28]Similarity
0.40470.55260.7341SinGAN [30] ESRGAN+ [35] GLEAN
Similarity
0.77180.95990.9678GLEAN in producing outputs with high fidelity, we extract
100 images from CelebA-HQ [15] and compute the cosine
similarity to the ground-truth on the ArcFace embedding
space [5]. As shown in Table 1, GLEAN achieves higher
similarity than the baseline methods, validating the superiority of GLEAN.
We additionally provide the quantitative comparison ondifferent categories in Table 2. For each category, we
select 100 images and compute their average PSNR and
LPIPS [42]. It is observed that mGANprior and PULSE
perform significantly worse as they fail to restore the original objects. GLEAN outperforms these methods in mostcategories, suggesting its effectiveness in generating images
with high quality and fidelity.

Table 2: Quantitative (PSNR/LPIPS) comparison on 16×
SR. GLEAN outperforms other methods in most categories.
ESRGAN+ denotes a larger version of ESRGAN [35] having similar FLOPs to GLEAN. Bolded texts represent the best performance.
Face [15]Cat [43]
Car [20]
Bedroom [41]Tower [41]mGANprior [11] PULSE [27] ESRGAN+ [35] GLEAN
23.66/0.4661 21.83/0.4600 26.76/0.2787 26.84/0.2681
17.01/0.5556 19.78/0.5241 19.99/0.3482 20.92/0.3215
14.53/0.7228 16.30/0.6491 19.42/0.3006 19.74/0.2830
16.38/0.5439 12.97/0.7131 19.47/0.3291 19.44/0.3310
15.96/0.4870 13.62/0.7066 17.86/0.3132 18.41/0.2850

images when the images are non-aligned and contain nonhuman faces despite it is trained on aligned human faces. In
contrast, the outputs of PULSE are biased to aligned humanfaces. Its outputs can only approximate the ground-truths
in low resolution. Such robustness enables GLEAN to be
applied to diverse categories and scenes such as cats, cars,
bedrooms, and towers. Examples are shown in Fig. 4 andmore results are provided in the appendix.

5. Ablation Studies
Importance of multi-resolution encoder features. We
demonstrate how the convolutional features generated fromthe encoder assist in the restoration of fine details and local
structures. We start with only the latent vectors and observe
the transition when features are gradually introduced to the

Quantitative comparison. To demonstrate the ability of6Resolution{32, 64}

Withoutbank features

LR

only the latent vectors. Similar to those methods, the network is able to synthesize realistic images given the latentvectors. However, guided only by low-dimensional vectors,
in which spatial information is not well-preserved, the network restores only the global attributes such as hair color
and poses, but fails to preserve finer details. When providing coarse (from 4×4 to 16×16) convolutional features
to the latent bank, more details are recovered and the outputs are better approximating the ground-truths. Further improvements in both quality and fidelity are observed whenfiner features are passed to the latent bank. The above observations corroborate our hypothesis that the convolutionalfeatures are pivotal in guiding the restoration of fine details
and local structures, which cannot be reconstructed with
only the latent vectors.

Resolution{32, 64, 128, 256}

Effects of latent bank features. To understand the contributions of the latent bank, we investigate the effects broughtby the latent bank features. We start by discarding all the latent bank features, and progressively pass the features to thedecoder. The comparison is shown in Fig. 8. Lacking appropriate prior information, the network is responsible forboth generating realistic details and maintaining fidelity to
the ground-truths. Such a demanding objective eventuallyleads to outputs that contain flaws in both structure restoration and texture generation. With the latent bank, the burdenof texture and details generation is reduced as the generator already captures rich image priors. Therefore, improvements in both structures and textures are observed when
passing finer features to the decoder.

Figure 8: Effects of the latent bank features. The rich texture
priors captured in the generator lift the burden of the encoder intexture generation. Improvements on both texture and structures
are observed when finer features are inserted into the decoder.
(Zoom-in for best view)

w/o decoder

w/ decoder

Importance of decoder. As shown in Fig. 9, without the
decoder, despite being perceptually convincing overall, theoutput image contains unpleasant artifacts when zoomed in.
The decoder allows the network to aggregate the information in a coarse-to-fine manner, leading to more natural details. In addition, the multi-scale skip-connections between
the encoder and decoder reinforce the spatial information
captured in the encoder features so that the latent bank could
focus more on detail generation, further enhancing the output quality.
Comparisons with reference-based methods. We assess
the efficacy of the new notion of GAN-based dictionary bycomparing GLEAN with two representative methods adopting an imagery dictionary for SR – SRNTT [47] and DFDNet [23]. Examples are shown in Fig. 10.
For DFDNet, we evaluate the performance on LR images
with unknown degradations4 . Through pre-constructing a
dictionary of facial components (e.g. eyes, lips), DFDNet
shows remarkable performance on face restoration. However, it cannot produce faithful results on parts absent in the
dictionary, such as skin and hair. Therefore, significant in-

Figure 9: Contributions of the decoder. The decoder reinforces
the spatial information captured in the encoder features and aggregate them in a coarse-to-fine manner, leading to enhanced quality.

latent bank as conditions. To discard the effects brought by
the decoder, we test with a variant of GLEAN where thegenerator directly produces the output images. The comparison is depicted in Fig. 7.
When all convolutional features are discarded, GLEAN
resembles the typical GAN inversion methods that learn

4 We further downsample the LR images to 64×64 to match the input sizeof GLEAN.

7

LR

DFDNetOriginal

GLEAN (ours)

Retouched

RefinedFigure 11: Image Retouching. GLEAN can be used to eliminate
unnatural artifacts introduced by amateur retouching. (Zoom-in
for best view)

the applicability to large-factor SR, where image matching is extremely difficult. More importantly, with no external images employed, GLEAN does not require any global
matching to search for suitable textures/details. This allows
GLEAN to be applied to images with larger resolutions,where global matching is computationally prohibitive.

(a) Comparison with DFDNet [23]
LR

SRNTT

GLEAN (ours)

6. Application – Image Retouching
In this section, we present one interesting application of
GLEAN. In interactive image retouching, users can manually edit the images based on their preference. However, aperfect output requires tedious and precise retouching. As aresult, artifacts are common in the outputs, especially thosefrom amateur retouching. As a powerful super-resolver,
GLEAN can be used as an image retouching tool to eliminate unpleasant artifacts.
As shown in Fig. 11, the blending operation in the interactive editing software produces a blurry and incoherent
output. Thanks to the capability of GLEAN in producing
high quality and fidelity images, GLEAN is able to eliminate the blurry region and generate a coherent output withnatural textures. Furthermore, with only a single forward
pass for generation, it can be easily incorporated into existing interactive editing software. More examples will beshown in the appendix.

(b) Comparison with SRNTT [47]Figure 10: Comparison to imagery dictionary. (a) DFDNet fails
to restore components absent in the dictionary (e.g. skin, hair),leading to incoherent outputs. (b) SRNTT is unable to produce
faithful fur textures.

coherence is observed in the outputs. Despite GLEAN is
trained on the bicubic kernel, it is still capable of producing appealing outputs. More importantly, GLEAN is notconfined to improving the visual quality of specific components. Instead, the entire image is super-resolved, leading tocoherent and pleasing results. The performance of GLEAN
could be further improved by employing multiple degradations during training.
For SRNTT, we follow the same settings and downsample the ground-truth images using the bicubic kernel. With
such low-resolution images (32×32), global matching becomes prohibitive, and hence SRNTT fails to transfer the
textures from HR reference images. As a result, SRNTT
tends to provide blurry textures. By capturing the distribution instead of specific imagery clues, GLEAN does not relyon any explicit textural transferal procedure. This enables

7. Conclusion
We have presented a new way to exploit pre-trained
GANs for the task of large-scale super-resolution, up to 64×
upscaling factor. We have shown that a pre-trained GAN
can be used as a generative latent bank in an encoder-bankdecoder architecture. Reconstructing photorealistic HR images requires just a single forward pass, thanks to effective ways in conditioning and retrieving rich priors from thebank. The generality of the notion of GAN-based dictionary allows GLEAN to be potentially extended to not only
diverse architectures but also various imaging tasks, such asimage denoising, inpainting and colorization.

8

Acknowledgement. This research was conducted in collaboration with SenseTime and supported by the Singapore
Government through the Industry Alignment Fund - Industry Collaboration Projects Grant. It is also partially supported by Singapore MOE AcRF Tier 1 (2018-T1-002-056)
and NTU SUG.

[17] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten,
Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. arXiv preprint
arXiv:1912.04958, 2019. 3, 4, 11[18] Deokyun Kim, Minseon Kim, Gihyun Kwon, and Dae-Shik
Kim. Progressive face super-resolution via attention to facial
landmark. In BMVC, 2019. 2[19] Diederik Kingma and Jimmy Ba. Adam: A method forstochastic optimization. In ICLR, 2015. 11[20] Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei Li.
3D object representations for fine-grained categorization. In
ICCV, 2013. 6, 11[21] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero,
Andrew Cunningham, Alejandro Acosta, Andrew P Aitken,
Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photorealistic single image super-resolution using a generative adversarial network. In CVPR, 2017. 1, 2, 4
[22] Mengyan Li, Yuechuan Sun, Zhaoyu Zhang, Haonian Xie,and Jun Yu. Deep learning face hallucination via attributes
transfer and enhancement. In ICME, 2019. 2[23] Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui
Lin, Wangmeng Zuo, and Lei Zhang. Blind face restoration via deep multi-scale component dictionaries. In ECCV,
2020. 4, 7, 8
[24] Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang,
Meng Wang, and Wangmeng Zuo. Enhanced blind facerestoration with multi-exemplar images and adaptive spatial
feature fusion. In CVPR, 2020. 4[25] Ilya Loshchilov and Frank Hutter.
SGDR: Stochastic gradient descent with warm restarts. arXiv preprint
arXiv:1608.03983, 2016. 11[26] Cheng Ma, Zhenyu Jiang, Yongming Rao, Jiwen Lu, and Jie
Zhou. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In
CVPR, 2020. 2[27] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi,
and Cynthia Rudin. PULSE: Self-supervised photo upsampling via latent space exploration of generative models. In
CVPR, 2020. 1, 2, 3, 4, 6, 11, 12[28] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin,
Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation.
In ECCV, 2020. 1, 2, 3, 4, 6
[29] Mehdi S M Sajjadi, Bernhard Schölkopf, and Michael
Hirsch. Enhancenet: Single image super-resolution throughautomated texture synthesis. In ICCV, pages 4501–4510,
2017. 2[30] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. SinGAN: Learning a generative model from a single natural image. In ICCV, 2019. 4, 6
[31] Taizhang Shang, Qiuju Dai, Shengchen Zhu, Tong Yang, andYandong Guo. Perceptual extreme super resolution networkwith receptive field block. In CVPRW, 2020. 2[32] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz,
Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan
Wang. Real-time single image and video super-resolution
using an efficient sub-pixel convolutional neural network. In
CVPR, pages 1874–1883, 2016. 4

References[1] David Bau, Hendrik Strobelt, William Peebles, Bolei Zhou,
Jun-Yan Zhu, Antonio Torralba, et al. Semantic photo manipulation with a generative image prior. TOG, 2020. 1, 2,
3[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large
scale gan training for high fidelity natural image synthesis.
arXiv preprint arXiv:1809.11096, 2018. 3[3] Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. Pixel
recursive super resolution. In ICCV, 2017. 2[4] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, andLei Zhang. Second-order attention network for single image
super-resolution. In CVPR, 2019. 2[5] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos
Zafeiriou. ArcFace: Additive angular margin loss for deep
face ... In CVPR, 2019. 6[6] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou
Tang. Learning a deep convolutional network for image
super-resolution. In ECCV, 2014. 2[7] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou
Tang. Image super-resolution using deep convolutional networks. TPAMI, 38(2):295–307, 2016. 2[8] Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In
ECCV, pages 391–407. Springer, 2016. 2[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing
Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, andYoshua Bengio. Generative adversarial nets. In NeurIPS,
2014. 1, 11[10] Klemen Grm, Walter J Scheirer, and Vitomir Štruc. Face hallucination using cascaded super-resolution and identity priors. TIP, 2019. 2[11] Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing
using multi-code GAN prior. In CVPR, 2020. 1, 2, 3, 4, 6
[12] Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan
Yang, and Jian Cheng. Ode-inspired network design for single image super-resolution. In CVPR, 2019. 2[13] Sangeek Hyun and Jae-Pil Heo. VarSR: Variational superresolution network for very low resolution images. In ECCV,
2020. 2[14] Justin Johnson, Alexandre Alahi, and Fei-Fei Li. Perceptual
losses for real-time style transfer and super-resolution. In
ECCV, pages 694–711. Springer, 2017. 4, 11[15] Tero Karras, Timo Ailo, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of GANs for improved quality, stability,and variation. In ICLR, 2018. 5, 6, 11, 12[16] Tero Karras, Samuli Laine, and Timo Aila. A style-based
generator architecture for generative adversarial networks. In
CVPR, 2019. 2, 3, 4, 11

9

[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR,
2015. 11[34] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.
Recovering realistic texture in image super-resolution by
deep spatial feature transform. In CVPR, 2018. 1, 4, 11[35] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu,
Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang.
ESRGAN: Enhanced super-resolution generative adversarial
networks. In ECCVW, 2018. 1, 2, 3, 4, 6, 11[36] Xiangyu Xu, Yongrui Ma, and Wenxiu Sun. Towards realscene super-resolution with raw images. In CVPR, 2019. 2[37] Xiangyu Xu, Deqing Sun, Jinshan Pan, Yujin Zhang,Hanspeter Pfister, and Ming-Hsuan Yang. Learning to superresolve blurry face and text images. In ICCV, 2017. 2[38] Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, andBolei Zhou. Generative hierarchical features from synthesizing images. arXiv preprint arXiv:2007.10379, 2020. 3[39] Xu Yan, Weibing Zhao, Kun Yuan, Ruimao Zhang, Zhen
Li, and Shuguang Cui. Towards content-independent multireference super-resolution: Adaptive pattern matching andfeature aggregation. In ECCV, 2020. 4[40] Jianchao Yang, John Wright, Thomas S Huang, and Yi
Ma. Image super-resolution via sparse representation. TIP,
19(11):20861–2873, 2010. 2[41] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas
Funkhouser, and Jianxiong Xiao. LSUN: Construction ofa large-scale image dataset using deep learning with humans
in the loop. arXiv preprint arXiv:1506.03365, 2015. 4, 6, 11[42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectivenessof deep features as a perceptual metric. arXiv preprint
arXiv:1801.03924, 2018. 6[43] Weiwei Zhang, Jian Sun, and Xiaoou Tang. Cat head detection - how to effectively exploit shape and texture features.
In ECCV, 2008. 6, 11[44] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng
Zhong, and Yun Fu. Image super-resolution using very deep
residual channel attention networks. In ECCV, 2018. 2[45] Yang Zhang, Ivor W Tsang, Yawei Luo, Changhui Hu, Xiaobo Lu, and Xin Yu. Copy and Paste GAN: Face hallucination from shaded thumbnails. In CVPR, 2020. 4[46] Yulun Zhang, Zhifei Zhang, Stephen DiVerdi, Zhaowen
Wang, Jose Echevarria, and Yun Fu. Texture hallucination
for large-scale painting super-resolution. In ECCV, 2020. 2[47] Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. Image super-resolution by neural texture transfer. In CVPR,
2019. 4, 7, 8
[48] Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo, andChen Change Loy. Cross-scale internal graph neural network
for image super-resolution. In NeurIPS, 2020. 2[49] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. Indomain GAN inversion for real image editing. In ECCV,
2020. 310

In all our experiments, we set αpercep =αgen =10−2 . For
the discriminator, we maximize

Appendix
We first provide the implementation details of GLEAN
in Sec. A. We then provide additional qualitative results on
various categories and scale factors in Sec. B.1. Finally, wedemonstrate the application of GELAN to the task of image
retouching in Sec. B.2.

Ld = log (1 − D(ŷ)) + log D(y).

We adopt Cosine Annealing Scheme [25] and Adam optimizer [19] in training. The number of iterations is 300K
and the initial learning rate is 10−4 . The batch size is 8
for human faces and 16 for other categories. We train ourmodels using two Nvidia V100 GPUs.

A. Training Details of GLEAN
We adopt pre-trained StyleGAN5 [16] or StyleGAN26 [17] as our generative latent bank. In this section,
we assume the latent bank is pre-trained and present thetraining details of GLEAN (i.e. the encoder-bank-decoder
network). Note that the weights of the latent bank are fixed
when training GLEAN to better employ the generative priorand to avoid biasing to the training distribution.
We train GLEAN on five categories including humanfaces, cats, cars, towers, and bedrooms. The training andtest datasets used in our experiments are summarized in Table 3. Since StyleGAN produces images with fixed size, we
resize the images in the datasets for our experiments.

B. Qualitative Results
B.1. Super-Resolution
Randomly-Selected Examples. In Fig. 12, we show the results of randomly-selected examples from CelebA-HQ [15].
By optimizing only the latent codes, PULSE [27] produces
outputs with low-fidelity. In contrast, guided by the encoder
features and our generative latent bank, GLEAN achieves
remarkable quality and fidelity, demonstrating the effectiveness of our designs.
Scale Factors and Categories. GLEAN is extensible tovarious scale factors (from 8× to 64×) and categories(e.g. faces, cats, cars, bedrooms, towers). From Fig. 13
to Fig. 18, we see that GLEAN outperforms DGP and
ESRGAN+ in both fidelity and quality. It is noteworthy thatthe performance of DGP and ESRGAN+ are less promisingon categories other than human faces.

Table 3: Datasets used in our experiments.

Human faces
Cats
Cars
Bedrooms
TowersTrain
Test
FFHQ [16]CelebA-HQ [15]LSUN-train [41]
CAT [43]
LSUN-train [41]Cars [20]LSUN-train [41] LSUN-validate [41]LSUN-train [41] LSUN-validate [41]B.2. Image Retouching
In interactive image retouching, users can manually edit
the images based on their preference. For instance, users
can change the facial expression of an object and performgeometric transformations for enlarging eyes. However, aperfect output requires tedious and precise retouching. As
a result, artifacts are common in the outputs from amateur
retouching.
GLEAN allows the possibility of performing realistic refinement of imperfect retouching. More specifically, given
a retouched image, we can first downsample the image
to a smaller resolution, where the artifacts vanished. We
can then upsample it back to the original resolution. With
GLEAN as a powerful super-resolver, we can obtain an output with unnatural artifacts suppressed.
As shown in Fig. 19, GLEAN is able to correct the unnatural artifacts introduced by amateur retouching while being
similar to the retouched images, realistic, and coherent withthe unaltered regions. In addition, since GLEAN requires
only a single forward pass, it can be used in interactive image editing software to allow a more flexible retouching.

Following previous works [34, 35], the objective function for GLEAN consists of three terms. MSE loss is used
to guide the fidelity of the output images:
Lmse =1||ŷ − y||22 ,N(5)where N , ŷ, and y denote the number of pixels, the output
image, and the ground-truth image, respectively. We further
incorporate perceptual loss [14] and adversarial loss [9] to
improve the perceptual quality:1||f (ŷ) − f (y)||22 ,N
Lgen = log (1 − D(ŷ)) ,Lpercep =(6)(7)

where f (·) denotes the feature embedding space of the
VGG16 [33] network, and D corresponds to the StyleGAN discriminator. The resulting objective function is aweighted mean of the three losses:
Lg = Lmse + αpercep ·Lpercep + αgen ·Lgen .

(9)

(8)

5 GenForce: https://github.com/genforce/genforce
6 BasicSR: https://github.com/xinntao/BasicSR

11

LR

PULSE

GLEAN

GT

LR

PULSE

GLEAN

GT

Figure 12: Comparison to PULSE on randomly-selected examples from CelebA-HQ [15]. By optimizing only the latent vectors, theoutputs of PULSE [27] differ significantly from the ground-truths. With our novel designs, GLEAN produces outputs highly similar to theground-truths.

12

LR

DGP

ESRGAN+GLEAN (ours)

GT

Figure 13: Comparison with DGP and ESRGAN+ . The outputs of DGP show noticeable identity differences to the ground-truths.
ESRGAN+ shows unpleasant artifacts for the fine details. (Zoom-in for best view)

13

GT

LR

GT

GLEAN (ours)

LR

GLEAN (ours)

16⇥<latexit sha1_base64="S0L0HaD2ZQGj2VXvLmfy7s2n8NU=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LFbBU0mkqMeCF48V7Ae2oWy2m3bpZhN2J0IJ/RdePCji1X/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6wEnC/YgOlQgFo2ilR+8q66GIuJn2yxW36s5BVomXkwrkaPTLX71BzNKIK2SSGtP13AT9jGoUTPJpqZcanlA2pkPetVRRu8XP5hdPyblVBiSMtS2FZK7+nshoZMwkCmxnRHFklr2Z+J/XTTG88TOhkhS5YotFYSoJxmT2PhkIzRnKiSWUaWFvJWxENWVoQyrZELzll1dJ67LquVXvvlapn+VxFOEETuECPLiGOtxBA5rAQMEzvMKbY5wX5935WLQWnHzmGP7A+fwBXlaQnQ==</latexit><latexit sha1_base64="49s0XcXq3VepUpcmDQglNkEnh+M=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LFbBU0mqoMeCF48V7Ae2oWy2m3bpZhN2J0IJ/RdePCji1X/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6wEnC/YgOlQgFo2ilx8ta1kMRcTPtlytu1Z2DrBIvJxXI0eiXv3qDmKURV8gkNabruQn6GdUomOTTUi81PKFsTIe8a6midoufzS+eknOrDEgYa1sKyVz9PZHRyJhJFNjOiOLILHsz8T+vm2J442dCJSlyxRaLwlQSjMnsfTIQmjOUE0so08LeStiIasrQhlSyIXjLL6+SVq3quVXv/qpSP8vjKMIJnMIFeHANdbiDBjSBgYJneIU3xzgvzrvzsWgtOPnMMfyB8/kDW0CQmw==</latexit><latexit sha1_base64="Lsx+X4yDi/aI2OYSamXpUf4XvG8=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LFbBU0mkqMeCF48V7Ae2oWy2m3bpZhN2J0IJ/RdePCji1X/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6wEnC/YgOlQgFo2ilx6ta1kMRcTPtlytu1Z2DrBIvJxXI0eiXv3qDmKURV8gkNabruQn6GdUomOTTUi81PKFsTIe8a6midoufzS+eknOrDEgYa1sKyVz9PZHRyJhJFNjOiOLILHsz8T+vm2J442dCJSlyxRaLwlQSjMnsfTIQmjOUE0so08LeStiIasrQhlSyIXjLL6+S1mXVc6vefa1SP8vjKMIJnMIFeHANdbiDBjSBgYJneIU3xzgvzrvzsWgtOPnMMfyB8/kDYv+QoA==</latexit>

32⇥

64⇥

Figure 14: Performance of GLEAN on 16×, 32×, and 64× SR. GLEAN is able to synthesize images well resembling the ground-truths
for up to 64× upsampling.

14

GT

<latexit sha1_base64="NvEctk66R5ZLHJAcjApTq20Rfto=">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBQ8hV0RzDHgxWME85BkCbOTSTJkZnaZ6RXCkq/w4kERr36ON//GSbIHTSxoKKq66e6KEiks+v63t7a+sbm1Xdgp7u7tHxyWjo6bNk4N4w0Wy9i0I2q5FJo3UKDk7cRwqiLJW9H4dua3nrixItYPOEl4qOhQi4FgFJ30WM26KBS3016p7Ff8OcgqCXJShhz1Xumr249ZqrhGJqm1ncBPMMyoQcEknxa7qeUJZWM65B1HNXVbwmx+8JRcOKVPBrFxpZHM1d8TGVXWTlTkOhXFkV32ZuJ/XifFQTXMhE5S5JotFg1SSTAms+9JXxjOUE4cocwIdythI2ooQ5dR0YUQLL+8SppXlcCvBPfX5dp5HkcBTuEMLiGAG6jBHdShAQwUPMMrvHnGe/HevY9F65qXz5zAH3ifP/BukGQ=</latexit>

GT

DGP

ESRGAN+DGPESRGAN+LR

GLEAN (ours)

LR

GLEAN (ours)

LR

GLEAN (ours)

LR

GLEAN (ours)

8⇥

Figure 15: (Top) Comparison with DGP and ESRGAN+ on Cats. DGP produces outputs with low fidelity; ESRGAN+ fails to synthesize
realistic textures. (Bottom) Performance of GLEAN on 8× SR. GLEAN produces realistic outputs that are highly similar to the groundtruths. (Zoom-in for best view)

15

GT

<latexit sha1_base64="NvEctk66R5ZLHJAcjApTq20Rfto=">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBQ8hV0RzDHgxWME85BkCbOTSTJkZnaZ6RXCkq/w4kERr36ON//GSbIHTSxoKKq66e6KEiks+v63t7a+sbm1Xdgp7u7tHxyWjo6bNk4N4w0Wy9i0I2q5FJo3UKDk7cRwqiLJW9H4dua3nrixItYPOEl4qOhQi4FgFJ30WM26KBS3016p7Ff8OcgqCXJShhz1Xumr249ZqrhGJqm1ncBPMMyoQcEknxa7qeUJZWM65B1HNXVbwmx+8JRcOKVPBrFxpZHM1d8TGVXWTlTkOhXFkV32ZuJ/XifFQTXMhE5S5JotFg1SSTAms+9JXxjOUE4cocwIdythI2ooQ5dR0YUQLL+8SppXlcCvBPfX5dp5HkcBTuEMLiGAG6jBHdShAQwUPMMrvHnGe/HevY9F65qXz5zAH3ifP/BukGQ=</latexit>

GT

DGP

ESRGAN+DGPESRGAN+LR

GLEAN (ours)

LR

GLEAN (ours)

LR

GLEAN (ours)

LR

GLEAN (ours)

8⇥

Figure 16: (Top) Comparison with DGP and ESRGAN+ on Cars. DGP produces outputs with low fidelity; ESRGAN+ fails to synthesize realistic textures. (Bottom) Performance of GLEAN on 8× SR. GLEAN produces realistic outputs that are highly similar to theground-truths. (Zoom-in for best view)

16

GT

<latexit sha1_base64="NvEctk66R5ZLHJAcjApTq20Rfto=">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBQ8hV0RzDHgxWME85BkCbOTSTJkZnaZ6RXCkq/w4kERr36ON//GSbIHTSxoKKq66e6KEiks+v63t7a+sbm1Xdgp7u7tHxyWjo6bNk4N4w0Wy9i0I2q5FJo3UKDk7cRwqiLJW9H4dua3nrixItYPOEl4qOhQi4FgFJ30WM26KBS3016p7Ff8OcgqCXJShhz1Xumr249ZqrhGJqm1ncBPMMyoQcEknxa7qeUJZWM65B1HNXVbwmx+8JRcOKVPBrFxpZHM1d8TGVXWTlTkOhXFkV32ZuJ/XifFQTXMhE5S5JotFg1SSTAms+9JXxjOUE4cocwIdythI2ooQ5dR0YUQLL+8SppXlcCvBPfX5dp5HkcBTuEMLiGAG6jBHdShAQwUPMMrvHnGe/HevY9F65qXz5zAH3ifP/BukGQ=</latexit>

GT

DGP

ESRGAN+DGPESRGAN+LR

GLEAN (ours)

LR

GLEAN (ours)

LR

GLEAN (ours)

LR

GLEAN (ours)

8⇥

Figure 17: (Top) Comparison with DGP and ESRGAN+ on Bedrooms. DGP produces outputs with low fidelity; ESRGAN+ fails tosynthesize realistic textures. (Bottom) Performance of GLEAN on 8× SR. GLEAN produces realistic outputs that are highly similar to
the ground-truths. (Zoom-in for best view)

17

GT

<latexit sha1_base64="NvEctk66R5ZLHJAcjApTq20Rfto=">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBQ8hV0RzDHgxWME85BkCbOTSTJkZnaZ6RXCkq/w4kERr36ON//GSbIHTSxoKKq66e6KEiks+v63t7a+sbm1Xdgp7u7tHxyWjo6bNk4N4w0Wy9i0I2q5FJo3UKDk7cRwqiLJW9H4dua3nrixItYPOEl4qOhQi4FgFJ30WM26KBS3016p7Ff8OcgqCXJShhz1Xumr249ZqrhGJqm1ncBPMMyoQcEknxa7qeUJZWM65B1HNXVbwmx+8JRcOKVPBrFxpZHM1d8TGVXWTlTkOhXFkV32ZuJ/XifFQTXMhE5S5JotFg1SSTAms+9JXxjOUE4cocwIdythI2ooQ5dR0YUQLL+8SppXlcCvBPfX5dp5HkcBTuEMLiGAG6jBHdShAQwUPMMrvHnGe/HevY9F65qXz5zAH3ifP/BukGQ=</latexit>

GT

DGP

ESRGAN+DGPESRGAN+LR

GLEAN (ours)

LR

GLEAN (ours)

LR

GLEAN (ours)

LR

GLEAN (ours)

8⇥

Figure 18: (Top) Comparison with DGP and ESRGAN+ on Towers. DGP produces outputs with low fidelity; ESRGAN+ fails tosynthesize realistic textures. (Bottom) Performance of GLEAN on 8× SR. GLEAN produces realistic outputs that are highly similar to
the ground-truths. (Zoom-in for best view)18

Original

Amateur-Retouched

Refined by GLEAN

Figure 19: Results on image retouching. GLEAN can be used to correct unpleasant artifacts introduced by amateur retouching. (Zoom-in
for best view)

19