Latent Convolutional Models

1 INTRODUCTION . Learning good image priors is one of the core problems of computer vision and machine learning . One promising approach to obtaining such priors is to learn a deep latent model , where the set of natural images is parameterized by a certain simple-structured set or probabilistic distribution , whereas the complexity of natural images is tackled by a deep ConvNet ( often called a generator or a decoder ) that maps from the latent space into the space of images . The best known examples are generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) and autoencoders ( Goodfellow et al. , 2016 ) . Given a good deep latent model , virtually any image restoration task can be solved by finding a latent representation that best corresponds to the image evidence ( e.g . the known pixels of an occluded image or a low-resolution image ) . The attractiveness of such approach is in the universality of the learned image prior . Indeed , applying the model to a new restoration task can be performed by simply changing the likelihood objective . The same latent model can therefore be reused for multiple tasks , and the learning process needs not to know the image degradation process in advance . This is in contrast to task-specific approaches that usually train deep feed-forward ConvNets for individual tasks , and which have a limited ability to generalize across tasks ( e.g . a feed-forward network trained for denoising can not perform large-hole inpainting and vice versa ) . At the moment , such image restoration approach based on latent models is limited to low-resolution images . E.g . ( Yeh et al. , 2017 ) showed how a latent model trained with GAN can be used to perform inpainting of tightly-cropped 64× 64 face images . Below , we show that such models trained with GANs can not generalize to higher resolution ( eventhough GAN-based systems are now able to obtain high-quality samples at high resolutions ( Karras et al. , 2018 ) ) . We argue that it is the limited dimensionality of the latent space in GANs and other existing latent models that precludes them from spanning the space of high-resolution natural images . To scale up latent modeling to high-resolution images , we consider latent models with tens of thousands of latent dimensions ( as compared to few hundred latent dimensions in existing works ) . We show that training such latent models is possible using direct optimization ( Bojanowski et al. , 2018 ) and that it leads to good image priors that can be used across a broad variety of reconstruction tasks . In previous models , the latent space has a simple structure such as a sphere or a box in a Euclidean space , or a full Euclidean space with a Gaussian prior . Such choice , however , is not viable in our ∗Currently at Stony Brook University . †Currently also with Samsung AI Center , Moscow . case , as vectors with tens of thousands of dimensions can not be easily used as inputs to a generator . Therefore , we consider two alternative parameterizations of a latent space . Firstly , as a baseline , we consider latent spaces parameterized by image stacks ( three-dimensional tensors ) , which allows to have “ fully-convolutional ” generators with reasonable number of parameters . Our full system uses a more sophisticated parameterization of the latent space , which we call a convolutional manifold , where the elements of the manifold correspond to the parameter vector of a separate ConvNet . Such indirect parameterization of images and image stacks have recently been shown to impose a certain prior ( Ulyanov et al. , 2018 ) , which is beneficial for restoration of natural images . In our case , we show that a similar prior can be used with success to parameterize high-dimensional latent spaces . To sum up , our contributions are as follows . Firstly , we consider the training of deep latent image models with the latent dimensionality that is much higher than previous works , and demonstrate that the resulting models provide universal ( w.r.t . restoration tasks ) image priors . Secondly , we suggest and investigate the convolutional parameterization for the latent spaces of such models , and show the benefits of such parameterization . Our experiments are performed on CelebA ( Liu et al. , 2015 ) ( 128x128 resolution ) , SUN Bedrooms ( Yu et al. , 2015 ) ( 256x256 resolution ) , CelebA-HQ ( Karras et al. , 2018 ) ( 1024x1024 resolution ) datasets , and we demonstrate that the latent models , once trained , can be applied to large hole inpainting , superresolution of very small images , and colorization tasks , outperforming other latent models in our comparisons . To the best of our knowledge , we are the first to demonstrate how “ direct ” latent modeling of natural images without extra components can be used to solve image restoration problems at these resolutions ( Figure 1 ) . Other related work . Deep latent models follow a long line of works on latent image models that goes back at least to the eigenfaces approach ( Sirovich & Kirby , 1987 ) . In terms of restoration , a competing and more popular approach are feed-forward networks trained for specific restoration tasks , which have seen rapid progress recently . Our approach does not quite match the quality of e.g . ( Iizuka et al. , 2017 ) , that is designed and trained specifically for the inpainting task , or the quality of e.g . ( Yu & Porikli , 2016 ) that is designed and trained specifically for the face superresolution task . Yet the models trained within our approach ( like other latent models ) are universal , as they can handle degradations unanticipated at training time . Our work is also related to pre-deep learning ( “ shallow ” ) methods that learn priors on ( potentiallyoverlapping ) image patches using maximum likelihood-type objectives such as ( Roth & Black , 2005 ; Karklin & Lewicki , 2009 ; Zoran & Weiss , 2011 ) . The use of multiple layers in our method allows to capture much longer correlations . As a result , our method can be used successfully to handle restoration tasks that require exploiting these correlations , such as large-hole inpainting . 2 METHOD . Let { x1 , x2 , . . . , xN } be a set of training images , that are considered to be samples from the distribution X of images in the space X of images of a certain size that need to be modeled . In latent modeling , we introduce a different space Z and a certain distribution Z in that space that is used to re-parameterize X . In previous works , Z is usually chosen to be a Euclidean space with few dozen to few hundred dimensions , while our choice for Z is discussed further below . The deep latent modeling of images implies learning the generator network gθ with learnable parameters θ , which usually has convolutional architecture . The generator network maps from Z to X and in particular is trained so that gθ ( Z ) ≈ X . Achieving the latter condition is extremely hard , and there are several approaches that can be used . Thus , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) train the generator network in parallel with a separate discriminator network that in some variants of GANs serves as an approximate ratio estimator between X and X+gθ ( Z ) over points inX . Alternatively , autoencoders ( Goodfellow et al. , 2016 ) and their variational counter-parts ( Kingma & Welling , 2014 ) train the generator in parallel with the encoder operating in the reverse direction , resulting in a more complex distribution Z . Of these two approaches , only GANs are known to be capable of synthesizing high-resolution images , although such ability comes with additional tricks and modifications of the learning formulation ( Arjovsky et al. , 2017 ; Karras et al. , 2018 ) . In this work , we start with a simpler approach to deep latent modeling ( Bojanowski et al. , 2018 ) known as the GLO model . GLO model optimizes the parameters of the generator network in parallel with the explicit embeddings of the training examples { z1 , z2 , . . . , zN } , such that gθ ( zi ) ≈ xi by the end of the optimization . Our approach differs from and expands ( Bojanowski et al. , 2018 ) in three ways : ( i ) we consider a much higher dimensionality of the latent space , ( ii ) we use an indirect parameterization of the latent space discussed further below , ( iii ) we demonstrate the applicability of the resulting model to a variety of image restoration tasks . Scaling up latent modeling . Relatively low-dimensional latent models of natural images presented in previous works are capable of producing visually-compelling image samples from the distribution ( Karras et al. , 2018 ) , but are not actually capable of matching or covering a rather high-dimensional distribution X . E.g . in our experiments , none of GAN models were capable of reconstructing most samples x from the hold-out set ( or even from the training set ; this observation is consistent with ( Bojanowski et al. , 2018 ) and also with ( Zhu et al. , 2016 ) ) . Being unable to reconstruct uncorrupted samples clearly suggests that the learned models are not suitable to perform restoration of corrupted samples . On the other hand , autoencoders and the related GLO latent model ( Bojanowski et al. , 2018 ) were able to achieve better reconstructions than GAN on the hold-out sets , yet have distinctly blurry reconstructions ( even on the training set ) , suggesting strong underfitting . We posit that existing deep latent models are limited by the dimensionality of the latent space that they consider , and aim to scale up this dimensionality significantly . Simply scaling up the latent dimensionality to few tens of dimensions is not easily feasible , as e.g . the generator network has to work with such a vector as an input , which would make the first fully-connected layer excessively large with hundreds of millions of parameters1 . To achieve a tractable size of the generator , one can consider latent elements z to have a threedimensional tensor structure , i.e . to be stacks of 2D image maps . Such choice of structure is very 1One can consider the first layer having a very thin matrix with a reasonable number of parameters mapping the latent vector to a much lower-dimensional space . This however would effectively amount to using lowerdimensional latent space and would defy the idea of scaling up latent dimensionality . natural for convolutional architectures , and allows to train “ fully-convolutional ” generators with the first layer being a standard convolutional operation . The downside of this choice , as we shall see , is that it allows limited coordination between distant parts of the images x = gθ ( z ) produced by the generator . This drawback is avoided when the latent space is parameterized using latent convolutional manifolds as described next . Latent convolutional manifolds . To impose more appropriate structure on the latent space , we consider structuring these spaces as convolutional manifolds defined as follows . Let s be a stack of maps of the size Ws × Hs × Cs and let { fφ |φ ∈ Φ } be a set of convolutional networks all sharing the same architecture f that transforms s to different maps of size Wz ×Hz × Cz . A certain parameter vector φ ∈ Φ thus defines a certain convolutional network fφ . Then , let z ( φ ) = fφ ( s ) be an element in the space of ( Wz ×Hz × Cz ) -dimensional maps . Various choices of φ then span a manifold embedded into this space , and we refer to it as the convolutional manifold . A convolutional manifold Cf , s is thus defined by the ConvNet architecture f as well as by the choice of the input s ( which in our experiments is always chosen to be filled with uniform random noise ) . Additionally , we also restrict the elements of vectors φ to lie within the [ −B ; B ] range . Formally , the convolutional manifold is defined as the following set : Cf , s = { z | z = fφ ( s ) , φ ∈ Φ } , Φ = [ −B ; B ] Nφ , ( 1 ) where φ serves as a natural parameterization and Nφ is the number of network parameters . Below , we refer to f as latent ConvNet , to disambiguate it from the generator g , which also has a convolutional structure . The idea of the convolutional manifold is inspired by the recent work on deep image priors ( Ulyanov et al. , 2018 ) . While they effectively use convolutional manifolds to model natural images directly , in our case , we use them to model the latent space of the generator networks resulting in a fully-fledged learnable latent image model ( whereas the model in ( Ulyanov et al. , 2018 ) can not be learned on a dataset of images ) . The work ( Ulyanov et al. , 2018 ) demonstrates that the regularization imposed by the structure of a very high-dimensional convolutional manifold is beneficial when modeling natural images . Our intuition here is that similar regularization would be beneficial in regularizing learning of high-dimensional latent spaces . As our experiments below reveal , this intuition holds true . Learning formulation . Learning the deep latent model ( Figure 2 ) in our framework then amounts to the following optimization task . Given the training examples { x1 , x2 , . . . , xN } , the architecture f of the convolutional manifold , and the architecture g of the generator network , we seek the set of the latent ConvNet parameter vectors { φ1 , φ2 , . . . , φN } and the parameters of the generator network θ that minimize the following objective : L ( φ1 , φ2 , . . . , φN , θ ) = 1 N N∑ i=1 ‖ gθ ( fφi ( s ) ) − xi ‖ , ( 2 ) with an additional box constraints φji ∈ [ −0.01 ; 0.01 ] and s being a random set of image maps filled with uniform noise . Following ( Bojanowski et al. , 2018 ) , the norm in ( 2 ) is taken to be the Laplacian-L1 : ‖x1−x2‖Lap-L1 = ∑ j 2 −2j |Lj ( x1−x2 ) |1 , where Lj is the jth level of the Laplacian image pyramid ( Burt & Adelson , 1983 ) . We have also found that adding an extra MSE loss term to the Lap-L1 loss term with the weight of 1.0 speeds up convergence of the models without affecting the results by much . The optimization ( 2 ) is performed using stochastic gradient descent . As an outcome of the optimization , each training example xi gets a representation zi = fφi on the convolutional manifold Cf , s . Importantly , the elements of the convolutional manifold then define a set of images in the image space ( which is the image of the convolutional manifold under learned generator ) : If , s , θ = { x |x = gθ ( fφ ( s ) ) , φ ∈ Φ } . ( 3 ) While not all elements of the manifold If , s , θ will correspond to natural images from the distribution X , we have found out that with few thousand dimensions , the resulting manifolds can cover the support of X rather well . I.e . each sample from the image distribution can be approximated by the element of If , s , θ with a low approximation error . This property can be used to perform all kinds of image restoration tasks . Image restoration using learned latent models . We now describe how the learned latent model can be used to perform the restoration of the unknown image x0 from the distribution X , given some evidence y . Depending on the degradation process , the evidence y can be an image x0 with masked values ( inpainting task ) , the low-resolution version of x0 ( superresolution task ) , the grayscale version of x0 ( colorization task ) , the noisy version of x0 ( denoising task ) , a certain statistics of x0 computed e.g . using a deep network ( feature inversion task ) , etc . We further assume , that the degradation process is described by the objective E ( x|y ) , which can be set to minus log-likelihood E ( x|y ) = − log p ( y|x ) of observing y as a result of the degradation of x. E.g . for the inpainting task , one can use E ( x|y ) = ‖ ( x − y ) m‖ , where m is the 0-1 mask of known pixels and denotes element-wise product . For the superresolution task , the restoration objective is naturally defined as E ( x|y ) = ‖ ↓ ( x ) − y‖ , where ↓ ( · ) is an image downsampling operator ( we use Lanczos in the experiments ) and y is the low-resolution version of the image . For the colorization task , the objective is defined as E ( x|y ) = ‖gray ( x ) − y‖ , where gray ( · ) denotes a projection from the RGB to grayscale images ( we use a simple averaging of the three color channels in the experiments ) and y is the grayscale version of the image . Using the learned latent model as a prior , the following estimation combining the learned prior and the provided image evidence is performed : φ̂ = arg min φ E ( gθ ( fφ ( s ) ) |y ) , x̂ = gθ ( fφ̂ ( s ) ) . ( 4 ) In other words , we simply estimate the element of the image manifold ( 3 ) that has the highest likelihood . The optimization is performed using stochastic gradient descent over the parameters φ on the latent convolutional manifold . For the baseline models , which use a direct parameterization of the latent space , we perform analogous estimation using optimization in the latent space : ẑ = arg min z E ( gθ ( z ) |y ) , x̂ = gθ ( z ) . ( 5 ) In the experiments , we compare the performance of our full model and several baseline models over a range of the restoration tasks using formulations ( 4 ) and ( 5 ) .

Summary: This paper proposes to increase the latent space dimensionality  of images, by stacking the latent representation vectors as a tensor. Then convolutional decoder and encoder networks are used to map the original data to latent space and vice versa. The learned latent representations can then be used in a universal framework for multiple tasks such as image inpainting, superresolution and colorization.