Manifold Modeling in Embedded Space: A Perspective for Interpreting "Deep Image Prior"

1 INTRODUCTION . The most important piece of information for image/tensor restoration would be the “ prior ” which usually converts the optimization problems from ill-posed to well-posed , and/or gives some robustness for specific noises and outliers . Many priors were studied in computer science problems such as low-rank representation ( Pearson , 1901 ; Hotelling , 1933 ; Hitchcock , 1927 ; Tucker , 1966 ) , smoothness ( Grimson , 1981 ; Poggio et al. , 1985 ; Li , 1994 ) , sparseness ( Tibshirani , 1996 ) , non-negativity ( Lee & Seung , 1999 ; Cichocki et al. , 2009 ) , statistical independence ( Hyvarinen et al. , 2004 ) , and so on . Particularly in today ’ s computer vision problems , total variation ( TV ) ( Guichard & Malgouyres , 1998 ; Vogel & Oman , 1998 ) , low-rank representation ( Liu et al. , 2013 ; Ji et al. , 2010 ; Zhao et al. , 2015 ; Wang et al. , 2017 ) , and non-local similarity ( Buades et al. , 2005 ; Dabov et al. , 2007 ) priors are often used for image modeling . These priors can be obtained by analyzing basic properties of natural images , and categorized as “ unsupervised image modeling ” . By contrast , the deep image prior ( DIP ) ( Ulyanov et al. , 2018 ) has been come from a part of “ supervised ” or “ data-driven ” image modeling framework ( i.e. , deep learning ) although the DIP itself is one of the state-of-the-art unsupervised image restoration methods . The method of DIP can be simply explained to only optimize an untrained ( i.e. , randomly initialized ) fully convolutional generator network ( ConvNet ) for minimizing squares loss between its generated image and an observed image ( e.g. , noisy image ) , and stop the optimization before the overfitting . Ulyanov et al . ( 2018 ) explained the reason why a high-capacity ConvNet can be used as a prior by the following statement : Network resists “ bad ” solutions and descends much more quickly towards naturally-looking images , and its phenomenon of “ impedance of ConvNet ” was confirmed by toy experiments . However , most researchers could not be fully convinced from only above explanation because it is just a part of whole . One of the essential questions is why is it ConvNet ? or in more practical perspective , to explain what is “ priors in DIP ” with simple and clear words ( like smoothness , sparseness , low-rank etc ) is very important . In this study , we tackle the question why ConvNet is essential as an image prior , and try to translate the “ deep image prior ” with words . For this purpose , we divide the convolution operation into “ embedding ” and “ transformation ” ( see Fig . 9 in Appendix ) . Here , the “ embedding ” stands for delay/shift-embedding ( i.e. , Hankelization ) which is a copy/duplication operation of image-patches by sliding window of patch size ( τ , τ ) . The embedding/Hankelization is a preprocessing to capture the delay/shift-invariant feature ( e.g. , non-local similarity ) of signals/images . This “ transformation ” is basically linear transformation in a simple convolution operation , and it also indicates some nonlinear transformation from the ConvNet perspective . To simplify the complicated “ encoder-decoder ” structure of ConvNet used in DIP , we consider the following network structure : Embedding H ( linear ) , encoding φr ( non-linear ) , decoding ψr ( non-linear ) , and backward embedding H† ( linear ) ( see Fig . 1 ) . Note that its encoder-decoder part ( φr , ψr ) is just a simple multi-layer perceptron along the filter domain ( i.e. , manifold learning ) , and it is sandwitched between forward and backward embedding ( H , H† ) . Hence , the proposed network can be characterized by Manifold Modeling in Embedded Space ( MMES ) . The proposed MMES is designed as simple as possible while keeping a essential ConvNet structure . Some parameters τ and r in MMES are corresponded with a kernel size and a filter size in ConvNet . When we set the horizontal dimension of hidden tensor L with r , each τ2-dimensional fiber in H , which is a vectorization of each ( τ , τ ) -patch of an input image , is encoded into r-dimensional space . Note that the volume of hidden tensor L looks to be larger than that of input/output image , but representation ability of L is much lower than input/output image space since the first/last tensor ( H , H′ ) must have Hankel structure ( i.e. , its representation ability is equivalent to image ) and the hidden tensor L is reduced to lower dimensions from H. Here , we assume r < τ2 , and its lowdimensionality indicates the existence of similar ( τ , τ ) -patches ( i.e. , self-similarity ) in the image , and it would provide some “ impedance ” which passes self-similar patches and resist/ignore others . Each fiber of Hidden tensor L represents a coordinate on the patch-manifold of image . It should be noted that the MMES network is a special case of deep neural networks . In fact , the proposed MMES can be considered as a new kind of auto-encoder ( AE ) in which convolution operations have been replaced by Hankelization in pre-processing and post-processing . Compared with ConvNet , the forward and backward embedding operations can be implemented by convolution and transposed convolution with one-hot-filters ( see Fig . 12 in Appendix for details ) . Note that the encoder-decoder part can be implemented by multiple convolution layers with kernel size ( 1,1 ) and non-linear activations . In our model , we do not use convolution explicitly but just do linear transform and non-linear activation for “ filter-domain ” ( i.e. , horizontal axis of tensors in Fig . 1 ) . The contributions in this study can be summarized as follow : ( 1 ) A new and simple approach of image/tensor modeling is proposed which translates the ConvNet , ( 2 ) effectiveness of the proposed method and similarity to the DIP are demonstrated in experiments , and ( 3 ) most importantly , there is a prospect for interpreting/characterizing the DIP as “ low-dimensional patch-manifold prior ” . 2 RELATED WORKS . Note that the idea of low-dimensional patch manifold itself has been proposed by Peyre ( 2009 ) and Osher et al . ( 2017 ) . Peyre had firstly formulated the patch manifold model of natural images and solve it by dictionary learning and manifold pursuit . Osher et al . formulated the regularization function to minimize dimension of patch manifold , and solved Laplace-Beltrami equation by point integral method . In comparison with these studies , we decrease the dimension of patch-manifold by utilizing AE shown in Fig . 1 . A related technique , low-rank tensor modeling in embedded space , has been studied recently by Yokota et al . ( 2018 ) . However , the modeling approaches here are different : multi-linear vs nonlinear manifold . Thus , our study would be interpreted as manifold version of ( Yokota et al. , 2018 ) in a perspective of tensor completion methods . Note that Yokota et al . ( 2018 ) applied their model for only tensor completion task . By contrast , we investigate here tensor completion , super-resolution , and deconvolution tasks . Another related work is devoted to group sparse representation ( GSR ) ( Zhang et al. , 2014a ) . The GSR is roughly characterized as a combination of similar patch-grouping and sparse modeling which is similar to the combination of embedding and manifold-modeling . However , the computational cost of similar patch-grouping is obviously higher than embedding , and this task is naturally included in manifold learning . The main difference between above studies and our is the motivation : Essential and simple image modeling which can translate the ConvNet/DIP . The proposed MMES has many connections with ConvNet/DIP such as embedding , non-linear mapping , and the training with noise . From a perspective of DIP , there are several related works . First , the deep geometric prior ( Williams et al. , 2019 ) utilises a good properties of a multi-layer perceptron for shape reconstruction problem which efficiently learn a smooth function from 2D space to 3D space . It helps us to understand DIP from a perspective of manifold learning . For example , it can be used for gray scale image reconstruction if an image is regarded as point could in 3D space ( i , j , Xij ) . However , this may not provide the good image reconstruction like DIP , because it just smoothly interpolates a point cloud by surface like a Volonoi interpolation . Especially it can not provide a property of self-similarity in natural image . Second , deep decoder ( Heckel & Hand , 2018 ) reconstructs natural images from noises by nonconvolutional networks which consists of linear channel/color transform , ReLU , channel/color normalization , and upsampling layers . In contrast that DIP uses over-parameterized network , deep decoder uses under-parameterized network and shows its ability of image reconstruction . Although deep decoder is a non-convolutional network , Authors emphasize the closed relationship between convolutional layers in DIP and upsampling layers in deep decoder . In this literature , Authors described ” If there is no upsampling layer , then there is no notion of locality in the resultant image ” in deep decoder . It implies the ” locality ” is the essence of image model , and the convolution/upsampling layer provides it . Furthermore , the deep decoder has a close relationship with our MMES . Note that the MMES is originally/essentially has only decoder and inverse MDT ( see Eq . ( 3 ) ) , and the encoder is just used for satisfying Hankel structure . The decoder and inverse MDT in our MMES are respectively corresponding linear operation and upsampling layer in deep decoder . Moreover , concept of under-parameterization is also similar to our MMES . From this , we can say the essence of image model is the ” locality ” , and its locality can be provided by ” convolution ” , ” upsampling ” , or ” delay-embedding ” . This is why the image restoration from single image with deep convolutional networks has highly attentions which are called by zero-shot learning , internal learning , or self-supervised learning ( Shocher et al. , 2018 ; Lehtinen et al. , 2018 ; Krull et al. , 2019 ; Batson & Royer , 2019 ; Xu et al. , 2019 ; Cha et al. , 2019 ; Laine et al. , 2019 ) . Recently , two generative models : SinGAN ( Shaham et al. , 2019 ) and InGAN ( Shocher et al. , 2019 ) learned from only a single image , have been proposed . Key concept of both papers is to impose the constraint for local patches of image to be natural . From a perspective of the constraint for local patches of image , our MMES has closed relationship with these works . However , we explicitly impose a low-dimensional manifold constraint for local patches rather than adversarial training with patch discriminators . 3 MANIFOLD MODELING IN EMBEDDED SPACE . Here , on the contrary to Section 1 , we start to explain the proposed method from the concept of MMES , and we systematically derive the MMES structure from it . Conceptually , the proposed tensor reconstruction method can be formulated by minimize X ||Y −F ( X ) ||2F , s.t . H ( X ) = [ h1 , h2 , ... , hT ] = : H , ( 1 ) ht ∈Mr for t = 1 , 2 , ... , T , where Y ∈ RJ1×J2×···×JN is an observed corrupted tensor , X ∈ RI1×I2×···×IN is an estimated tensor , F : RI1×I2×···×IN → RJ1×J2×···×JN is a linear operator which represents the observation system , H : RI1×I2×···×IN → RD×T is padding and Hankelization operator with sliding window of size ( τ1 , τ2 , ... , τN ) , and we impose each column of matrixH can be sampled from an r-dimensional manifoldMr in D-dimensional Euclid space ( see Appendix B for details ) . We have r ≤ D. For simplicity , we puttedD : = ∏ n τn and T : = ∏ n ( In+τn−1 ) . For tensor completion task , F : = PΩ is a projection operator onto support set Ω so that the missing elements are set to be zero . For superresolution task , F is a down-sampling operator of images/tensors . For deconvolution task , F is a convolution operator with some blur kernels . Fig . 2 shows the concept of proposed manifold modeling in case of image inpainting ( i.e. , N = 2 ) . We minimize the distance between observation Y and reconstruction X with its support Ω , and all patches in X should be included in some restricted manifold Mr . In other words , X is represented by the patch-manifold , and the property of the patch-manifold can be image priors . For example , low dimensionality of patch-manifold restricts the non-local similarity of images/tensors , and it would be related with “ impedance ” in DIP . We model X indirectly by designing the properties of patch-manifoldMr .

Summary: This paper introduces a transformation from the deep image prior (DIP) to an embedding with an autoencoder (MMES). The authors aim to use this transformation to explain ("in words") why the DIP works so well and explain why convolutions are needed in the DIP. The contributions are summarised as a) providing an interpretable analogue to the convnet, b) demonstration of the proposed method's effectiveness, and c) characterisation of the DIP as a "low-dimensional patch-manifold prior".