Manifold Modeling in Embedded Space: A Perspective for Interpreting "Deep Image Prior"

1 INTRODUCTION . The most important piece of information for image/tensor restoration would be the “ prior ” which usually converts the optimization problems from ill-posed to well-posed , and/or gives some robustness for specific noises and outliers . Many priors were studied in computer science problems such as low-rank representation ( Pearson , 1901 ; Hotelling , 1933 ; Hitchcock , 1927 ; Tucker , 1966 ) , smoothness ( Grimson , 1981 ; Poggio et al. , 1985 ; Li , 1994 ) , sparseness ( Tibshirani , 1996 ) , non-negativity ( Lee & Seung , 1999 ; Cichocki et al. , 2009 ) , statistical independence ( Hyvarinen et al. , 2004 ) , and so on . Particularly in today ’ s computer vision problems , total variation ( TV ) ( Guichard & Malgouyres , 1998 ; Vogel & Oman , 1998 ) , low-rank representation ( Liu et al. , 2013 ; Ji et al. , 2010 ; Zhao et al. , 2015 ; Wang et al. , 2017 ) , and non-local similarity ( Buades et al. , 2005 ; Dabov et al. , 2007 ) priors are often used for image modeling . These priors can be obtained by analyzing basic properties of natural images , and categorized as “ unsupervised image modeling ” . By contrast , the deep image prior ( DIP ) ( Ulyanov et al. , 2018 ) has been come from a part of “ supervised ” or “ data-driven ” image modeling framework ( i.e. , deep learning ) although the DIP itself is one of the state-of-the-art unsupervised image restoration methods . The method of DIP can be simply explained to only optimize an untrained ( i.e. , randomly initialized ) fully convolutional generator network ( ConvNet ) for minimizing squares loss between its generated image and an observed image ( e.g. , noisy image ) , and stop the optimization before the overfitting . Ulyanov et al . ( 2018 ) explained the reason why a high-capacity ConvNet can be used as a prior by the following statement : Network resists “ bad ” solutions and descends much more quickly towards naturally-looking images , and its phenomenon of “ impedance of ConvNet ” was confirmed by toy experiments . However , most researchers could not be fully convinced from only above explanation because it is just a part of whole . One of the essential questions is why is it ConvNet ? or in more practical perspective , to explain what is “ priors in DIP ” with simple and clear words ( like smoothness , sparseness , low-rank etc ) is very important . In this study , we tackle the question why ConvNet is essential as an image prior , and try to translate the “ deep image prior ” with words . For this purpose , we divide the convolution operation into “ embedding ” and “ transformation ” ( see Fig . 9 in Appendix ) . Here , the “ embedding ” stands for delay/shift-embedding ( i.e. , Hankelization ) which is a copy/duplication operation of image-patches by sliding window of patch size ( τ , τ ) . The embedding/Hankelization is a preprocessing to capture the delay/shift-invariant feature ( e.g. , non-local similarity ) of signals/images . This “ transformation ” is basically linear transformation in a simple convolution operation , and it also indicates some nonlinear transformation from the ConvNet perspective . To simplify the complicated “ encoder-decoder ” structure of ConvNet used in DIP , we consider the following network structure : Embedding H ( linear ) , encoding φr ( non-linear ) , decoding ψr ( non-linear ) , and backward embedding H† ( linear ) ( see Fig . 1 ) . Note that its encoder-decoder part ( φr , ψr ) is just a simple multi-layer perceptron along the filter domain ( i.e. , manifold learning ) , and it is sandwitched between forward and backward embedding ( H , H† ) . Hence , the proposed network can be characterized by Manifold Modeling in Embedded Space ( MMES ) . The proposed MMES is designed as simple as possible while keeping a essential ConvNet structure . Some parameters τ and r in MMES are corresponded with a kernel size and a filter size in ConvNet . When we set the horizontal dimension of hidden tensor L with r , each τ2-dimensional fiber in H , which is a vectorization of each ( τ , τ ) -patch of an input image , is encoded into r-dimensional space . Note that the volume of hidden tensor L looks to be larger than that of input/output image , but representation ability of L is much lower than input/output image space since the first/last tensor ( H , H′ ) must have Hankel structure ( i.e. , its representation ability is equivalent to image ) and the hidden tensor L is reduced to lower dimensions from H. Here , we assume r < τ2 , and its lowdimensionality indicates the existence of similar ( τ , τ ) -patches ( i.e. , self-similarity ) in the image , and it would provide some “ impedance ” which passes self-similar patches and resist/ignore others . Each fiber of Hidden tensor L represents a coordinate on the patch-manifold of image . It should be noted that the MMES network is a special case of deep neural networks . In fact , the proposed MMES can be considered as a new kind of auto-encoder ( AE ) in which convolution operations have been replaced by Hankelization in pre-processing and post-processing . Compared with ConvNet , the forward and backward embedding operations can be implemented by convolution and transposed convolution with one-hot-filters ( see Fig . 12 in Appendix for details ) . Note that the encoder-decoder part can be implemented by multiple convolution layers with kernel size ( 1,1 ) and non-linear activations . In our model , we do not use convolution explicitly but just do linear transform and non-linear activation for “ filter-domain ” ( i.e. , horizontal axis of tensors in Fig . 1 ) . The contributions in this study can be summarized as follow : ( 1 ) A new and simple approach of image/tensor modeling is proposed which translates the ConvNet , ( 2 ) effectiveness of the proposed method and similarity to the DIP are demonstrated in experiments , and ( 3 ) most importantly , there is a prospect for interpreting/characterizing the DIP as “ low-dimensional patch-manifold prior ” . 2 RELATED WORKS . Note that the idea of low-dimensional patch manifold itself has been proposed by Peyre ( 2009 ) and Osher et al . ( 2017 ) . Peyre had firstly formulated the patch manifold model of natural images and solve it by dictionary learning and manifold pursuit . Osher et al . formulated the regularization function to minimize dimension of patch manifold , and solved Laplace-Beltrami equation by point integral method . In comparison with these studies , we decrease the dimension of patch-manifold by utilizing AE shown in Fig . 1 . A related technique , low-rank tensor modeling in embedded space , has been studied recently by Yokota et al . ( 2018 ) . However , the modeling approaches here are different : multi-linear vs nonlinear manifold . Thus , our study would be interpreted as manifold version of ( Yokota et al. , 2018 ) in a perspective of tensor completion methods . Note that Yokota et al . ( 2018 ) applied their model for only tensor completion task . By contrast , we investigate here tensor completion , super-resolution , and deconvolution tasks . Another related work is devoted to group sparse representation ( GSR ) ( Zhang et al. , 2014a ) . The GSR is roughly characterized as a combination of similar patch-grouping and sparse modeling which is similar to the combination of embedding and manifold-modeling . However , the computational cost of similar patch-grouping is obviously higher than embedding , and this task is naturally included in manifold learning . The main difference between above studies and our is the motivation : Essential and simple image modeling which can translate the ConvNet/DIP . The proposed MMES has many connections with ConvNet/DIP such as embedding , non-linear mapping , and the training with noise . From a perspective of DIP , there are several related works . First , the deep geometric prior ( Williams et al. , 2019 ) utilises a good properties of a multi-layer perceptron for shape reconstruction problem which efficiently learn a smooth function from 2D space to 3D space . It helps us to understand DIP from a perspective of manifold learning . For example , it can be used for gray scale image reconstruction if an image is regarded as point could in 3D space ( i , j , Xij ) . However , this may not provide the good image reconstruction like DIP , because it just smoothly interpolates a point cloud by surface like a Volonoi interpolation . Especially it can not provide a property of self-similarity in natural image . Second , deep decoder ( Heckel & Hand , 2018 ) reconstructs natural images from noises by nonconvolutional networks which consists of linear channel/color transform , ReLU , channel/color normalization , and upsampling layers . In contrast that DIP uses over-parameterized network , deep decoder uses under-parameterized network and shows its ability of image reconstruction . Although deep decoder is a non-convolutional network , Authors emphasize the closed relationship between convolutional layers in DIP and upsampling layers in deep decoder . In this literature , Authors described ” If there is no upsampling layer , then there is no notion of locality in the resultant image ” in deep decoder . It implies the ” locality ” is the essence of image model , and the convolution/upsampling layer provides it . Furthermore , the deep decoder has a close relationship with our MMES . Note that the MMES is originally/essentially has only decoder and inverse MDT ( see Eq . ( 3 ) ) , and the encoder is just used for satisfying Hankel structure . The decoder and inverse MDT in our MMES are respectively corresponding linear operation and upsampling layer in deep decoder . Moreover , concept of under-parameterization is also similar to our MMES . From this , we can say the essence of image model is the ” locality ” , and its locality can be provided by ” convolution ” , ” upsampling ” , or ” delay-embedding ” . This is why the image restoration from single image with deep convolutional networks has highly attentions which are called by zero-shot learning , internal learning , or self-supervised learning ( Shocher et al. , 2018 ; Lehtinen et al. , 2018 ; Krull et al. , 2019 ; Batson & Royer , 2019 ; Xu et al. , 2019 ; Cha et al. , 2019 ; Laine et al. , 2019 ) . Recently , two generative models : SinGAN ( Shaham et al. , 2019 ) and InGAN ( Shocher et al. , 2019 ) learned from only a single image , have been proposed . Key concept of both papers is to impose the constraint for local patches of image to be natural . From a perspective of the constraint for local patches of image , our MMES has closed relationship with these works . However , we explicitly impose a low-dimensional manifold constraint for local patches rather than adversarial training with patch discriminators . 3 MANIFOLD MODELING IN EMBEDDED SPACE . Here , on the contrary to Section 1 , we start to explain the proposed method from the concept of MMES , and we systematically derive the MMES structure from it . Conceptually , the proposed tensor reconstruction method can be formulated by minimize X ||Y −F ( X ) ||2F , s.t . H ( X ) = [ h1 , h2 , ... , hT ] = : H , ( 1 ) ht ∈Mr for t = 1 , 2 , ... , T , where Y ∈ RJ1×J2×···×JN is an observed corrupted tensor , X ∈ RI1×I2×···×IN is an estimated tensor , F : RI1×I2×···×IN → RJ1×J2×···×JN is a linear operator which represents the observation system , H : RI1×I2×···×IN → RD×T is padding and Hankelization operator with sliding window of size ( τ1 , τ2 , ... , τN ) , and we impose each column of matrixH can be sampled from an r-dimensional manifoldMr in D-dimensional Euclid space ( see Appendix B for details ) . We have r ≤ D. For simplicity , we puttedD : = ∏ n τn and T : = ∏ n ( In+τn−1 ) . For tensor completion task , F : = PΩ is a projection operator onto support set Ω so that the missing elements are set to be zero . For superresolution task , F is a down-sampling operator of images/tensors . For deconvolution task , F is a convolution operator with some blur kernels . Fig . 2 shows the concept of proposed manifold modeling in case of image inpainting ( i.e. , N = 2 ) . We minimize the distance between observation Y and reconstruction X with its support Ω , and all patches in X should be included in some restricted manifold Mr . In other words , X is represented by the patch-manifold , and the property of the patch-manifold can be image priors . For example , low dimensionality of patch-manifold restricts the non-local similarity of images/tensors , and it would be related with “ impedance ” in DIP . We model X indirectly by designing the properties of patch-manifoldMr .

Summary: In this paper, the authors present a natural image model based on the manifold of image patches.  It is similar to the Deep Image Prior in that it is untrained and has a convolutional-like structure.  It leads to an optimization problem with a reconstruction loss term and an auto encoding term.  The authors show empirical results in time series recovery, non-semantic inpainting, and super resolution.  In the image processing tasks, the performance of the proposed algorithm is on par (sometimes slightly worse, sometimes slightly better) than that of DIP.