Manifold Modeling in Embedded Space: A Perspective for Interpreting "Deep Image Prior"
Authors: Tatsuya Yokota
Hidekata Hontani
Qibin Zhao
Andrzej Cichocki
Venue:  (2020)
Keywords: Deep image prior
Manifold model
Auto-encoder
Convolutional neural network
Delay-embedding
Hankelization
Tensor completion
Image inpainting
Supperresolution

Abstract: Deep image prior (DIP), which utilizes a deep convolutional network (ConvNet) structure itself as an image prior, has attracted huge attentions in computer vision community.  It empirically shows the effectiveness of ConvNet structure for various image restoration applications.  However, why the DIP works so well is still unknown, and why convolution operation is essential for image reconstruction or enhancement is not very clear. In this study, we tackle these questions. The proposed approach is dividing the convolution into ``delay-embedding'' and ``transformation (\ie encoder-decoder)'', and proposing a simple, but essential, image/tensor modeling method which is closely related to dynamical systems and self-similarity. The proposed method named as manifold modeling in embedded space (MMES) is implemented by using a novel denoising-auto-encoder in combination with multi-way delay-embedding transform. In spite of its simplicity, the image/tensor completion and super-resolution results of MMES are quite similar even competitive to DIP in our extensive experiments, and these results would help us for reinterpreting/characterizing the DIP from a perspective of ``low-dimensional patch-manifold prior''.

Under review as a conference paper at ICLR 2020
MANIFOLD MODELING IN EMBEDDED SPACE: A PER- SPECTIVE FOR INTERPRETING "DEEP IMAGE PRIOR"
Anonymous authors Paper under double-blind review
ABSTRACT
Deep image prior (DIP) (Ulyanov et al. 2018), which utilizes deep convolutional network (ConvNet) structure itself as an image prior, has attracted huge attentions in computer vision community. It empirically shows the effectiveness of ConvNet structure for various image restoration applications. However, why the DIP works well is still unknown, and why convolution operation is useful for image recon- struction or enhancement is not very clear. In this study, we tackle these questions. The proposed approach is dividing the convolution into "delay-embedding" and "transformation (i.e, mecoder-decoded image/tensor modeling method which is closely related to dynamical systems and self-similarity. The proposed method named as manifold modeling in embedded in
and proposing a simple, but essential,
space (MMES) is implemented by using a novel sernosing--auto-nneee bination with multi-way delay- gelay-embedding transform. In spite of its simplicity, the image/tensor completion, super-resolution, and deconvolution results of MMES are quite similar even competitive to DIP in our extensive experiments, and these results would help us for connttrpretin/ccaaacterrinn the DIP from a perspective of "low-dimensional patch-manifold prior".
INTRODUCTION
The most important piece of information for image/tensor restoration would be the "prior" which usually converts the optimization problems from ill ill-posed to well-posed. and/or gives some robust- ness for specific noises and outiers. Many priors were studied in computer science problems such as
low-rank representation (Pearson 1901 Hotelling 1933 Hitchcock] 1927 Tucker 1966), smooth-
ness (Grimson) 1981 Poggio et al. 1985; Lij 1994), sparseness (Tibshirani 1996). non-negativity
(Lee & Seung | Cichocki et al. 2009), statistical independence (Hyvarinen et al. 2004), and so
Particularly in today's s computer vision problems, total variation (TV) (Guichard & Malgouyres
1998
Vogel & Oman 1998), low-rank representation (Liu et al., 2013 Ji et al.) 2010) Zhao et al,,
2013
Wang al.] (017), and non-local similarity (Buades al. 2005 Daboy al.] 2007) priors
are often used for image modeling. These priors can be obtained by analyzing basic properties of natural images, and categorized "unsupervised image modeling".
By contrast, the deep image prior (DIP) (Ulyanov et al. 2018) has been come from a part of "super- vised" or "data-driven" image modeling framework (i.e., deep learning) although the DIP itself is one of the state-of-the-arr unsupervised image restoration methods. The method of DIP can be sim- ply explained only optimize an untrained (i.e., randomly initialized) fully convolutional generator network (ConvNet) for minimizing squares loss between its generated image and an observed image (e.g., noisy image), and stop the optimization before the overfitting. Ulyanov et al. (2018) explained the reason why high-capacity ConvNet can be used as prior by the following statement: Nei- work resists "bad" solutions and descends much more quickly towards naturally-lookin images, and its phenomenon of "impedance of ConvNet" was confirmed by toy experiments. However, most researchers could not be fully convinced from only above explanation because it is just a part of whole. One of the essential questions is why is it ConvNet? in more practical perspective, to explain what is "priors in DIP" with simple and clear words (like smoothness, sparseness, low-rank etc) is very important.
In this study, we tackle the question why ConvNet is essential as an image prior, and try to translate the "deep image prior" with words. For this purpose, we divide the convolution operation into

---

Under review as a conference paper at ICLR 2020
ConvNet
MMES
H
H
H
44-
7
convy layer with kernel size (k.k) activation down/upsample
skip connection concatenation to next domain
conv layer with kernel size (1,1) activation
forward
embedding with window (F,1)
Figure 1: Comparison of typical auto-encoder ConvNet and the proposed MMES network.
"embedding" and "transformation" (see Fig. 9 in Appendix). Here, the "embedding" stands for delay/shift-embeeddin (i.e., Hankelization) which a copy/duplication operation of image-patches by sliding window of patch size (T, The Habbedding/Hnnkliizzttio is preprocessing to capture the delay/shift-invariant feature (e.g,, non-local similarity) of signals/images. This "transformation" basically linear transformation in simple convolution operation, and it also indicates some linear transformation from the ConvNet perspective.
To simplify the complicated "encoder-decoderr" structure of ConvNet used in DIP, we consider the following network structure: Embedding Ht (linear), encoding Φ, (non-linear), decoding th, (non-ling mon--ineear), and backward embedding Ht (linear) (see Fig. D. Note that its encoder-decoder part (ør, 2))) is just a simple multi- perceptron along the filter domain (i.e., manifold learning), and it sandwitched between forward and backward embedding (H. H²), Hence, the proposed network can be characterized by Manifold Modeling in Embedded Space (MMES). The proposed MMES is designed as simple as possible while keeping a essential ConvNet structure. Some parameters r and in MMES are corresponded with kernel size and filter size in ConvNet.
When we set the horizontal dimension hidden tensor with r, each 7"-dimensional fiber in H. which is vectorization of each (7, -patch of input image, is encoded into r-dimensional space. Note that the volume hidden tensor looks to be larger than that of input/output image, but representation ability of L is much lower than input/output image space since the first/last tensor (H,H') must have Hankel structure (i.e., its representation ability is equivalent to image) and the hidden tensor L is reduced lower dimensions from H. Here, assume r 72, and its low- dimensionality indicates the existence of similar (r, -aatcchs (i.e., self-similarity) in the image, and it would provide some "impedance" which passes self-similar patches and resist/ignore others. Each fiber of Hidden tensor C represents a coordinate on the patch-maninh manifold of image.
should be noted that the MMES network is special case of deep neural networks. In fact, the proposed MMES can be considered as new kind of auto-encoder (AE) in which convolution operations have been replaced by Hankelization in pre-processing and post-processing. Compared with ConvNet, the forward and backward embedding operations can be implemented by convolution and transposed convolution with one-hot-filter (see Fig. 12 in Appendix for details). Note that the encoder-decoder part can be implemented by multiple convolution layers with kernel size (1.1) and non-linear activations. In our model, we not use convolution explicitly but just do linear transform and non-linear activation for "filte-domain" (i.e., horizontal axis of tensors in Fig. [.
The contributions in this study can be summarized as follow: (1) A new and simple approach of image/tensor modeling is proposed which translates the ConvNet, (2) effectiveness of the proposed method and similarity to the DIP are demonstrated experiments, and (3) most importantly, there prospect for interprefing/charattriiinngg the DIP as "low-dimensional patch-manifold prior".
2

---

Under review as a conference paper at ICLR 2020
2 RELATED WORKS
Note that the idea of low-dimensional patch manifold itself has been proposed by Peyre (2009) and Osher et al. (2017). Peyre had firstly formulated the patch manifold model of natural images and solve it by dictionary learning and manifold pursuit. Osher al. formulated the regularization function to minimize dimension of patch manifold, and solved Laplace-Beltann Beltrami equation by point integral method. In comparison with these studies, we decrease the dimension of patch-manifold by utlizzig AE shown in Fig.
A related technique, low-rank tensor modeling in embedded space, has been studied recently by Yokota et al. (2018). However, the modeling approaches here are different: multi-linear vs non- linear manifold. Thus, our study would be interpreted as manifold version of (Yokota et al.] (2018) in a perspective of tensor completion methods. Note that Yokota et al. (2018) applied their model for only tensor completion task. By contrast, we investigate here tensor completion, super-resolution, and deconvolution tasks.
Another related work devoted to group sparse representation (GSR) (Zhagg et al. 2014a). The
GSR is roughly characterized as combination of similar patch-grouping and sparse modeling which similar to the combination of embedding and manifold manifold-modeling. However, the computational cost of similar patch-groupin is obviously higher than embedding, and this task is naturally included in manifold learning.
The main difference between above studies and our is the motivation: Essential and simple image modeling which can translate the ConwNet/DIP. The proposed MMES has many connections with ConvNet/DIP such as embedding, non-linear mapping, and the training with noise.
From a perspective of DIP, there are several related works. First, the deep geometric prior (Williams et al. 2019) utilises good properties of multi-layer perceptron for shape reconstruction problem which efficiently learn a smooth function from 2D space to 3D space. It helps us to understand DIP from a perspective of manifold learning. For example, it can be used for gray scale image reconstruction if an image is regarded as point could in 3D space (i.j. Xiy). However, this may not provide the good image reconstruction like DIP, because just smoothly interpolates a point cloud by surface like a Volonoi interpolation. Especially it can not provide a property of self-similarity in natural image.
reconstructs natural images from noises by non- convolutional networks which consists linear channel/color transform, ReI.U, channel/color nor- malization, and upsampling layers. In contrast that DIP uses wwrr-parameterized network, deep decoder uses uuder-parameteriin deep decoder is non-convolutional network, Authors emphasize the closed relationship between convolutional layers in DIP and upsampling layers in deep decoder. In this literature, Authors de- scribed "If there is upsampling layer, then there is notion of locality in the resultant im- age" in deep decoder. It implies the "locality" is the essence of image model, and the convolu- tion/upsampling layer provides it. Furthermore, the deep decoder has close relationship with our MMES. Note that the MMES is originally/essentially has only decoder and inverse MDT (see 33. ). and the encoder just used for satisfying Hankel structure. The decoder and inverse MDT in our MMES are respectively corresponding linear operation and upsampling layer in deep decoder. is also similar to our MMES.
Second, deep decoder (Heckel & Hand 2018)
network and shows its ability of image reconstruction. Although
Moreover, concept of mnder-parameterizatti
From this, we can say the essence of image model is the "locality", and its locality can be provided by "ccovolution"", "upsampling", or ddelay-embeddingg single image with deep convolutional networks has highly attentions which are called by zero-shot learning, internal learning, or self-supervised learning (Shocher et al. 2018 [ehtinen et al. 2018
This is why the image restoration from
Krull et al. 2019: Batson & Royer 2019 Xu et al. 2019 Cha al. [019 Laine et al. 2019)
Recently, two generative models: SinGAN (Shaham et al. 2019) and InGAN (Shocher et al. 2019) leamned from only a single image, have been proposed. Key concept of both papers is to impose the constraint for local patches of image to be natural. From a perspective of the constraint for local patches of image, impose a low- now-dimensional manifold constraint for local patches rather than adversarial training with patch discriminators.
MMES has closed relationship with these works. However, explicitly
3

---

Under review as a conference paper at ICLR 2020
Y
X
M,
embed
optimize
represent
observed image
reconstructed image
patch-manifold
Figure 2: Conceptual illustration of MMES for image inpainting task.
MANIFOLD MODELING IN EMBEDDED SPACE
Here, on the contrary to Section 1 we start to explain the proposed method from the concept of MMES, and systematically derive the MMES structure from it. Conceptually, the proposed tensor reconstruction method can formulated by
minimize 11 - F(x)))
s.t. H(X) = [h1, h2 ..., hTl =: H, h1 E M, for L = 1, ..... T.
(I)
where 1=11x is an observed corrupted tensor, X' 11xlgx-xxx is an estimated tensor, F R/ xIx - R X J2xx-XJ JN is linear operator which represents the observation system, IR. //1x/xx.xxxxx + DxT padding and Hankelization operator with sliding window of size (Ti, T.,..., T,), and we impose each column of matrix H can be sampled from an r-dimensional manifold M, D-dimensional Euclid space (see Appendix for details). simplicity, putted D IIn and T = II, (L,
have r S D. For + 1). For tensor completion task, F := Po a projection operator onto support set Q so that the missing elements are set to be zero. For super- resolution task, FF is down-sampling operator of images/tensors. For deconvolution task, F is convolution operator with some blur kernels. Fig. 2 shows the concept of proposed manifold mod- eling in case of image inpainting (i.e,, N 2). We minimize the distance between observation y and reconstruction with its support f and all patches X' should be included in some restricted manifold M,. In other words, X is represented by the patch-manifold, and the property of the patch- manifold can be image priors. For example, low dimensionality of patch-manifold restricts the non- non-local similarity of images/tensors, and it would related with "impedance" in DIP. We model X' indirectly by designing the properties of patch neench-maifold M,.
3.1 DEFINITION OF LOW-DIMENSIONAL MANIFOLD
We consider an AE define the r-dimensional manifold M, in (I,, Tn dimeensnni Euclidean space as follows:
M, = (1) ( J',, (6r argmin (v...r)
1h, 3, or
t=1
where o, RDD + R" is an encoder, v. R' - RD is decoder, and V RDD + RD is an auto-encoder constructed from (he Note that, in general. the use of AE models is a widely accepted approach for manifold learning (Hinton & Salakhutdinov 2006). The properties of the manifold M, are determined by the properties o and v/,, By employing multi-layer perceptrons (neural networks) for dr and thy, encoder-decoder may provide a smooth manifold.
3.2 PROBLEM FORMULATION
In this section, we combine the conceptual formulation T and the AE guided manifold constraint to derive equivalent and more practical optimization problem. First, we redefine tensor X' as an output of generator:
X : [h1, h2...., be....,,rr,, where b4 E Mr
(1)))
(3)
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Optimization algorithm for tensor reconstruction
input: 3 E R 1R-1x..xx XJ= (corrupted tensor), F, T,
initialize: Z E Rh XIN auto-encoder A,, 5.0; repeat
H < 2( R DXT with T:
generate noise E E RXT with Ø;
LA < H ( + E
fre
E)
update (Z, A,) by Adam for free + ALABi
if Lm: < CAE then 1.11; else +- 0.99X;
until converge
output: X Ht A, H() R/xxx.xx (reconstructed tensor);
where 4 R", and Ht is a pseudo inverse of H. At this moment, X' is function of (4)TT how- ever Hankel structure of matrix H can not be always guaranteed under the unconstrained condition of 4. For guaranteeing the Hankel structure of matrix H, we further transform follow:
X =
= 7t Ór & or (92). 2
= -Ht AHA,199 92. ....... 9r
HH A,H(((,,
where we put Ar DxT + DxT as an operator which auto-encodes each column of a input matrix with (vr, ør), and 92, gr as matrix, which has Hankel structure and is transformed by Hankelization of some input tensor E R2xx1xx-xxxx Note that Z the most compact representation for Hankel matrix [91. 92..... grl grl- Eq. describes the MMES network shown in Fig. H. dr. 3ه and H are respectively corresponding to forward embedding, encoding, decoding. and backward embedding, where encoder and decoder can be defined e.g. by multi-layer perceptrons (i.e., repetition of linear transformation and non-linear activation).
From this formulation, Problem 11 is transformed as minimizez || - DyF((((A A where A, AE which defines the manifold M,, In this study, the AE/manifold is learned from an observed tensor y itself, thus the optimization problem is finally formulated as
minimize 2.Ar
1 - F(H
A, J
H( +A | A, (
(5)
where we refer respectively the first and second terms by reconstruction loss and an eecoding loss. and À 0 is trade-off parameter for balancing both losses.
3.3 OPTIMIZATION ALGORITIIM
Optimization problem (5) consists two terms: a reconstruction loss, and an auto-encoding loss. Hyperparameter À is balance both losses. Basically, À should be large because auto-encoding loss should be zero. However, very large À prohibits minimizing the reconstruction loss, and may lead to local optima. Therefore, we adjust gradually the value of À in the optimization process.
shows an optimization algorithm for tensor reconstruction and/or enhancement. For AE learning, we employs strategy of denoising-auto-mmcoder (see Appendix in detail), Adaptation of À is just an example, and it can be modified appropriately with data. Here, the trade-off parameter is adjusted for keeping Lree > LAE, but for no large gap between both losses. By exploiting the convolutional structure of H and H (see Appendix I. the calculation flow of free and LaF can be easily implemented by using neural network libraries such as TensorFlow We employed Adam Kingma& & Ba 2014) optimizer for updating (Z,A)).
Algorithm
4 EXPERIMENTS
Here, we show the selective experimental results to demonstrate the close similarity and some slight differences between DIP and MMES. First, toy examples with time-series signal and gray-scale
5

---

Under review as a conference paper at ICLR 2020
Original
Corrupted (incompleteSmoiss)
Reconstructed
subspace modeling manifold modeling
Signal space
x
www
w
WiV
WNV
Embedded space
zit + 2)
x +)
x00
Figure 3: Time series signal recovery of subspace and manifold models in embedded space.
Learn
1st
10
12
Reconstruct
14
2nd axis
Figure 4: Distribution of two-dimensional -8tt)- on manifold learned from a 50% missing gray-scale image of 'Lena'.
image were recovered by the proposed method show basic behaviors. Thereafter, we show the main results by comparison with DIP and other selective methods color-image inpainting, super- resolution, and deconvolution tasks. Optional results of optimization behavior, hyper-parameter sensitivity, and volumetric/3D image completion are shown in Appendix.
4.1 TOY EXAMPLES
In this section, we apply the proposed method into a toy example of signal recovery. Fig. shows result of this experiment. A one-dimensional time-series signal is generated from Lorentz sys- tem, and corrupted by additive Gaussian noise, random missing, and three block occlusions. The corrupted signal was recovered by the subspace modeling (Yokota al. 2018). and the proposed manifold modeling in embedded space. Window size of eelay-embedding was 64, the lowest dimension of auto-encoder was r 3, and additive noise standard deviation was set to a 0.05. Manifold modeling catched the structure of Lorentz attractor much better than subspace modeling.
Fig. visualizes two-dimensional (8, 8p-ttth manifold learned by the proposed method from (8, Similar patches are located near each other, and the smooth change of patterns can be observed. implies the relationship between non-local similarity based methods (Buades et al.) 2005 Dabov
50% missing gray-scale image of 'Lena". For this figure, we set
2, = 0.05.
et al. 2007 Gu et al. 2014; Zhang et al. 2014a), and the manifold modeling (i.e. DAE) plays
key role of sattch-grouping in the proposed method. The difference from the non-local similarity based approach is that the manifold modeling is "global" rather than "non-local" which finds similar patches of the target patch from its neighborhood area.

---

Under review as a conference paper at ICLR 2020
Test images:
Test images:
Averages of PSNR
Averages SSIM
Averages PSNR
Averages of SSIM
35
JTTV
VOTT
27
0.95
30
Toppored
20
01
0.0
150 518 (64)
258 (M4)
912 (96)
120 852 (M4)
856 (4))
04 012 (9)
completion task
(b) superresolution task
Figure 5: Comparison of performance by averages of PSNR and SSIM for color image completion
and super-resolution tasks with various settings.
Original
Missing
HaURTC
TMac
Tucker
LRTV
MOT-Tucker
GSR
DIP
NMES
(proposed)
Figure 6: Illustration of image inpainting results from nature images with 99% missing pixels by HaLTRC, TMac, tSVD, Tucker inc., LRTV, GSR, MDT-Tucker, DIP and the proposed MMES. Note that DIP and MMES approaches provide the best performance in comparison state-of-the- arts methods.
4.2 COLOR IMAGE COMPLETION, ESPECIALLY FOR EXTREMELY HIGH NUMBER OF MISSING PIXELS
In this section, we compare performance the proposed method with several selected unsupervised image inpainting methods: low-rank tensor completion (HaLRTC) (Liu et al. 2013),, parallel low- rank matrix factorization (TMac) (Xu et al. 2015). tubal nuclear norm regularization (tSVD) (Zhang eet al. 2014b), Tucker decomposition with rank increment (Tucker inc.) Yokota al. 2018). low- rank and total-variation (LRTV) regularization (Yokota & Hontani] 2017 2019). smooth PARAFAC
tensor completion (SPC) (Yokota al. 2016), GSR (Zhang al. 2014a). multi-way delay embed- ding based Tucker modeling MMDT-Tucken (Yokota et al. 20188) and DIP (Ulyanov et al. 2018).
Implementation and detailed hyper-parameter settings are explained in Appendix. Basically, carefully tuned the noise ratio (PSNR) and structural similarity (SSIM).
for all methods to perform the best scores of peak- seau--sggal-to-
Fig. 5(a) shows the eight test images and averages of PSNR and SSIM for various missing ratio (50%, 70%, 90%, 95% 99% and for selective competitive methods. The proposed method is quite competitive with DIP. Fig. shows the illustration of results. The 99% of randomly selected voxels are removed from 3D (256,256.3) -teesosss and the tensors were recovered by various methods. Ba- sically low-rank priors (HaLRTC, TMac, (SVD, Tucker) could not recover such highly incomplete image. In piecewise smoothness prior (LRTV), ovor-smoothed images were reconstructed since the essential image properties could not be captured. There was somewhat jump from them by SPC (i.c., smooth prior of basis functions in low-rank tensor decomposition). MDT-Tucker further improves it by exploiting the shift-invariant multi-linear basis. GSR nicely recovered the global

---

Under review as a conference paper at ICLR 2020
Original
low reS.
Sproposedi
Figure 7: Comparison of approach with other methods for super-resolution task. The first line 'Leaves' were up-scaled from (64,64.3) to (256,256.3), and the second line 'Airplane' was up-scaled from (64,64,3) (512,512.3).
pattern of images but details were insufficient. Finally, the reconstructed images by DIP and MMES recovered both global and local patterns of images.
4.3 COLOR IMAGE SUPER-RESOLL RESOLUTION
In this section, we compare the proposed method with selected unsupervised image super-resolution methods: Bicubic interpolation, GSR (Zhang al. 2014a), 7SSR (Shocher et al.] 2018) and DIP (Ulyanov et al. 2018). Implementation and detailed hyper-parameter settings are explained in Ap- pendix. Basically, we carefully tuned the syper--arameters for all methods to perform the best scores of PSNR and SSIM.
Fig. b)b) shows values of PSNR and SSIM of the computer simulation results. We used three (256,256.3) color images, and six (512,512,3) color images. Super resolution methods scaling up them from four or eight times down-scaled images of them with Lanczos2 kernels. According to this quantitative evaluation, bicubic interpolation was clearly worse than others. 7SSR worked well for up-scaling from 128,3), however the performances were substantially decreased for up- scaling from (64,64,3). Basically, GSR, DIP, and MMES were very competitive. In detail, DIP was slightly better than GSR, and the proposed MMES was slightly better than DIP. More detailed PSNR/SSIM values given by Table Appendix. Fig. F shows selected high resolution images reconstructed by four super-resolution methods. In general, bicubic method reconstructed blurred images and these were visually worse than others. GSR results had smooth outlines in all images, but these were slightly blurred. ZSSR was weak for very low-resolution images. DIP reconstructed visually sharp images but these images had jagged artifacts along the diagonal lines. The proposed MMES reconstructed sharp and smooth outlines.
4.4 COLOR IMAGE DECONVOLUTION
In this section, we compare the proposed method with DIP for image deconnolltton//debbuing task. Three (256,256.3) color images are prepared and blurred by using three different Gaussian filters. For DIP we choose the best early stopping timing from {1000, 2000, ... 10000} iterations. For MMES, we employed the fixed AE structure as [32r2,r r, 327"], and parameters as т 4, r = 16, Fig. 8 shows the reconstructed deblurring images by DIP and MMES. Tab. shows the PSNR and SSIM values of these results. We can see that the similarity of the methods qualitatively and quantitatively.
and 0.01 for all nine
8

---

Under review as a conference paper at ICLR 2020
A9
A15
A31
B15
B31
C15
PewerPait
Blur
PoinerPoltt
PawerPPakt
2011
2002
DIP
MMES
Figure 8: Comparison our approach with DIP for deconvolution//eebbriig task. Three color images were blurred by three Gaussian windows of different sizes. These were recovered by the DIP and the proposed MMES.
Table 1:
PSNR/SSIM values for deconvolution results
PSNR
A9
A15
A30
B9
B15
B31
C9
C15
C31
DIP
28.85
24.53
18.21
26.58
19.80
16.21
27.18
22.07
15.74
MMES
29.24
24.72
18.92
27.46
20.89
16.45
29.15
22.38
16.55
SSIM
A9
A15
A30
B9
B15
B31
C9
C15
C31
DIP
.9346
.8586
.5922
.9438
.7871
.5628
.9438
.8276
.4568
MMES
.9436
.8599
.6234
.9512
.8088
.5805
.9636
.8382
.5284
INTERPRETATION OF MMES TOWARD EXPLAINING DIP
It is well known that there is no mathematical definition of interpretability in machine learning and there is no one unique definition of interpretation. We understand the interpretability as a degree to which human can consistently predict the model's results or performance. The higher the in- terpretability of a deep learning model, the easier it is for someone to comprehend why certain performance or predictions or expected output can be achieved. We think that a model is better in- terpretable than another model if its performance or behaviors are easier for a human to comprehend than performance of the other models.
5.1 FROM A PERSPECTIVE OF DIMENSIONALITY REDUCTION//ANIFOR LEARNING
The manifold learning and associated auto-encoder (AE) can be viewed as the generalized non-linear version of principal component analysis (PCA). In fact, manifold learning solves the key problem of dimensionality reduction very efficiently. In other words, manifold learning (modeling) is an approach non-linear dimensionality reduction. Manifold modeling for this task are based on the idea that the dimensionality of many data sets is only artificially high. Although the patches of images (data points) consist hundreds/thousands pixels, they may represented as function of only few or quite limited number underlying parameters. That is, the patches are actually samples from a low-dimensional manifold that embedded high-dimensional space. Manifold learning algorithms attempt to uncover these parameters in order to find a low dimensional representation of the images.
In MMES approach to solve the problem applied original embedding via multi-way delay embedding transform (MDT or Hankelization). Our algorithm is based on the optimization of cost function and it works towards extracting the www-dinensional manifold that is used to describe the high-dimensional data. The manifold is described mathematically by Eq. (2) and cost function is formulated by Eq. (5)

---

Under review as a conference paper at ICLR 2020
5.2 REGARDING OUR ATTEMPT TO INTERPRET "NOISE IMPEDANCE IN DIP" VIA MMES
As mentioned at introduction, Ulyanov et al. (2018) reported an important phenomenon of noise impedance of ConvNet structures. Here, provide prospect for explaining the noise impedance in DIP through the MMES.
Let us consider the sparse-land model, i.c. noise-free images are distributed along low-dimensional manifolds in the high-dimensional Euclidean space and images perturbed by noises thicken the man- ifolds (make the dimension of the manifolds higher). Under this model, the distribution of images can be assumed to be higher along the low- www-dmensional noise-free image manifolds. When we as- sume that the image patches are sampled from -ww-dimesioonl manifold like sparse-land model, is difficult to put noisy patches on the low-dimensional manifold. Let us consider to fit the network for noisy images. In such case the fastest way for decreasing squared enror (loss function) to learn "similar patches" which often appear in large set of image-patches Note that finding sim- ilar image-patches for denoising is well-known problem solved, c.g., by BM3D algorithm, which find similar image patches by template matching. In contrast, auto-encoder automatically maps similar- similar-patches into close points on the dimensnoalonal manifold. When -attcrrs have some noise, the oow-dimensional representation tries keep the common components of similar patches, while reducing the noise components. This has been proved by Alainn & (2ngg))(0014) so that a (de- noising) auto-encoder maps input image patches toward higher density portions in the image space. In other words, a (denoising) auto-encoder has kind of a force to reconstruct the low-dimensional patch manifold. and this is our rough explanation of noise impedance phenomenon. Although the proposed MMES and DIP are not completely equivalent, we see many analogies and similarities and we believe that our MMES model and associated learning algorithm give some new insight for DIP.
DISCUSSIONS AND CONCLUSIONS
beautiful manifold representation of complicated signals in embedded space has been originally discovered in a study of dynamical system analysis (i.e., chaos analysis) for time-series signals (Packard et al.) 1980). After this, many signal processing and computer vision applications have been studied but most methods have considered only linear approximation because of the difficulty
of non-linear modeling VVnn Overschee De Moor 1991; Szummer Picard 1996; Li et al,,
1997 Ding et al. [2077] Markovsky (208)) However nowadays, the study of non- non-linear/man linear/manifold modeling has been well progressed with deep learning, and it was successfully applied in this study. Interestingly, could apply this non-linear system analysis not only for time-series signals but also natural color images and tensors (this is an extension from delay-embedding to multi-way shift- embedding). The best of knowledge, this is the first study apply Hankelization with AE into general tensor data reconstruction.
MMES is novel and simple image reconstruction model based the low-dimensional patch- manifold prior which has many connections to ConvNet. We believe it helps us to understand how work ConvNet/DIP through MMES, and support use DIP for various applications like ten- sor/image reconstruction or enhancement (Gong et al. 2018 Yokota et al.) 2019 Van Veen et al.,
2018 Gandelsman al. 2019).
Finally, we established bridges between quite different research areas such as the dynamical system analysis, the deep learning, and the tensor modeling. The proposed method is just a prototype and can be further improved by incorporating other methods such regularizations, multi-scale extensions, and adversarial training.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3663-359) 2014.
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In Proceedings of ICMI,, pp. 524-533, 2019
Chris Bishop. Training with noise equivalent Tikhonov regularization. Neural Computation, 7(1):108-114) 116, 1995.
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In Proceedings of CVPR, volume 2, pp. 60-65. IEEE, 2005.
Sungmin Cha, Taeeon Park, and Taesup Moon. Gan2gan: Generative noise learning for blind image denoising with single noisy images. arXiv preprint arXiv: 1905. 10488, 2019.
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-ichi Amari. Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation. John Wiley & Sons, 2009.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16 (8), 2007.
Max Welling Diederik P Kingma. Auto-encoding variational bayes. In Proceedings of ICLR, 2014.
Tao Ding, Mario Sznaier, and Octavia I Camps. A rank minimization approach video inpainting. In Proceedings of ICCV, pp. 1-8. IEEE, 2007.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of ICML, pp. 1050-1059, 2016.
Yosef Gandelsman, Assaf Shocher, and Michal Irani. "double-dip": Unsupervised image decompo- sition via coupled deep-imageepriors. In Proceedings of CVPR, June 2019.
Kuang Gong, Ciprian Catana, Jinyi Qi, and Quanzheng Li. PET image reconstruction using deep image prior. IEEE Transactions Medical Imaging, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016.
William Eric Leifur Grimson. From Images to Surfaces: A Computational Study of the Human Early Visual System. MIT Press, 1981.
Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application image denoising. In Proceedings of CVPR, pp. 2862-28222 2662-2869, 2014.
Frédéric Guichard and François Malgouyres. Total variation based interpolation. In Proceedings of EUSIPCO, pp. 1-4. IEEE, 1998.
Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non-convolutional networks. arXiv preprint arXiv:1810.03992, 2018.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-500 2006.
Frank I. Hitchcock. The expression of tensor polyadic as sum of products. Journal of Mathematics and Physics, 6(1):164-189 1927.
Harold Hotelling. Analysis of complex of statistical variables into principal components. Journal of Educational Psychology, 24(6):417, 1933.
Aapo Hyvarinen, Juha Karhunen, and Erkki Oja. Independent Component Analysis, volume 46.
John Wiley & Sons, 2004.
11

---

Under review as a conference paper at ICLR 2020
Hui Ji, Chaoqiang Liu, Zuowei Shen, and Yuhong Xu. Robust video denoising using low rank matrix completion. Proceedings of CVPR, pp. 1791-1798. IEEE, 2010.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint UrXii::442.660 2014.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. chise2void-learinge denoising from single noisy images. In Proceedings of CVPR, pp. 2129-2137, 2019.
Samuli Laine, Jaakko Lehtinen, and Timo Aila. Self-supervised deep image denoising. arXiv preprint arXiv:1901.10277 2019.
Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factor- ization. Nature, 401(6755):788, 1999.
Jaakko Lehtinen, Jacob Munkherg, Jon Hasselgren, Samuli Laine, Tero Karras, Miik Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In Proceedings of ICML, pp. 2971-2980, 2018.
Stan Z Li. Markov random field models in computer vision. In Proceedings of ECCV, pp. 361-370. Springer, 1994.
Ye Li, KJ Ray Liu, and Javad Razavilar. A parameter estimation scheme for damped sinusoidal signals based on low-rank Hankel approximation. IEEE Transactions on Signal Processing, 45 (2):481-486, 1997.
Liu, Przemyslaw Musialski, Peter Wonka, and Jieping Ye. Tensor completion for estimating missing values in visual data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):208-220, 2013.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv: 2015.
Ivan Markovsky. Structured low-rank approximation and its applications. Automatica, 44(4):891- 909, 2008
Stanley Osher, Zuoqiang Shi, and Wei Zhu. Low dimensional manifold model for image processing. SIAM Journal on Imaging Sciences, 1666-16666 2017.
Norman H Packard, James P Crutchfield, J Doyne Farmer, and Robert Shaw. Geometry from a time series. Physical Review Letters, 45(9):712, 1980.
Karl Pearson. LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901.
Gabriel Peyre. Manifold models for signals and images. Computer Vision and Image Understanding, 113(2):249-260. 2009.
Tomaso Poggio, Vincent Torre, and Christof Koch. Computational vision and regularization theory. Nature, 317:26, 1985.
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint arXiv: 1706.04987, 2017.
Tamar Rott Shaham, Tali Dekel, and Tomer Michacli. Singan: Learning generative model from single natural image. In Proceedings of ICCV, pp. 4570-4580, 2019.
Assaf Shocher, Nadav Cohen, and Michal Irani. Zero-shot super-resolution using deep internal learning. Proceedings of CVPR, pp- 3118-3126, 2018.
Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. Ingan: Capturing and retargeting the DNA of natural image. In Proceedings of ICCV, pp. 4492-4501, 2019.
12

---

Under review as a conference paper at ICLR 2020
Sho Sonoda and Noboru Murata. Transportation analysis of denoising autoencoders: A novel method for analyzing deep neural networks. arXiv preprint arXiv:1712.04145. 2017.
Martin Szummer and Rosalind W Picard. Temporal texture modeling. In Proceedings of ICIP, volume 3. pp. 823-826. IEEE, 1996.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), (8((1)267-228) 1996.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(:): 279-311, 1966.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of CVPR, pp. 9446-9454, 2018.
Peter Van Overschee and Bart De Moor. Subspace algorithms for the stochastic identification prob- lem. In Proceedings of IEEE Conference on Decision and Control, pp. 1321-1326. IEEE, 1991.
Dave Van Veen, Ajil Jalal, Mahdi Soltanolkotabi, Eric Price, Sriram Vishwanath, and Alexandros G Dimakis. Compressed sensing with deep image prior and learned regularization. arXiv preprint arXiv: 180:: 06438, 2018.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of ICML, pp. 1096-1 103, 2008
Curtis R Vogel and Mary E Oman. Fast, robust total variation-based reconstruction of noisy, blurred images. IEEE Transactions Image Processing, 7(6):813-824, 1998.
Wenqi Wang, Vaneet Aggarwal, and Shuchin Aeron. Efficient low rank tensor ring completion. In Proceedings of ICCV, pp. 5697-5705, 2017.
Francis Williams, Teseo Schneider, Claudio Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. Deep geometric prior for surface reconstruction. In Proceedings of CVPR, pp. 10130-10139, 2019.
Jun Xu, Yuan Huang, Li Liu, Fan Zhu, Xingsong Hou, and Ling Shao. Noisy-as-clean: Learning unsupervised denoising from the corrupted image. arXiv preprint 190iv:9006.0000 2019.
Yangyang Xu, Ruru Hao, Wotao Yin, and Zhixun Su. Parallel matrix factorization for low-rank tensor completion. Inverse Problems & Imaging, 9(2), 2015.
Noam Yair and Tomer Michaeli. Multi-scale weighted nuclear norm image restoration. In Proceed- ings of CVPR, pp. 3165-3174, 2018
Tatsuya Yokota and Hidekata Hontani. Simultancous visual data completion and denoising based on tensor rank and total variation minimization and its primal-dual splitting algorithm. In Pro- ceedings of CVPR, pp. 3732-3740, 2017.
Tatsuya Yokota and Hidekata Hontani. Simultaneous tensor completion and denoising by noise inequality constrained convex optimization. IEEE Access, 2019.
Tatsuya Yokota, Qibin Zhao, and Andrzej Cichocki. Smooth PARAFAC decomposition for tensor completion. IEEE Transactions on Signal Processing, 64(20);5423-5433 2016.
Tatsuya Yokota, Burak Erem, Seyhmus Guler, Simon K Warfield and Hidekata Hontani. Missing slice recovery for tensors using low-rank model in embedded space. In Proceedings of CVPR, pp. 8251-8259, 2018.
Tatsuya Yokota, Kazuya Kawai, Muneyuki Sakata, Yuichi Kimura, and Hidekata Hontani. Dynamic PET image reconstruction using nonnegative matrix factorization incorporated with deep image prior. In Proceedings of ICCV, pp. 3126-3135, 2019.
Jian Zhang, Debin Zhao, and Wen Gao. Group-based sparse representation for image restoration. IEEE Transactions Image Processing, 233)):3336-351, 2014a.
13

---

Under review as a conference paper at ICLR 2020
Zemin Zhang, Gregory Ely, Shuchin Aeron, Ning Hao, and Misha Kilmer. Novel methods for multilinear data completion and de-noising based on tensor-SVD. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3842-38499 2014b.
Qibin Zhao, Liqing Zhang, and Andrzej Cichocki. Bayesian CP factorization of incomplete ten- sors with automatic rank determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3709)1751-1763, 2015.
14

---

Under review as a conference paper at ICLR 2020
A
HANKELIZATION OF ONE- AND TWO-DIMENSIONAL ARRAYS
For example, Hankelization of one-dimensional array f = [fi. f2. ... .r. with window size 7 3 given by
f
f2
f3
f4
Js
f2
f4
Js
f6
f3
f4
fb
f6
fr/
We can see the anti-diagonal elements of above matrix are equivalent. Such matrix is called as "Hankel matrix".
For two-dimensional array
f11
f12
f13
f2
f22
f23
fs
f32
fa3
consider unfold of it and inverse folding by
J11
J21
f21
f31
f31
fi1
f12
f13)
f12
f11
f12
f13
f12
unfold
f21
f22
f2
=
f22
and
fz1
f22
f23
fold
f2
(8)
fa
fa2
faa
f32
fsi
f32
f33
fs2
f13
f13
S23
f23
f33
f33
The point here is that we scan matrix elements column-wise manner. Hankelization of this two- dimensional array (matrix) with T = [2, 2]
given by scanning a matrix with local (2.2)-window column-wis wise manner, and unfold and stack each local patch left foft-to-righ.. Thus, it is given
fu
f21
f22
fii
f21
f12
f22
f21
fai
f22
faz f23 f33/
fzi
fai
f22
fsz
f12
J22
f13
f12
f22
f1a
f23
f22
f32
/23
fm
fa2
f23
fa3
We can see that is a Hankel matrix. However, is a "block Hankel matrix" in perspective of block matrix, matrix that its elements are also matrices. We can see the block matrix itself is Hankel matrix and all elements are Hankel matrices, too. Thus, Hankel matrix is special case of block Hankel matrix in case of that all elements are scalar. In this paper, say simply "Hankel structure" for block Hankel structure.
Figure 9 shows illustrative explanation of valid convolution which is decomposed into delay- smnhedding/Hankelizattoo and linear transformation. ID valid convolution of f with kernel [h1, f22 ha] can be provided by matrix-vector product of the Hankel matrix and h. In similar way, 2D valid convolution can be provided by matrix-vector product the block Hankel matrix and unfolded kernel.
B MULTIWAY-DELAY EMBEDDING FOR TENSORS
Multiway-delay embedding transform (MDT) is a multi-way generalization of Hankelization pro- posed by Yokota et al. (2018).
In Yooktaa et al. 2018), MDT is defined by using the multi-linear tensor product with multiple duplication matrices and tensor reshaping. Basically, use the same operation, but a padding operation is added. Thus, the multiway-dele embedding used in this study is defined by
H(x) == unfold (pad (X) x1 S ... XN SN),
(10)
15

---

Under review as a conference paper at ICLR 2020
h1
fi
fah2 fahs
f1 fa fs
h2
h2
F2h + falt + faht
f2 fa f4
h2
ha
fah fale faha
fa fo f0
ha
(1D case)
fah; + fah2 + feha
fa fo
foh: + feh2 + frhs
fo fr
fm
Delay-embedding or Hankelization + Linear transform
J7
Ji f12 fi3
hn h12
fi
Arx Aet Age
Asa
fa f2z
fix fax
Aga Agg
(2D case)
fax fzz fa
bun lzz
fai fa2 f33
fa f22
Aaz A1z has Roo
fzr fax
Au Asz
fa Isa
Far fax
Aza Aag
fu f21 f12 fzz
ha
J21 fai J2a f32
frg
fold
fiu fza fia fus
x
hgg
J2a f32 Ja3 Sa3
hzg
Figure 9: Decomposition of 1D and 2D convolutions: Valid convolution can be divided into delay- cmbecddng/Faankelizatton and linear transformation.
where pad, R xxxxxxxxx + 111++ 1)) is a N N-dimensional reflection padding
operator of tensor, un 1
- R DDT is unfolding opera-
which outputs matrix from input N-th order tensor, and S, IRTa(4atT- - a duplication matrix. Fig. shows the duplication matrix with т.
For example, Hankelization with reflection padding of f [S1, f2. -. S7] with is given by
Fa
f2
f
f2
F3
f4
fz
fe
JT
12
_1
f2
f3
f4
Js
f6
JT
f6
(11)
f1
f2
fs
f1
fs
f6
fr
f6
fs/
Fig. [1 shows an example of our multiway- dellay embedding in case of second order tensors. The overlapped patch grid is constructed by multi-line linear tensor product with S.,. Finally, all patches are splitted, lined up. and vectorized.
The Moore-Penrose pseudo inverse of H. is given by
Ht H trim, (fold (H) X1 S xN S)y)
(12)
where St. = -x) is pseudo inverse of S,, fold(D,T) unfold and trim, - pad a trimming operator for removing (T, -1) elements at start and end of each mode. Note that H oH an identity map, but H. H is not, that is kind of projection.
B.1 DELAY EMBEDDING USING CONVOLUTION
Delay embedding and its pseudo inverse be implemented by using convolution with all
hot-tensor windows of size (71, T2, ..., Ty). The one-hot-tensor windows can be given by folding a
D-dimensional identity matrix Ip R DDD into In E X Fig. 12 shows calculation
flow of multi-way delay embedding using convolution in a case of N 2. Multi-linear tensor product is replaced with convolution with one-hot-tensor windows.
For
one
dimensional
array
[KI,..., my]"
we have pad (w)
Do 21
21
16

---

Under review as a conference paper at ICLR 2020
Pseudo inverse of the convolution with padding is given by its adjoint operation, which is called as the "transposed convolution" in some neural network library, with trimming and simple scaling with D
Z.
S
-U-r+1)
Figure 10: Duplication matrix. In case that have I columns, consists of (1 T + 1) identity matrices of size (T,
(12
pod
1)-
duplicate
unfald
pseudo inverse St Sg!
(1
1072
04
matricize
vectorize
(1s
(4 4 310
Figure 11: Flow of mulivay--dday--mmeeddng operation (N 2).
filters
72
pad
convolution
X
X
trim
conv_ transpose division 7173
fold
unfold
7172
(/1 + 1)(22
Figure 12: Multivay--dlay-cmbeddng using convolution (N 2).
C DESIGN OF AUTO-ENCODER
In this section, we discuss how design the neural network architecture auto-encoder for restrict- ing the manifold M,. The simplest way is controlling the value of and it directly restricts the dimensionality of latent space. There are many other possibilities: Tikhonov regularization (Good- fellow et al.) 2016), drop-out (Gal & Ghahramani] 2016). denoising auto-encoder (Vincent et al.] 2008). variational auto-encoder (Diederik Kingma, 2014), adversarial auto-encoder (Makhzani al. 2015), alpha-GAN (Rosca et al.) 2017). and on. All methods have some perspective and promise, however the cost is not low. In this study, we select an attractive and fundamental one: "denoising auto-encoder" relationship with Tikhonov regularization (Bishop) 1995), and decreases the entropy of data (Soodda & Murata 2017). Furthermore, learning with noise is also employed in the deep image prior.
/incent et al. 2008). The DAE is attractive because it has strong
Finally, we designed an auto-encoder with controlling the dimension r and the standard deviation o of additive zero-mean Gaussian noise. Fig. 13 shows the illustration of example of architecture of
17

---

Under review as a conference paper at ICLR 2020
auto-encoder which used in this study. In this case. it consists of five hidden variables of which sizes are D,,,,,,,,D) with leaky ReLU activation.
input D
output
dense layer leaky relu dense layer
Figure 13: An example of architecture of auto-encoder.
D A SPECIAL SETTING FOR COLOR-IMAGE RECOVERY
In case of multi-channel color image recovery case, because spacial pattern of individual channels are similar and the patch-manifold can be shared. Fig. [4 shows an illustration of the auto-encoder shared version of MMES in case of color image recovery. In this case, we put three channels of input and each channel input is embedded, indepen- dently. Then, three block Hankel matrices are concatenated, and auto-encoded simultaneously. In- verted three images are stacked as color-image (third-order tensor), and finally color-transformed. The last color-transform can be implemented by convolution layer with kernel size (1.1), and it is also optimized as parameters. It should be noted that the input three channels are not necessary to correspond to RGB, but it would be optimized as some compact color-representation.
special setting of generator network
colur-adjustment
normalization
Zn
Zg
ZB
x
embed
embed
auto-encoder
invert
invert
Ar
HR
HG
HB
HR
Hc
H
Figure 14: Generator network a case of color-image recovery.
E OTHER DETAILS OF MAGE-INPAINTING EXPERIMENTS
Here, we explain detailed experimental settings in Section
In this section, we compared performance of proposed method with several selected unsuper- vised image inpainting methods: low-rank tensor completion (HaLRTC) (Liu et al. 2013), parallel low-rank matrix factorization (TMac) (Xu et al.) 2015), tubal nuclear norm regularization (ISVD) (Zhang et al. 2014b). Tucker decomposition with rank increment (Tucker inc.)
Yokotta et al.
2018), low-rank and total-variation (LRTV) regularization" Yokota & Hontani]
2017
2019). smooth
PARAFAC tensor completion (SPC Yokota et al. 2016), GSR (Zhang al. 201 4a
multi-way
Foor LRTVV, MATLAB software was downloaded from
https://sites.. www.///............oo.oo.t/oo.o.
yt www.suwwaaaaaaa/aeea /ttp///ae//ttttrd/aaa//eee
For SPC, the MATLAB software downloaded from httpps:///ite....ooll.....
smooth
For GSR, each color channel was recovered, independently. using the MATLAB softwre downloaded
from ttombbccpee////ttub..oooo//// hwtp:///.....c/aaaaaaaaaa///////
18

---

Under review as a conference paper at ICLR 2020
Table 2: Parameter settings for MMES in image completion experiments
[T.
airplane
baboon
barbara
facade
house
lena
peppers
saiboat
50 %
(16.4)
(10,4)
(6,4)
(10,4)
(16,4)
(6,4)
(6,4)
(6,4)
70 %
(16,4)
(10,4)
(6,4)
(16,4)
(16,4)
(6,4)
(16,4)
(6,4)
90 %
(16,4)
(4,8)
(6,4)
(16,4)
(16,4)
(8,4)
(16,4)
(4,4)
95 %
(16,4)
(4,6)
(6,4)
(16,4)
(16,4)
(6,8)
(16,4)
(6,8)
99 %
(8,32)
(4,4)
(6,4)
(4.1)
(8,16)
(10,32)
(8,8)
(6,4)
delay embedding based Tucker modeling (MDT-Tucker Yokota et al. 2018). and DIF Ulyanov
al. 2018).
For this experiments, signal-to-noise ratio (PSNR) and for structural similarity (SSIM), although it would not be perfect. For DIP, we did not try the all network structures with various kernel sizes, filter sizes, and depth. We just employed "default architecture", which the details are available in supplemental materia of (Ulyanov et al., 2018), and employed the best results at the appropriate intermediate iterations in optimizations based on the value of PSNR. For the proposed MMES method, we adaptively selected the patch-size and dimension Table 2 shows parameter settings of r [T, 7] and r for MMES. Noise level of denoising auto- encoder was set as a = 0.05 for all images. For auto-encoder, same architecture shown in Fig. [3] was employed. Initial learning rate of Adam optimizer was 0.01 and we decayed the learning rate with 0.98 every 100 iterations. The optimization was stopped after 20,000 iterations for each image.
of all methods were tuned manually to perform the best peak-
F OTHER DETAILS OF
EXPERIMENTS
Here, we explain detailed experimental settings in Section
In this section, we compare performance of the proposed method with several selected unsupervised image super-resolution methods: bicubic interpolation, GSR ZZhang et al. 20142)0))))))) ZSSR" and DIP (Ulyanov et al. 2018).
In this experiments, DIP conducted with the best number of iterations from {1000, 2000, 3000, 9000). For four times (x4) up-scaling in MMES, we set r = 6, r = 32, and o 0.1. For eight times (x8) up-scaling in MMES, we set 7 6, 16, and a 0.1. For all images in MMES, the architecture of auto-encoder consists of three hidden layers with sizes of [8r2, rr.7... We assumed the same Lanezos2 kernel for wwwn--amplig system for all super-resolution methods.
Tab. 3 shows values of PSNR and SSIM of the results. We used three (256,256.3) color images, and six (512,512,3) color images. Super resolution methods scaling up them from four eight times down-scaled images of them. According to this quantitative evaluation, bicubic interpolation was clearly worse than others. ZSSR was good for (128.128.3) color images, however the performance were substantially decreased for (64,64.3) color image. Basically, GSR, DIP, and MMES were very competitive, In detail, DIP was slightly better than GSR, and the proposed MMES was slightly better than DIP.
FFor MDT-Tucker,
the MATLAB
software
downloaded
from
https: / / http://.w...ooeeeooooee//ee/caaa mdt www.ucceeeecooooocceeeeeooeoaeee coop:ooooooooootoooooooooooopoeo
For DIP, we implemented by ourselves in Python with TensorF
httpps://mii http:/mmmmtyulyyyoyyoo
For GSR, each color channel was recovered, independently, using the MATLAB software downloaded
from https: / thub. cem/
We slightly modified MATLAB code for applying
to super-resolution task.
For 7SSR, software was downloaded from
www.ccccccsooooooooo shocher/sSS
We set
the Lanezos2 kerne for this super-resolution task.
19

---

Under review as a conference paper at ICLR 2020
Table 3: Values of PSNR and SSIM in super-resolution task
PSNR/SSIM
Ricubic
GSR
7SSR
(proposed)
Sturfish 256)
23.98 .7124
25.737 .7922
25.13/.7748
25.79/.7930
26.18/ .8099
House (64 to 256)
26.21 178899
28.05
8394
26.89 8202
28.33 /8220
28.79 8448
Leaves (64 256)
.673
22,60/
20.191.744 1.706
22.54/ 8535
/
Aiplanee (128 to 512)
26.30/ .9766
27.74/ 94887
27.53/ 9430
27.49 .9355
28.40 .503
Airplane
512)
22.93 17545
23,79
22.74 .7629
23.83 8155
24.10 .207
Baboon (128 512)
20.61 .694
20,93/ / .7542
20.94 .7489
20.52 .7260
20,92/ / 74486
Baboon (64 512)
19.38 .5005
19.61 .5039
19.54 4926
19.64 .3085
19.64 .0024
Lena (122 512)
28.64 .1722
30.36/ / 9481
29.56/ .9417
29.91 .9406
29.76 / .4006
Lena (64 to 512)
25.23 .7110
26.47/ 8271
25.56/ 7946
26.71 8340
26,68 8327
Monarch (128 512)
26.00/
.576
Monarch (64 to 512)
20.65 76977
22.13/ 8393
21.22/ 8018
22.65 8594
23,01/ 8627
Peppers (128 to 512)
27.27 .9922
29.19 .9642
28.601 .589
28.78 9578
28.85 9584
Peppers (64 512)
24.15 8173
25.52/ .5753
24.35/ .299
26.07 8904
25.75 .794
Sailboal (128 to 512)
24.38 8885
25.43/ -9262
25.33/ 9228
25.13 9130
25.72/ .273
Sailboat (64 to 512)
21.22 6898
21,94/ 77633
21,55/ 77276
22.32/ 7664
23.37 / .7705
Average
23.66 7801
25.147 .3933
24.357 81411 8141
25.19 8401
25,53 .874
iter.
10 iter.
100 iter.
1000 iter.
10000 iter.
20000 iter.
10
rec
10
DAE
10
10
100
10
102
10
10
105
6
4
2
0
10
10
102 Iterations
10
10
10
Figure 15: Optimization behavior.
G OTHER EXPERIMENTAL RESULTS
G.1 OPTIMIZATION BEHAVIOR
For this experiment, we recovered 50% missing gray-scale image of 'Lena'. We stopped the opti- mization algorithm after 20,000 iterations. Learning rate set 0.01, and we decayed the learn- ing rate with 0.98 every 100 iterations. was adapted by Algorithm I every iterations. Fig. [5] shows optimization behaviors of reconstructed image, reconstruction loss Lee auto-encoding loss LDAE and trade-off coefficient By using trade-off adjustment, the reconstruction loss and the auto-encoding loss intersected around 1,500 iterations, and both losses jointly decreased after the intersection point.
G.2 HYYPER-PARMEER SENSITIVITY
We evaluate the sensitivity of MMES with three patch-size as 8), and dimension and noise standard deviation were varied. Fig. 17 shows
r, a, and т. First, we fixed the
20

---

Under review as a conference paper at ICLR 2020
Original
(999i
With noise
Without noise
Figure 16: Reconstruction of 'home' image by training with/without noise in deep image prior.
the reconstruction results of a 99% missing image of 'Lena' by the proposed method with different settings of (r, The proposed method with very low dimension = 1) provided blurred results, and the proposed method with very high dimension (r = 64) provided results which have many peaks. Furthermore, some appropriate noise level (a 0.05) provides sharp and clean results. For reference, Fig. shows the difference of DIP optimized with and without noise. From both results, the effects of learning with noise can be confirmed.
Next, we fixed the noise level as a = 0.05, and the patch-size were varied with some values of Fig. 18 shows the results with various patch-size settings for recovering a 99% missing image. The patch sizes 7 of (8,8) or (10,10) were appropriate for this case. Patch size is very important because depends on the variety of patch patterns. patch size is too large, then patch variations might expand and the structure of patch-manifold is complicated. By contrast, if patch size is too small, then the information obtained from the embedded matrix is limited and the reconstruction becomes difficult in highly missing cases. The same problem might be occurred in all patch-based image reconstruction methods (Buades et al.) 2003; Dabov et al.) 2007] Gu et al.] 2014; Zhang al. 2014a). However, good patch sizes would be different for different images and types/levels of corruption, and the estimation of good patch size is an open problem. Multi-scale approach (Yair & Michaalll (2118) may reduce part of this issue but the patch-size is still fixed or tuned hyper-parameler.
G.3 VOLUMETRIC/3D IMAGE/TENSOR COMPLETION
In this section, we show the results MR- JiR-imagc/DD-tensoo completion problem. The size of MR image is (109,91,91). We randomly remove 50%, 70%, and 90% voxels of the original MR-image and recover the missing MR-images by the proposed method and DIP. For DIP, we implemented the 3D version of default architecture in TensorF low, but the number of filters of shallow layers were slightly reduced because of the GPU memory constraint. For the proposed method, 3D patch-size [4, 4, 4), the lowest dimension was r and noise level was 0.05. Same
was architecture shown in Fig. 13 was employed.
Fig. shows reconstruction behavior of PSNR with final value of PSNR/SSIM in this experiment. From the values of PSNR and SSIM, the proposed MMES outperformed DIP in low-rate missing cases, and it quite competitive in highly missing cases. The some degradation of DIP might be occurred by the insufficiency of filter sizes since much more filter sizes would be required for 3D ConvNet than 2D ConvNet. Moreover, computational times required for our MMES were signifi- cantly shorter than that of DIP in this tensor completion problem.
21

---

Under review as a conference paper at ICLR 2020
noise levels of denoising autoencoder
0.2
0.1
0.05
0.01
0.001
r=1
r=4
r=16
r=64
Performance of reconstruction of color image of 'Lena" with 99% pixels missing for various parameter setting.
Figure 17:
(4,4), F=4 20.0915 0.5654
(6,6), r=8 20.6534 0.6372
(8,8), 16
(10,10), r=32
(12,12), r=48
(16,16), r=
PSNR SSIM
20.9763
21.2370 0.6626
20.4509
19.2145
0.6568
0.6545
0.5827
Figure 18: Reconstruction of 'Lena' image for various patch sizes
Methods (missing rate, PSNR/SSIM)
35
DIP(50%,31.588 9475)
30
DIP(70%.29.788)
PSNR [dB]
DIP(90%, 26.16 1.8613)
25
Proposed(50%, 33.38 .9676
20
Proposed(70% 30.57 9448 )
Proposed(90%..25.96 .8636
15
0
10 Computational time [h]
15
Figure 19: Results of MRI completion: Optimization behaviors of PSNR with final values of PSNR/SSIM by DIP and proposed MMES.
22


Reviews:
[{"review_id": "r1xJqwuRKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a natural image model based on the manifold of image patches.  It is similar to the Deep Image Prior in that it is untrained and has a convolutional-like structure.  It leads to an optimization problem with a reconstruction loss term and an auto encoding term.  The authors show empirical results in time series recovery, non-semantic inpainting, and super resolution.  In the image processing tasks, the performance of the proposed algorithm is on par (sometimes slightly worse, sometimes slightly better) than that of DIP.\n\nI think the perspective of image patch analysis is a useful addition to the knowledge base for unlearned image priors.  That said, the paper says it tackles the questions for why the DIP \"works so well\" and why convolution operations are \"essential for image reconstruction or enhancement\".  After reading the paper, it is unclear to me how this work addresses these questions.  In particular, demonstrating a similar convolutional system does not rule out the possibility that there are non-convolutional systems that also explain the effect.  For example, the Deep Geometric Prior paper is nonconvolutional (it is entirely a MLP), and it also has the effect of fitting a smooth signal without training (subject to early stopping).  The DGP could be applied to images as well, resulting in a nonconvolutional deep image prior.  I think the authors should address more clearly and thoroughly the logical connection between their results and the explanation of the DIP, especially in light of the DGP.\n\nThe paper claims that the proposed method is more interpretable, and it would be nice if they could demonstrate this interpretability and the benefits it brings in solving image reconstruction problems.  \n\nAs a result, I am inclined to recommend a weak reject, but if the concerns above are addressed, I envision my rating could improve upon rebuttal.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1g1383viS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a transformation from the deep image prior (DIP) to an embedding with an autoencoder (MMES). The authors aim to use this transformation to explain (\"in words\") why the DIP works so well and explain why convolutions are needed in the DIP. The contributions are summarised as a) providing an interpretable analogue to the convnet, b) demonstration of the proposed method's effectiveness, and c) characterisation of the DIP as a \"low-dimensional patch-manifold prior\".\n\nI think the MMES approach is interesting and potentially a good analogue to the DIP, and explicitly draws out the locality prior the authors claim is integral to DIP. The good results and comparison to DIP demonstrates that this locality prior may be important to the task.\n\nI disagree that this method is \"interpretable\"/\"explainable\", at least without any evidence toward this presented in the paper. There is still fundamentally a deep network as in DIP. The discussion on interpretability is limited and mostly provided through comparison with other methods.\n\nAll up I think this is a useful paper, even though the paper overstates its contributions. I would like to see the clarity improved: it took me a long while to make the connection between the method presented in the paper and the implications for DIP. This connection should have been more explicit in the paper. I would also like to see the \"interpretability\" statement either clearly explained or removed (I am not convinced that this method is interpretable).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkllj_e0tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposed a low-dimensional patch-manifold prior perspective for reinterpreting the deep image prior. I think this is very interesting work that could have a lot of impact in vision and beyond, since effective construct the problem in reconstruction tasks are highly relevant to a number of tasks. I was initially quite excited about this paper, but as I drilled into the details of this work. \n\nFor the manifold modeling, though the authors defined each part of the formulation in equations (1)-(5), but it is not clear how to design corresponding efficient structures for different low-level problems. In other words, the application ability is not clear. For example, if the proposed MMES scheme is used in image deconvolution tasks, how to design the corresponding structure?\n\nFor the parameters, there is no description of the parameters set in subsection 4.2 and 4.3. And what are the criteria for selecting these parameters?\n\nFor the experimental part, the comparison experiments in subsection 4.1(Toy examples) and subsection 4.3 (Color image superresolution) lack comparisons with the latest methods. I think, more comparison experiments should be provided. \n\n\n[Update after rebuttal period]\nThe revision and responses have addressed most of my concerns, so I keep my original score.\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a combination of a delay embedding as well as an autoencoder to perform representation learning. The proposed algorithm shows competitive performance with deep image prior, which is a convnet structure. The paper claims that the new approach is interpretable and provides explainable insight into image priors.

The discussion period was used constructively, with the authors addressing reviewer comments, and the reviewers acknowledging this an updating their scores.

Overall, the proposed architecture is good, but the structure and presentation of the paper is still not up to the standards of ICLR. The current presentation seems to over-claim interpretability, without sufficient theoretical or empirical evidence.

Decision: Reject