BOOSTING ENCODER-DECODER CNN FOR INVERSE PROBLEMS
Authors: Eunju Cha
Jaeduck Jang
Junho Lee
Eunha Lee
Jong Chul Ye
Venue:  (2020)
Keywords: Prediction error
Boosting
Encoder-decoder convolutional neural network
Inverse problem

Abstract: Encoder-decoder convolutional neural networks  (CNN)  have been extensively used for various  inverse problems. However,  their prediction error  for unseen test data is difficult to estimate  a priori, since the neural networks are trained using only selected data and their architectures are largely considered blackboxes. This poses a fundamental  challenge in improving the performance of  neural networks. Recently, it was shown that Stein’s unbiased risk estimator (SURE) can be used as an unbiased estimator of the prediction error for denoising problems. However, the computation of the divergence term in SURE is difficult to implement in a neural network framework, and the condition to avoid trivial identity mapping is not well defined. In this paper, inspired by the finding that  an encoder-decoder CNN can be expressed as a piecewise linear representation, we provide a  close form expression of  the unbiased estimator for the prediction error. The close form representation leads to a novel boosting scheme to prevent a neural network from converging to an identity mapping so that it can enhance the performance. Experimental results show that the proposed algorithm provides consistent improvement in various inverse problems.

Under review as a conference paper at ICLR 2020
IMPROVING ENCODER-DECODER CNN FOR INVERSE PROBLEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Encoder-decoder convolutional neural networks (CNN) have been extensively used for various inverse problems. However, their prediction error for unseen test data is difficult to estimate priori, since the neural networks are trained us- ing only selected data and their architectures are largely considered blackboxes. This poses fundamental challenge in improving the performance of neural net- works. Recently, was shown that Stein's unbiased risk estimator (SURE) can be used However, the computation of the divergence term in SURE is difficult to imple- ment in neural network framework, and the condition avoid trivial identity mapping is not well defined. In this paper, inspired by the finding that an encoder- decoder CNN can be expressed close form expression of the unbiased estimator for the prediction error. The close form representation leads to a novel bootstrap and aggregation scheme to prevent a neural network from converging to an identity mapping so that it can enhance the performance. Experimental results show that the proposed algorithm provides consistent improvement in various inverse problems.
unbiased estimator of the prediction error for denoising problems.
piecewise linear representation, we provide a
INTRODUCTION
Suppose that the measurement æ is corrupted with additive noises:
w. w ~ N((,
(1)
where μ E R denotes the unknown mean vector, and o² is the variance. Consider deep neural network model F(x) == Fe (æ) with the weight θ which is trained with the data producing an estimate ç Fa (x). Then, our goal is to estimate the prediction error.
Eir(ft) Fx. ||=
/ Faa F(x))?
(2)
which quantifies how well ja can predict a test data x", independently drawn from the same distribu- tion of (Efron) 2004 Tibshirani & Rosset 2018). The problem of estimating the prediction error is closely related to the generalizability of neural network (Anthony & Bartlett 2009). Moreover, this problem is tightly linked to the classical approaches for model order selection in statistics literature
(Stoica & Selen, (2044). One of the most investigated statistical theories to address this question is so-called covariance penalties approaches such as Mallow's 's Cp (Mallows 1973). Akaike's informa- tion criterion (AIC) (Akaike 1974), Stein's 's unbiased risk estimate (SURE) (Donoho & Johnstone) 1995), etc.
This paper is particularly interested in estimating the prediction error of encoder-decoder convo- lutional neural networks (F-D CNNs) such as U-Net (Ronneberger et al. 2015 Han Yel 2018 Ye & Sung 2019). E-D CNNs have been extensively used for various inverse problems (Ye et al. 2018). Recent theoretical results showed that, thanks to the ReLU nonlinearities, the input space is partitioned into non-overlapping regions that input images in each region share the same linear representation, but not across different partitions Ye & Sung 2019).
One of the most important contributions of this paper is that this property be exploited to derive a simplified form of prediction error estimator that can be used for neural network training. In par- ticular, provide close form expression for the divergence term in the SURE estimator, which

---

Under review as a conference paper at ICLR 2020
suggests that the divergence term be neglected if proper batch normalization is used. Further- more, the simplified form of the SURE estimate for the prediction error leads to a bootstrap and aggregation scheme to prevent the CNN from converging the neural network performance. The applicability of the new method is demonstrated using various inverse problems such accelerated MRI, energy-dispersive X-ray spectroscopie imaging, decon- volution, and etc, which clearly show that our method can significantly improve the image quality compared the existing approaches.
trivial solution so that it can improve
RELATED WORKS
2.1 STEIN RISK UNBIASED ESTIMATION FOR PREDICTION ERROR
The Stein Risk Unbiased Estimator (SURE) for the prediction error can be represented by (Dnnhho & Johnstone] 1995):
(x) = x - F(x)]1 div
where div(.) denotes the divergence. Recently, the authors in (Soltanayev & Chun 2018 employed the SURE for CNN-based image denoising without references by minimizing the following loss:
(SURE( = Σ (0) (21 20 2o
where [P denotes the training samples. Although the application of SURE for unsupervised denoising is an important advance in theory, there are several practical limitations. In particular, due to the difficulty of calculating the divergence term, the authors relied on MonteCarlo SURE using MonteCarlo simulation. This
(Ramani et al.) 2008) which calculates the divergence introduces additional hyperparameters, on which the final results critically depend.
2.2 PIECEWISE LINEAR REPRESENTATION BY E-D CNN WITH RELUS
Recently, the authors in Ye & Sung 2019) showed that the output of a CNN composed of и layer of encoder and decoder without skipped connection can be represented by the following basis-like representation:
y :- F(x)
()) - B(x)
(5)
where (x:)) and (æx) denote the i-th column of the following frame basis and its dual:
B(x) = E' (x) B(x) = (' ... (x))
(6)
(7)
where (x) and (x)) denote the diagonal matrix with 0 and I values that are determined by the ReLU output in the previous convolution steps. E' and D' refer the encoder and decoder matrices. respectively, which are determined by pooling (resp. unpooling) operation and convolution filters CNN with the skipped connection, similar basis expression can be obtained Ye & Sung (019))
(Ye & Sung 2019). For the case
Note that the expression (5) is nonlinear due to the dependency on the input signal æ. Moreover, as the ReLUU nonlinearity is applied after the convolution operation, the on-and-off activation pattern of each ReLU determines a binary partition of the feature space across the hyperplane that is deter- mined by the convolution. Accordingly, one of the most important observation in Ye & Sung (2019) that the input space is partitioned into multiple non-overlapping regions so that input images for each region share the same linear representation, but overall representation is still non-linear. This implies that two different input images may automatically switched to two distinct linear representations that different from each other.
2

---

Under review as a conference paper at ICLR 2020
3 MAIN CONTRIBUTION
In 33 and 4 the calculation the divergence term is not trivial for general neural networks. This is why the authors in (Soltanayev & Chun 2018) employed the MonteCarlo SURE. Thanks to the basis-like expression in (5), here we show that there exists simple explicit form the divergence term for the case of E-D CNNs. Then, the batch normalization is shown to make the divergence term trivial.
3.1 DIVERGENCE SIMPLIFICATION IN SURE
In Proposition 6 of Ye & Sung 2019), the authors show that (F(x)/8x B(x)) for the case of E-D CNN with ReLUs, Accordingly, we have
div div(F(x)) = Tr
Ox (
(FF(z)
= Tr (B())B(a)T) Tr (B(())
b(b(()),
(8)
where (AA denotes the trace of A.
This close form representation (8) leads to further simplification of divergence term by exploiting
the property of the batch normalization. Recall that batch normalization has been extensively used to make the training stable (loffe & Szegedy 2015 Hoffer et al. 2018 Cho & Lee, 2017 Miyato al., 2018: [Ulyanvv et al., 2016). It has been consistently shown that the batch normal ization closely related to the norm of the Jacobian matrix OF x (Ox, which is equal to (()) For example, in their original paper (loffe & Szegedy 2015). the authors conjectured that "Batch Normalization may lead the layer Jacobians to have singular values close 1, which is known to be beneficial for training". By extending the idea in (loffe & Szegedy 2015) to multiple layers, the batch normalization can be understood as to make the covariance of the network output and input similar. For example, for the uncorrelated input with (ov(=(!) = o² I, the batch normalization may result Cov F( (a() = I²I. Furthermore, for sufficiently smaller a, we have
o² ~ Cov [o Cov B(x B(x())(((()
(1)) (z) T( ()) (()T)
(1) B (1) ( Cov
(1) (()
B(x )(a()) (()())
o² ((x (1) B(x
(1)
because the corresponding piecewise linear representation does not change for the small perturbation have
of the input (Ye & Sung 2019). Therefore,
B(x B(x I,
since (xx( ((x is a square matrix. This suggests that (b.(2 (20( = Tr
LivEr (i
n,
(9)
where n is the dimension of the input signal. As the resulting divergence term just constant, the contribution of the divergence term is considered trivial and can be neglected.
3.2 BAGGING ESTIMATOR
Another important drawback of SURE-based denoising network 44) is that is difficult prevent the network from learning trivial identity mapping. More specifically, if Fo(z)) æc, the cost function in 4 becomes zero. One way to avoid this trivial solution to guarantee that the divergence term at the optimal network parameter should be negative. However, given that the divergence term comes from the degree of the freedom (Efron) 2004) and the amount of excess optimism in estimating the prediction enror (Tibshirani & Rosset 2018), enforcing negative value may be unnatural. Moreover, with the divergence simplification in (9) the cost function 44)
can be simplified as
( ( =
x ()) Fo(x 202n
(10)
p
3

---

Under review as a conference paper at ICLR 2020
so that it is much apparent that trivial solution for (00) is Fe(x)
To prevent the neural network from converging to this trivial solution, here we propose a bootstrap aggregation (bagging) scheme (Breiman (996). Bagging is classical machine learning technique which uses bootstrap sampling and aggregation of the results to reduce the variance and to improve the accuracy of the base learner. The rationale for bagging is that it may be easier train several simple weak learners and combine them into a more complex learner than to learn a single strong learner. More specifically, we use the following bagging estimator:
K
(
wk (L)
(11)
and the corresponding prediction error estimate:
K wk Fe (L
Err(p) == Eag, ot x jall Em, w x="
where {L LK denotes the diagonal matrix whose diagonal elements are either 0 or depending on the bootstrap subsampling patterns, and {wx) ) is the corresponding weights. The corresponding empirical prediction error estimate is then given by
e (wx))
1 p
Σ
x
Σ wk Fe (1)
(12)
Note that (1) that minimizes 12) cannot be the trivial identity mapping. Furthermore, the following proposition shows that the prediction error for the bagging estimator individual error estimate.
(1) due to the subsampling patterns so that the optimal neural network Fe
[II) is not bigger than the average of the
Proposition 1. Let wk 2 0 and Ee wh = 1. Then, we have
P
Wk ((0) - )12
2 P
Σ
((()) - -E w (()
P 1m1
Proof. For any x E (1) 1=11 we have
Σ wk | 2(Lx2))1
T x 2x
Σ
Wv F (L1z) + Σ wk [(L=x))]
x
k
k
2 8 T x 2x
T
Wk F (L+x))
+ Σ wy. (L
k
x Σ W (L
where use the Jensen's inequality for the inequality. By summing up for all training data {P Ji=1, conclude the proof.
The gap increases more when the F(L:)) provides diverse output for each realization of the index La (Breiman] 1996). In fact, the authors in (Ye Sung 2019) showed that the input space results in non-overlapping partitions with different linear representations so that by changing the subsampling pattern Lg. the distinct representation may be selected, which can make the corresponding bagging estimate more accurate.
4 IMPLEMENTATION DETAILS
The final loss fiucctinn for our bagging estimator is given by
K
(0,
E)
= P
wk wy Fo 12)
(13)
4

---

Under review as a conference paper at ICLR 2020
Bootstrup subsampling
Deighad
Regression
Network
Attention Network
Figure I:
Overall network architecture of the proposed method. The acquired data is split into multiple subset using bootstrap subsampling. The data is then processed using regression network followed by an aggregation module using an attention network.
where we also parameterize the weighting using a neural network with parameter Although the standard way of aggregation in bagging estimator is simple average of the overall results of the regression network, this may not the best method when the number of bootstrap subsampling is limited. Instead, propose weighted average scheme whose weight is calculated by data attention module so that it efficiently combines all data by adaptively incorporating output from various bootstrap sub-sampling patterns. schematic diagram of the proposed method is illustrated in Fig. [ which consists of three building blocks: bootstrap subsampling, a regression network, and an attention network to calculate the weight parameters.
10:4
20V5
512
S12
Joca b4
3x3 Comy, ReLU
1xL Comv
Skip
Concat
Average Pooling
Decom,
Figure 2: Architecture for the regression network.
In particular, U-net [Ronneberger et al. 2015) in Fig. 2 was used as our regression network. The network is composed of four stages with convolution, batch normalization, ReLU, and skip connec- tion with concatenation. Fach stage is composed of three 3 x 3 convolution layers followed by batch normalization and ReLU,, except for the last layer, which 1 x convolution layer. The number of convolutional filters increases from 64 in the first stage to 1024 in the final stage. To estimate the weight Wk. our attention network consists two fully connected layer. The input dimension of the attention network is IR 1x 1xK followed by the average pooling of the concatenated output of regres- sion network. The number hidden node is 64, and the final dimension of the output is IR xx for aggregation.
The overall network was trained using Adam optimization with the momentum 31 = 0.9 and B3 0.999. The proposed network was implemented in Python using TensorFlow library and trained using an NVidia GeForce GTX 1080-Ti graphics processing unit.
EXPERIMENTAL RESULTS
Experiments were conducted for various inverse problems such as accelerated magnetic resonance imaging (MRI), energy-dispersive X-ray spectroscopy (EDX) (Solé et (007), and image super- resolution in Appendix.
5.1 ACCELERATED MRI
accelerated MRI, the goal to recover high quality MR images from sparsely sampled k-space data to reduce the acquisition time. This problem has been extensively studied using compressed sensing (Lustig et al. 2007). but recently deep learning approaches have been the main research interest due its excellent performance at significantly reduced run time complexity (Hammernik et al. 2018; Han & Ye 2019). A standard method for neural network training for accelerated MRI based the supervised learning, where the MR images from fully sampled k-space data is used
5

---

Under review as a conference paper at ICLR 2020
reference and subsampled and zero-filled k-space data is used as the input for the neural network. Specifically, we use the R = 13.45 subsampled k-space data as neural network input and the goal to obtain high resolution images from the sub-sampled k-space data. As baseline algorithm, the state-of-the-ar deep learning algorithm called k-space deep learning (Han & Ye 2019) is employed, which interpolates the missing k-data using CNN.
In the proposed method, the accelerated k-space data is further subsampled using bootstrap sam- pling with the subsampling ratio of 91 %. and the number of bootstrap sample set K was set to 10. The same k-space deep learning network (Han& & Ye 2019) was used for our regression net- work using bootstrap samples. We trained the baseline algorithm and our method under the same conditions, except that the proposed network was trained by minimize the loss (13). The initial learning rate was set to 10-2, and it was divided by half at every 50 epochs until it reaches to around 10-1 The training data was generated from Human Connectome Project (HCP) MR dataset (https:/ddb.ummamn www.hcomeetttnne.orgg....o..
Among the 34 subject data sets, 28 subject data sets were used for training and validation. The other subject data sets were used for test. We also provide re- construction results by GRAPPA (Griswold et al. 2002), which is a standard k-space interpolation method in MRI.
Ground-truth
GRAPPA
Baseline
Ours
34.213 / 0.719
40.515 / 0.920
47.445/ / 0.951
Figure 3: Reconstruction results using accelerated MR data at the acceleration factor of R 13.45. The PSNR and SSIM index values for each images are written at the corner.
Thanks to the bagging, the proposed method provided nearly perfect reconstruction results compared to other algorithms as shown in Fig. 3 Moreover, its average peak signal cormall-t-noise ratio (PSNR) and structural similarity (SSIM) index for entire test dataset significantly outperform existing methods as shown in Table 1
Table 1: Quantitative comparison using PSNR and SSIM index.
GRAPPA Baseline Han Yo 2019
Ours
PSNR (dB)
34.52
38.67
42.72
SSIM
0.72
0.89
0.91
To demonstrate the optimality of the proposed architecture, we performed extensive ablation study. For the ablation study, we used 13 subject data sets from HCP MR dataset, from which seven subject data sets were used for training and validation and the remaining dataset were used for test. The ablation study was performed by excluding some structures from the network and applying the same training processes. First. our theory says that the performance improvement of our method increases with the number of bootstrap subsamples K. To verify this, we performed comparative studies with different K values such as 1, 2, and 10. As shown in Table the proposed network provided better reconstruction results as the number of bootstrap sample set K increased. This clearly confirmed the importance of the bootstrap subsampling. Second, our derivation suggests that the batch normalization plays a key role to simplify the divergence term. As shown in Fig. 4 and Table 3 the lack of batch normalization produces a significantly degraded image with an average PSNR drop from 40.047 dB 25.290 dB. This verified the important role of batch normalization.
Table 2: Quantitative comparison using PSNR and SSIM index with respect to number of bootstrap subsample sets.
K
4
10
PSNR (dB)
38.68 0.89
39.81
40.05
40.27
SSTM
0.90
0.91
0.93

---

Under review as a conference paper at ICLR 2020
We also evaluated the importance of the attention network calculate the weights. With standard bagging, the final output a simple average of the entire network output for each bootstrap subsam- ples without the attention network. As shown in Fig. 4 and Table 3] the simple average produced blurry results than by the attention network and the PSNR and SSIM values were worse. This con- firmed that the important of attention network for the weight calculation.
Ground-trutie
w/o BN
Standard bagging
Proposed
24.093 / 0.445
37.061 / 0.889
44.897 / 0.955
Figure 4: Ablation study for the accelerated MR experiments at the acceleration ratio R = 13.45. The PSNR and SSIM index values for each images written the corner. w/o BN: reconstruc- tion results from an ablated network without batch normalization, Standard bagging: reconstruction results from ablated network without an attention module.
Table Quantitative comparison in terms of PSNR and SSIM index in our ablation study.
without BN
Standard bagging
Ours
PSNR (dB)
25.29 0.49
35.91 0.867
40.05
SSIM
0.914
5.2 ENERGY-DISPERSIVE X- RARAY SPECTROSCOPY IMAGING
As for another experiment, we use the energy-dispersive X-ray spectroscopy (EDX) (oolé et al. 2007) data set which were measured by scanning transmission electron microscopy. EDX (Solé al. 2007) is an analytical technique used for spectroscopic characterization of a sample. relies on the emission of characteristic X-rays from specimen that is generated by high-energy beam of electrons. Specifically, the incident beam may eject an electron from the inner shell of atoms. Then, an electron from an outer shell then fills the hole, and the energy difference may be released in the form of an X-ray. As the energies of the X-rays are characteristic of the energy differences that are determined by the atomic structure of the emitting element, EDX is widely used for nano-scale quantitative and qualitative elemental composition analysis by measuring X-ray radiation from the interaction with high energy electron and the material (McDowell et al., 2012).
However, in EDX, the specimens can be quickly damaged by the high energy electrons, so the acquisition time should be reduced to its minimum. This usually results in very noisy and even incomplete images as shown Fig. a). Therefore, the goal of this experiment was to remove the noise and reconstruct high resolution EDX images. The main technical difficulty is that there are no noiseless reference data. Due to the lots of missing photons, a widely used approach for EDX analysis kernel regression method using an average kernel as shown in Fig. (b)) Unfortunately, this often results in severe blurring. Existing unsupervised learning methods such as SURE network (Soltanayev & Chun 2018) and Noise2Noise (Lehtinen al.) 2018) cannot be used for this purpose, since there are no specific noise models for the EDX and the noiseless clean data is not available for Noise2Noise training. In fact, this difficulty of EDX denoising was our original motivation for this work. For network training and inference, use bootstrap subsampled images from the measurement image in Fig. 5 a). In addition, we used the measurement image in Fig. 5(a) as the label for training.
We used 28 cases from the EDX dataset. The specimen are composed of quantum-dots, where core and shell consist of Cadmium (Cd), Selenium (Se), Zine (Zn), and Sulfur (S). The regression network was trained using image patches, whose sizes was 128 x 128. For stable training. the intensity of data normalized between [0, 1]. The initial learning rate was set to 10-3, and it was divided in
7

---

Under review as a conference paper at ICLR 2020
half every 50 epochs until it reaches around 10- For the bootstrap subsampling, the number of random subsampling mask was K = 30. The regression network was optimized to minimize the loss (13) with respect to ④ first, after which the attention network was trained to properly aggregate the entire interpolated output. As shown in Fig. d),,, our method successfully produced high resolution images which significantly better than existing kernel regression methods. Thanks the bagging procedure, the regression network is believed to learn the measurement statistics so that the network not only removes the noise but also filled in the unmeasured data. We also compared the performance between the standard bagging and the proposed method. As shown in Fig. ((c) and (d), the weighted averaging from the attention module provides better performance over the the standard bagging that uses the simple average. This confirmed the efficacy of the proposed method.
Measured data
Kernel regression Standard bagging
Proposed
(b)
(d)
1st Sample
Measured data
Kernel regression Standard bagging
Proposed
(a)
(b)
(d)
2nd Sample
Figure 5: EDX experimental results. Green and blue particles refer to Zn and Cd, respectively. (a) Input data, and the reconstruction results using (b) the existing kernel regression method, and the proposed methods (c) with simple average (standard bagging) and (d) the attention based weighted average.
CONCLUSION
In this paper, we proposed a novel bagging scheme of encoder-decoder CNNs for various inverse problems. The algorithm was derived by the observation that the existing SURE-based unsuper- vised denoising networks have several practical limitations due the divergence term and potential to converge trivial solution. Inspired by the recent discovery of basis-like representation of the encoder-decoder CNN, we provide a simple approximation of the divergence term. This also led to novel bagging scheme to prevent from converging the trivial identity mapping. In the im- plementation, multiple input data were generated by bootstrap subsampling, after which final result are obtained by aggregating the entire output of network using an attention network. Experimen- tal results with accelerated MRI and EDX experiments showed that the proposed method provides consistent improvement for various inverse problems.
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1265135, 2017.
8

---

Under review as a conference paper at ICLR 2020
Hirotugu Akaike. A new look at the statistical model identification. In Selected Papers of Hirotugu Akaike, pp. 215-222. Springer, 1974.
Martin Anthony and Peter L Bartlett. Neural network learning: Theoretical cambridge university press, 2009.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems, pp. 5225-5235, 2017.
David L Donoho and lain M Johnstone. Adapting to unknown smoothness via wavelet shrinkage. Journal of the american statistical association, 90(432):1200-122)) 1995.
Bradley Efron. The estimation of prediction error. covariance penalties and cross cross-validation. Jour- nal of the American Statistical Association, 99(467):619-632. 2004.
Mark A Griswold, Peter M Jakob, Robin M Heidemann, Mathias Nitka,, Vladimir Jellus, Jianmin Wang, Berthold Kiefer, and Axel Haase. Generalized autocalibrating partially parallel acquisi- tions (GRAPPA). Magn. Reson. Med., 4766):1002-1210, 2002.
Kerstin Hammemnik, Teresa Klatzer, Erich Kobler, Michael P Recht, Daniel K Sodickson, Thomas Pock, and Florian Knoll. Learning variational network for reconstruction of accelerated MRI data. Magnetic resonance in medicine, 79(6):305533071 2018.
Yoseob Han and Jong Chul Ye. Framing U-Net via deep convolutional framelets: Application to sparse-view CT. IEEE transactions on medical imaging, 37(6):111-1129, 2018.
Yoseob Han and Jong Chul Ye. k-Space Deep Learning for Accelerated MRI. IEEE Transactions on Medical Imaging (in press), also available as arXiv preprint urXiv:1805.0377 2019.
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back-projection networks for super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1664-16733 2018.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, pp. 2160-2170, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 148-456 2015.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2Noise: learning image restoration without clean data. In International Confer- ence on Machine Learning, pp. 2971-2980, 2018.
Michael Lustig, David Donoho, and John M Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn. Reson. Med., 58(6):1182-1183 2007.
Colin L Mallows. Some comments on pp Technometrics, 15(4):661-675. 1973.
Matthew T McDowell, IIl Ryu, Seok Woo Lee, Chongmin Wang. William D Nix. and Yi Cui. Studying the kinetics of crystalline silicon nanoparticle lithiation with in situ transmission electron microscopy. Advanced Materials, 24445))6034-6041 2012.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 2018.
Sathish Ramani, Thierry Blu, and Michael Unser. Monte-Carlo SURE: black-box optimization
of regularization parameters for general denoising algorithms. IEEE Transactions on image pro- cessing, 154) 2008.

---

Under review as a conference paper at ICLR 2020
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, Springer. 2015.
Solé, Papillon, Cotte, Walter, and Susini. multiplatform code for the analysis of energy energy-dispersive X-ray fluorescence spectra. Spectrochimica Acta Part B: Atomic Spectroscopy, 62(1))633-8, 2007.
Shakarim Soltanayev and Se Young Chun. Training deep learning based denoisers without ground truth data. In Advances in Neural Information Processing Systems, pp. 3257-3267, 2018.
Petre Stoica and Yngve Selen. Model-orderr selection: review of information criterion rules. IEEE Signal Processing Magazine, (4) 2004.
Ryan Tibshirani and Saharon Rosset. Excess optimism: How biased is the apparent error of an estimator tuned by SURE? Journal of the American Statistical Association, pp. 1-16, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing gredient for fast stylization. arXiv preprint arXiv: 1o0vv:160..08022 20166
Jong Chul Ye and Woon Kyoung Sung. Understanding geometry of encoder-decoder CNNs. Pro- ceedings of the 2019 International Conference on International Conference on Machine Learning (ICML). also available arXiv preprint
19110006777, 2019.
Jong Chul Ye, Yoseob Han, and Eunju Cha. Deep convolutional framelets: A general deep learning framework for inverse problems. SIAM Journal Imaging Sciences, 11(2):991-1048, 2018.
A APPENDIX: SUPER-RESOLUTE APPLICATION
Single image super-resolution (SR) is task to estimate high-resolution (HR) image from its low- resolution (LR) image. To demonstrate effectiveness of proposed method, proposed bagging scheme was also applied to these problems. Deep Back-Projection Network (DBPN) (Haris) ct al. 2018) is a recent statc-of-the-es method for this applications, so we employed DBPN as bascline model improve its performance by the proposed method. The bascline network trained using DIV2K (Agustsson & Timofte 2017) with totally 800 training images on the x2 and x4 (in both horizontal and vertical directions) super-resolution tasks. The network consists of three modules: initial feature extraction, back- pock-projection stages, and reconstruction. The initial LR feature maps were extracted using the convolution layer of the initial feature extraction module. By applying the sequence of projection units which alters between configuration of LR and HR feature maps, the extensive HR features maps can be obtained. The final HR image was reconstructed using the overall HR feature maps from the projection units.
Ground Truth
Bicubic
DBPN
Proposed
Figure 6: Comparisons between DBPN and the proposed method for x4 super-resolution task.
Training conditions of the baseline algorithm and our method were followed as described in (Haris et al. 2018). Specifically, we used patch processing whose size is 32x32 for LR image. The initial learning rate was set to 10 14-4, and was divided by 10 at every x 10" iterations for total 10 iterations. For the training of our bagging network, the number of random subsampling mask K was set to 32 and for x2 and x4 task, respectively. The subsampling ratio for bootstrap sampling was set to 80%. In addition, the entire networks were trained simultaneously minimize the loss [3)
10

---

Under review as a conference paper at ICLR 2020
Thanks to the bootstrapping and aggregation using attention network, the data distribution can be fully exploited to restore the high resolution components, which results in the properly reconstructed details of the image shown in Fig. 6 As described in Table 4 our method can also improve the quantitative performance of the super-resolution task.
Table 4: Comparison of PSNR and SSIM index for super-resolution task
seil4
bst100
seil4
hsd100
Algoritlu
Scale
PSNR
SSIM
PSNR
SSIM
Algorithu
Scale
PSNR
SSIM
PSNR
SSIM
DBUN
iaris
0,950
Haris 2018
Proposea
30.802
0.939
31.720
0,951
Propores
26.222
0.837
26.855
0.855


Reviews:
[{"review_id": "SyegTqoCFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a piecewise linear close form expression for the Stein’s unbiased risk estimator and use this formulation to construct a new Encoder-decoder convolutional neural network. The author claimed that this closely related to bagging. Improved experimental results on two inverse problems are presented. Overall, the experiment results are encouraging but the paper need clarification on a few points.\n\n1. In the model description part, the intuition behind the attention modules is never mentioned. It will be nice to explain the intuition and possibly attached the derivation of the loss function the attention modules. \n\n2. the author seems misunderstand the difference between boosting and bagging. The way described in the paper is bagging and in order to do boosting, a sequential type of network structure probably need to be proposed.\n\n3. How will be model performance compared with a simple bagging for the baseline compared in the experiment part?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJejsuDCtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors consider an encoder decoder setup for linear deblurring problem and propose efficient boosting estimators. Specifically, they use the Stein's unbiased risk estimator for the problem when the noise is gaussian. In the case when the encoder and decoder is represented by a convolutional neural network with RELU activations, they show how they can exploit the recent theoretical results that show the kernel type results to make their procedure efficient. They then propose using a set of models (boosting) and prove that the boosted loss function lower bounds the \"nonboosted\" loss function.\n\n1. I think Proposition 1 has minor errors, there is no need to apply Jensen's inequality since there's nothing random, but I think the claim is correct -- it is trivial. In experiments, they use attention network which is not a CNN, so I'm not sure how any of the theory applies to this case, can you please clarify?\n\n2. Experimental focus of the paper is to analyze biomedical datasets -- HCP, EDX and the authors compared their method to *only* one baseline. I suggest that they perform some more comparisons on natural images like http://vllab.ucmerced.edu/wlai24/cvpr16_deblur_study/", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkg9o73nKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\nThe authors address the problem of efficiently employing the SURE estimator as a network training regularizer. They show that for CNN autoencoders this can be efficiently computed. Their other contribution is a bagging/boosting technique which is proved to avoid trivial solutions. The proposed architecture, motivated by the theoretical statements, is shown to outperform classic and 2019 state of the art image reconstruction algorithms in MRI and EDX.\n2. Decision and arguments\nUnfortunately this paper is outside my expertise so I can’t evaluate the novelty of the theoretical accomplishments. However taking that as a given, they well-motive the proposed architecture and achieve impressive experimental results. The experiments are well described.\n3. Questions \na) Why do Table 1 and Figure 3 provide different PSNR and SSIM values?\nb) Is there any way to measure accuracy to ground-truth with the EDX data? Or are the results just qualitative? \nc) With respect to Figure 2, and in general for autoencoders, the input and output have the same dimension. So how do you reconcile this with undersampled MRI and EDX data? I understand you train on fully sampled data—then how do you input undersampled data? Are the unknown samples set to zero?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a closed-form expression for the Stein’s unbiased estimator for the prediction error, and a boosting approach based on this, with empirical evaluation. While this paper is interesting, all reviewers seem to agree that more work is required before this paper can be published at ICLR.  
Decision: Reject