Self-supervised regression learning using domain knowledge: Applications to improving self-supervised image denoising
Authors: Il Yong Chun
Dongwon Park
Xuehang Zheng
Se Young Chun
Yong Long
Venue: ICLR 2022 Submitted (2022)
Keywords: Self-supervised learning
Regression
Image denoising
Deep learning

Abstract: Regression that predicts continuous quantity is a central part of applications using computational imaging and computer vision technologies. Yet, studying and understanding self-supervised learning for regression tasks -- except for a particular regression task, image denoising -- have lagged behind. This paper proposes a general self-supervised regression learning (SSRL) framework that enables learning regression neural networks with only input data (but without ground-truth target data), by using a designable operator that encapsulates domain knowledge of a specific application. The paper underlines the importance of domain knowledge by showing that under some mild conditions, the better designable operator is used, the proposed SSRL loss becomes closer to ordinary supervised learning loss. Numerical experiments for natural image denoising and low-dose computational tomography denoising demonstrate that proposed SSRL significantly improves the denoising quality over several existing self-supervised denoising methods.

Under review as a conference paper at ICLR 2022
SELF-SUPERVISED IN LEARNING USING DO- MAIN KNOWLEDGE: APPLICATIONS TO IMPROVING SELF-SUPERVISED IMAGE DENOISING
Anonymous authors Paper under double-bling review
ABSTRACT
Regression that predicts continuous quantity is central part of applications using computational imaging and computer vision technologies. Yet, studying and un- derstanding self-supervised learning for regression tasks except for a particular regression task, image denoising have lagged behind. This paper proposes gen- eral self- self-suuer regression learning (SSRL) framework that enables learning regression neural networks with only input data (but without ground-truth target data), by using designable operator that encapsulates domain knowledge of a specific application. The paper underlines the importance of domain knowledge by showing that under some mild conditions, the better designable operator is used, the proposed SSRL loss becomes closer to ordinary supervised learning loss. Numerical experiments for camera image denoising and low-dose computational tomography denoising demonstrate that proposed SSRL significantly improves the denoising quality over several existing self-supervised denoising methods.
INTRODUCTION
Deep regression neural network (NN)-based methods that can accurately predict real- or complex-
valued output have been rapidly gaining popularity in a wide range computational imaging and
computer vision applications including image denoising (Vincent et al. 2010 Xie et al. 2012 Zhang
jet al. 2017), image deblurring (Xu et al.] (014)). image super-resolution (Dong et al. [016 Kim
al. 2016). light-field reconstruction (Chun al. 2020 Huang et al. 2020), object localization (Szegedy et al. 2013), end-to-end autonomous driving (Bojarski et al. 2016). Yet, they lack a general self-supervised learning framework.
In training a regression NN f : N >> TR-M, the most prevalent supervised learning approach minimizes the mean square error (MSE) between what predicts from an input æ R and ground-truth target E MM;
min (f))) v1ž.
Learning a denoising or refining NN uses 1 with M = N dubbed Noise2True where is a corrupted image and y is a clean (i.e., ground-truth) image. However, challenging or even impossible to collect many clean images in many practical applications, motivating research
self-supervised learning for image denoising (Ulyanov et al.] 2018) Sootanayev & Chun 2018 Krull
let al. 2019 Batson & Royer 2019 Laine al.) 2019; Moran al. 2020 Quan et al. 2020 Xu
et al.
2020; Hendriksen et al. 2020 Xie et al. 2020: Huang et al. (021) called self self-supervised
image denoising. To learn denoiser f with single noisy images, popular celf-supervised image
denoising method, Noise2Self (Batson & Royer 2019) (see also the concurrent work (Krull et al.,
2019),, and its sophisticated relaxation, Noise2Same (Xie al. 2020), study the following MSE minimization problem:
min Bxllf(r) - x1
These methods some partitioning schemes in (2) avoid that its optimal solution is just the identity mapping I. Noisc2Noise (Lehtinen et al. 2018) that learns a denoiser with pairs of two
This article has appendix and supplement. The appendix and supplement number sections, figures, and tables with the prefix "A" and "S", respectively.

---

Under review as a conference paper at ICLR 2022
independent noisy images, is pioneer work for self-supervised image denoising. Motivated by Noise2Noise, several self-supervised image denoising methods such as Noise2Inverse (Hendriksen
et al. 2020), Neighbor2Neighbor (Huang et al., 202ID, and (Soltanayev & Chun 2018 Moran al.) 2020 Xu et al. 2020) emulate pairs of two independent noisy images, by applying partitioning or adding simulated noise to single noisy measurements. All the aforementioned methods have been developed based on some noise assumptions including pixel-wise independent noise (Krull et al.)
2019 Xie al. 2020. Huang et al. 2021) and zero-mean noise (Lehtinen al.) 2018 Batson & Royer 2019] Xie et al., 2020). Yet, they lack design flexibility that might relax such noise as- sumptions and further improve the denoising performance of NNs. Some works estimate statistical parameters of noise, such as noise histogram (Krull et al. 2020) and parameters of Gaussian mixture noise model (Prakash et al. 2021).
This paper presents new insights on this topic. The paper proposes general self-supervised learning framework for regression problems, which we refer as self-supervised regression learning (SSRL). Proposed SSRL enables learning regression NNs with only imput samples, by using designable op- crator that can encapsulate domain knowledge of a specific application. Our main results show that under some mild conditions (e.g., in image denoising, statistical noise properties in z), the better desinable operator is used, the proposed SSRL loss becomes closer to ordinary supervised learning loss. In addition, designable operator with good domain knowledge existing self-supervised denoising methods. Numerical experiments for camera image and low-dose computational tomography (CT) denoising with both simulated and real-world datasets corrupted by only single noise realization demonstrate that the proposed SSRL framework significantly improves denoising quality compared several existing self-supervised denoising methods. Put together, our findings provide new insights into how using good domain knowledge can improve self-supervised denoising, underscoring the benefits of understanding application-spee knowl- edge in SSRL. (Section S.I further elaborates the contributions of the paper.)
relax noise assumptions of
SSRL USING DOMAIN KNOWLEDGE
The proposed SSRL loss is given by
(.llf(c)- - g(x)
(3)
where g RN + RM is designable operator encapsulating domain knowledge of specific application. We will incorporate some sophisticated setups in (3) such that f obtained by minimizing ( cannot merely be g. Although related theorems (see later) hold for any M, we mainly focus on practical image denoising applications with pseudo-target g(x) & y.
2.1 MOTIVATION
This section empirically shows that understand- ing domain knowledge is important for design- ing g in the proposed SSRL loss (3) The following camera image denoising examples demonstrate that well-designed g with good do- main knowledge improves the denoising perfor- mance of learned via (3)
PSNR 25.8 dB
PSNR 23.1 dB
Suppose that camera images are corrupted by salt-and-pepper noise. Consider two example setups for g(-). median filtering and BM3D denoiser (Mäkinen et al. 2020), denoted by median(.) and BM3D(.), respectively. Fig- ure [ compares the denoising performance of minimum f with the two aforementioned y se- tups: f* with median filtering significantly im- proved that with BM3D denoiser. This is not surprising, as median filtering is widely known to be effective in reducing halt-and-pepper noise (Bovik) 2010 $3.2). This result emphasizes the impor- tance of understanding domain knowledge of specific applications in proposed SSRL.
Figure 1: Error map comparisons of denoised im- ages from 3 using g(z) = median(z) (left) and g(x)
BM3D(z) (right) (blue and yellow de- note 0 and 50 absolute errors, respectively). Peak signal signal-to-noise to-noise ratio (PSNR) values are averaged.
2

---

Under review as a conference paper at ICLR 2022
2.2 PRELIMINARIES
We first introduce the J-complement between two functions f and g:
Definition 1. For a given partition J = {J .............................. JB) +1 |JBl= N) of the dimensions of input x. E IR functions f R + R& and g IR - RRd are called I -complementary, if Ja) does not depend on g(x3) for all denotes a vector restricted to J.
(J)....,
where J denotes the complement of J, and (-),
That is, and g use information from outside and inside of J predict output and give pseudo- target, respectively. In denoiser learning (where M N), Definition I specializes to the J- invariance of f (Batson & Royer 2019). by setting 9 I. Incorporating Definition [ into the SSRL loss (3 is straightforward approach to avoid that optimal f is just y in (3). The proposed SSRL framework assumes the followings:
Assumption 1) 2 and ro e are conditionally independent given i.e., p(r|y) = p(z>lu)p(z>-lv)
y,- Assumption 3) f and g are -Beasullme
Assumption 2) 18 [lg(()))))
In denoiser learning, Assumption holds if noise in each subset E I is conditionally independent from that in Js, given y. Assumption 1 can be satisfied in general regression NN learning, by adding small randomized perturbations (independent of y) to either J or J similar to Moran et al. (2020). Assumption 2 suggests direction for designing gi suppose that x has non-zero-mean noise; one then can design g to make noise zero-mean using the domain knowledge. Assumption 3 is satisfied if and g are continuous. This condition is mild because many regression NNs f are continuous where their modules, convolution, matrix-vector multiplication, rectified linear unit activation, max pooling, etc. continuous and one can design with measurable continuous function.
Finally, observe that the proposed SSRL loss can be rewritten by
[x]l((r) - | E (fx)) vll + |o(()) v11ž 2(f(x) - y, (()) y)-
(4)
We aim to either remove or control the third term, incorporating the J J-complement and/or Assump- tions 1-3 introduced above.
2.3 SSRL USING DOMAIN KNOWLEDGE WITH J -COMPLEMENT
This section studies SSRL loss (3) minimization over f that is J- J-complementary of g. Our first
main result shows that under Assumptions 1-3, the SSRL loss (3) with the J J-complement is the
sum of the ordinary supervised learning loss and variance of g(x) vanishes.
i.e., the third term in (4)
Theorem 2. Under Assumptions 1-3, the SSRI loss becomes
3 with the I -complement in Definition
o EE llf(x)) -1122 (lg(()) v11..
(5)
The following equality similarly holds for any K E K.: (x)))(() - g(x) = (f)))
yK 113 [[((z)) - yk 112, where K.
is a partition of (1,..., MM. and f(-) and g(-) denote f())
and 9() restricted to K, respectively. The optimal solution for is given by
( E(g(())) Eyy
Proof. See Section A.I in the appendix.
The result (6) suggests another direction for designing g: we aim to design good that can make g(æy) close to y. Using such g. optimal solution of LHS in (6). [z)())) becomes close to its supervision counterpart, [F(y] xr v.l. Consequently, such reduces the second term Br.yl[ v11 in RHS of (5). leading 3 closer to D.. If ideal such that ( = y, the SSRL loss (3) becomes the usual supervised learning loss. In designing g, domain knowledge of specific application is crucial.
Domain knowledge includes noise properties in x and pre-trained NN by existing self-supervised denoising, such Noise2Self (Batson & Royer 2019) and Noise2Noise (Lehtinen et al. 2018).
Specifically, the proposed SSRL loss using the I J-complement is given by
Lina (f) A Σ
Ex Exll(( g(x)))l
JEJ
3

---

Under review as a conference paper at ICLR 2022
Je
Neural network f
Predicted output f(zre
Lnd f
Noisy imput x:
Pre-designed with domain knowledge
Pseudo-targe target g(x.)
Mask on J
Neural network f
Predicted output f(xxe)
(
Noisy input æ:
Frredesigned with domain knowledge
ZJ
Pseudo target g(æs),
Figure 2: Proposed SSRL models using the J-complement in denoiser learning. Top: f and use almost equal amount of information from input, i.e., |J] [J%] where J and J are comple- mentary checkerboard masks. Bottom: f and use unbalanced amount of information from input, specifically, [J]] >> [J].
Figure 2 illlusrrates with complementary checkerboard masks J and J, where f and use almost equal amount of information, and its variant, where g uses much less information than f. The variant computes MSE only on J J: in this setup, it is challenging for g to predict the entire image.
Relation to previous self- denoising works. In denoiser learning, the proposed SSRL. loss (3) with the J J-complement of and g specializes to ( with the J-invariance of f. i.e, Noise2Self. Noise2lnverse (Hendriksen et al. (020) and Neighbor2Neighbor (Huang et al.) 2021 that emulate pairs of two independent noisy images by partitioning single measurements (e.g., CT ray measurements with independent noise and corrupted images with pixel-wise inde- pendent noise) can be viewed as Noise2Self. Thus, SSRL loss (3) with the J-complement of f and ceeeeeooiii-mmiivvaieee
g I specializes to the aforementioned
self-super denoising methods. (Noise2Noise also can be viewed by SSRL (3) with the setup above, by constructing with stacking two independent noisy images, where image is corrupted by independent noise realizations.)
2.4 SSRL USING DOMAIN KNOWLEDGE WITHOUT J T-COPPLEMENT
This section studies the SSRL loss serve by (4) that Ee.w]]f(2> v13
without using J-complement in Definition T Ob- (9(x,)) y/ Exllf(x) 9(x))113 + (f y. gx)) y). Inspired by Noise2Same (Xie et al. 2020). the second proposed SSRL approach approximation of the right hand side in this equation that does that f and are J-complementary. Our second main result finds an upper bound for the term En (f(x)) - y, g(x)) y) without relying on the J-complement (remind that this term vanishes If and are J J-complementary; see Theorem
to minimize
assume
Theorem 3. Assume that Var(g(a.)).. o?, Vm. Under Assumptions 1-3, the following bound holds:
Frr.y((()) (.g())) y) < SOMM (Ez]I(<)) {(x))-)))) 1/2
(8)

---

Under review as a conference paper at ICLR 2022
The following bound similarty holds for any K K: Bz,y (f.g((()) yk g(x))x - -9x) S ov /Kl
(B=l)((z) (((x==)))]l 1/2 where K and K. are defined in Theorem 2
Proof. See Section A.2 in the appendix.
Using Theorem 3 the proposed SSRL loss that does not rely on the J -complement is given by
C(f) Σ -
20y M. (E,llf(z) (z)))) 1/2
JEJ
where is given in Theorem 3 Here, a regression NN can information from the entire
input v; whereas (/) in uses only partial input ze in f. The intuition for designing g in Section 2.3 similarly applies here Our aim to design good g such that g(x.) is closer y, consequenty leading (9) close to ① Similar to the variant of Cina(f) (see Section 2.3). if the
amount of information between two partitions and J is unbalanced in denoiser learning, one can modify (9) to compute the MSE only on J or J in either both terms or the right term in (9) (These
variants use the second result in Theorem
3)
We conjecture that how well designed g is, i.e., how close is pseudo-target g(x)) to ground-truth y, captured by a defined in Theorem 3 We support the conjecture with examples Section |A.3 of the appendix. Then, this "goodness" of g balances the two terms in (9) via σ. If is well-designed such that (xx)) ) is close to y. i.c., o² is small, then the SSRL loss (9) relies more on the first term with good pseudo-target. If is poorly-designed such that o² is large, then (9) puts more weight more on the second term that can implicitly promote the T J-invariance of f.
Relation to previous self-supervised denoising work. The proposed SSRL loss (9) becomes the Noise2Same loss (Xie et al., 2020 Thm. 2), by replacing g(zy) with x and adding randomness to K J in the second term. In practice, can tune in (9) without knowing its exact value, similar to Noise2Same. One might use the aforementioned conjectured behavior of ( with expected performance of g.
2.5 RELAXING NOISE ASSUMPTIONS OF EXISTING SELF-SUPERT SUPERVUDERVIEED DENOISING METHODS
This section explains how proposed SSRL can relax noise assumptions of existing self- supervised denoising methods. Noise assumptions of existing self-supervised denoising methods include additive white Gaussian noise (AWGN with known variance) (Soltanayev & Chun 2018),
pixel-wise independent noise (Krull et al. 2019 Xie al. 2020 Huang et al. (001]). and zero-mean noise (Lehtinen et al. 22118] Quan et al. 2020) or more generally E Jz
(Batson & Royer 2019
Xie et 2020). Assumption I of proposed SSRL relaxes the AWGN and pixel-wise independent noise assumptions, and is identical to the first assumption of 2olse (Batson & Royer 2019) Assumption proposed SSRL can relax the second assumption of Noise2Self, E a y y, that is also (implicitly) used in Noise2Same (Xie et al. 2020) and Noise2Noise (Lehtinen al. 2018 8), by using a desinable function g. For example, x is corrupted by additive non-zero-men noise e that is independent of y, i.e., x y y then one can design g as follows: be estimated from calibration of imaging systems. The next section will explain how one can design g using domain knowledge and select g if domain knowledge is unavailable in image denoising.
g(x) x E(e], where E[e] can
3 EXAMPLES OF HOW TO DESIGN g USING DOMAIN KNOWLEDGE, AND EMPIRICAL-LOSS APPROACH FOR SELECTING 9 IN IMAGE DENOISING
Understanding noise statisties or properties the first step towards accurate image recovery in com- putational imaging. First, this section describes how to use domain knowledge for designing g in two imaging applications with practical noise models: /) camera image denoising in mixed Poisson- Aeassian-Berroulle noise, 2) low-dose CT denoising. Both applications have complicated noise models or strong noise, where Assumptions 1 1-2 in Section 2.2 are not completely satisfied. Noisy images in the first application are corrupted by independent and identically distributed (i.i.d.) noise with zoro mean. The noise in the second application is approximately zero-mean but is likely non-i.i.d. In designing for each application, we will use the suggestions in Sections 2.2 23 in- vestigating Assumptions 1 1-3. Second, we propose a g-selection approach in image denoising that calculates some empirical measure related to (6) using only input training data.
5

---

Under review as a conference paper at ICLR 2022
3.1 DESIGNING 9 USING DOMAIN KNOWLEDGE: CAMERA IMAGE DENOISING
The major noise sources in camera imaging (using charge coupled device) include object-dependent photoelectrons in image sensors, readout in camera electronics, and analog-to-digital converter and transmission errors that can be modeled by Poisson noise, AWGN, and Bernoulli (i.e., salt-and- pepper) noise models (Snyder et al., [193]. (Bovik) 2010, p. 90). We use the following practical mixed
noise model (Snyder et al. 1993 Batson & Royer 2019):
Ft II [0.235] [Berrnullli
+ Gn)), <~ (~N(0,,22))
1, N, (10)
where [0.255] performs 8-bit quantization and clips pixel values outside of 255], Bernoulli sub- stitutes a pixel value with either or 255 with probability p (0 and 255 are coined with equal prob- ability), Poisson generates pixel intensity-depender
Poisson noise with gain parameter X, and is AWGN. ( shows a noisy image corrupted by the mixed noise model (TO). an image corrupted only by the mixed
Gausssan noise, Ex[ y in Assumption 2 can be satisfied (o(soossn))Ay))) / y = you Vn). However, if Bernoulli noise is additionally considered as given in 10 the assumption E[ [ y will not be satisfied (E [Bernoulli, (3m))l)
p)y, + 127.5p, vn). The quantization operator [00255)] also makes it hard to satisfy E[ = y.
Following the suggestion based Assump- tion 2 (see Section (.2). we handcraft to "approximately" satisfy E [x](())l)) = y with a simple operator. We interpret aforemen- tioned Bernoulli noise and clipping artifact as salt-and-pepper noise. Median filtering is computational efficient method that is effec- tive in reducing celt-andd-eppeer and impulse noises (Bovik) 2010, $3.2). We design g by ap- plying weighted median filtering (Brownrigg) 1984) to a pixel with intensity either 0 or 255 at each color channel, aiming that this g design "approximately" satisfy [[g(x))y] = y by sup- pressing the salt-and-pepper noise effects cased by II 2.255 and Bernoulli,- (This is supported
PSNR 9.8
RMSE = 48.3 HU
Figure An input noisy image in camera im- age denoising in mixed noise (left) and low-dose CT (right). PSNR and root mean square error (RMSE) values were averaged across all test sam- ples.
by empirical results in Section SS.4)
Assumption is satisfied because the noise in (0) is i.i.d. and independent of In Assumption 3, conjecture that the above g design is measurable (median operator is measurable under some conditions (Rustad] ).04))
3.2 DESIGNING USING DOMAIN KNOWLEDGE: LOW-DOSE CT DENOISING
In X-ray CT (with monoenergetic source), the pre-log measurement data usually modeled by the Poisson model, i.e., Poisson (Po exp( exp(-[Ay):) photons per ray, A E LRN a CT projection system matrix, and I. the number of measured rays. Using the quadratic approximation to the log-likelihood of Poisson model, the post-log measurement z < R²- given y can be approximated as the following Gaussian model (Sauer & Bouman 1993 Fessler 2000 : zly [Ay, C) where C E R L is a diagonal covariance matrix and its diagonal elements become more nonuniform in lower-dose CT. This model suggests that post-log measurement may be modeled by
..... J., where Po is the number of incident
Ay + S. where e ,N( C). The filtered back-projection (FBP) method (Kak & Slaney 1988 $3) performs computationally efficient CT reconstruction and has been widely used in commercial CT scanners (Pan et al.) 2009). In low-dose
CT, however. reconstructed image = Fz suffers from strong noise and streak artifacts. where F N denotes linear FBP operator. motivating research on learning denoising NNs. Figure 3 (right) shows a noisy FBP image in low-dose CT. Using the statistical results above, we model that a reconstructed image by F is corrupted by an arbitrary additive noise e:
I = + e, =(FA- - D) + Fe.
(11)
Low-dose CT uses all projection rays similar to standard-doss CT (but with substantially reduced dose) where F approximately inverts A, i..., FA z I, so we conclude that under (II). E(e] z 0 and B(x[y] v y. (See empirical results in Section S.4 that support E/e] x 0.)

---

Under review as a conference paper at ICLR 2022
The above domain knowledge in low-dose CT indicates that handcrafting 9 to have zero-me cero-mean to satisfy Assumption can be redundant. Following the suggestion motivated by Theorem (see Section 2.3). we set g pre-trained denoiser by the existing self-supervised denoising methods (Batson& & Royer 2019; Hendriksen et al., 2020; Xie et al. 2020). Since such pre-trained will have some denoising capability and give better reference than g = I to 77 and (9) (see empirical results
in Section (S.)). we expect that proposed SSRL losses 7 and ( improve the denoising quality over the aforementioned existing self-supervised denoising methods.
Assumption is unlikely satisfied because in FBP images, neighboring noise components are likely to be correlated, i.e., Var(e) R FCFT using FA- = I and noise model (11). Assumption 3 is satisfied as we use the conventional denoisiong NN, DnCNN (Zhang et al. 2017) and (modified) U-Net (Ronneberger et al. 2015), that are a continuous function.
3.3 EMPIRICAL.-LOSS APPROACH FOR SELECTING 9 IF DOMAIN KNOWLEDGE UNAVAILABLE
If accurate domain knowledge of a specific application is unavailable, it would be challenging to explicitly design g. In such cases in denoising, our general suggestion is to measure existing self- supervised denoising loss, an upper bound of [1())))))) -yl 2 or its variant that measure the quality of g,
only with input training data. The lower quantity implies that g is better and implicitly encapsulates better domain knowledge. In camera image denoising with the real-world dataset (Abdelhamed et al. 2018), the empirical measure of the Neighbor2Neighhor loss (Huang al.] 2021) BBlg (z L with setting ! as I and median filtering are 0.0052 and 0.0048, respectively. In low-dose CT denoising with the real-world dataset (Moen et al. 202T), the empirical measure of the Noise2Self loss (Batson & Royer 2019) B [z - Je 2 with setting as I and pre-trained DnCNN by Noise2Self are 22044.5 and 1 17062.0 (in HU2 where HU stands for modified Hounsfield unit), respectively. We expect better SSRL performance with the selected g designs over I.
4 EXPERIMENTAL RESULTS AND DISCUSSION
We evaluated proposed SSRL in two practical imaging applications in Section with both simu- lated and real-world datasets. For these applications, we mainly focuses on comparisons with self- supervised denoising methods using single noisy input samples, particularly when statistical noise parameters are unavailable. We compared the performances of the following methods: Noise2Self (Batson & Royer 2019) Noise2Noise- motivvate methods that emulate pairs of two independent noisy images- Neighbor2Neighbon (Huang et al. (02I] or Noise2Inverse (Hendriksen et al.) (020) -Noise2Same (Xie et al. 2020), and corresponding SSRL to each aforementioned method. We also included Noise2True ① results as baseline. For f or g in all the methods, we used the conventional denoising NN architecture, DnCNN (Zhang et al. (0017) or modified U-Net used in Noise2Self. We include experiment setup, and image and numerical results for/from real-world datasets in Sec. A.4
4.1 EXPERIMENTAL SETUP FOR SIMULATED DATASETS
Camera image denoising in mixed noise. We evaluated the proposed SSRL framework with three RGB camera image datasets, ImageNet ILSVRC 2012 Val (Russakovsky et al. 2015). BSD 300 (Martin et al., 2001). Set 5 (Bevilacqua et al. 2012). For training, we used the ImageNet ILSVRC 2012 Val dataset with 20,000 images; for tests, we used the BSD 300 and Set 5 datasets consisting of 300 and 5 images, respectively. We simulated noisy images with the following imaging parameters introduced in [T): X = 30, a. 60, and 0.2. We evaluated the denoising quality by the most conventional error metric in camera image denoising, PSNR and structural similarity index measure (SSIM). In Neighbor2Neighh
setup, we emulated two independent noisy images from single noisy images by random neighbor sub-sampling with 5x2-windows (Huang et al. 2021
Low-dose CT denoising. We evaluated the proposed SSRL framework with The 2016 Low Dose CT Grand Challenge data (McCollough) 2016). We selected 200 regular-dose chest images of size N = 512 x 512 and the 3 mm slice thickness from four patients. For training, we used 170 (85% chest images from three patients; for tests, used 30 (15%) chest images from the other patient. We simulated low-dose sinograms using the Poisson model with the selected regular- chest datasets. In particular, we simulated sinograms of size I 736 x 1 1152 ('detectors" x 'projection views"),

---

Under review as a conference paper at ICLR 2022
Propased SSRL in
Reference
Noise2True
Noise2Self'
Noise setup
PSNR 25.2 cBB SSIM 0,.19
PSNR =2 20.6 dB
PSNR 22.1 dB
SSIM =
SSIM 0.748
Proposed SSRL in
Proposed SSRL in
Neighbor2Neighhor
Neighbor Neighbor setup
Noise2Same
Noise setup
PSNR 20.2 dB
PSNR 22..
PSNR 20.1
PSNR 21.2
SSIM = -..703
SSIM = =0.56
SSIM =0.690
SSIM = =0.705
Figure 4: Comparisons of denoised images (left) via DrCNNs from different learning methods and their saturation maps (right) in camera image denoising (blue and yellow denote 0 and 0.5 absolute error, respectively). PSNR & SSIM values were averaged across all BSD 300 test samples.
no-scatter monoenergetic source with Po 5 10a. We used FBP (Kak & Slaney) 1988, $3) to reconstruct images with resolution 0.69 mm x 0.69 mm. We evaluated the denoising quality by the most conventional error metric in CT application, RMSE in HU. In Noise2Inverse setup, we emulated two independent noisy images by partitioning single sinograms with odd and even views and applying FBP two partitioned independent sinograms.
with fan-beam geometry corresponding
4.2 COMPARISONS BETWEEN DIFFERENT SELF-SUPERVISED DENOISING METHODS
Compare each existing self supevisee denoising method to its corresponding SSRL setup in Fig- ures 45 and Figures S.4 and Tables S.3. see three comparison sets, each grouped by red box. For both applications, proposed SSRL achieves significantly better image denoising qual- ity, i.e., closer to the Noise2True quality, compared to the existing methods, Noise2Self, Neigh- bor2Neighbor, Noise2Inverse, and Noise2Same, regardless of the regression NN architecture. We
show DrCNN prediction uncertainty of all the methods in Figure SS.
Camera image denoising in mixed noise (simulated data). Figures 4 and S.3 show that in all the three comparison sets, SSRL gives closer image quality, particularly color saturation, to Noise2True than existing methods, Noise2Self, Neighbor2Neighboo, and Noise2Same. Setting g as noise. Yet, compared Noise2True, denoised images obtained by proposed SSRL lack saturation and detail preservation.
weighted median filtering avoids bias in SSRL loss caused by salt-and-
For saturation and detail preservation comparisons, see Figures S.T and S.3
Comparing the three comparison sets Figures and S33 and Tables S. S.2 shows that all the Noise2Self, Neighbor2Neighbor, and Noise2Self setups have comparable results in terms of PSNR values. The potential reason that in this application, all the three setups similarly satisfy Assump- tions 1-3 (see Section 2.2 in particular, Assumption is well-satisfied by pixel-wise i.i.d. noise.
In this application, the zero-mean noise assumption of the existing self-supervised denoising meth- ods is violated, whereas its counterpart in proposed SSRL, Assumption 2, is "approximately" satis- fied by g in Section 3.1 This suggests the importance of satisfying Assumption with good g.
Low-dose CT denoising (simulated data). In all the three comparison sets, SSRL better recovers low-contrast regions (e.g., soft tissues) and small details, and significantly reduces noise and artifacts throughout the image, existing methods, Noise2Self, Noise2Inverse, and Noise2Same. See
8

---

Under review as a conference paper at ICLR 2022
Propased SSRL in Noise2Sell setup
Reference
Noise2True
Noise2Self'
RMSE 16.3 HU
RMSE 36.2 HU
RMSE 25.0 HU
Proposed SSRL in Noise2Inverse setup
Proposed SSRL in Noise2Same setup
Noise2Inverse
Noise2Same
RMSE 22.9 HU
RMSE 21.9 HU
RMSE 28.1 HU
RMSE = 26.0 HU
Figure 5: Comparisons of denoised images via DnCNNs from different learning methods in low- dose CT (display window is [800, 1200) HU). RMSE values averaged across all test samples.
zoom-ins and circled small details in Figures 3 and S.4 and eπrоr images in Figure S.2 particularly in 'Noise2Self vs. Propose SSRL Noise2Self setup' and 'Noise2Same vs. Proposed SSRL in Noise2Same setup.' The results might imply that simply setting g as pre-trained NN by existing self- supervised denoising methods works like a charm in SSRL. Proposed SSRL in the Noise2Inverse setup can provide images with image quality that is comparable to conventional FBP at 10 times higher dose (when Pa = 5 x 10 RMSE 20.5 HU on average; see DrCNN results in Figure 5).
Next, comparing Nois and Noise2Same result sets to that of Noise2Inverse in Figures S.2 and (S.4] and Table S.3 shows that Noise2lnverse setup significantly improves the denoising qual- ity, compared to Noise2Self and Noise2Same setups. We conjecture that violation of Assump- tion 1 1- that is satisfied in the Noise2lnverse setup but unlikely to be satisfied in the Noise2Self and Noise2Same setups in this application degrades the performance.
Camera image denoising and low-dose CT denoising with real-world datasets. Denoised im- age results in Figures A. (A.3 and S.8 from the two real-world datasets (Abdelhamed et al.) 2018 Moen et al. 2021) demonstrate that SSRL improves existing self-supervised denoising meth- ods, particularly Neighbor2Neighhor, 2oise and Noise2Same, without having their exact noise properties. The results well correspond to our expectation in Section 3.3
5 CONCLUSION
It is important to develop SSRL that enables comparable prediction performances to supervised learning, because it is extremely challenging to collect many ground-truth target samples in many practical computational imaging and computer vision applications. The proposed SSRL framework bridges the gap between SSRL and supervised regression learning via domain knowledge of ap- plications. To achieve closer prediction performance to supervised learning, SSRL uses domain knowledge to design a better
g such that g(x.)) becomes closer to y. For camera image denoising and low-dose CT denoising with both simulated and real-world datasets, SSRL achieves more accurate prediction compared to the existing self-supervised denoising methods (Bat-) son & Royer 2019 Huang et al. 2021 1 Hendriksen et al.) 2020 Xie et al.) 2020). Remark, however, that applying SSRL to other regressions problems may need careful investigations about based on their domain knowledge. Our future work is extending proposed SSRL to other machine learning problems such as teacher-student models (see Section S.7)) and meta- leaanin. On the application side, our future work is applying SSRL to regression problems beyond image denoising.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
Section 2.2 specifies all the theoretical assumptions. Section A. and Section A.2 in the appendix in- clude the complete proofs of Theorem 2 and Theorem respectively. Section S in the supplement includes the complete implementation details including the hyperparameter selection strategies and chosen hyperparameters.
included an anonymized zip file that includes test codes, test data, trained models, and instructions and codes that provide complete description of the data processing steps, as supplementary materials. We will make our codes (for data construction, training, and test) and trained models publicly available on GitHub if the paper is accepted.
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown. high-quality denoising dataset for smartphone cameras. In Proc. IEEE CVPR, pp. 1692-1700, Salt Lake City, Utah, Jun. 2018.
Joshua Batson and Loic Royer. Noise2Self: Blind denoising by self-supervision. Proc. ICML, pp. 524-533, Long Beach, CA, Jun. 2019.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Maric Line Alberi-Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. In Proc. BMVC, pp. 135.1-135.10, Surrey, UK, Sep. 2012.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. NIPS Deep Learning Symposium, Dec. 2016. arXiv:1604.07316
Alan Bovik. Handbook of image and video processing. Academic Press, Florida, 2010.
David RK Brownrigg. The weighted median filter. Commun. of the ACM, 27:807-818 1984.
Yong Chun, Zhengyu Huang, Hongki Lim, and Jeffrey A Fessler. Momentum-Net. Fast and convergent iterative neural network for inverse problems. early access in IEEE Trans. Pattern Anal. Mach. Intell., 2020. doi: 10.1 T119///AAM.202003122200
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell., 38(2):295-307, Feb. 2016.
Jeffrey A Fessler. Statistical image reconstruction methods for transmission tomography. In M Sonka and J Michael Fitzpatrick (eds.). Handbook of Medical Imaging, Volume 2. Medical Image Processing and Analysis. pp. 1-70. SPIE, Bellingham, WA, 2000.
Allard Adriaan Hendriksen, Daniël Maria Pelt, and K Joost Batenburg, Noise2Inverse: Self- supervised deep convolutional denoising for tomography. IEEE Trans. Comput. Imag., 6:1320- 1335, Aug. 2020.
Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2Neighbor: Self- supervised denoising from single noisy images. In Proc. IEEE/CVF CVPR, pp. 14781-14790. Virtual, Jun. 2021.
Zhengyu Huang, Jeffrey A Fessler, Theodore B Norris, and II Yong Chun. Light-field reconstruction and depth estimation from focal stack images using convolutional neural networks. In Proc. IEEE Barcelona, Spain, May 2020.
ICASSP, pp.
A. C. Kak and M. Slaney. Principles of computerized tomographic imaging. IEEE Press, New York, NY, 1988.
Jiwon Kim, Kwon Jung Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks, In Proc. IEEE CVPR, Las Vegas, NV, Jun. 2016.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2 Void Learning denoising from single noisy images. In Proc. IEEF/CVF CVPR, pp. 2129-2137, Long Beach, CA, Jun. 2019.
10

---

Under review as a conference paper at ICLR 2022
Alexander Krull, Tomás Vičar, Mangal Prakash, Manan Lalit, and Florian Jug. Probabilistic noise2void: Unsupervised content-aware denoising. Frontiers in Computer Science, 2:5, 2020.
Samuli Laine, Jaakko Lehtinen, and Timo Aila. High-quality self- self-suuervised deep image denoising. In Proc. NIPS, Vancouver, Canada, Dec. 2019.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miik Aittala, and Timo Aila. Noise2Noise: Learning image restoration without clean data. In Proc. ICML, pp. 2974,,974, Stockholm, Sweden, Jul. 2018.
Ymir Mäkinen, Lucio Azzari, and Alessandro Foi. Collaborative filtering of correlated noise: Ex- act transform-domain variance for improved shrinkage and patch matching. IEEE Trans. Image Process., 29:8339-83355 8354, Aug. 2020.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application evaluating segmentation algorithms and measuring ecological statistics. In Proc. IEEE ICCV, pp. 416-423, Vancouver, Canada, July 2001.
C. McCollough. TU-FG-207A-04: Overview of the low dose CT grand challenge. Med. Phys., 43 (6Pat35):3755--7666 3760, Jun. 2016.
Taylor R Moen, Baiyu Chen, David R Holmes III, Xinhui Duan, Zhicong Yu, Lifeng Yu, Shuai Leng. Joel G Fletcher, and Cynthia H McCollough. Low-dose CT image and projection dataset. Med. Phys., 48(2):902-911, Feb. 2021.
Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. Noisier2Noise: Learning denoise from unpaired noisy data. In Proc. IEEE/CVF CVPR, pp. 12072, Virtual, Jun. 2020.
Xiaochuan Pan, Emil Y Sidky, and Michael Vannier. Why do commercial CT scanners still employ traditional, filtered back-projection for image reconstruction? Inverse Probl., 25(12): 2009.
Mangal Prakash, Alexander Krull, and Florian Jug. Fully unsupervised diversity denoising with convolutional variational autoencoders. In Proc. ICLR, Virtual, May. 2021.
Yuhui Quan, Minggin Chen, Tongyao Pang. and Hui Ji. Self2Self with dropout: Learning self- supervised denoising from single image. In Proc. IEEE/CVF CVPR, pp. 1898,1898, Virtual, Jun. 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. Proc. Med. Image Computing and Computer Assist. Interven., pp. 234-241, Munich, Germany, Oct. 2015.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. Int. Comput. Vision, 115(3):211-555 2015.
Alf B Rustad. The median in multidimensional spaces. Adv. App. Math., 33(2):366-396, Mar. 2004.
Ken Sauer and Charles Bouman. A local update strategy for iterative reconstruction from projec- tions. IEEE Trans. Signal Process., 41(2):334-544 -488 Feb. 1993.
Donald L. Snyder, Abed M. Hammoud, and Richard L. White. Image recovery from data acquired with a charge-coupled-devic. camera. Opt. Soc. Am. A, 1065):1014-0023, May 1993.
Shakarim Soltanayev and Se Young Chun. Training deep learning based denoisers without ground truth data. In Proc. NIPS, volume 31, Montreal, Canada, Dec. 2018.
Christian Szegedy, Alexander Toshev, and Dumitru Erhan. Deep neural networks for object detec- tion. 2, Dec. 2013.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proc. IEEE/CVF CVPR, pp. 9446 9454, Salt Lake City, Utah, June 2018.

---

Under review as a conference paper at ICLR 2022
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. J. Mach. Learn. Res., 11(12):337-33408 Dec. 2010.
Junyuan Xie, Linli Xu, and Enhong Chen. Image denoising and inpainting with deep neural net- works. In Proc. NIPS, pp. 341-349, Lake Tahoe, NV, Dec. 2012.
Yaochen Xie, Zhengyang Wang, and Shuiwang Ji. Noise2Same: Optimizing a self-supervised bound for image denoising. In Proc. NIPS, volume 33, pp. 20320- 20330, Vancouver, Canada, Dec. 2020.
Jun Xu, Yuan Huang, Li Liu, Fan Zhu, Xingsong Hou, and Ling Shao. Noisy-As- Learning unsupervised denoising from the corrupted image. IEEE Trans. Image Process., 29:9316-9329, Sep. 2020.
Li Xu, Jimmy SJ Ren, Ce Liu, and Jiaya Jia. Deep convolutional neural network for image decon- volution. In Proc. NIPS, pp. 1790-1798, Montreal, Canada, Dec. 2014.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7):31 1422- 3155, Feb. 2017.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A.1 PROOFS FOR THEOREM 2
Observe first that combining Assumptions & 3 and the J-complement implies that f()) and g( are conditionally independent, given i.e.,
f ml TT g(z))mll. Vm.
Using this result with Assumption 2 and reminding that the I J-complement implies E (x)))())
g(x) E (x ( we obtain the following result from 4
E., (x g En.llf(ary))
21l2 + (19(x)) 21l2
where the equality uses
(x - y, g(x)) - 1) = Ey (f - y g(x)) (((())))9 - y)
Ey (Exlp (x. yml) (Err(ggE))) vmml)
0
in which the second equality uses the first result above and the third equality holds by Assumption 2.
This completes the proofs.
A.2 PROOFS FOR THEOREM 3
We first obtain the following bound:
Ex (. (f))) y. ()((()) y)
Eey's [(((z) Um (m)(g())) Um)
By (f) yn (g(w)())) Vna) 1--- EE (f(x), Yn (o(or)) vm))
Ey [Cov(f(z),,. Vm. (( vmlvv]
By [Cov(f(x) m(
Ey [Cov(s(x)) (x g(
By (var(((=) ((xJ=mml)) Varl)(=))mlu))1²
Ey (Var(f(x)). - ((xs=)mll)) Var(g(=>)mlu))1?
m
1/2
M Ey (Var(f(x)m (am00mmm((aasmm)a.a) f(aJ=)mll) Var(g( l/)]
M
m=l
M Ey (Var([(x) (arr))ll)) o2]
1/2
M Σ
where the second equality holds by Assumption 2, the third cquality uses Cov(X, E| E where X, Y, and Z are random variables or vectors, the fifth equal- ity holds because f( does not correlate with g(z.)m, Vm (due to Assumptions 1 and 3), so subtracting f( )m from f(x), does not change the covariance. Now, the first inequality uses the Pearson correlation coefficient bound, the second inequality uses the Jensen's inequality EVX & VEX, the third inequality uses the Jensen's inequality Em Vam /M¹> En, Om for
13

---

Under review as a conference paper at ICLR 2022
any R" and the last inequality holds by the conditional variance bound specified in Theo- rem 3 We bound and rewrite the final result above and this completes the proof:
1/2
(f(()(()) v. v.g(æs) 4) SOVM-
Σ
By [Var(f(x) - f(xJ=)mlb))
1/2
S SoVM-
Σ By Enjyy [f(x) f((
mm
M Σ
1/2
= -VMM-
Eg [f(z)m - -f(xx=mml
m==1
where the equality uses the filtration property of conditional expectation.
A.3 EXAMPLES THAT SUPPORT CONJECTURED BEHAVIOR OF (9) WITH
The first example in general regression models the pseudo-target follows: g(x)) y e1. where 61 E MRM is some arbitrarily additive noise independent of y. This gives Var (()g()))lll)) = Var(ym + (e1) 1y) Var(c1) I o², 1, ,,..., M. Under this model, how close is g(x.) is captured by a.
The second example in image denoising assumes that æ is corrupted by AWGN 62 E R. that is independent of Setting as a linear mapping G E IR J gives Var(g(=z)a/l)) = Var((Gys) + (G(()))))ll) Var((G(e2)))))) S a², Vn I,..., N. Under this model, how close is g( to is captured by a.
A.4 EXPERIMENTAL SETUP AND RESULTS FOR/FROM REAL-WORLD DATASETS
A.4.1 EXPERIMENTAL SETUP FOR WORLD DATASETS
We also evaluated the proposed SSRL framework with real-world camera image and low-dose CT datasets, where we do not have their complete noise properties/statistics. We chose the publicly available SIDD sRGB Data (Abdelhamed et al.] (0188) and Low Dose CT Image and Projection Data (Moen et al. 202T), where both the datasets include high-quality images or standard-dose FBP images so that one can run Noise2True experiments and obtain quantitative results. We used the DnCNN architecture for all experiments. For each experiment, we used the same implementation setup (such as hyperparameters and masking scheme) as that in the corresponding simulated data experiment. In particular, is median filter and pre-trained denoiser via existing self-supervised denoising methods in camera image denoising and low-dose CT denoising experiments, respectively.
Camera image denoising with SIDD sRGB Data. We used the SIDD sRGB training and validation datasets for training and tests, re- spectively. We chose the representative com- parison setup, Neighbor2Neighbor, from the camera image denoising experiments using simulated data in Section 4.1
PSNR 23.7 JB
RMSE = 147.9 HU
Low-dose CT denoising with Low Dose CT Image and Projection Data. We followed the data construction setup in Section 4.1 that was used in simulated data experiments; we mark that chest CT scans in the Low Dose CT Image and Projection Data the mm slice
Figure A. An input intrinsically image in camera image denoising (left) and low-dose CT (right). PSNR and RMSE values were averaged across all test samples.
14

---

Under review as a conference paper at ICLR 2022
thickness. We cannot run Noise2lnverse experiments because the Low Dose CT Image and Pro- jection Data does not provide two independent half-view FBP images. We thus chose the other comparison setups, Noise2Self and Noise2Same.
A.4.2 MAIN EXPERIMENTAL RESULTS FROM WORLD DATASETS
This section includes main experimental results such as denoised images and calculated performance measure, from the real-world datasets. Section S.3.2 in the supplement includes their supplementary results.
Proposed SSRL in Neighbor setup
Reference
Noise2True
Neighbor2Neighbor
PSNR 31.1 dB SSIM 0.841
PSNR 27.2 dB SSIM =0.672
PSNR 28.2 dB SSIM 00.688
Figure A.2: Comparisons of denoised images (top) via DnCNNs from different learning methods and their saturation error maps (bottom) in camera image denoising (blue and yellow denote 0 and 0.5 absolute errors, respectively). PSNR and SSIM values were averaged across all SIDD sRGB validation samples.
Proposed SSRL in Noise2Self setup
Reference
Noise2True
Noise2Self
RMSE 59.2 HU
RMSE 106.8 HU
RMSE 71. HU
Proposed SSRL in
2ome
Noise2Same setup
RMSE = 90.5 HU
RMSE 71.0 HU
Figure A.3: Comparisons of denoised images via DnCNNs from different learning methods in low-
dose CT (display window (800, 1200] HU). RMSE values were averaged across all test samples.
15

---

Under review as a conference paper at ICLR 2022
SELF-SUPERVISED N LEARNING USING DO- MAIN KNOWLEDGE: APPLICATIONS TO IMPROVING SELF-SUPERVISED IMAGE DENOISING (SUPPLEMENT)
Anonymous authors Paper under double-blind review
S.1 DETAILED PAPER CONTRIBUTIONS
This section elaborates the contributions of the proposed SSRL framework:
1. The paper applies the proposed SSRL generalization to several recent representative self-
supervised denoising methods, Noise2Self (Batson Royer 2019). Noise2Same (Xie
et al. (020). Noise2Noise (Lehtinen et al. 2018). Noise2lnverse (Hendriksen et al. (020),
and Neighbor2Neighbor (Huang al. 2021). See Sections 2.3 2.4 With camera image
and low-dose CT denoising experiments using both real and synthetic datasets, the paper demonstrates the outperforming performance of SSRL extensions over aforementioned self-supervised denoising methods.
2. Section 2.5 shows that designable ppeuooppredicton g in SSRL can relax noise assumptions of existing self-supervised denoising methods. In addition, Section 4.2 includes experi- ments studying how denoising performances change with satisfying noise assumption(s) (i.e., Assumptions 1-2).
3. The paper explains how incorporate domain knowledge into self-supervised denoising methods via and why more accurate domain knowledge can improve them. See examples in Sections 3.2 In addition, Section 3.3 proposes an empirical approach for selecting
g if domain knowledge of specific application is unavailable.
4. The proposed SSRL framework in Section 2 considers regression NN learning beyond denoiser learning, by using a desinable operator g. The paper is the first step towards self-supervised learning in regression problems, by showing that the proposed framework extends well to image denoising problem.
S.2 IMPLEMENTATION DETAILS, DATA AND CODE LICENSES, AND LIBRARY VERSIONS
This section describes implementation details, lists hyperparameters, and specifies license of datasets and codes used in this study.
S.2.1 DATA AND CODE LICENCES, AND LIBRARY VERSIONS
The ImageNet ILSVRC 2012 Val and BSD 300 datasets have the Custom license (re- search, non-commercial), the Set 5 data has the Unknown license, and the SIDD sRGB Data (Abdelhamed et al. 2018) has the MIT license. We obtained The 2016 Low Dose CT Grand Challenge data (McCollough 2016) from https::/aapm app box. com/ s/ eaw4 [aw4jddb53kegllee httplllpttaavvvoo www.bppttvvllllls......... httaa///eeeeeeeeeeeee2233335
and Low Dose CT Im- 7937/ 9npp-2 2637 The 2016 Low Dose CT Grand Challenge data and Low Dose CT Image and Projection Data have
age and Projection Data (Moen et al. 2021 from hommhttps::. // org/10.
the Custom license and the patient information is fully redacted.
We implemented all the methods specified Section 4 by modifyyng the Noise2Self code (Batson http://gtthbb.ccmmmc//.iooo///// se2sel with
& Royer 2019) (GitHub repository:

---

Under review as a conference paper at ICLR 2022
version Dec. 17, 2019) that is licensed under the MIT license. For all training and testing experi- ments, we used Pytorch 1.0.0 or 1.7.0 (Paskk et al. 2019) with the BSD-style license. For simu- lating low-dose FBP images, used the Michigan imgge reconstruction toolbox (MIRT) (Fessler 2016) of which license information is declared on its release page. For sinogram generation and FBP reconstruction,
used the "Gt.omo2_de" the "bpp.. mm" routine (updated on Dec. 21, 2005), respectively.
m" routine (updated Dec. 10, 2006) and
S.2.2 COMMON IMPLEMENTATION DETAILS IN BOTH APPLICATIONS
For all the existing self-supervised denoising methods specified in Section and Noise2True, we finely tuned their nyperparameters, including the initial learning rate, learning rate decay parameters, minibatch size, number of DrCNN layers, and balancing parameter a (see, e.g., (9. to achieve the best numerical results. We simply applied the chosen hyperparameter sets corresponding SSRL setups. We applied the chosen learning rate decay parameters, minibatch size, and number of DnCNN layers in Noise2 experiments to all the self-supervised denoising methods.
For the existing self-supervised denoising methods, Noise2Self (Batson & Royer 2019) and Noise2Same (Xie et al. 2020), we used their default masking setups. The Noise2Self default setup uses the deterministic masking scheme for each J that equi-spacedly samples 6. 6.25% of the num- ber of pixels in each training image (i.c., a single pixel is selected in each 4 x 4 window). The Noise2Same default setup uses the saturated sampling scheme (Xie et al. 2020 Krull et al. 2019) for each J that randomly samples z 0.5% of the number of pixels in each traiiing image (i.e., a single pixel is sampled in each 14 x 14 window). In training denoising NNs, both methods inter- polate missing pixels in Ie by applying weighted average to their 8 neighboring pixels, and use interpolated y input denoisers.
For the existing Noise2Noisee ages, Neighbor2Neighhoo (Huang et al. (022)) and Noise2Inverse (Hendriksen al.) 2020). we calculated their loss with non-masked images as proposed.
methods that emulate pairs of two independent noisy im-
We tested all trained regression NNs non-masked images rather than masked images with J as this setup gave higher denoising accuracy than prediction with masked images (Batson & Royer 2019 Xie et al. 2020p.
S.2.3
IMPLEMENTATION DETAILS FOR EXPERIMENTS WITH SYNTHETIC CAMERA IMAGE DENOISING DATA
The common hyperparameters for all learning methods were defined as follows. (In this applica- tion, these gave good image denoising performance across all existing self-supervised denoising methods and Noise2True, since we rescaled or normalized training images; see details below.) We used the default 17-layer DnCNN (Zhang et al., 2017) and the modified U-Net used in Noise2Self (Batson& & Royer 2019), and trained all DnCNNs and U-Nets with the mini-batch version of Adam (Kingma & Bal 2015). We selected the initial learning rate, the batch size, and the number of epochs x 10-1 8, and 190, respectively, and decayed the learning rates by factor of 0.5 every 50,000 iterations. (For Neighhor2Neighhor (Huang et al., 2021), we set the batch size as 32, as it reduces training image size with 2 x 2 sub-sampling window.) We used the data augmentations in Noise2Same (Xie et al., 2020). i.e., random crops with size 256 x 256, rotation and flipping. Except for Noise2Same, we rescaled all images to [0, 1]. following (Batson & Royer 2019 Huang et al., 2021]. In Noise2Same experiments (including Saiee we normalized each image by subtracting its mean and dividing by its standard deviation at each channel, following (Xie et al.) 2020).
For proposed SSRL in the Noise2Self and 2ome setups (referred to as SSRL-NNise2Selff and SSRL-Noise2Same, respectively), we used the deterministic masking scheme in Figure (boooom for each J with x 11.1% and 2s 1.2% sampling ratio, respectively - i.e., single pixel is selected in each 3 x 3 and 9 x 9 window, respectively. These setups gave more appealing results than the default masking parameters (i.e., 4 4 window in Noise2Self and 14x 14 window in Noise2Same): compare Figure So6 corresponding results in Figure 4 We observed in this application that using sufficient amount of information for a linear interpolation in of is useful for giving good prediction. For weighted median filtering (Brownrigg 1984) g in all SSRL setups, we used the following weights:
2

---

Under review as a conference paper at ICLR 2022
[1, 1;2, 9 2, where box denotes the central weight. The dilation rates of weighted median filtering for SSRL in the Noise2Self, Noise2Same, and Neighbor2Neighbor setups are 3, 9, and 1, respectively, corresponding to the distances between pixels in J. For SSRL-Noise2Self, computed Lind in 7 only on J (see Figure 2 b)ottom)). For SSRI-Noise2Same, we used the same balancing parameter as Noise2Same used, i.e., 1, and computed & (both terms) in ( only on J.
The DrCNN and U-Net training time for each experiment was less than 72 hours with NVIDIA
TITAN V GPU.
S.2.4 IMPLEMENTATION DETAILS FOR EXPERIMENTS WITH SYNTHETIC LOW-DOSE CT DENOISING DATA
We used fan-beam geometry for sinogram simulation, where width of each detector column is 1 1.2858 mm, source to detector distance is 1085.6 mm, and source to rotation center distance is 595 mm. For the FBP method, we used a ramp filter because in general, it better preserves the sharpness of edges on reconstructed images than Hanning filter (but overall noise increases).
The common hyperparameters for all learning methods were defined as follows. We used 8-layer DnCNN (Zhang et al.) 2017) with its default setup and the modified U-Net used in Noise2Self (Bat-) son & Royer] 2019). and trained all DnCNNs and U-Nets with the mini-batch version of Adam (Kingma & Ba, 2013). We selected the batch size and the number of epochs as 2 and 1,000, re- spectively, and decayed the learning rates by a factor of 0.95 every 10 epochs. In training DnCNNs and U-Nets, we selected the initial learning rates as 0.1 and 5 x 10 respectively, unless stated otherwise.
used complementary checkboard masks and JJ in Figure too.) We observed in this application that if is set to use small amount of information, i.e.. |J < [J%. then pre-trained, g makes poor prediction. For SSRL-Noise2Self, we set as pre-trained NN by Noise2Self with complementary checkerboard masks (see its inference results with 8-layer DnCNN and U-Net in Figures (S.(a) (a) and SS.4, respectively). In training DnC- NNs, we used the same initial learning rate as that used in pre-training 0.01. We computed Lind as given in (7) (see Figure (acuure@too). too)).
For proposed SSRL-Noise2Seff and SSRL-Noise2Same
For SSRL in the Noise2Inverse setup, we set f and as f/2 and I - g/2, respectively, where g pre-trained NN by Noise2Inverse. In training DnCNNs, we used the same initial learning rate as that used in pre-training 0.001. In inference, we averaged the predictions from f and g. as this corresponds to training setup above. all Noise2lnverse inferences (including SSRL- Noise2lnverse), we input full-view FBP images since this is consistent with other experiments and gave better denoising performance than denoising half-view FBP images. For SSRL-Noise2Same, set g pre-trained NN by Noise2Same with complementary checkerboard masks (see its test results with DrCNN and U-Net in Figures (.7 and S.4 respectively), and computed L (both 15, setting the ratio of the first in (9) as 10. For Noise2Same with either the default setup and complementary checkerboard masks, we chose the balancing parameter as 500. For self-supervised denoising methods with complementary checkerboard masks, interpolated missing pixels in both I=-= and æJ. by averaging their 4 neighboring pixels.
terms) in (9) only J We chose the balancing parameter term to the squared second term 20 (=) | -
The DrCNN and U-Net training time for existing self-supervised denoising methods and proposed SSRL methods was less than 10 hours and 12 hours, respectively, with an NVIDIA TITAN Xp GPU. took total less than 22 hours train both f and g.
S.3 SUPPLEMENTARY EXPERIMENTAL RESULTS FOR SECTION
This section mainly includes supplementary materials to Section 4
S.3. SUPPLEMENTARY EXPERIMENTAL RESULTS WITH SYNTHETIC DATASETS
Figures{S.. and S.2 show error maps of denoised images via DnCNNs from different learning meth- ods in camera image denoising in mixed noise and low-dose CT, respectively. These show for both
3

---

Under review as a conference paper at ICLR 2022
applications that in all the three comparison setups, SSRL significantly reduces enrors and artifacts across the entire image, over existing self selessueee denoising methods. Red boxes compare exist- ing sulf--upe denoising method to proposed SSRL in the corresponding setup.
Proposed SSRL in Noise2Self setup
Input noise image
Noise2True
Noise2Self
PSNR 9.8 dB SSIM O. 134
PSNR= 25.2 dB SSIM 0.819
PSNR 20.66 JB SSIM 00.711
PSNR 22.1 dB SSIM =0.748
Proposed SSRL in Neighbor2Neighbor setup
Proposed SSRL in Noise2Same setup
Neighbor2Neighbbor
Noise2Same
PSNR = 20.2 dB
PSNR= 22.3 dB SSIM 0.756
PSNR -20.1 dB SSIM -0.690
PSNR = 21.2 dB SSIM -0.705
SSIM 0.703
Figure S.1: Error map comparisons of denoised images via DnCNNs from different learning meth- ods in camera image denoising (blue and yellow denote O and 50 absolute errors, respectively). PSNR and SSIM values were averaged across all BSD 300 test samples.
Propased SSRL in 2ois setup
Input noisy image
Noise2True
Noise2Self
RMSE- - 18.3 HU
RMSE 16.3 HU
RMSE 36 HU
RMSE - 25.0 HU
Proposed SSRL in
Proposed SSRL in
Noise2Inverse
Noise2Inverse setup
Noise2Same
2oise2Same setup
RMSE- 22.9 IIU
RMSE 21.91IU
RMSE 28. 1 IIU
RMSE 26.0 II
Figure S.2: Error map comparisons of denoised images via DnCNNs from different learning meth- ods in low-dose CT (blue and yellow denote O and 50 absolute errors in HU, respectively). RMSE values were averaged across all test samples from The 2016 Low Dose CT Grand Challenge data.
4

---

Under review as a conference paper at ICLR 2022
Figures S.3 and S.4 show denoised images via U-Nets from different learning methods in camera image denoising in mixed noise and low-dose CT. respectively. These demonstrate for both appli- cations that in all the three comparison setups, SSRL significantly improves existing self-supervised denoising methods regardless of the regression neural network architecture.
Propased SSRL in
Reference
Noise2True
Noise2Self
Noise2Self setup
PSNR = 26.0 AB SSIM C.819
PSNR 20.9 dB SSIM 00.711
PSNR 22.5 dB SSIM 7..788
Proposed SSRL in
Proposed SSRL in
Neighbor2Neighbor
Srighbor2Neighben setup
2oise2Same
Noise2Same setup
PSNR 20.8 dB
PSNR 22.4 dB
PSNR 19.5
PSNR 20.7
SSIM = 0.703
SSIMI = -0.756
SSIM 0.6112
SSIM = 0.660
Figure S.3: Comparisons of denoised images (left) via U-Nets from different learning methods and their saturation error maps (right) in camera image denoising (blue and yellow denote 0 and 0.5 absolute errors, respectively). PSNR and SSIM values were averaged across all BSD 300 test samples.
Noise2Self (checkerboard mask)
Proposed SSRL Noise2Self setup
Reference
Noise2True
RMSE 18.5 HU
RMSE 32.3 HU
RMSE = 26.7 7 HU
Proposed SSRL in Noise2Inverse setup
Noise2Same (checkerboard mask)
Proposed SSRL in Noise2Same setup
Noise2lnverse
RMSE 24.0 HU
RMSE 23.5 HU
RMSE = 31.1 HU
RMSE = 28.2 HU
Figure S.4: Comparisons of denoised images via U-Nets from different learning methods in low- dose CT (display window is [800, 1200] HU). RMSE values were averaged across all test samples from The 2016 Low Dose CT Grand Challenge data.
5

---

Under review as a conference paper at ICLR 2022
S.3 report quantitative image denoising results with DrCNN and U-Net with BSD 300 and Set 5 data in camera image denoising in mixed noise, and with chest slices of The 2016 Low Dose CT Grand Challenge data in low-dose CT. Red boxes in Tables S.I S.3 compare existing self-supervised denoising method to proposed SSRL in the corresponding setup.
Tables S.I
Table S.1: Averaged test PSNR (dB) (first and third rows) and SSIM (second and fourth rows) comparisons with from different learning methods with DnCNN (first and second rows) and U-Net (third and fourth rows) in camera image denoising (simulated noisy BSD 300 dataset).
Proposed SSRL in Noise2Self setup
Proposed SSRL in Neighbor2- Neighbor setup
Proposed SSRL in Noise2Same setup
Neighbor2- Neighbor
Noise2Self 2ruse2True
Noise2Same
25.2
20.6
22.1
20.2
22.3
20.1
21.2
0.819
0.711
0.748
0.703
0.756
0.690
0.705
26.0
20.9
22.5
20.8
22.4
19.5
20.7
0.849
0.730
0.765
0.728
0.767
0.612
0.660
Table S.2: Averaged test PSNR (dB) (first and third rows) and SSIM (second and fourth rows) comparisons from different learning methods with DrCNN (first and second rows) and U-Net (third and fourth rows) in camera image denoising (simulated noisy Set dataset).
Proposed SSRL in Nels sctup
Proposed SSRL in Neighbor2- Neighbor setup
Proposed SSRL in 2oise2Same sctup
Neighhor2- Neighbor
Noise2True
Noise2Self
Noise2Same
26.4
19.3
21.2
19.0
21.9
18.9
20.0
0.890
0.743
0.785
0.744
0.806
0.729
0.753
27.2
19.5
21.8
19.4
21.6
17.7
19.3
0.910
0.752
0.802
0.751
0.800
0.626
0.712
Averaged test RMSE (HU) comparisons from different learning methods with DnCNN (first row) and U-Net (second row) in low-dose CT denoising (simulated low-dose CT dataset).
Table S.3:
Proposed SSRL
Proposed SSRL in Noise2- Inverse setup
Proposed SSRL in Noise2Same setup
Noise2- Inverse
Noise2True
Noise2Self
in Noise2Self sctup
Noise2Same
16.3
36.2
25.0
22.9
21.9
28.4
26.0
18.5
32.3
26.7
24.0
23.5
31.1
28.2
Figure S.5 compares prediction uncertainty of trained DnCNN denoisers via different learning meth- ods in both applications. The error bar graphs in Figure [S.5 show that for both applications, in all the three comparison sets, proposed SSRL gives similar or lower prediction uncertainty over the existing self-supervised denoising methods, Noise2Self, Neighbor2Neighhbr, Noise2Inverse, and Noise2Same.
Figure shows denoised camera images from SSRL and Rrr.R--ooseeaae using the default masking parameters in Noise2Self and Noise2Same (see Section (.2.3). Compare the results in Figure S.6 with the corresponding ones in Figure 4 using the designed setups (see Section S.2.3). The comparisons demonstrate that the default and designed setups give very similar PSNR results, ie., < (0.1 dB, but the designed setups gives slightly more visually appealing results than the default ones in Nolse2Self and Noise2Same.
Figure S.7 shows denoised images from Noise2Self and Noise2Same with DnCNN and comple- mentary checkerboard masks and J and reports the corresponding quantitative test results, in low-dose CT denoising. Comparing the results in Figure those of Nelf and 2oie

---

Under review as a conference paper at ICLR 2022
using the default masking setups in Figures 5 and S.2 shows that checkerboard masking improves denoising quality over default masking in Noise2Self, and achieves comparable denoising perfor- mance to default masking in Noise2Same. (See their default masking setups in Section [S.2.2] SFRL-NNise2Self with checkerboard masking achieves significantly better denoising quality com- pared Noise2Self with checkerboard masking. We used pre-trained DnCNN from these setups as in the corresponding SSRL setup.
28
40
PSNR (dB)
RMSE (HU)
24
300
25
20
18
N2T
N2Self
SSRI. in Nel2Nei 8SRI. in N2Same SSRL in
N2T
N2Self SSRL in N2tw 8SR.. in N28ame SSRL in N2Self setup
setup
Nei2Nei sccup
sctup
actup
sctup
Methods
Methods
Figure S.5: Denoising performance error bars for different learning methods with DrCNN in camera image denoising mixed noise (with BSD 300) (left, 300 test images) and low-dose CT (right, 30 test images). Red asterisks denote the averaged test PSNR or RMSE values. Error bar represents one standard deviation of test PSNR or RMSE values.
(a) SSRL-Noise2Selff with the default setup in Noise2Self (i.e., 4 x window)
(b) SSRL-Noise2Same with the default setup
in 2oise2Same (i.e., 14 x 14 window)
PSNR 22.0
PSNR 21.1 JB
Figure S.6: Denoised images via DnCNNs and their corresponding error maps from SSRL- Noise2Self and SSRL-Noise2Same with default masks setup camera image denoising. (In error maps, blue and yellow denote 0 and 50 absolute errors, respectively.) PSNR values were averaged across all BSD 300 test samples.
(a) Noise2Self with complementary checkerboard masks
(b) Noise2Same with complementary checkerboard masks
RMSE 30.9 HU
RMSE = 28.7 HU
maps from Noise2Self and Noise2Same with complementary checkerboard masks in low-dose CT. (The display window of denoised images is (800, 1200] HU; in error maps, blue and yellow denote 0 and 50 absolute errors in HU, respectively.) RMSE values were averaged across all test samples.
Figure S.7: Denoised images via DrCNNs and their corresponding

---

Under review as a conference paper at ICLR 2022
S.3.2 SUPPLEMENTARY EXPERIMENTAL RESULTS WITH REAL- WORLD DATASETS
Figures(S.8]an show error maps of denoised images via DnCNNs from different learning meth- ods in camera image and low-dose CT denoising with real-world datasets, respectively. These show for both applications that in Neighbor2Neighhor, Noise2Self, or Noise2Same comparison setups, SSRL significantly improve the entire image without having their exact noise properties.
Proposed SSRL in Neighhor2Neighten setup
Input noisy image
Noise2True
Neighbor2Neighhor
PSNR= 23.7 dB SSIM 0.485
PSNR 31.1 dB SSIM 0.841
PSNR = 27. dB SSIM 00662
PSNR = 28.2 dB SSIM 0..88
Figure S.8: Error map comparisons of denoised images via DcCNNs from different learning meth- ods in camera image denoising with real-world dataset (blue and yellow denote 0 and 50 absolute errors, respectively). PSNR and SSIM values were averaged across all SIDD sRGB validation sam- ples.
Proposed SSRL in 2oise setup
Input noisy image
Noise2Self
RMSE +147.9 HU
RMSE 1O6.8 HU
RMSE 71.2 HU
Proposed SSRL in
2oise2True
Noise2Same
2oise2Same setup
RMSE- 559.2 HU
RMSK- 90.5 HU
RMSE 4710 HU
Figure S.9: Error map comparisons of denoised images via DcCNNs from different learning meth- ods in low-dose CT with real-world dataset (blue and yellow denote 0 and 100 absolute errors in HU, respectively). RMSE values were averaged across all test samples.
S.4 EMPIRICAL RESULTS TO SUPPORT SOME CLAIMS IN MAIN PAPER
The following empirical results support that proposed SSRL loss better approximates the supervision (Noise2True) loss than existing self-supervised learning, particularly when Assumptions 1-2 are not completely satisfied. We used the representative self supervised denoising setup, Noise2Self. In the camera image denoising experiments in Section 4.1 the empirical loss values (at the last epoch) of [Noise2Self, SSfL-Noise2Self, Noise2True) are {0.296, 0.244, 0.170) (in RMSE); in the low-dose CT denoising experiments in Section 4.1 those are (2186.4, 329.1, 270.2] (in HU²).
8

---

Under review as a conference paper at ICLR 2022
empirically supports claim in Section 3.1 that g design "approximately" satisfy Elg(2) 8/ in camera image denoising, with both simulated and real-word datasets. We calcu- lated empirical E(x - ylv] and B[g(z) ylyl with simulated noisy BSD 300 test samples (using noise model 10 and the real-world noisy dataset (specifically, SIDD sRGB validation samples) in Section A.4.1 where g median filtering. In the simulated dataset, the empirical measures for {avg( | x ylyll), avg([[[[((x) ylul)) are (0.0201, 0.0098}, where avg denotes averaging across pixels. In the real-word dataset, those are {0.0113, 0.0083J. These support our claim that [[g(=)]y = y approximately satisfied with median filtering
Figure S.10
Figure S.11 empirically supports our claim in Section 3.2 that noise of FBP-reconstructed images in low-dose CT, i.c., e in (TT), has approximately zero-mean. The position of the patient table base is similar across FBP images, gave higher errors in the calculated sample mean; see the bottom of the image in Figure S.11
The following empirical results support the claim in Section 3.2 that pre-trained g will give better reference than I in low-dose CT denoising: the empirical measure of the Noise2Self loss (Batson Royer 2019 (()g())) - x= [12 with simulated noisy CT test samples by setting as Z and pre-trained Dn CNN by Noise2Self are 2455.7 and 1839.1 (in HU²), respectively.
(a) Simulated noisy dataset
(b) Real-world noisy dataset (SIDD sRGB validation samples)
(BSD 300 test samples)
Figure S.10: Empirical observations of EEz ylyil (left) and [E[g(z) yluil (right) in camera image denoising ((a) Blue and yellow denote 0 and 0.1 absolute errors, respectively. (b) Blue and yellow denote 0 and 0.02 absolute errors, respectively.)
0
Figure $.11: Sample mean of noise in low-dose FBP images e in (1). We calculated the sample mean with 200 samples.
S.5 EXPERIMENTAL RESULTS WITH GAUSSIAN AND POISSON+GAUSSIAN NOISE MODELS
This section studies the performance of the proposed SSRL framework with benchmark noisy datasets, MIT-Adobe FiveK data (Bychkovsky et al. 201I), corrupted by sole Gaussian and Pois- son+Gaussian noises.

---

Under review as a conference paper at ICLR 2022
S.5.1 EXPERIMENTAL SETUP
In Guassian denoising experiments, we simulated AWGN with the standard deviation value o, = 25 (uang et al., 2021: Xu et al. 2020), and selected ! Wiener filtering that is known be optimal in the sense of minimum MSE in Gaussian denoising. In Poisson+Gaussian denoising experiments, we followed the noise simulation setup (Byun al. 2021 Tab. 5: (a, or) (0.05, 0.02)) that cor- responds to (A, o.) (20,5, 5.1) where À and are defined in (TO). To better visualize the results, we enhanced the brightness with gamma correction (with the parameter 3). We choose the repre- sentative comparison setup. Neighbor2Neighbor, from the camera image denoising experiments in Section 4.1 This is also a state-of-the-art self-supervised denoising method, particularly when only single noisy images are available (Huang et al. 2021
S.5.2 COMPARISONS BETWEEN DIFFERENT LEARNING METHODS
Two observations in Figure S.12 12 First, in both Gaussian and Poisson+Gaussian denoising, proposed SSRL using "good" g further improved a state-of-the-art self-supervised denoising method, Neigh- bor2Neighbor. Second, in both Gaussian and Poisson+Gaussian denoising, the performance gap between Noise2True and Neighbor2Neighbor spond to existing literature (Huang et al. 2021 Byun et al. (02TT) We conjecture that this is because the noise assumptions of self-supervised denoising method are well satisfied in the aforementioned two experiments. This is connected to our conjecture in Section 4.2 that self-supervised denoising performance degrades, its assumptions are not completely satisfied.
small, where the PSNR gap numbers well corre-
Reference image
Noise2True
Neighhor2Neighhor
Proposed SSRL in Neighbor2Neighbnr setup
PSNR 34.61 dB
PSNR 34.34 dB
PSNR 34..5 dB
(a) Gaussian denoising (i..d. with a, = 25)
PSNR 37.77 dB
PSNR 37 dB
PSNR 37.59 dB
(b) Poisson-Gaussian denoising (i.id. with (A, a.) = 5.1)
Figure S.12:
Comparisons of denoised images via DnCNNs from different learning methods in camera image denoising. PSNR vlaues averaged across all MIT-Adobe FiveK test samples.
S.6 COMPARISONS TO SELF2SELF
This section compares the proposed SSRL framework with saate-of-thheart blind image denoising (a.k.a. self-supervised denoising with a single image) method, Self2Self (Quan et al., 2020). with synthetic and real-world noisy datasets for each application (see Section (We used the authors'
Self2Self implementation.)
First, in Figure S.13 and S.14 compare Self2Self with Neighbor2Neighbor and Noise2Self (the rep- resentative comparison setup in each simulated imaging experiment in Section 4.I), respectively. The comparisons show that Self2Self gives comparable results to
10

---

Under review as a conference paper at ICLR 2022
Remind, however, that Self2Self is blind denoising method, so it needs significantly larger com- putations than Neighbor2Neighbor, when one denoises a new noisy image. With good g designs (see Sections 3.1 3.2p. proposed SSRL significantly outperformed Self2Self in both applications, regardless of whether their dataset is real or simulated.
Proposed SSRL in Neighbor2Neighbor setup
Noise2Truc
Self2Self
Neighbor2Neighhor
PSNR 27.2 dB
PSR
19.0 dB
PSNR -19.4 dB
PSNR 21.6 dB
(a) Synthetic noisy dataset (Set using imaging simulation in Section
4.1
PSNR = 31.1 dB
PSNR
included)
PSNR 27.2 dB
PSNR 28.2 dB
(b) Real-world noisy dataset (SIDD sRGB validation)
Figure S.13: Comparisons of denoised images (right) from different learning methods and their error maps (left) in camera image denoising (blue and yellow 0 and 50 absolute errors, respectively). PSNR values were averaged across all test samples.
Proposed SSRL in Noise2Self setup
Noise2True
Sel12Self
Noise2Self
RMSE 18.5 HU
RMSE HU
RMSE 32.3 HU
RMSE 26.7 HU
(a) Synthetic dataset (The 2016 Low Dose CT Grand Challenge data) using imaging sim. Section
RMSE 59.2 HU
RMSE 101.1 HU
RMSE 106.8 HU
RMSE 71.2 HU
Real-world dataset (Low Dose CT Image and Projection Data)
Figure S.14: Comparisons of denoised images from different learning methods in low-dose CT with synthetic and real-world datasets (display window is [800, 1200] HU). RMSE values were averaged across all test samples.
11

---

Under review as a conference paper at ICLR 2022
S.7 PRELIMINARY RESULTS WITH THE TEACHER-STUDERT LEARNING PERSPECTIVE
The proposed SSRL framework is applicable to teacher-student learning (Wang & Yoon 2021 Hin- ton et al. 2015 Buciluá et al. 2006) that aims to learn smaller student network from bigger teacher network(s)) s). We ran preliminary experiments in self-supervised low-dose CT denoising (using sim- ulated data). The teacher model g is the pre-trained 8-layer DrCNN by Noise2Self (with checker- board masking), and we set the student model f as {8, 7, 6, 5, 4, 3 3J-layer DrCNN, Applying sSRL- Noise2Self, we obtained the following numerical results: the RMSE (HU) values of student models with {8, 7, 6, 5, 4, 3l-layer DnCNNs are {25.0, 25.3, 25.5, 25.4, 25.2, 27.5). respectively. The stu- dent DnCNNs that have the equal or lower complexity compared to its teacher network, significantly improves its teacher model of which RMSE value is 30.9 (in HU). The results might imply that stu- dent models learned from SSRL can outperform their teacher model, if they retain sufficiently high network complexity as compared to their teacher's ' (c.g.. 3-layer DnCNN), In addition, we have addi- tional SSRL experiment in low-dose CT denoising with the "iterative" teacher-student perspective. The teacher model 9 is pre-trained 5-layer DrCNN from the non-iterative teacher-student SSRL method above, and we set the student model f as 3-layer DnCNN. We obtained 27.3 RMSE (in HU), implying only marginal improvement over the 3-layer DrCNN obtained by the non-iterative stadeet-ttudent SSRL method. We conjecture that iterative teacher-student SSRL needs more so- phisticated g-setups, such as an ensemble of teacher models (Hinton et al. 2015).
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown. A high-quality denoising dataset for smartphone cameras. In Proc. IEEE CVPR, pp. 1692-1700, Salt Lake City, Utah, Jun. 2018.
Joshua Batson and Loic Royer. Noise2Self: Blind denoising by self-supervision. Proc. ICML, pp. 524-55443333 Long Beach, CA, Jun. 2019.
David RK Brownrigg. The weighted median filter. Commun. of the ACM, 27:807-818, 1984.
Cristian Buciluă, Rich Caruana, and Alexandru Nieulescu-Mizil. Model compression. In Proc. ACM. SIGKDD., pp. 535-541, 2006.
Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. Learning photographic global tonal adjustment with a database of input output image pairs. In Proc. IEEE CVPR, pp. 97-104, Colorado Springs, CO, Jun. 2011.
Jaeseok Byun, Sungmin Cha, and Taesup Moon. Fbi-denoiser: Fast blind image denoiser for poisson-gaussian noise. Proc. IEEE CVPR, pp. 5768-5777, Virtual, Jun. 2021.
J. A. Fessler. Michigan image reconstruction toolbox (MIRT) for Matlab. Available from http: //web. eecs. umich. edu/ 2016.
Allard Adriaan Hendriksen, Daniël Maria Pelt, and K Joost Batenburg. Noise2Inverse: Self- supervised deep convolutional denoising for tomography. IEEE Trans. Comput. Imag., 6:1320- 1335, Aug. 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Proc. NIPS Workshop, pp- 1790-1798, Montreal, Canada, Dec. 2015.
Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2Neighbor: Self- supervised denoising from single noisy images. Proc. IEEE/CVF CVPR, pp. 14781-14790, Virtual, Jun. 2021.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In Proc. ICLR 2015, pp. 1-15, San Diego, CA, May 2015.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2 Void Learning denoising from single noisy images. In Proc. IEEF/CVF CVPR, pp. 2129-2137, Long Beach, CA, Jun. 2019.
12

---

Under review as a conference paper at ICLR 2022
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Mika Aittala, and Timo Aila. Noise2Noise: Learning image restoration without clean data. In Proc. ICML, pp- 2965-2974, Stockholm, Sweden, Jul. 2018.
C. M.Coollough TU-FG-207A-04: Overview of the low dose CT grand challenge. Med. Phys., 43 (6Part35):3559-376)) Jun. 2016.
Taylor R Moen, Baiyu Chen, David R Holmes III, Xinhui Duan, Zhicong Yu, Lifeng Yu, Shuai Leng, Joel G Fletcher, and Cynthia H McCollough. Low-dose CT image and projection dataset. Med. Phys., 48(2):022-111, Feb. 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, nigh-performance deep learning library. In Proc. NIPS, volume 32, pp. Canada, Dec. 2019.
Vancouver,
Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2Self with dropout: Learning self- supervised denoising from single image. In Proc. IEEE/CVF CVPR, pp. 1890-1898, Virtual, Jun. 2020.
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelli- gence: A review and new outlooks. IEEE Trans. Pattern Anal. Mach. Intell., Jan. 2021.
Yaochen Xie, Zhengyang Wang, and Shuiwang Ji. Noise2Same: Optimizing a self-supervised bound for image denoising. In Proc. NIPS, volume 33, pp. 20320-20330. Vancouver, Canada, Dec. 2020.
Jun Xu, Yuan Huang, Li Liu, Fan Zhu, Xingsong Hou, and Ling Shao. Noisy-As-Cleam Learning unsupervised denoising from the corrupted image. IEEE Trans. Image Process.. 29:9316-9329, Sep. 2020.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Trans. Image Process., 26(7):3142- 3155, Feb. 2017.
13


Reviews:
[{"review_id": "rxWa5CTSnEJ", "reviewer": "Reviewer_1ABd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Examples in Motivation are not convincing. There is no doubt that ground-truth clean image can provide better denoising performance, and any other modifications will yield denoising performance decreases. \n\n2. If I understand right, when training denoiser f, the supervision is from complementary denoising results by pre-designed g. The masks on J and Jc have no intersection. From Fig. 2, it is confused to understand the training procedure. Predicted output in above should be masked on J? Why pseduo-target is entire image while the one in bottom is masked? Also in experiments, the results by unbalanced masks are not given, so why give this setting in Fig. 2? \n\n3. The proposed method is evaluated on their designed settings, which is not consistent with most exisitng self-supervised denoising methods.", "summary_of_the_paper": "This paper proposed a self-supervised denoing method using domain knowledge. The proposed method seems somewhat like knowledge distillation, where a pre-trained denoiser or handcrafted designed noise models g can provide better initial results than original noisy images, and then a better denoiser f can be trained based on the results of g instead of noisy images. By forcing the complement of f and g, f will not be same with g, and the denoising performance is bossted.  ", "summary_of_the_review": "This work proposed an interesing self-supervised denoising method and its improvements are significant. But it has several issues to be addressed listed above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SQ6xqp5AKgm", "reviewer": "Reviewer_iEcr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strong points*\n\n1. The technical writing of the paper is fairly well. The derivation can be easily  followed.\n\n2. The paper presented a generalization of the noise2self, which conceptually can handle more tasks than noise2self.\n\n*Weak points*\n\n1. The novelty is limited, it is essentially a very minor generalization of noise2self, the un-bias estimator of loss function under J-invariance. There is no convincing example to show such a generalization can be realized with a constructive scheme.\n2. The contribution is over-claimed. While the paper is written in the way that it seems to be able to  go beyond image denoising, it is not the case. The paper does not show how  operator $g$ can be defined for other image recovery problems with non-identity measurement matrix.\n3.  Experiments are limited with very specific synthesized noise. \n4.  Its practical value for self-supervised image denoising is not well justified in the experiments.  While the proposed method is a minor incremental work built on noise2self. its performance is rather well below recent existing self-denoising methods,, in terms of the performance gap to the supervised counterpart.\n\n*Comments and Questions*\n\n1. The paper claim a regression network which can be self-supervised. Can the author provide some examples on $g$ for other tasks that goes beyond image denoising?\n2. The examples used in the experiments are removing mixture of noise. The paper did not clearly how the proposed SSRL loss function has its advantage over noise2self, in terms of its approximation to the loss function of noise2truth.\n3. The proposed methods showed better performance over noise2self in the case of Gaussian+poisson+pepper-and-salt. The first 2 noises are independent, thus it fits the assumption of noise2self. The pepper-and-salt noise can be trivial treated in noise2self by excluding all pixels with value $0$ or $1$. The experiments should be conducted with such modifications on noise2self to justify its value. \n4. Since noise2self, there has been a rapid progress on self-supervised image denoising which much better performance than noise2self. Indeed, the performance of recent many methods, e.g.  noiser2noise[CVPR'20], self2self [CVPR'20], Noise-as-clean [TIP'21], \nPartially-linear denoiser [PAMI'21], R2R [CVPR'21], are close to the supervised counterparts. These all are based on the same idea, designing an approximate unbiased estimator to loss function.  To list some, However, From the experiments shown in this paper, the gap between the proposed one and the supervised one is not small. More experiments should be included.", "summary_of_the_paper": "This paper present an self-supervised deep learning method for image denoising, which is a generalization of the concept proposed in noise2self. The basic idea is to approximate the unbiased estimation of the supervised loss function, which utilized the noise independence of the input and the output of the network. Some experiments are conducted on 2 image denoising tasks.\n", "summary_of_the_review": "The case for this paper is quite clear to me. The paper present a minor generalization of the noise2self, which is conceptually more general, but not constructive scheme is developed to make it more applicable that noise2self. There is no concrete example to show its advantage over noise2self in terms of generalization to different problems or different noise. This paper is not a theoretical paper and its value largely lies in its empirical performance.  There have been many recent works on the unbiased estimator of the supervised loss function, the same as the proposed one. e.g.  noiser2noise[CVPR'20], self2self [CVPR'20], Noise-as-clean [TIP'21], partially-linear denoiser [PAMI'21], R2R [CVPR'21]. These methods provide competitive performance to the representative supervised denoising network, such as DnCNN, for various noise types including Gaussian, Poisson and real-world noise. The experiments conducted in this paper does not provide any comparison to these methods on standard benchmarking dataset, and the available experiments seems that the gap between the proposed one and the supervised one is quite big. In other words, the practical benefit of the proposed DDSL is not impressive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "4X7PvrECs_a", "reviewer": "Reviewer_vDqQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think it is a good paper to improve the performance of existing unsupervised denoising methods. However, I have several concerns:\n1. One statement of this paper is the domain knowledge is important which can further improve the performance of unsupervised denoising. This is based on that Eq. 3 is closer to Eq. 1 if the 2nd term of Eq. 5 is close to zero. However, the second term of Eq. 5 is not relevant to f(x). In this way, is the back propagation of Eq. 3 and Eq. 1 the same when optimizing f?\n2. A g(x) closer to the ground truth can lead to a better result for f(x) according to my understanding. In this way, g(x) can also be treated as a denoiser. Can the authors explain why f(x) has a better denoising ability than g(x)?\n3. The visual improvement is too marginal even though the PSNR is higher e.g. in Fig. 4.\n4. It seems spatially correlated noises may violate the assumptions in this paper. The authors provide some experiments to show the effectiveness of the proposed method in real image denoising dataset (Abdelhamed et al., 2018). Can the authors provide more analysis about why the proposed method still works well for spatially correlated noises?\n5. The motivation is strange to me in Sec. 2.1. It is strange to use norm(y) as supervision in Fig.1 (right).\n6. In sec. 3.1 and 3.2, Fig. A should be revised to Fig. 3?", "summary_of_the_paper": "This paper proposes a method for unsupervised image denoising. It shows that a better designed operator based on domain knowledge can benefit unsupervised image denoising task. The provided experimental results show the proposed method outperforms existing unsupervised denoising ones and achieves similar performance to supervised methods.", "summary_of_the_review": "This paper proposes a novel method to improve the performance of unsupervised denoising methods. But I still hope the authors address my concerns above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on unsupervised image denoising and proposes a method to do so. It shows that using a designed operator based on domain knowledge can help improve unsupervised image denoising. The authors also provide experimental results demonstrating that the proposed methods outperform existing unsupervised denoising and behave similar in performance to supervised methods. The reviewers liked the improvements but (1) limited novelty/simple extension of noise2self, (2) example not convincing, (3) lack of clarity in 2.3, (4) a variety of other technical concerns. The authors partially addressed these concerns. However, I concur with the reviewers that the paper still requires more work and is not ready for publication in its current form.
Decision: Reject