Self-supervised regression learning using domain knowledge: Applications to improving self-supervised image denoising

1 INTRODUCTION . Deep regression neural network ( NN ) -based methods that can accurately predict real- or complexvalued output have been rapidly gaining popularity in a wide range of computational imaging and computer vision applications including image denoising ( Vincent et al. , 2010 ; Xie et al. , 2012 ; Zhang et al. , 2017 ) , image deblurring ( Xu et al. , 2014 ) , image super-resolution ( Dong et al. , 2016 ; Kim et al. , 2016 ) , light-field reconstruction ( Chun et al. , 2020 ; Huang et al. , 2020 ) , object localization ( Szegedy et al. , 2013 ) , end-to-end autonomous driving ( Bojarski et al. , 2016 ) . Yet , they lack a general self-supervised learning framework . In training a regression NN f : RN → RM , the most prevalent supervised learning approach minimizes the mean square error ( MSE ) between what f predicts from an input x ∈ RN and a ground-truth target y ∈ RM : min f Ex , y‖f ( x ) − y‖22 . ( 1 ) Learning a denoising or refining NN uses ( 1 ) with M = N – dubbed Noise2True – where x is a corrupted image and y is a clean ( i.e. , ground-truth ) image . However , it is challenging or even impossible to collect many clean images y in many practical applications , motivating research on self-supervised learning for image denoising ( Ulyanov et al. , 2018 ; Soltanayev & Chun , 2018 ; Krull et al. , 2019 ; Batson & Royer , 2019 ; Laine et al. , 2019 ; Moran et al. , 2020 ; Quan et al. , 2020 ; Xu et al. , 2020 ; Hendriksen et al. , 2020 ; Xie et al. , 2020 ; Huang et al. , 2021 ) – called self-supervised image denoising . To learn a denoiser f with single noisy images , a popular self-supervised image denoising method , Noise2Self ( Batson & Royer , 2019 ) ( see also the concurrent work ( Krull et al. , 2019 ) ) , and its sophisticated relaxation , Noise2Same ( Xie et al. , 2020 ) , study the following MSE minimization problem : min f Ex‖f ( x ) − x‖22 . ( 2 ) These methods use some partitioning schemes in ( 2 ) to avoid that its optimal solution is just the identity mapping I. Noise2Noise ( Lehtinen et al. , 2018 ) that learns a denoiser with pairs of two This article has appendix and supplement . The appendix and supplement number sections , figures , and tables with the prefix “ A ” and “ S ” , respectively . independent noisy images , is a pioneer work for self-supervised image denoising . Motivated by Noise2Noise , several self-supervised image denoising methods such as Noise2Inverse ( Hendriksen et al. , 2020 ) , Neighbor2Neighbor ( Huang et al. , 2021 ) , and ( Soltanayev & Chun , 2018 ; Moran et al. , 2020 ; Xu et al. , 2020 ) emulate pairs of two independent noisy images , by applying partitioning or adding simulated noise to single noisy measurements . All the aforementioned methods have been developed based on some noise assumptions including pixel-wise independent noise ( Krull et al. , 2019 ; Xie et al. , 2020 ; Huang et al. , 2021 ) and zero-mean noise ( Lehtinen et al. , 2018 ; Batson & Royer , 2019 ; Xie et al. , 2020 ) . Yet , they lack design flexibility that might relax such noise assumptions and further improve the denoising performance of NNs . Some works estimate statistical parameters of noise , such as noise histogram ( Krull et al. , 2020 ) and parameters of Gaussian mixture noise model ( Prakash et al. , 2021 ) . This paper presents new insights on this topic . The paper proposes a general self-supervised learning framework for regression problems , which we refer to as self-supervised regression learning ( SSRL ) . Proposed SSRL enables learning regression NNs with only input samples , by using a designable operator that can encapsulate domain knowledge of a specific application . Our main results show that under some mild conditions ( e.g. , in image denoising , statistical noise properties in x ) , the better desinable operator is used , the proposed SSRL loss becomes closer to ordinary supervised learning loss . In addition , a designable operator with good domain knowledge can relax noise assumptions of existing self-supervised denoising methods . Numerical experiments for camera image and low-dose computational tomography ( CT ) denoising with both simulated and real-world datasets – corrupted by only single noise realization – demonstrate that the proposed SSRL framework significantly improves denoising quality compared to several existing self-supervised denoising methods . Put together , our findings provide new insights into how using good domain knowledge can improve self-supervised denoising , underscoring the benefits of understanding application-specific knowledge in SSRL . ( Section S.1 further elaborates the contributions of the paper . ) 2 SSRL USING DOMAIN KNOWLEDGE . The proposed SSRL loss is given by Ex‖f ( x ) − g ( x ) ‖22 , ( 3 ) where g : RN → RM is a designable operator encapsulating domain knowledge of a specific application . We will incorporate some sophisticated setups in ( 3 ) such that f obtained by minimizing ( 3 ) can not merely be g. Although related theorems ( see later ) hold for any M , we mainly focus on practical image denoising applications with pseudo-target g ( x ) 6≈ y . 2.1 MOTIVATION This section empirically shows that understanding domain knowledge is important for designing g in the proposed SSRL loss ( 3 ) . The following camera image denoising examples demonstrate that well-designed g with good domain knowledge improves the denoising performance of learned f via ( 3 ) . Suppose that camera images are corrupted by salt-and-pepper noise . Consider two example setups for g ( · ) , median filtering and BM3D denoiser ( Mäkinen et al. , 2020 ) , denoted by median ( · ) and BM3D ( · ) , respectively . Figure 1 compares the denoising performance of minimum f ? with the two aforementioned g setups : f ? with median filtering significantly im- proved that with BM3D denoiser . This is not surprising , as median filtering is widely known to be effective in reducing salt-and-pepper noise ( Bovik , 2010 , §3.2 ) . This result emphasizes the importance of understanding domain knowledge of specific applications in proposed SSRL . 2.2 PRELIMINARIES . We first introduce the J -complement between two functions f and g : Definition 1 . For a given partition J = { J1 , . . . , JB } ( |J1| + . . . + |JB | = N ) of the dimensions of input x ∈ RN , functions f : RN → RM and g : RN → RM are called J -complementary , if f ( xJc ) does not depend on g ( xJ ) for all J ∈ J , where Jc denotes the complement of J , and ( · ) J denotes a vector restricted to J . That is , f and g use information from outside and inside of J to predict output and give pseudotarget , respectively . In denoiser learning ( where M = N ) , Definition 1 specializes to the J - invariance of f ( Batson & Royer , 2019 ) , by setting g = I . Incorporating Definition 1 into the SSRL loss ( 3 ) is a straightforward approach to avoid that optimal f is just g in ( 3 ) . The proposed SSRL framework assumes the followings : Assumption 1 ) xJ and xJc are conditionally independent given y , i.e. , p ( x|y ) = p ( xJ |y ) p ( xJc |y ) . Assumption 2 ) E [ g ( x ) |y ] = y . Assumption 3 ) f and g are ( Borel- ) measurable . In denoiser learning , Assumption 1 holds if noise in each subset J ∈ J is conditionally independent from that in Jc , given y . Assumption 1 can be satisfied in general regression NN learning , by adding small randomized perturbations ( independent of y ) to either J or Jc , similar to Moran et al . ( 2020 ) . Assumption 2 suggests a direction for designing g : suppose that x has non-zero-mean noise ; one then can design g to make noise zero-mean using the domain knowledge . Assumption 3 is satisfied if f and g are continuous . This condition is mild because many regression NNs f are continuous – where their modules , convolution , matrix-vector multiplication , rectified linear unit activation , max pooling , etc . are continuous – and one can design g with measurable or continuous function . Finally , observe that the proposed SSRL loss ( 3 ) can be rewritten by Ex‖f ( x ) − g ( x ) ‖22 = Ex , y‖f ( x ) − y‖22 + ‖g ( x ) − y‖22 − 2〈f ( x ) − y , g ( x ) − y〉 . ( 4 ) We aim to either remove or control the third term , incorporating the J -complement and/or Assumptions 1-3 introduced above . 2.3 SSRL USING DOMAIN KNOWLEDGE WITH J -COMPLEMENT . This section studies SSRL loss ( 3 ) minimization over f that is J -complementary of g. Our first main result shows that under Assumptions 1–3 , the SSRL loss ( 3 ) with the J -complement is the sum of the ordinary supervised learning loss and variance of g ( x ) − y , i.e. , the third term in ( 4 ) vanishes . Theorem 2 . Under Assumptions 1–3 , the SSRL loss ( 3 ) with the J -complement in Definition 1 becomes Ex‖f ( x ) − g ( x ) ‖22 = Ex , y‖f ( x ) − y‖22 + ‖g ( x ) − y‖22 . ( 5 ) The following equality similarly holds for any K ∈ K : Ex‖f ( x ) K − g ( x ) K‖22 = Ex , y‖f ( x ) K − yK‖22 + ‖g ( x ) K − yK‖22 , where K is a partition of { 1 , . . . , M } , and f ( · ) K and g ( · ) K denote f ( · ) and g ( · ) restricted to K , respectively . The optimal solution for ( 5 ) is given by f ? ( xJc ) = E [ g ( xJ ) |xJc ] = E [ y|xJc ] . ( 6 ) Proof . See Section A.1 in the appendix . The result ( 6 ) suggests another direction for designing g : we aim to design good g that can make g ( xJ ) close to y . Using such g , optimal solution of LHS in ( 6 ) , E [ g ( xJ ) |xJc ] , becomes close to its supervision counterpart , E [ y|xJc ] . Consequently , such g reduces the second term Ex , y‖g ( x ) −y‖22 in RHS of ( 5 ) , leading ( 3 ) closer to ( 1 ) . If g is ideal such that g ( xJ ) = y , the SSRL loss ( 3 ) becomes the usual supervised learning loss . In designing g , domain knowledge of specific application is crucial . Domain knowledge includes noise properties in x and pre-trained NN by existing self-supervised denoising , such as Noise2Self ( Batson & Royer , 2019 ) and Noise2Noise ( Lehtinen et al. , 2018 ) . Specifically , the proposed SSRL loss using the J -complement is given by Lind ( f ) =∆ ∑ J∈J Ex‖f ( xJc ) − g ( xJ ) ‖22 . ( 7 ) Figure 2 illustrates ( 7 ) with complementary checkerboard masks Jc and J , where f and g use almost equal amount of information , and its variant , where g uses much less information than f . The variant computes MSE only on J ∈ J ; in this setup , it is challenging for g to predict the entire image . Relation to previous self-supervised denoising works . In denoiser learning , the proposed SSRL loss ( 3 ) with the J -complement of f and g = I specializes to ( 2 ) with the J -invariance of f , i.e. , Noise2Self . Noise2Inverse ( Hendriksen et al. , 2020 ) and Neighbor2Neighbor ( Huang et al. , 2021 ) that emulate pairs of two independent noisy images by partitioning single measurements ( e.g. , CT ray measurements with independent noise and corrupted images with pixel-wise independent noise ) can be viewed as Noise2Self . Thus , SSRL loss ( 3 ) with the J -complement of f and g = I specializes to the aforementioned Noise2Noise-motivated self-supervised denoising methods . ( Noise2Noise also can be viewed by SSRL ( 3 ) with the setup above , by constructing x with stacking two independent noisy images , where an image is corrupted by two independent noise realizations . )

Summary: This paper present an self-supervised deep learning method for image denoising, which is a generalization of the concept proposed in noise2self. The basic idea is to approximate the unbiased estimation of the supervised loss function, which utilized the noise independence of the input and the output of the network. Some experiments are conducted on 2 image denoising tasks.