Self-supervised regression learning using domain knowledge: Applications to improving self-supervised image denoising

1 INTRODUCTION . Deep regression neural network ( NN ) -based methods that can accurately predict real- or complexvalued output have been rapidly gaining popularity in a wide range of computational imaging and computer vision applications including image denoising ( Vincent et al. , 2010 ; Xie et al. , 2012 ; Zhang et al. , 2017 ) , image deblurring ( Xu et al. , 2014 ) , image super-resolution ( Dong et al. , 2016 ; Kim et al. , 2016 ) , light-field reconstruction ( Chun et al. , 2020 ; Huang et al. , 2020 ) , object localization ( Szegedy et al. , 2013 ) , end-to-end autonomous driving ( Bojarski et al. , 2016 ) . Yet , they lack a general self-supervised learning framework . In training a regression NN f : RN → RM , the most prevalent supervised learning approach minimizes the mean square error ( MSE ) between what f predicts from an input x ∈ RN and a ground-truth target y ∈ RM : min f Ex , y‖f ( x ) − y‖22 . ( 1 ) Learning a denoising or refining NN uses ( 1 ) with M = N – dubbed Noise2True – where x is a corrupted image and y is a clean ( i.e. , ground-truth ) image . However , it is challenging or even impossible to collect many clean images y in many practical applications , motivating research on self-supervised learning for image denoising ( Ulyanov et al. , 2018 ; Soltanayev & Chun , 2018 ; Krull et al. , 2019 ; Batson & Royer , 2019 ; Laine et al. , 2019 ; Moran et al. , 2020 ; Quan et al. , 2020 ; Xu et al. , 2020 ; Hendriksen et al. , 2020 ; Xie et al. , 2020 ; Huang et al. , 2021 ) – called self-supervised image denoising . To learn a denoiser f with single noisy images , a popular self-supervised image denoising method , Noise2Self ( Batson & Royer , 2019 ) ( see also the concurrent work ( Krull et al. , 2019 ) ) , and its sophisticated relaxation , Noise2Same ( Xie et al. , 2020 ) , study the following MSE minimization problem : min f Ex‖f ( x ) − x‖22 . ( 2 ) These methods use some partitioning schemes in ( 2 ) to avoid that its optimal solution is just the identity mapping I. Noise2Noise ( Lehtinen et al. , 2018 ) that learns a denoiser with pairs of two This article has appendix and supplement . The appendix and supplement number sections , figures , and tables with the prefix “ A ” and “ S ” , respectively . independent noisy images , is a pioneer work for self-supervised image denoising . Motivated by Noise2Noise , several self-supervised image denoising methods such as Noise2Inverse ( Hendriksen et al. , 2020 ) , Neighbor2Neighbor ( Huang et al. , 2021 ) , and ( Soltanayev & Chun , 2018 ; Moran et al. , 2020 ; Xu et al. , 2020 ) emulate pairs of two independent noisy images , by applying partitioning or adding simulated noise to single noisy measurements . All the aforementioned methods have been developed based on some noise assumptions including pixel-wise independent noise ( Krull et al. , 2019 ; Xie et al. , 2020 ; Huang et al. , 2021 ) and zero-mean noise ( Lehtinen et al. , 2018 ; Batson & Royer , 2019 ; Xie et al. , 2020 ) . Yet , they lack design flexibility that might relax such noise assumptions and further improve the denoising performance of NNs . Some works estimate statistical parameters of noise , such as noise histogram ( Krull et al. , 2020 ) and parameters of Gaussian mixture noise model ( Prakash et al. , 2021 ) . This paper presents new insights on this topic . The paper proposes a general self-supervised learning framework for regression problems , which we refer to as self-supervised regression learning ( SSRL ) . Proposed SSRL enables learning regression NNs with only input samples , by using a designable operator that can encapsulate domain knowledge of a specific application . Our main results show that under some mild conditions ( e.g. , in image denoising , statistical noise properties in x ) , the better desinable operator is used , the proposed SSRL loss becomes closer to ordinary supervised learning loss . In addition , a designable operator with good domain knowledge can relax noise assumptions of existing self-supervised denoising methods . Numerical experiments for camera image and low-dose computational tomography ( CT ) denoising with both simulated and real-world datasets – corrupted by only single noise realization – demonstrate that the proposed SSRL framework significantly improves denoising quality compared to several existing self-supervised denoising methods . Put together , our findings provide new insights into how using good domain knowledge can improve self-supervised denoising , underscoring the benefits of understanding application-specific knowledge in SSRL . ( Section S.1 further elaborates the contributions of the paper . ) 2 SSRL USING DOMAIN KNOWLEDGE . The proposed SSRL loss is given by Ex‖f ( x ) − g ( x ) ‖22 , ( 3 ) where g : RN → RM is a designable operator encapsulating domain knowledge of a specific application . We will incorporate some sophisticated setups in ( 3 ) such that f obtained by minimizing ( 3 ) can not merely be g. Although related theorems ( see later ) hold for any M , we mainly focus on practical image denoising applications with pseudo-target g ( x ) 6≈ y . 2.1 MOTIVATION This section empirically shows that understanding domain knowledge is important for designing g in the proposed SSRL loss ( 3 ) . The following camera image denoising examples demonstrate that well-designed g with good domain knowledge improves the denoising performance of learned f via ( 3 ) . Suppose that camera images are corrupted by salt-and-pepper noise . Consider two example setups for g ( · ) , median filtering and BM3D denoiser ( Mäkinen et al. , 2020 ) , denoted by median ( · ) and BM3D ( · ) , respectively . Figure 1 compares the denoising performance of minimum f ? with the two aforementioned g setups : f ? with median filtering significantly im- proved that with BM3D denoiser . This is not surprising , as median filtering is widely known to be effective in reducing salt-and-pepper noise ( Bovik , 2010 , §3.2 ) . This result emphasizes the importance of understanding domain knowledge of specific applications in proposed SSRL . 2.2 PRELIMINARIES . We first introduce the J -complement between two functions f and g : Definition 1 . For a given partition J = { J1 , . . . , JB } ( |J1| + . . . + |JB | = N ) of the dimensions of input x ∈ RN , functions f : RN → RM and g : RN → RM are called J -complementary , if f ( xJc ) does not depend on g ( xJ ) for all J ∈ J , where Jc denotes the complement of J , and ( · ) J denotes a vector restricted to J . That is , f and g use information from outside and inside of J to predict output and give pseudotarget , respectively . In denoiser learning ( where M = N ) , Definition 1 specializes to the J - invariance of f ( Batson & Royer , 2019 ) , by setting g = I . Incorporating Definition 1 into the SSRL loss ( 3 ) is a straightforward approach to avoid that optimal f is just g in ( 3 ) . The proposed SSRL framework assumes the followings : Assumption 1 ) xJ and xJc are conditionally independent given y , i.e. , p ( x|y ) = p ( xJ |y ) p ( xJc |y ) . Assumption 2 ) E [ g ( x ) |y ] = y . Assumption 3 ) f and g are ( Borel- ) measurable . In denoiser learning , Assumption 1 holds if noise in each subset J ∈ J is conditionally independent from that in Jc , given y . Assumption 1 can be satisfied in general regression NN learning , by adding small randomized perturbations ( independent of y ) to either J or Jc , similar to Moran et al . ( 2020 ) . Assumption 2 suggests a direction for designing g : suppose that x has non-zero-mean noise ; one then can design g to make noise zero-mean using the domain knowledge . Assumption 3 is satisfied if f and g are continuous . This condition is mild because many regression NNs f are continuous – where their modules , convolution , matrix-vector multiplication , rectified linear unit activation , max pooling , etc . are continuous – and one can design g with measurable or continuous function . Finally , observe that the proposed SSRL loss ( 3 ) can be rewritten by Ex‖f ( x ) − g ( x ) ‖22 = Ex , y‖f ( x ) − y‖22 + ‖g ( x ) − y‖22 − 2〈f ( x ) − y , g ( x ) − y〉 . ( 4 ) We aim to either remove or control the third term , incorporating the J -complement and/or Assumptions 1-3 introduced above . 2.3 SSRL USING DOMAIN KNOWLEDGE WITH J -COMPLEMENT . This section studies SSRL loss ( 3 ) minimization over f that is J -complementary of g. Our first main result shows that under Assumptions 1–3 , the SSRL loss ( 3 ) with the J -complement is the sum of the ordinary supervised learning loss and variance of g ( x ) − y , i.e. , the third term in ( 4 ) vanishes . Theorem 2 . Under Assumptions 1–3 , the SSRL loss ( 3 ) with the J -complement in Definition 1 becomes Ex‖f ( x ) − g ( x ) ‖22 = Ex , y‖f ( x ) − y‖22 + ‖g ( x ) − y‖22 . ( 5 ) The following equality similarly holds for any K ∈ K : Ex‖f ( x ) K − g ( x ) K‖22 = Ex , y‖f ( x ) K − yK‖22 + ‖g ( x ) K − yK‖22 , where K is a partition of { 1 , . . . , M } , and f ( · ) K and g ( · ) K denote f ( · ) and g ( · ) restricted to K , respectively . The optimal solution for ( 5 ) is given by f ? ( xJc ) = E [ g ( xJ ) |xJc ] = E [ y|xJc ] . ( 6 ) Proof . See Section A.1 in the appendix . The result ( 6 ) suggests another direction for designing g : we aim to design good g that can make g ( xJ ) close to y . Using such g , optimal solution of LHS in ( 6 ) , E [ g ( xJ ) |xJc ] , becomes close to its supervision counterpart , E [ y|xJc ] . Consequently , such g reduces the second term Ex , y‖g ( x ) −y‖22 in RHS of ( 5 ) , leading ( 3 ) closer to ( 1 ) . If g is ideal such that g ( xJ ) = y , the SSRL loss ( 3 ) becomes the usual supervised learning loss . In designing g , domain knowledge of specific application is crucial . Domain knowledge includes noise properties in x and pre-trained NN by existing self-supervised denoising , such as Noise2Self ( Batson & Royer , 2019 ) and Noise2Noise ( Lehtinen et al. , 2018 ) . Specifically , the proposed SSRL loss using the J -complement is given by Lind ( f ) =∆ ∑ J∈J Ex‖f ( xJc ) − g ( xJ ) ‖22 . ( 7 ) Figure 2 illustrates ( 7 ) with complementary checkerboard masks Jc and J , where f and g use almost equal amount of information , and its variant , where g uses much less information than f . The variant computes MSE only on J ∈ J ; in this setup , it is challenging for g to predict the entire image . Relation to previous self-supervised denoising works . In denoiser learning , the proposed SSRL loss ( 3 ) with the J -complement of f and g = I specializes to ( 2 ) with the J -invariance of f , i.e. , Noise2Self . Noise2Inverse ( Hendriksen et al. , 2020 ) and Neighbor2Neighbor ( Huang et al. , 2021 ) that emulate pairs of two independent noisy images by partitioning single measurements ( e.g. , CT ray measurements with independent noise and corrupted images with pixel-wise independent noise ) can be viewed as Noise2Self . Thus , SSRL loss ( 3 ) with the J -complement of f and g = I specializes to the aforementioned Noise2Noise-motivated self-supervised denoising methods . ( Noise2Noise also can be viewed by SSRL ( 3 ) with the setup above , by constructing x with stacking two independent noisy images , where an image is corrupted by two independent noise realizations . )

Summary: This paper proposed a self-supervised denoing method using domain knowledge. The proposed method seems somewhat like knowledge distillation, where a pre-trained denoiser or handcrafted designed noise models g can provide better initial results than original noisy images, and then a better denoiser f can be trained based on the results of g instead of noisy images. By forcing the complement of f and g, f will not be same with g, and the denoising performance is bossted.