A Holistic Approach to Cross-Channel Image Noise Modelingand its Application to Image Denoising
Seonghyeon Nam*
Yonsei University

Youngbae Hwang*
KETI

Abstract
Modelling and analyzing noise in images is a fundamental task in many computer vision systems. Traditionally, noise has been modelled per color channel assuming that the color channels are independent. Although thecolor channels can be considered as mutually independent
in camera RAW images, signals from different color channels get mixed during the imaging process inside the camera due to gamut mapping, tone-mapping, and compression.
We show the influence of the in-camera imaging pipeline onnoise and propose a new noise model in the 3D RGB space
to accounts for the color channel mix-ups. A data-driven
approach for determining the parameters of the new noisemodel is introduced as well as its application to image denoising. The experiments show that our noise model represents the noise in regular JPEG images more accurately
compared to the previous models and is advantageous inimage denoising.

1. Introduction
Noise is one of the most fundamental problems in computer vision and image processing. In computer vision literature, many works mention noise as one of the main sourcesthat explain the error of their systems and emphasize the necessity of establishing robustness against noise. But whatreally is image noise and how can we explain it?
Informally, noise explains the uncertainty of the lightmeasurement in an image. A low noise value would indicate that the observed intensity value is highly likely to be
the ground truth intensity and vice versa. Since many computer vision algorithms rely on accurate light measurement,
it is important to have a noise model that explains the properties of image noise accurately.
What is interesting is that all of the existing image noisemodels fall short of explaining what is really happening
to noise in the images that most people (both regular consumers and vision researchers) use today - JPEG images* Authors contributed equally to this work.

Yasuyuki Matsushita
Osaka University

Seon Joo Kim
Yonsei University

in the sRGB color space. A key assumption used in existing noise models is that the noise is independent between
different color channels. While this channel independency
is valid for linear vision cameras or RAW images, which
are unprocessed sensor-level measurements, the assumption
breaks down as the R,G,B values are heavily mixed during the in-camera image processing [3, 15]. Furthermore,
image compression, typically in JPEG format, significantly
affects the noise characteristics; however, the effect of thecompression noise has not been explicitly considered in the
past1 .
This work attempts to seek deeper understanding about
image noise and to come up with a better explanation about
the noise compared to the previous noise models. Previous
noise models either only fit to a limited number of cases,or they were only validated with synthetic images createdwith their own models. Even in the image denoising works,
the quantitative performance of denoising is evaluated withsynthetic images created with a per channel independent
Gaussian model [5], which does not describe the noise inreal photographs as we will see later in this paper. Therefore, we argue for a new image noise model that better explains the properties of noise in images that most people use
today.
The contributions of this paper are as follows:• We provide observations and analysis on the effect of
the in-camera imaging process on noise and introduce
a new cross-channel noise model. We develop a colordepenent noise model in the 3D RGB space that cansimultaneously take into account the correlation between the color channels and the JPEG compression
effect.
• We further propose a data-driven approach for automatically determining the noise in the 3D RGB spacefrom observed color images. Specifically, we use a
simple multi-layer perceptron (MLP) to infer the parameters of the noise model. Note that we use the neu1 Note that our problem is to analyze the effect of compression on thenoise level, which is different from dealing with the blocky JPEG noise or
artifacts[18].

ral network (NN) to compute the noise model, which
is fundamentally different from using NNs for image
denoising [25, 2].
• We validate our model and the parameter estimation
method using real images instead of synthetic images
and show that applying our new noise model can improve the image denoising performance.

2. Related Work
The most common noise model used in computer vision
is the channel-independent Gaussian model [21, 22, 24] because of its simplicity. However, the Gaussian noise model
is found inflexible to describe the actual noise in real images, and therefore, several more sophisticated noise models have been recently proposed. In [6], Foi et al. proposeda Poissonian-Gaussian noise model for single-image raw
data to consider signal-dependent and signal-independent
noise components separately. Granados et al. [8] presented
a noise model that takes into account both temporal and thespatial noise for reconstructing high-dynamic range (HDR)
images. Their weighting function produces statistically optimal estimates under the assumption of compound Gaussian noise. Hwang et al. [12] presented a difference basednoise model using the Skellam distribution to represent the
distribution of intensity differences. They showed that thedifference-based modeling has a more significant linear relationship between the intensity and the noise parameters.
The methods described above all operate with RAW images or images from a linear vision camera [12]. Whilenoise modeling of RAW image data is useful for some specific tasks, most of the images used in computer vision go
through an in-camera imaging pipeline. In a seminal workby Healey and Kondepudy [10], five main sources for image noise in the camera imaging process were identified as
photon shot, fixed pattern, dark current, readout, and quantization noise, and they presented a statistical model in which
the variance of noise is linearly proportional to the observed
intensity.
In [19], Liu et al. presented a more general noise modelthat fits the in-camera imaging pipeline, which includes processes such as white balancing and camera response functions (gamma correction). They used the in-camera imaging
model from [23] and defined the noise level function (NLF)
as the variation of standard deviation of the noise distribution according to image brightness. Using the space of
camera response functions [9], they used a Bayesian MAP
estimation to infer the NLF from a single image. In spiteof insufficient information to estimate image noise, their
method showed good performance when applied to noise
removal [19] and image deblurring [13].
While the methods described above are effective, eachcolor channel is still treated independently in these works.

In [15], Kim et al. described a new in-camera imaging
model that well fits modern cameras by showing the effectof the gamut mapping step, which is a 3D nonlinear mapping (RGB to RGB). Their work intrinsically indicates thelimitation of channel-independent noise modeling becauseof the mixture of color channels due to the gamut mappingand color space transformations. In addition to such mixtures, JPEG compression process [4, 18] adds further mixing of color channels. Our goal is to accurately model anddetermine such cross-channel image noise.

3. Noise Model in the 3D RGB SpaceThis section analyzes the effects of the in-camera imaging process and JPEG compression on noise and propose anoise model that can accurately represent them.

3.1. Noise through the In-camera Imaging Process
Figure 1 shows the influence of each procedure in the incamera imaging pipeline on noise. The top row of the figure lists the imaging steps described in [15]. To verify how
the noise characteristics are altered through the procedures,
we first took a RAW image with many homogeneous color
patches with a Canon EOS-5D Mark III camera. We thensimulated the imaging pipeline using the calibrated camera parameters from [15] and observed the changes of thenoise distributions. As shown in the first plots in Fig. 1(b) and (c), both the Skellam parameter [12] and the variance [19] linearly increase with the intensity value in the
RAW image. While this linear relationship is still largely
maintained after going through the demosaicing and whitebalancing/linear color transformations, the linear relationship drastically breaks down with the gamut mapping andtone mapping processes.
We ran another experiment to see how the mix-up of
R,G,B channel values (mainly due to 3D gamut mapping)influence the image noise. We took images as shown in
Fig. 2 (a) with a Nikon D800 camera, which records images
in three different formats: RAW, uncompressed TIFF, andJPEG. The uncompressed TIFF image allows us to analyze
the effect of the whole imaging pipeline without the compression effect. With these different formats of images, we
computed the covariance matrices for each pixel from 1,000
temporal images of a static scene, some of which are shownin Fig. 2 (b-d). They show the magnitude of elements inthe covariance matrix; from the variances of R,G,B to thecovariance R/G, R/B, and G/B. At the RAW level, the noise
in different channels are indeed independent. However, we
can observe that the covariance scores significantly increaseas the image goes through the imaging pipeline, and reachthe point where we cannot simply ignore the cross-channel
noise.

Demosaicing

White Balance andColor Space Transform

RAW image RGB image

Channel-wise 3x3 Linear
Scalingtransform

Camera RAW

Gamut
mappingTone
mapping

JPEG compression

0

100 200
R intensity

10
0

0

30
20
10
0

100 200
R intensity

0

100 200
R intensity

30
20
10
0

0

100 200
R intensity

30

Skellam Param.

020

Skellam Param.

1030

Skellam Param.

20

Skellam Param.

30

Skellam Param.

Skellam Param.

(a) In-camera imaging pipeline

20
10
0

0

100 200
R intensity

30
20
10
0

0

100 200
R intensity

0

100 200
R intensity(b) Changes in the Skellam distribution of noise through the pipeline00100 200
R intensity

10
0

0

20
10
0

100 200
R intensity

0

100 200
R intensity

20
10
0

0

100 200
R intensity

30
Variance

20

30
Variance

1030
Variance

20

30
Variance

30
Variance

Variance

30

20
10
0

0

100 200
R intensity

20
10
0

(c) Changes in the noise variance distribution through the pipeline

pixel 3Magnitude(a) Test image
70
60
50
40
30
20
10
0 R

RAW
Before compression
After compression

G

B

R/G R/B G/B

Covariances(c) Covariance changes of pixel 2

120
100
80
60
40
20
0 R

RAW
Before compression
After compression

G

B

R/G R/B G/B

Covariances(b) Covariance changes of pixel 1

Magnitude

pixel 1
pixel 2Magnitude

Figure 1. In-camera imaging pipeline and changes in noise distribution through the pipeline. (a) shows block diagram sequences of incamera imaging pipeline. (b) and (c) show the changes in the Skellam and the variance distribution through the pipeline, respectively.
Noise chracteristic drastically changes with the gamut mapping and the tone mapping processes.

70
60
50
40
30
20
10
0 R

RAW
Before compression
After compression

G

B

R/G R/B G/B

Covariances(d) Covariance changes of pixel 3

Figure 2. Covariance magnitude changes of some pixels after incamera imaging pipeline and JPEG compression. (a) is the test
image scene which was captured in a RAW, an uncompressed TIFF
and a JPEG format by Nikon D800. (b)-(d) shows the changes in
the covariance terms during the imaging pipeline.

3.2. Effect of the JPEG Compression on Noise
Figures 1 and 2 show that JPEG compression has a significant effect on the noise characteristics. With a typical
JPEG compression, an image is compressed by dividing theimage region into 8×8 patches and processing those patches
separately. Therefore, the level of compression may differ patch by patch, and the influence of the compression tonoise would also depend on the patch content. The noisecharacteristics depend not only on a single pixel RGB value
but also on other pixels in the patch. Therefore, even for aparticular RGB value, the noise characteristics would vary
according to the other pixels in the patch.
The examples of the effect of the JPEG compression onnoise are visualized in Fig. 3. After recording 1,000 JPEG
images of a static scene, we fit covariance matrices to different pixels. Figure 3 shows the covariance matrices ofseveral pixels that have the same RGB value in the form of
ellipsoids. As expected, the pixels (having the same RGB
value) that are located in similar patches show the similar covariance structures (Fig. 3 (a)), while pixels in visually different patches exhibit diverse covariance structures(Fig. 3 (b)). These examples indicate that a good noisemodel should be able to explain noise’s dependence on boththe scene and the pixel color.

120

100

110

90

100

80

B

B

Group 2 (White)

Group 1 (Red)

110

70

90
80

Group 2 (White)

60
50
80
60
G 40
170
20 110 120 130 140 150 160

Group 1 (Red)

70
60
150

G

100 110

120

R

160

150

140

130

R

(a) Similar patches(b) Different patches

25

20

20

15
10
5
0

130 140 150 160 170

R intensity

10

15

10
5
0

20

0

5

10

15
10
5
0

15

20
15
10
5
0

0 2 4 6 8 10 12 14

0

2

4

6

8 10 12 14Squared Mahalanobis
Distance

Squared Mahalanobis
Distance

Squared Mahalanobis
Distance

Squared Mahalanobis
Distance(b) Q-Q plot of pixel 1(c) Q-Q plot of pixel 2(d) Q-Q plot of pixel 3(e) Q-Q plot of pixel 470 80 90 100 110 120

G intensity

75

75
105

65

95

B

65

B

700
600
500
400
300
200
100
0

G

800
700
600
500
400
300
200
100
0

5

Number of Pixels

700
600
500
400
300
200
100
0 120

Number of Pixels

Number of Pixels(a) Test image

0

15

25

Chi-Square
Quantile

25

20

Chi-Square
Quantile

25

Chi-Square
Quantile

pixel 1
pixel 3 pixel 2 pixel 4

Chi-Square
Quantile

Figure 3. JPEG compression effect on 8×8 patch. (a) shows the ellipsoids of covariance matrices of the same RGB value in similar patches
have similar shapes. (b) shows the ellipsoids vary according to the patch the color is in. The ellipsoids represent 95% confidence intervalof distribution.

55

85
45

45

35

35 75 80 85 90 95100 105110

75
30 40 50 60 70 80

B intensity

125

135

145

155

55

R

125

135

145
R

155

G

(f) Univariate and bivariate Gaussian distributions of pixel 4

Figure 4. Multivariate Gaussian fitting test on real data. (a) A test image captured 10, 000 times using a Samsung Galaxy S6 smartphone
camera (ISO 800, 80% compression). (b)-(e) Multivariate Q-Q plot of four selected pixels. The linear relationships indicate that the
samples follow multivariate Gaussian distribution [11]. (f) Shows the color distribtion of pixel 4. It empirically shows that the noise should
be modelled as a multivariate (3D) Gaussian distribution.

3.3. Noise Model in the 3D RGB SpaceTo properly account for the image noise discussed in previous subsections, we propose a noise model that is characterized by a covariance matrix in the RGB color space.
Based on the observations as shown in Fig. 4, we model thenoise as the signal-dependent multivariate Gaussian distribution. In Fig. 4 (b)-(e), the Q-Q plots show the orderedsquared Mahalanobis distance of samples versus the estimated quantiles from a chi-square distribution with 3 degrees of freedom. The linear relationships as shown on theplots mean that the samples follow a multivariate Gaussian
distribution [11]. The empirical example in Fig. 4 (f) also
supports for the multivariate (3D) Gaussian model.
In addition to the multivariate Gaussian model, we also
consider the local patch contents in our model to deal withthe content dependency due to the JPEG compression. With
this consideration, the noise of an image pixel (x, y) is determined by the (R,G,B) values of the pixel I(x, y) as wellas the 8 × 8 patch in which the pixel (x, y) is located. We
ignore the pixel position in the patch for simplicity.
Putting it altogether, our noise model in the 3D RGB
space is written mathematically as:I(x, y) = I(x, y)+N 0, Σ(I(x, y), pxy ) , 2
σr σrg σrb(1)
Σ(I(x, y), pxy ) = σrg σg2 σgb  ,σrb σgb σb2where I(x, y) is the true intensity of I(x, y), and pxy is the
8 × 8 color patch value. N (0, Σ(I(x, y), pxy )) is the zeromean multivariate Gaussian distribution of noise with thecovariance matrix Σ(I(x, y), pxy ) that is the function of thetrue intensity I(x, y) and its corresponding patch pxy .

4. Data-driven Noise Estimation Algorithm
In theory, the proposed model in Eq. (1) should be defined for every possible patch with each pixel value in the

3D RGB space, which is computed as 256(8×8×3) . It is unrealistic to compute and store the noise model parameters
for all those colors and patch values. Therefore, we employ
a data-driven approach based on a multi layer perceptron(MLP) for determining the noise parameters of pixels in agiven image.

4.1. Data Collection
For the MLP to perform well, collecting a large numberof quality data is essential. We captured training images for
11 static scenes, 500 JPEG images per scene, and computed
the mean image of each scene to generate the ground truth
noise-free images.2 Some of the captured scenes are shownin Fig. 5. For each dataset, the covariance for each pixel,
which is computed using the temporal stack of images, is
fed into the system for training along with its (R,G,B) values
and the 8 × 8 × 3 patch information. The training is done
per camera model and settings such as ISO, and the total
amount of data per set is about 13 million patches for animage with resolution 7360 × 4912 (98% of the data is used
for the training and 2% for the validation).

4.2. MLP based Noise Estimation Method
A multi-layer perceptron (MLP) is a feed-forward neural network that trains a nonlinear transformation of vectorvalued input layer. The input layer is mapped to the output
layer via several hidden layers. Formally, MLPs are definedasx(n+1) = g(b(n) + W(n) x(n) ),
(2)
(n+1)

(1)

where xis the value of (n + 1)-th layer (x is the
input layer). W(n) and b(n) are trainable weights and abias. For the nonlinear activation function g, a sigmoid, atanh, or ReLU [16] is used. When n is more than 2, theMLPs can be used as a universal approximator, which isable to learn any nonlinear mapping. Therefore, we use an
MLP with our training data to find the complex nonlinear
mapping from the RGB value of a pixel and its surrounding
patch to its corresponding covariance matrix.
At first, we had expected that the MLP for our problem
would require a large number of layers and units. However,
we have found that a single hidden layer MLP is enoughto learn the nonlinear correlation of our data. Specifically,
the structure of our MLP is (195, 200, 6), which are thenumber of units in each layer3 . Note that the number ofparameters of the MLP is considerably small compared tothe original problem space where we need a covariance matrix per combination of color and patch (256(8×8×3) ). With
the small number of parameters and its regression power,
2 Using the mean of temporal images has been used as noise-free image [19, 20].
3 The input layer is the concatenation of RGB color (3) and vectorized
8 × 8 color patch (192).

Figure 5. Some samples of the scenes in our dataset.

MLP serves as an efficient and accurate modeling tool forour noise modelling problem.
The trained MLP can be seen as a regressor, which canpredict the covariance matrix for any given (R,G,B) value
and its patch. We can formally express the MLP-basednoise estimation method asΣ(I(x, y), pxy ) = h(f (I(x, y), pxy ))),

(3)

where the input I(x, y) and pxy are the same as thosein Eq. (1). Because the output covariance matrix is symmetric, we only use the half of the matrix. h is a function thatconverts the 6 dimensional output to the 3 × 3 covariancematrix. To ensure that the covariance matrix is positivedefinite, we replace zero or negative eigenvalues of the matrix with a small positive value. Our MLP f is trained byminimizing the following cost function:L=

1 X −1kh (Σ(Ii (x, y), pxy,i ))
N i

(4)

2

− f (Ii (x, y), pxy,i )k ,where h−1 is the inverse function of h. In our implementation, we use the ReLU as the activation function andstochastic gradient descent [17] as the optmization method.
Also, the learning rate was set to 0.0001 and the training
iteration was set to a million with 64 batch size. On average, the learning took 20 minutes with the machine withNVIDIA GTX Titan X GPU.

5. Experiments
5.1. Experimental Results
To verify the accuracy of our noise estimation, we compare the estimated noise covariance with the ground truth
covariance computed from training set. We used the following distance measure introduced in [7] as the similarity
between covariance matrices:
v
u n
uX 2
d(A, B) = t
ln λi (A, B),(5)i=1

Image #Camera1
2
3

Camera Settings
ISO

JPEG

1600

Normal3200

Normal6400

Normal3200

Normal3200

Fine

Nikon D800
Average4
5
6

Nikon D800
Average7
8
9

Nikon D800
Average10
11
12

Nikon D600
Average13
14
15

Canon 5D Mark III
Average

NLF [19] from GT
MeanMedian
4.86
4.88
4.60
4.49
5.61
5.41
5.02
4.93
5.32
5.19
5.60
5.46
5.50
5.50
5.47
5.38
5.80
5.72
6.05
6.16
6.04
6.04
5.96
5.97
5.99
6.05
5.60
5.66
5.07
4.98
5.55
5.56
3.72
3.39
4.75
4.65
4.80
4.81
4.42
4.28

Ours
Mean
1.82
1.71
3.16
2.23
2.39
2.03
2.31
2.24
1.99
2.28
2.06
2.11
1.68
1.83
1.59
1.70
2.36
2.75
2.66
2.59

Median
1.76
1.58
2.43
1.92
1.89
1.89
1.92
1.90
1.80
2.20
1.97
1.99
1.63
1.70
1.44
1.59
2.24
2.55
2.53
2.44

Table 1. Noise model evaluations for test images shown in Fig. 6. The values are the median and the mean value of covariance matrix
errors in Eq. 5. Small value means better performance. Regardless of the scene, ISO and the camera, our model represent the noise moreaccurately.

(a)

(b)

(c)

(d)

(e)

Figure 6. Test images used in Table 1. From left to right, 1, 4, 7, 10, and 13 (bold).
50
40
30
20
10
00

Red

35
30
25
20
15
10
5
50 100 150 200 250 00

Green

35
30
25
20
15
10
5
50 100 150 200 250 00

Blue

50 100 150 200 250

Figure 7. The NLF computed from our ground truth. The points
are minimum variances of each intensity. We obtain the NLF byfitting the intensity-variance pairs of each channel.

where λi (A, B) is the i-th generalized eigenvalue of Ax =λBx.
Since no previous noise model can be fit to real dataas ours, direct comparisons with previous models are difficult. The closest model that can be used is the noise level
function (NLF) in [19], so we compared our model to NLF
as shown in Table 1. For NLF, we computed NLFs separately for each color channel by obtaining the lower boundof intensity-variance pairs from 500 static images, whichis the upper bound of noise as shown in Fig. 7. From thevariances of three channels, we generated the covariancematrix with zero off-diagonal terms. The table validates ourmodel esstimation process and also shows that it represents
the noise in real images better than the NLF.
Figure 8 shows a qualitative analysis of our new noisemodel and its parameter estimation. It verifies that our multivariate Gaussian model which is an ellipsoid fits well withthe observed data samples.

5.2. Image Denoising Application
To study the effectiveness of our model and its estimation
method, we apply our method to image denoising, which
is a process of estimating the true intensity corresponding
to the scene radiance from the noisy observations. For
image denoising, we adopt the Bayesian non-local means(BNLM) method in [14], which is an extension of the nonlocal means denoising algorithm [1] with more robust similarity measures. Non-local means algorithms denoise the

110

150

70

145

100

60

140

90

135

50

B
110

120

R

130 100

140

120

G

160

70
60

115

100

10
0
130 140

125
120

20

100

B

40
30

90

80

130

B

B

115
110
105
100
95
90
85
80
75
70
65
80

50
150

160

170

180

190 200 0

R

RGB=(106, 130, 90)

RGB=(166, 62, 33)

110

G

50

105
100

110 120

130

140

R

150

160 120

140

160

G

RGB=(134, 147, 123)

180

40

50

100

R

150 80

90

110

100

120

130

140

G

RGB=(117, 108, 76)

Figure 8. Some noise estimation results of our model. Each shows two covariance ellipsoids which are our estimation (red) and the groundtruth (white). Also, black dots are actual color points from 500 temporal images. Our estimation is quite accurate compared with theground truth and fits well with the noise distribution of real JPEG image.

images based on the nature of self-similarity in natural images.
Let I be the noisy observation of a pixel (r, g, b) and I be
the denoised color. We can apply the BNLM to compute I,
which is expressed as√2√d2 (i,j)− 2M −1

P

− 12

P

− 21

j∈Ni eIi =j∈Ni eIj√2√d2 (i,j)− 2M −1,(6)

6. Conclusion

where i and j are pixel positions, Ni is the set of neighboring pixels of i, and M is the number of pixels in patch times
the number of channels. The squared dissimilarity measured2 (i, j) is originally the squared Euclidean distance normalized by σ 2 . But, in our problem it is changed to the squared
Mahalanobis distance, which is expressed as
d2 (i, j) = (IPi − IPj )T Σ−1
Pi (IPi − IPj )X=(Ii+d − Ij+d )T Σ−1
i+d (Ii+d − Ij+d ),the qualitative examples shown in Figure 9. These experiments support the need for a new image noise model thatare both color and content dependent. We would also liketo point out that the experiments in Table 2 and Figure 9 aremeaningful as the first noise evaluation on real image data
(to the best of our knowledge) as most previous noise evaluations were done on either RAW images or on simulated
data.

(7)

d∈P

where IPi and IPj are M dimensional vectorized patches
whose center pixel is i and j, respectively and ΣPi is thecovariance matrix of IPi . d2 (i, j) can be rewritten as thesum of each pixel distance, where P is the set of disparities
d from center pixel within patch. For each pixel i + d, Σi+d
is the 3 × 3 covariance matrix of the pixel.
Table 2 and Figure 9 show the experimental resultsof image denoising both quantitatively and qualitatively.
BNLM denoising with our noise model is compared with
BM3D [5], original BNLM [14], and BNLM with NLF
noise model [19]. For BM3D and original BNLM, σ is computed by averaging the ground truth (GT) σ of every pixels
in the whole image. For all noise models applied to BNLM,
5 × 5 patch and 35 × 35 search window are used, which are
sufficient for both the quality and the time complexity.
For vast majority of cases, denoising using our noisemodel outperformed other models quantitatively. The quality of denoising using our noise model is more apparent inIn this paper, we presented a new noise model in the
3D RGB space considering both the cross-channel dependency and the scene dependency of the noise in consumer
camera images. We empirically showed that the noise characteristics change through the in-camera imaging process
and the JPEG compression. The observations showed thata new noise model and the estimation method are necessary as previous noise models cannot explain those factors.
To estimate the noise, we collected training image sets forvarious scenes and proposed a data-driven noise estimation
algorithm using a multi layer perceptron. We validated our
method using real images and applied it to image denoising,which showed large improvement over the previous work.
In the future, we are interested in applying our work to othercomputer vision applications including radiometric calibration and HDR imaging.

Acknowledgement
This work was supported by Global Ph.D. Fellowship Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2015H1A2A 1033924), the center for Integrated Smart Sensors funded by the Ministry of Science, ICT & Future Planning as Global Frontier Project(CISS-2013M3A6A6073718), and Institute for Information & communications Technology Promotion (IITP) grant
funded by the Korea government (MSIP) (B0101-16-0552,
Development of Predictive Visual Intelligence Technology).

Camera Settings

Image #
1
2
3

Noisy Image

CameraISO

Nikon D800

1600

Average4
5
6

Nikon D800

3200

Average7
8
9

Nikon D800

6400

Average10
11
12

Nikon D600

3200

Average13
14
15

Canon 5D Mark III
Average

3200

PSNR
35.47
35.71
34.81
35.33
33.26
32.89
32.91
33.02
29.63
29.97
29.87
29.82
33.28
33.77
35.21
34.09
37.00
33.88
33.83
34.90

SSIM
0.957
0.954
0.989
0.967
0.978
0.988
0.951
0.972
0.862
0.921
0.914
0.899
0.968
0.990
0.939
0.966
0.976
0.983
0.977
0.979

BM3D [5]σ from GT
PSNR
SSIM
36.15
0.964
36.57
0.964
35.47
0.991
36.06
0.973
34.00
0.982
33.43
0.989
33.53
0.957
33.65
0.976
29.97
0.872
30.33
0.928
30.21
0.921
30.17
0.907
33.70
0.972
34.33
0.992
35.75
0.954
34.59
0.973
37.79
0.984
34.34
0.986
34.27
0.979
35.47
0.983

σ from GT
PSNR
SSIM
37.59
0.980
39.42
0.990
37.40
0.995
38.14
0.988
38.10
0.992
35.17
0.995
38.33
0.987
37.20
0.991
33.35
0.954
32.25
0.967
32.67
0.962
32.76
0.961
34.74
0.978
36.20
0.995
40.57
0.987
37.17
0.987
38.44
0.986
35.27
0.988
34.78
0.982
36.16
0.985

BNLM [14]NLF [19] from GT
PSNR
SSIM
36.61
0.972
37.61
0.981
35.91
0.993
36.71
0.982
35.99
0.988
33.84
0.991
35.92
0.976
35.25
0.985
31.91
0.933
30.94
0.950
31.13
0.940
31.33
0.941
34.27
0.975
35.54
0.995
38.42
0.979
36.08
0.983
37.97
0.987
34.39
0.986
34.13
0.979
35.50
0.984

Ours
PSNR
SSIM
37.99
0.982
40.36
0.992
38.30
0.996
38.89
0.990
39.01
0.993
36.75
0.996
39.06
0.990
38.27
0.993
34.61
0.963
33.21
0.970
33.22
0.970
33.68
0.968
34.98
0.979
35.95
0.995
41.15
0.989
37.36
0.988
38.37
0.988
35.37
0.990
34.91
0.983
36.22
0.987

Table 2. Denoising performance comparisons. In vast majority of cases, our noise model outperforms other models for denoising in boththe PSNR and the SSIM.

Noisy Image

BM3D + σ

Noisy Image

BM3D + σ

BNLM + σ

BNLM + NLF

BNLM + σ

BNLM + NLF

BNLM + Ours

Mean ImageBNLM + Ours

Mean Image

(a)

(b)

Figure 9. Qualitative denoising performance comparisons. (a) and (b) are data 7 and 8 in Table 2, respectively. In addition to the quantitativevalues, we can observe that denoising with our noise model outperform others in this example.

References[1] A. Buades, B. Coll, and J.-M. Morel. A non-local algorithm
for image denoising. In Proc. IEEE Conference on Computer
Vision and Pattern Recognition, 2005.
[2] H. Burger, C. Schuler, and S. Harmeling. Image denoising:Can plain neural networks compete with BM3D? In Proc.
IEEE Conference on Computer Vision and Pattern Recognition, 2012.
[3] A. Chakrabarti, Y. Xiong, B. Sun, T. Darrell, D. Scharstein,
T. Zickler, and K. Saenko. Modeling radiometric uncertainty for vision with tone-mapped color images. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
36(11):2185–2198, 2014.
[4] I. Choi, S. Kim, M. Brown, and Y.-W. Tai. A learning-based
approach to reduce JPEG artifacts in image matting. In Proc.
IEEE International Conference on Computer Vision, 2013.
[5] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing,16(8):2080–2095, Aug 2007.
[6] A. Foi, M. Trimeche, V. Katkovnik, and K. Egiazarian.
Practical poissonian-gaussian noise modeling and fitting forsingle-image raw-data. IEEE Transactions on Image Processing, 17(10):1737–1754, 2008.
[7] W. Förstner and B. Moonen. A metric for covariance matrices. In Geodesy-The Challenge of the 3rd Millennium, pages
299–309. Springer, 2003.
[8] M. Granados, B. Ajdin, M. Wand, and C. Theobalt. Optimal
HDR reconstruction with linear digital cameras. In Proc.
IEEE Conference on Computer Vision and Pattern Recognition, 2010.
[9] M. Grossberg and S. Nayar. Modeling the space of camera
response functions. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 26(10):1272–1282, Oct 2004.
[10] G. E. Healey and R. Kondepudy. Radiometric CCD camera calibration and noise estimation. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 16(3):267–276,
1994.
[11] M. J. R. Healy. Multivariate normal plotting. Journal of
the Royal Statistical Society. Series C (Applied Statistics),
17(2):157–161, 1968.
[12] Y. Hwang, J. S. Kim, and I. S. Kweon. Difference-based
image noise modeling using skellam distribution. IEEE
Transactions on Pattern Analysis and Machine Intelligence,34(7):1329–1341, 2012.
[13] N. Joshi, C. L. Zitnick, R. Szeliski, and D. Kriegman. Image deblurring and denoising using color priors. In Proc.
IEEE Conference on Computer Vision and Pattern Recognition, pages 1550–1557. IEEE, 2009.
[14] C. Kervrann, J. Boulanger, and P. Coup. Bayesian non-local
means filter, image redundancy and adaptive dictionaries fornoise removal. In Proc. Conf. Scale-Space and Variational
Meth., 2007.
[15] S. J. Kim, H. T. Lin, Z. Lu, S. Susstrunk, S. Lin, and M. S.
Brown. A new in-camera imaging model for color computer vision and its application. IEEE Transactions on Pat-

tern Analysis and Machine Intelligence, 34(12):2289–2302,
2012.
[16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Proc. Advances in Neural Information Processing Systems,
2012.
[17] Y. LeCun, L. Bottou, G. Orr, and K. Muller. Efficient backprop. In G. Orr and M. K., editors, Neural Networks: Tricksof the trade. Springer, 1998.
[18] Y. Li, F. Guo, R. T. Tan, and M. S. Brown. A contrast enhancement framework with JPEG artifacts suppression. In
Proc. European Conference on Computer Vision, 2014.
[19] C. Liu, R. Szeliski, S. B. Kang, C. L. Zitnick, and W. T.
Freeman. Automatic estimation and removal of noise froma single image. IEEE Transactions on Pattern Analysis andMachine Intelligence, 30(2):299–314, 2008.
[20] X. Liu, M. Tanaka, and M. Okutomi.
Practical
signal-dependent noise parameter estimation from a single
noisy image. IEEE Transactions on Image Processing,23(10):4361–4371, Oct 2014.
[21] N. Ohta. A statistical approach to background subtraction
for surveillance systems. In Proc. IEEE International Conference on Computer Vision, pages 481–486, 2001.
[22] P. L. Rosin. Thresholding for change detection. In Proc.
IEEE International Conference on Computer Vision, pages
274–279, 1998.
[23] Y. Tsin, V. Ramesh, and T. Kanade. Statistical calibration of
ccd imaging process. In Proc. IEEE International Conference on Computer Vision, July 2001.
[24] C. Wren, A. Azarbaygaui, T. Darrel, and A. Pentland.
Pfinder: Real-time tracking of the human body. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
19:780–785, July 1997.
[25] J. Xie, L. Xu, and E. Chen. Image denoising and inpainting with deep neural networks. In Proc. Advances in Neural
Information Processing Systems, 2012.