NMF vs. ICA for Light Source Separation under AC Illumination
Ruri Oya, Ryo Matsuoka and Takahiro Okabe
Department of Artificial Intelligence, Kyushu Institute of Technology, Japanokabe@ai.kyutech.ac.jp

Keywords:

Light Source Separation, Alternating Current, Flicker, NMF, ICA.

Abstract:

Artificial light sources powered by an electric grid change their intensities in response to the grid’s alternating
current (AC). Their flickers are usually too fast to notice with our naked eyes, but can be captured by using
cameras with short exposure time settings. In this paper, we propose a method for light source separation
under AC illumination on the basis of Blind Source Separation (BSS). Specifically, we show that light source
separation results in matrix factorization, since the input images of a scene illuminated by multiple AC lightsources are represented by the linear combinations of the basis images, each of which is the image of the scene
illuminated by only one of the light sources, with the coefficients, each of which is the intensity of the lightsource. Then, we make use of Non-negative Matrix Factorization (NMF), because both the pixel values of thebasis images and the intensities of the light sources are non-negative. We experimentally confirmed that our
method using NMF works better than Independent Component Analysis (ICA), and studied the performanceof our method under various conditions: varying exposure times and noise levels.

1

INTRODUCTION

Artificial light sources in our surroundings are often powered by an electric grid, and therefore theirintensities rapidly change in response to the grid’s
alternating current (AC). Their flickers are usually
too fast to notice with our naked eyes, but canbe captured by using cameras with short exposure
time settings (Vollmer and Möllmann, 2015). Such
rapid flickers could make auto white balance unnatural (Hsu et al., 2008).
Sheinin et al. (Sheinin et al., 2017) propose amethod for light source separation under AC illumination. Their proposed method decomposes an image
sequence of a scene illuminated by multiple AC lightsources into the images of the scene, each of which isilluminated by only one of the light sources, and thetemporal intensity profiles of the light sources. They
make use of their self-build coded-exposure camera
synchronized to AC and the dataset of temporal intensity profiles of various light sources, and then achieve
light source separation even for dark scenes such as acity-scale scene at night. Later, Sheinin et al. (Sheinin
et al., 2018) achieve light source separation under AC
illumination by using consumer rolling-shutter cameras, but still require the dataset of temporal intensity
profiles of various light sources.
In this paper, we propose a method for light source
separation under AC illumination on the basis of

Blind Source Separation (BSS). Specifically, we showthat light source separation results in matrix factorization, since the input images are represented by the linear combinations of the basis images, each of whichis the image of the scene illuminated by only one of
the light sources, with the coefficients, each of whichis the intensity of the light source. Then, we make useof Non-negative Matrix Factorization (NMF) (Berryet al., 2007) for BSS, because both the pixel valuesof the basis images and the intensities of the lightsources are non-negative.
We conducted a number of experiments andconfirmed that our proposed method using NMF
works better than Independent Component Analysis (ICA) (Hyvärinen and Oja, 2000). In addition,
we studied the performance of our method, which isbased on fast flickers of light sources’ intensities, under various conditions: varying exposure times andnoise levels.
Our proposed method based on BSS does not require the dataset of light sources’ temporal intensity
profiles nor the self-build camera synchronized to AC
in contrast to Sheinin et al. (Sheinin et al., 2017;
Sheinin et al., 2018). Therefore, our method is applicable to image sequences captured by using consumer
cameras and applicable to unknown light sources thatare not included in the dataset, although it is not suitedfor dark scenes because the images captured by using
those cameras have low S/N ratios in general.

460
Oya, R., Matsuoka, R. and Okabe, T.
NMF vs. ICA for Light Source Separation under AC Illumination.
In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP, pages
460-465
ISBN: 978-989-758-402-2
Copyright © 2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved

NMF vs. ICA for Light Source Separation under AC Illumination

The main contributions of this study are twofold.
First, we propose a method for light source separation
under AC illumination on the basis of BSS; it does not
require the dataset of light sources’ temporal intensity profiles nor the self-build camera synchronized to
AC. Second, we experimentally confirmed that NMF
works better than ICA for light source separation under AC illumination, and studied the behavior of our
proposed method using NMF under varying exposure
times and noise levels.

2

PROPOSED METHOD

According to the superposition principle of light, animage of a scene taken under multiple light sources
is represented by a convex combination of the basis
images, each of which is the image taken under oneof the light sources. Specifically, the pixel value i pc fof an input image sequence at the p-the pixel (p =1, 2, 3, ..., P) in the c-th channel (c = 1, 2, 3) and in
the f -th frame ( f = 1, 2, 3, ..., F) is represented asNi pc f = ∑ b pcn an f .

(1)

n=1

Here, N is the number of light sources, b pcn is the
pixel value of the n-th basis image at the p-th pixel
and in the c-th channel, and an f is the intensity of then-th light source in the f -th frame.
We can rewrite eq.(1) in a matrix form as
I = BA,

Figure 1: The first three images of the synthetic image sequences of (a) Scene 1, (b) Scene 2, and (c) Scene 3.

subject to the conditions that all the elements of thematrices B and A are non-negative. Here, || ||Fr stands
for the Frobenius norm of a matrix.
Since the matrix I is represented by the productof the matrices B and A , there are the ambiguities inscale and order between those matrices. In addition,
since B A = BU −1U A , the ambiguity represented by
the N × N regular matrix U , that keeps all the elements of the matrices BU −1 and U A non-negative,
could occur.

3

EXPERIMENTS(2)

where I is the 3P × F matrix consisting of the pixel
values of the input image sequence, B is the 3P × N
matrix consisting of the pixel values of the N basis
images, and A is the N × F matrix consisting of theintensities of the N light sources. Therefore, lightsource separation results in the problem of matrix factorization; factorizing the matrix I of the input image
sequence into the matrix B of the basis images and thematrix A of the light sources’ intensities.
The necessary condition for light source separation is that the number of equations is larger than orequal to the number of unknowns: (3P × F) ≥ (3P ×
N + N × F). When the number of pixel P is large
enough compared to the number of light sources N,
light source separation is possible if F > N. Our proposed method makes use of NMF (Berry et al., 2007)
for matrix factorization, because both the pixel values of the basis images and the intensities of the lightsources are non-negative. Specifically, our method estimates the matrices B and A by the minimization;1
min ||II − B A||2Fr
(3)
B,A
A} 2
{B

To confirm the effectiveness of our proposed method,
we conducted qualitative and quantitative evaluationby using both synthetic and real images.

3.1 NMF vs. ICA
We compared the following three methods for lightsource separation under AC illumination.
• NMF (our proposed method) decomposes the input images into the basis images and the lightsources’ intensities according to eq.(3).
• ICA-1 decomposes the input images into the basis
images and the light sources’ intensities so thatthe basis images are independent of each other.
• ICA-2 decomposes the input images into the basis images and the light sources’ intensities so thatthe light sources’ intensities are independent ofeach other.
In our implementation, we used the alternating leastsquares algorithm (Berry et al., 2007) for NMF, andFastICA (Hyvärinen and Oja, 2000) for ICA-1 andICA-2.
461

VISAPP 2020 - 15th International Conference on Computer Vision Theory and Applications

Table 1: The PSNRs of the basis images estimated by using NMF, ICA-1, and ICA-2 for Scenes 1, 2, and 3. The numericalvalues in each cell are the PSNRs for the first/second basis images.

method\scene
NMF
ICA-1
ICA-2

1
52.52/42.22
17.57/31.06
23.32/42.20

244.70/53.72
39.16/20.53
27.46/46.27

3
30.97/24.46
24.39/33.28
23.38/37.40

Table 2: The RMSEs of the intensity profiles of the light sources estimated by using NMF, ICA-1, and ICA-2 for Scenes 1, 2,
and 3. The numerical values in each cell are the RMSEs for the first/second light sources.

method\scene
NMF
ICA-1
ICA-2

1
0.001/0.003
0.023/0.045
0.036/0.012

Figure 2: The results of light source separation for Scene 1:(a) the ground truth of the basis images, and the basis images estimated by using (b) NMF, (c) ICA-1, and (d) ICA-2.

Figure 3: The results of light source separation for Scene 2:(a) the ground truth of the basis images, and the basis images estimated by using (b) NMF, (c) ICA-1, and (d) ICA-2.

Synthetic Images:
We synthesized three input image sequences by using
the two datasets; one is for basis images under varying light source directions (Barron and Malik, 2015)
and the other is for light sources’ temporal intensity profiles sampled at 26 points per cycle (Sheinin
et al., 2017). We tested three illumination conditions;a scene is illuminated by two light sources with different colors and from different directions (Scene 1),
those with the same color and from different directions (Scene 2), and those with different colors and462

20.001/0.004
0.017/0.017
0.036/0.012

3
0.012/0.014
0.010/0.014
0.034/0.001

Figure 4: The results of light source separation for Scene 3:(a) the ground truth of the basis images, and the basis images estimated by using (b) NMF, (c) ICA-1, and (d) ICA-2.

from the same direction (Scene 3) as shown in Figure 1.
Figures 2, 3, and 4 show the results of light source
separation for Scenes 1, 2, and 3 respectively: (a) the
ground truth of the basis images, and the basis images
estimated by using (b) NMF, (c) ICA-1, and (d) ICA2. Tables 1 and 2 summarize the PSNRs of the estimated basis images and the RMSEs of the estimatedintensity profiles of the light sources for those scenes
by using NMF, ICA-1, and ICA-2 respectively.
We can see that NMF works well and that its
performance is better than those of ICA-1 and ICA2, when both the colors and directions of the lightsources are different (Scene 1) and when only the directions of the light sources are different (Scene 2).
On the other hand, all the methods do not necessarily
work well when the directions of the light sources are
the same (Scene 3). This is because there are shadowsin Scenes 1 and 2. Specifically, since the directionsof the light sources are different in Scenes 1 and 2,
there are some areas that are illuminated by only oneof the light sources. Because we can directly observe
the flickers due to the single light source there, thoseshadows could be important clues for light source separation.

NMF vs. ICA for Light Source Separation under AC Illumination

Table 3: The PSNRs of the basis images estimated by using NMF, ICA-1, and ICA-2 for Scenes 4, 5, and 6. The numericalvalues in each cell are the PSNRs for the first/second basis images.

method\scene
NMF
ICA-1
ICA-2

4
31.34/30.44
23.11/20.47
26.13/21.31

5
32.56/31.97
31.72/25.95
19.52/25.74

6
29.72/30.88
21.39/18.98
15.61/26.10

Figure 7: The results of light source separation for Scene 5:(a) the ground truth of the basis images, and the basis images estimated by using (b) NMF, (c) ICA-1, and (d) ICA-2.
Figure 5: The first three images of the real image sequenceof (a) Scene 4, (b) Scene 5, and (c) Scene 6.

Figure 8: The results of light source separation for Scene 6:(a) the ground truth of the basis images, and the basis images estimated by using (b) NMF, (c) ICA-1, and (d) ICA-2.
Figure 6: The results of light source separation for Scene 4:(a) the ground truth of the basis images, and the basis images estimated by using (b) NMF, (c) ICA-1, and (d) ICA-2.

Real Images:
We captured the image sequences of three scenes, i.e.
Scenes 4, 5, and 6, each of which is illuminated by
two light sources powered by an electric grid with 60
Hz as shown in Figure 5. We used the high-speed
camera FASTCAM Mini UX50 from Photolon withthe frame rate of 2,500 fps/1,000 fps and with the exposure time of 0.4 ms/1 ms for Scenes 4 and 5/Scene
6 respectively.
Figures 6, 7, and 8 show the results of light source
separation for those scenes: (a) the ground truth of thebasis images, and the basis images estimated by using(b) NMF, (c) ICA-1, and (d) ICA-2. We consider theimages captured when turning only one of the lightsources on as the ground truth of the basis images.
Table 3 summarizes the PSNRs of the basis images

estimated by using NMF, ICA-1, and ICA-2.
We obtained the results consistent to those usingsynthetic images. Figure 6, 7, and 8 qualitatively
show that NMF works better than ICA-1 and ICA2. In particular, the colors and shadows in the basis
images estimated by using ICA-1 are often inaccurate, and the basis images estimated by using ICA-2
are often darker than the ground truth due to negative
pixel values1 . Table 3 quantitatively shows that NMF
works better than ICA-1 and ICA-2 significantly.

3.2 Sensitivity Analysis
The performance of our proposed method using NMF
depends on exposure times as well as image noises,
since light source separation under AC illumination
1 Since the scales of the basis images estimated by using
NMF and ICA are ambiguous, we adjusted the scales of theestimated basis images so that the PSNRs are maximized
for fair quantitative evaluation

463

VISAPP 2020 - 15th International Conference on Computer Vision Theory and Applications

Table 4: The PSNRs of the basis images estimated by using NMF for varying standard deviations of Gaussian noises andvarying exposure times. The numerical values in each cell are the PSNRs for the first/second basis images.

σ=0
σ=1
σ=2
σ=4
σ=8

0.25 ms
50.03/34.89
46.22/35.07
42.13/34.46
41.08/33.49
39.89/31.78

0.5 ms50.02/34.88
46.19/35.06
42.11/34.47
41.08/33.48
39.85/31.87

1 ms
50.01/34.83
46.05/35.02
42.08/34.41
41.03/33.46
39.83/31.74

2 ms
49.90/34.62
45.58/34.86
41.86/34.30
40.89/33.37
39.76/31.59

4 ms
49.29/33.87
43.91/34.19
41.24/33.80
40.57/33.03
39.36/31.89

8 ms
43.21/32.44
40.02/32.58
39.80/32.54
39.43/32.21
38.22/31.13

Table 5: The RMSEs of the intensity profiles of the light sources estimated by using NMF for varying standard deviationsof Gaussian noises and varying exposure times. The numerical values in each cell are the RMSEs for the first/second lightsources.

σ=0
σ=1
σ=2
σ=4
σ=8

0.25 ms
0.003/0.000
0.002/0.002
0.003/0.008
0.003/0.012
0.007/0.013

0.5 ms
0.003/0.000
0.002/0.002
0.003/0.008
0.004/0.012
0.009/0.015

0.5

0.5

0.5

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0

0.1

0
0

0.01

0.02

0
0

0.01

0.02

0

(b)

(a)
0.5

0.5

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0

0

0

0.01

(d)

0.02

0

0.01

(e)

0.01

0.02

(c)

0.5

0

1 ms
0.003/0.000
0.002/0.002
0.003/0.008
0.003/0.012
0.008/0.015

0.02

0

0.01

4 ms
0.006/0.000
0.004/0.003
0.005/0.010
0.006/0.012
0.015/0.013

8 ms
0.023/0.000
0.013/0.010
0.011/0.012
0.011/0.013
0.013/0.015

worse as not only the standard deviation of the Gaussian noises but also the exposure time increase. Thisis because the intensity profiles of the light sources
are getting smoother and the flickers also becomes
invisible as the exposure time increases as shown in
Figure 9.

4

CONCLUSION

0.02

(f)

Figure 9: The intensity profiles of the first (solid lines) andsecond (dotted lines) light sources when the exposure times
are (a) 0.25 ms, (b) 0.5 ms, (c) 1ms, (d) 2 ms, (e) 4 ms, and(f) 8 ms respectively.

makes use of fast flickers of light sources’ intensities. Accordingly, we studied the sensitivity of our
method using NMF to exposure times and noises byusing synthetic images.
Specifically, we interpolated the temporal intensity profiles (Sheinin et al., 2017), which are sampled at 26 points per cycle, by spline interpolation,and computed the intensity profiles with the exposure times from 0.25 ms to 8 ms. Then, we synthesized the images with varying exposure times bycombining the basis images for Scene 1 with thoselight sources’ intensities, and added zero-mean Gaussian noises, whose standard deviation σ is from 1 to 8
for 8-bit pixel values, to them.
Tables 4 and 5 summarize the PSNRs of the basis images and the RMSEs of the intensity profilesof the light sources estimated by using NMF. We cansee that the PSNRs of the basis images are getting
464

2 ms
0.004/0.000
0.002/0.002
0.003/0.008
0.004/0.012
0.006/0.014

In this paper, we proposed a method for light source
separation under AC illumination on the basis of BSS;
we decompose the image sequence of a scene illuminated by multiple AC light sources into the basis images and the light sources’ intensities. Specifically,
we show that light source separation results in matrix factorization, and make use of NMF for BSS because both the pixel values of the basis images andthe intensities of the light sources are non-negative.
We experimentally confirmed that our method using
NMF works better than ICA, and studied the performance of our method under various conditions: varying exposure times and noise levels. Our future work
includes light source separation under more than twolight sources, the improved separation by taking noise
removal into consideration, and the application of theseparation results.

ACKNOWLEDGMENTS
This work was partially supported by JSPS KAKENHI Grant Numbers JP18H05011 and JP17H01766.

NMF vs. ICA for Light Source Separation under AC Illumination

REFERENCES
Barron, J. and Malik, J. (2015). Shape, illumination, andreflectance from shading. IEEE TPAMI, 38(7):1670–
1687.
Berry, M., Browne, M., Langville, A., Pauca, V., and Plemmons, R. (2007). Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics & Data Analysis, 52(1):155–173.
Hsu, E., Mertens, T., Paris, S., Avidan, S., and Durand, F.
(2008). Light mixture estimation for spatially varying
white balance. ACM TOG, 27(3):1–7.
Hyvärinen, A. and Oja, E. (2000). Independent component
analysis: algorithms and applications. Neural Networks, 13(4–5):411–430.
Sheinin, M., Schechner, Y., and Kutulakos, K. (2017).
Computational imaging on the electric grid. In Proc.
IEEE CVPR2017, pages 6437–6446.
Sheinin, M., Schechner, Y., and Kutulakos, K. (2018).
Rolling shutter imaging on the electric grid. In Proc.
IEEE ICCP2018, pages 1–12.
Vollmer, M. and Möllmann, K.-P. (2015). Flickering lamps.
European Journal of Physics, 36(035027).

465