E-Journal of Advanced Maintenance Vol.7-1(2015) 50-58
Japan Society of Maintenology

Correction of Over and Under Exposure Images Using Multiple Lighting
SystemJonghoon IM1*, Hiromitsu FUJII1, Atsushi YAMASHITA1, and Hajime ASAMA1
1

The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan

ABSTRACT
When images are acquired in bright condition, it can cause a loss of highlight details (over exposure) in brightarea and a loss of shadow details (under exposure) in dark area. Over and under exposure causes a big problem
when people investigate dangerous place like Fukushima nuclear power plant through the camera attachedremote control robot. In this paper, we propose a method to correct the over and under exposure image to solve
the problem. The image processing consists of four steps. Firstly, multiple images are acquired by alternately
turning on and off each illumination which set in different positions. Then the image obtained first is defined as
input image 1, the image obtained second is defined as input image 2 and the image obtained N-th is defined as
input image N. Secondly, luminance of the images is corrected. Thirdly, over and under exposure regions in theimage are extracted from the input image 1. Finally, the over and under exposure regions in the input image 1
are compensated by other images. The results show that the over and under exposure regions in the input image
are recovered by our proposed method.
KEYWORDS

ARTICLE INFORMATION

Image processing, Multi-exposure images fusion, Image correction, Overand under exposure, Multiple lighting system

Article history:
Received 5 November 2014
Accepted 26 February 2015

1.

Introduction

It is necessary to use remote control robots when people investigate dangerous place where theycannot enter. Especially, it is necessary to use some lightings which are attached to the robot in darkarea without external lighting in order to obtain the information of surrounding as shown in Fig. 1. In
this case, the problem is that a portion of the image is unclear because of over and under exposure. In
order to solve this problem, Multi-Exposure Fusion (MEF) techniques[1][2][3][4] and High Dynamic
Range (HDR) imaging techniques[5][6][7] have been proposed. These techniques are obtaining one
image by combining a plurality of images with different exposure level. In the HDR imaging, a highdynamic range image is compressed into a low dynamic range image using tone mapping

Fig. 1. Investigation using remote controlled robot.

*

Corresponding author, E-mail: im@robot.t.u-tokyo.ac.jp

ISSN-1883-9894/10 © 2010 – JSM and the authors. All rights reserved.

50

J. Im, et al ./ Correction of Over and Under Exposure Images
Using Multiple Lighting System

methods[8][9] for viewing on ordinary displays. And it needs the calibration of the camera response
function (CRF) which is required in HDR imaging if the CRF is not linear. However, MEF does notneed these process, and has low computational complexity. So, we propose a method of correction
over and under exposure which is based on MEF.
In the general HDR imaging, the magnitude of the luminance values between the pixels ispreserved before and after tone mapping. For this reason, the available dynamic range in each region
is limited in the environment that has large contrast. And it is difficult to store all the texture
information to the HDR image because of the constraint condition. And in the previous HDR imagingand MEF methods use a beam splitter[10][11] or adjust camera’s shutter speed[12][13][14] to acquire
multiple images having different exposure value. In the former case, it is need to specific andexpensive device. And it is difficult to acquire multiple images whose exposure values are appropriate
by adjusting camera’s shutter speed in the environment that has large contrast in the latter case. To
solve these problem, we use multiple lightings to acquire multiple images whose locations of the overand under exposure areas are different from each other.
In this paper, we propose a method to correct the over and under exposure region in the image
using multiple lighting system. We assume the situation that operator investigates in a dark
environment using a remote control robot like Fig. 1. Multiple lightings, such as the headlights of the
car are attached to the front of the robot, and each lighting can be turned on and off by the operator.
The rest of the paper is organized as follows. Section 2 describes proposed method, and Section 3
depicts experiment and the result. Finally, Section 4 concludes the paper.

2.

Proposed method

Our method consists of image acquisition, luminance correction, extraction of over and under
exposure areas and compensation of over and under exposure areas. The proposed method is shown in
Fig. 2. Each step is described in detail and three prepared sample images are used for explaining theprocess of our proposed method.

Fig. 2. The schematic view of our proposed method
for correction of over/under exposure.

51

J. Im, et al ./ Correction of Over and Under Exposure Images
Using Multiple Lighting System

(a)

(b)

(c)

(d)

Fig. 3 Background image and input images: (a) Background image, (b) Input image 1 I01,(c) Input image 2 I02 and (d) Input image 3 I03.

2.1. Image acquisition
In this process, multiple photographs that over and under exposure appeared in different location
are captured by changing lighting condition at the same background (Fig. 3(a)). In this research,
multiple images are acquired by alternately turning on and off each illumination which set in different
positions. The image obtained first is defined as input image 1 I01, the image obtained second isdefined as input image 2 I02 and the image obtained N-th is defined as input image N I0N. Fig. 3 shows
sample images which were taken with a real camera to explain the process of our proposed method.
The subjects in the images are a 1.29m × 0.73m printed picture which describes inside of Fukushima
nuclear power plant. The over exposure is appeared in the lower left and the under exposure isappeared in the upper right in the input image 1 (Fig. 3(b)). On the other hand, the over exposure isappeared in the lower right and the under exposure is appeared in the left in the input image 2(Fig. 3(c)). In the input image 3 (Fig. 3(d)), the over exposure is appeared in the upper middle and theunder exposure is appeared in the lower left and right.
2.2. Luminance correction
The brightness distribution of the acquired images are changed by the illumination. It can be
confirmed by comparing Fig. 3 (b), (c) and (d) that is acquired by changing light condition. In order toobtain a clear image, it is necessary to correct the brightness distribution of the acquired images. In
this step, firstly we find the center of irradiated area, secondly correct the brightness of each pixel byconsidering distance from the center of the irradiated area.
In order to find the center of irradiated area, firstly the over exposure areas that are caused bylightings are extracted. One threshold value τC which is set in advance is used for the extraction asshown equation (1).
1
C n (i, j ) = 0(a)if I 0n (i, j ) > t C,otherwise

(b)

(1)

(n = 1,2,  N )(c)Fig. 4. Irradiated area images: (a) Irradiated area image C1, (b) Irradiated area image C2 and (c)Irradiated area image C3. The over exposure areas are displayed in green and non-over

exposure areas are displayed in black. The position of center of irradiated area is displayed in
red circle.

52

J. Im, et al ./ Correction of Over and Under Exposure Images
Using Multiple Lighting System

where n is the number of input image, I0(i, j) is the pixel value of an original input image. I0n(i, j) isthe pixel value of the original input image n coordinates (i, j).The pixel value describes how brightthat pixel is. C n is the obtained image as a result of luminance correction process, and the image isdefined as an irradiated area image C n. Color images are converted into grayscale image before
process. If the pixel value of the image n coordinates (i, j) is over τC which is set in advance byconsidering environment and lighting condition, it is determined over exposure area and the pixel
value is converted to 1. Otherwise it is determined non-over exposure area and the pixel value isconverted to 0. As a result, the irradiated area C n is acquired (Fig. 4(a)), we convert grayscale image
to color image in order to make it easier to view the extracted area. The over exposure areas are
displayed in green and non-over exposure areas are displayed in black in the Fig. 4(a). Next, thecenter of the irradiated area in the over exposure areas is searched by calculating the average value ofi and j coordinates of the green pixels as shown equation (2). M is the number of green pixels in theirradiated area image C n. Acquired position of the center of the irradiated area set to (Xc, Yc). The
position is displayed in red circle in the Fig. 4(a). The same procedure is performed in other input
images (Fig. 4(b), (c)).
Xc =∑ x ∈ {i | C (i, j) = 1} x , Yc = ∑ y ∈ { j | C (i, j) = 1} y ,
M

M

(2)

Then, we correct the luminance of the images using distance between the center of the irradiatedarea and each pixel. In Fig. 3(b), (c) and (d), around the center of the irradiated area is the brightest.
And the luminance become gradually darker when the distance between the center of the irradiatedarea and each pixel become longer. In order to correct the luminance of the images, equations (3), (4)and (5) are used.
(3)

L' (i, j ) = f ( L(i, j )),f ( L(i, j )) = α ⋅ L(i, j ) ⋅ r 2 ,

(4)

2

(5)

2

r = (i − X c ) + ( j − Yc ) ,where f(L(i, j)) is a luminance correction function, L(i, j) is a luminance value at the coordinates (i, j)
before luminance correction and L’(i, j) is a luminance value at the coordinates (i, j) after luminancecorrection. α is correction factor which is set in advance by considering environment condition and it
keep the luminance value from becoming very large. r is the distance between the center of theirradiated area and each pixel. Xc and Yc are the position of the center of the irradiated area. We use
equation (4) as the luminance correction function in this time. Equation (4) is an approximate
equation for correcting the luminance of the input image using the principle that the brightness isinversely proportional to the square of the distance from the center of irradiated areas to each pixel.
The brightness is equalized by multiplying the brightness of each pixel from input image and thesquare of the distance. The luminance correction value in equation (4) is determined depending on the
distance from the center of the over exposure area, regardless of the shape of the object. As a result of
the luminance correction, luminance corrected image Ip is acquired. Ipn is luminance corrected imageof input image I0n , Fig. 5(a), (b) and (c) shows Ip1, Ip2 and Ip3.
2.3. Extraction of over and under exposure areas
This process extracts the over and under exposure areas in an image in order to get rid of theseareas. Two threshold values τH and τL which are set in advance are used for the extraction as shown
equation (6).
1
D (i, j ) = 
0

if t L < I (i, j ) < t H,otherwise

53(6)

J. Im, et al ./ Correction of Over and Under Exposure Images
Using Multiple Lighting System

(a)

(b)

(c)

(d)

Fig. 5 Luminance corrected images and determination map: (a) Luminance corrected image Ip1, (b)Luminance corrected image Ip2, (c) Luminance corrected image Ip3 and (d) determination map D1.

where I(i, j) is a pixel value of the input image coordinates (i, j). D is the reference matrix fordetermining over and under exposure areas, which is obtained as a result of the extraction process ofover and under exposure. This reference matrix is defined as a determination map. If the pixel
value of the input image coordinates (i, j) is over τH which is set in advance, it is determined over
exposure areas and the pixel value is converted to 0. If the pixel value of the input image coordinates(i, j) is under τL which is set in advance, it is determined under exposure areas and the pixel values isconverted to 0. Otherwise, if the pixel value of the image coordinates (i, j) is included between τH andτL, it is determined non-over and under exposure areas and the pixel value is converted to 1. As aresult, the determination map D is acquired. Fig. 5(d) shows determination map of the input image 1(Fig. 3(b)). The over and under exposure areas are displayed in black and non-over and under
exposure areas are displayed in white in the Fig. 5(d).
2.4. Compensation of over and under exposure areas
In this step, the over and under exposure areas of the image 1 are compensated by other images.
The determination map D which is obtained in Section 2.3 is used. The rule of the synthesis is shownin equations (7) and (8).
 I n -1 (i, j )I nf (i, j ) =  fn
 I p (i, j )D n -1 (i, j ) = 1, (n ≥ 2)
D n -1 (i, j ) = 0(7)

I 1f (i, j ) = I 1p (i, j ) ,∑ I (i, j) − I (i, j) ≤ β∑ I (i, j)
n
f

n −1
f

n
f

(8)

where D n is a determination map of luminance corrected image n Ipn except D 1. D 1 is a determination
map of input image 1 I0n. If is an output image whose over and under exposure areas werecompensated. If n is the output image when n images are used. The areas where D n-1 (i, j)=1 in thedetermination map are compensated by the image If n-1, and the areas where D n-1 (i, j)=0 in thedetermination map are compensated by the image Ipn. As a result of this process, compensated image
If n is obtained. The process will proceed until n=N. However, it is not efficient to use all of the images
for the compensation because the computational time becomes longer. In order to reduce thecomputational time, we set a termination condition of the compensation algorithm. If the rate ofchange of pixel value between If n and If n-1 is less than β which was decided on the basis of previous
experimental result as shown in equation (8), the process is over. For example, when N=3, the areas
where D1 (i, j)=1, which are shown as white in Fig. 5(d), are compensated by the image Ip1 (Fig. 5(a)).
The areas where D1 (i, j)=0, which are shown as black in Fig. 5(d), are compensated by the image Ip2(Fig. 5(b)). As the result of the compensation, the output image 2 If 2 (Fig. 6(a)) is obtained. Next,
output image 2 If 2 is compensated by the image Ip3(Fig. 5(c)) and determination map D 2(Fig. 6(b)) in
the same way. As the result of the compensation, the output image 3 If 3 (Fig. 6(c)) is obtained.

54

J. Im, et al ./ Correction of Over and Under Exposure Images
Using Multiple Lighting System(a) Output image2 Iｆ2(b) Determination map D2(c) Output image3 Iｆ3Fig. 6. Output images when N=3.
Table 1. Photographing conditions.

CameraDefinitionISO
F-number
Shutter speed
Focal length
Luminance of lighting

NIKON D200
640 ×480
2,500
f/8
1/40sec
24mm
1,000lm

3. Experiment3.1. Experiment environment
In the experiment, we used four lightings in order to acquire four images that over and under
exposure appeared in different location. For the experiment, we prepared two signs in advance(Fig. 7(a)). ‘DANGER Radiation risk’ letters were written on the left side sign, and ‘DO NOTENTER, STAFF ONLY’ letters were written on the right side sign. Fig. 7(b) shows the distance
between the signs and the camera, and Fig. 7(c) shows the distance between the camera and thelightings. The height of the camera was 8.50 × 10-1m from the floor, and the distance between thesigns and camera was 2.00m as shown in Fig. 7(b). The lighting 1 was located in 2.00 × 10-1m under
3.00 × 10-1m left side of the camera, the lighting 2 was located in 2.00 × 10-1m under 3.00 × 10-1m rightside of the camera. The lighting 3 was located in 2.00 × 10-1m upper of the camera and the lighting 4
was located in 2.00 × 10-1m under of the camera as shown in Fig. 7(c). All of the photographing
conditions were fixed except the distance between the camera and the lightings during the image
acquisition process. The photographing conditions are depicted in Table 1. Finally, four images are
acquired by alternately turning on and off each lighting which set in different positions.
3.2. Experiment results
All experiments are simulated on an Intel CoreTM i5-2520M CPU 2.50 GHz PC with 8 GB RAM.
The computing time of the proposed method is about 0.19 seconds by single thread coding in C++.
Fig. 8 shows the experiment results. Fig. 8(a), (b), (c) and (d) shows the input image 1, 2, 3 and 4.
Figures 8(e), (f) and (g) described the output image 2, 3 and 4 which were combined by our proposed
method. In this experiment, we set the τH=200, τL=5, τC=200, α=1.3 × 10-4 and β=0.1.
In the input image 1, the over exposure appeared on the left side sign and the under exposure
appeared on the right side of the image. In consequence, the letters which were written on the left sign
became unclear. Also the right side of the image cannot identify because of the under exposure. On
the other hand, in the input image 2, the over exposure appeared on the right side sign and the under
exposure appeared on the left side of the image. As a result, the letters which were written on the right55

J. Im, et al ./ Correction of Over and Under Exposure Images
Using Multiple Lighting System(a) Two signs(b) Distance between signs and camera(c) Distance between camera and lightings

Fig. 7. Experiment environment.

(a) Input image 1 I01(b) Input image 2 I02(c) Input image 3 I03(d) Input image 4 I03(e) Output image 2 If2(f) Output image 3 If3(g) Output image 4 If4(h) Contrast compensation(i) HDR imaging[7]Fig. 8. Experiment results.

sign became unclear. The left side of the image cannot identify because of the under exposure. In the
input image 3 and 4, the over exposure appeared on the center of the image and the under exposure
appeared on the edge of the image. As a result of applying the proposed compensation algorithm tothe input image 1, the over exposed areas in the input image1 became clear, and ‘DANGER Radiation
risk’ letters became visible. Also, the under exposed areas in the input image 1 were compensated andthe areas became clear. The rate of the pixel change of output image 2, 3 and 4 were 0.59, 0.07 and0.01. The process terminated when the output image 3 was synthesized. The process time of each case56

J. Im, et al ./ Correction of Over and Under Exposure Images
Using Multiple Lighting Systemwere 0.12, 0.19 and 0.26 seconds. As the results of the termination condition, the computation time
reduced by 0.07 seconds.
Next, we compared the result of the proposed method (Fig. 8(f)), contrast compensation(Fig. 8(h)) and HDR imaging (Fig. 8(i)) [7]. Fig.9 shows hue histogram of original image, the resultof contrast compensation, the result of HDR imaging and the result of our proposed method. X-axis of
the histogram represents a hue value [0,180] and Y-axis represents a frequency of each hue value. The
hue value is more evenly distributed in the result of the proposed method than the original image, theresult of HDR imaging and contrast compensation. In order to view whole scene in the environmentthat has large contrast, it is need to the hue values which are uniformly distributed. The experimental
result shows that our proposed method is better than the previous contrast compensation and HDR
imaging.

4. Conclusion
In this paper, we proposed the method of correction the over and under exposure images using
multiple lighting system. We assume the situation that operator investigates in a dark environment
using a remote control robot, and in this time, multiple lightings, such as the headlights of the car are
attached to the front of the robot, and each lighting can be turned on and off by the operator. And webuild experimental environment that we assumed. Next, multiple images were acquired using multiple
lighting system. Finally, the over and under exposure areas were compensated by the proposed
method. The experimental results showed the effectiveness of our proposed method.
In this research, we fixed the threshold value τc, and correction factor α. And we decide the value
appropriately by considering experiment environment. However, the appropriate value is changed by
the environment and lighting condition. So we are considering to decide the value automatically for
future work. And we plan to apply our method to real situation such as Fukushima nuclear power
plant. And it is necessary to control brightness of the light or adjust the angle of the light not only
ON/OFF of the light in order to acquire more images in which the exposure values are different.

Fig. 9. Hue histogram of original image, the result of HDR imaging, the result of contrast compensationand the result of our proposed method.

57

J. Im, et al ./ Correction of Over and Under Exposure Images
Using Multiple Lighting System

Acknowledgement
This research was supported by the Agency for Natural Resources and Energy and International
Research Institute for Nuclear Decommissioning, Japan, Fiscal Year 2013.

References[1] T. Mertens, J. Kautz and F. Van Reeth: “Exposure Fusion: A Simple and Practical Alternative to HighDynamic Range Photography”, Computer Graph. Forum, Vol. 28, No. 1, pp. 161-171(2009).
[2] R. Shen, I. Cheng and A. Basu: “QoE-Based Multi-Exposure Fusion in Hierarchical Multivariate
Gaussian CRF”, IEEE Transactions on Image Processing, Vol. 22, No. 6, pp. 2469-2478(2013).
[3] A. Agrawal, R. Raskar, S. Nayar and Y. Li: “Removing Photography Artifacts Using Gradient Projection
and Flash-Exposure Sampling”, Proceedings of ACM SIGGRAPH 2005,Vol. 24, No. 3, pp. 828-835(2005).
[4] J. Kong, R. Wang, Y. Lu, X. Feng and J. Zhang: “A Novel Fusion Approach of Multi-exposure Image”,
Proceedings of International Conference on “Computer as a Tool” 2007 EUROCON, pp. 163-169(2007).
[5] S. Mann and R. W. Picard: “On being ‘Undigital’ with Digital Cameras : Extending Dynamic Range by
Combining Differently Exposed Pictures”, Proceedings of the 48th Annual Conference of the Imaging
Science and Technology, pp. 442-448(1995).
[6] K. Agusanto, L. Li, C. Zhu and S. W. Ng: “Photo Realistic Rendering for Augmented Reality Using
Environment Illumination”, Proceedings of the Second IEEE and ACM International Symposium on Mixed
and Augmented Reality, pp. 208-216(2003).
[7] P. Debevec and J. Malik: “Recovering High Dynamic Range Radiance Maps from Photographs”,
Proceedings of ACM SIGGRAPH 1997, pp. 369-378(1997).
[8] E. Reinhard, M. Stark, P. Shirley and J. Ferwerda: “Photographic Tone Reproduction for Digital Images”,
Proceedings of ACM SIGGRAPH 2002, pp. 267-276(2002).
[9] J. Kuang, G. M. Johnson and M. D. Fairchild: “ICAM06: A Refined Image Appearance Model for HDR
Image Rendering”, Journal of Visual Communication and Image Representation, Vol. 18, No. 5,
pp. 406-414(2007).
[10] M. D. Tocci, C. Kiser, N. Tocci and P. Sen: “A Versatile HDR Video Production System”, ACM
Transactions on Graphics, Vol. 30, No. 4, pp. 41-50(2011).
[11] M. Aggarwal and N. Ahuja: “Split Aperture Imaging for High Dynamic Range”, Proceedings of IEEE
International Conference on Computer Vision 2001, pp. 10-17(2001).
[12] D. C. H. Schleicher and B. G. Zagar: “High Dynamic Range Imaging by Varying Exposure Time, Gain and
Aperture of a Video Camera”, Proceedings of IEEE Instrumentation and Measurement Technology
Conference 2010, pp. 486-491(2010).
[13] Y. Piao and W. Xu: “Method of Auto Multi-Exposure for High Dynamic Range Imaging”, Proceedings ofInternational Conference on Computer, Mechatromics, Control and Electronic Engineering 2010, Vol. 6,
pp. 93-97(2010).
[14] N. Barakat and T. E. Darcie: “Minimal Capture Sets for Multi-Exposure Enhanced-Dynamic-Range
Image”, Proceedings of IEEE International Symposium on Signal Processing and Information Technology
2006, pp. 524-529(2006).

58