RM-002

FIT2010（第 9 回情報科学技術フォーラム）LED Traffic Light Detection Using a High-speed-camera for aRoad-to-vehicle Visible Light Communication System
H. Chinthaka N. Premachandra, Tomohiro Yendo, Mehrdad Panahpour Tehrani, Takaya Yamazato,
Hiraku Okada, Toshiaki Fujii, and Masayuki Tanimoto
Abstract— A new visible light road-to-vehicle communication
system at intersections is proposed for the area of ITS. In thissystem, the communication between a vehicle and an LED traffic light is conducted using an LED traffic light as a transmitter,and an on-vehicle high-speed camera as a receiver. The LEDs in
the transmitter emit light in high frequency and those emitting
LEDs are captured by the high-speed camera for making
communication. Here, the luminance value of LEDs in thetransmitter should be captured in consecutive frames to achievecommunication. For this purpose, finding the transmitter andtracking it in consecutive frames by processing the images fromthe high-speed camera is important. We propose new effective
algorithms for finding and tracking the transmitter, which resultin a increased communication speed, compared to the previous
methods. Experiments using appropriate images showed theeffectiveness of the proposals.

I. INTRODUCTION
We conduct research mainly on assistance for safe driving
as an area of Intelligent Transport Ssystem(ITS). On-vehicle
cameras play an important role in capturing the informationof external environment in many driver assistance systems.
Some studies have been conducted for detecting traffic
signs, and signal lights so on[1][2][3][4][8][9]. In this study,
we propose a road-to-vehicle Visible Light Communication(VLC) system using an on-vehicle high-speed camera as
a receiver and LED traffic light as a transmitter.
VLC is a wireless communication method using luminance, transmitting data by emitting light. It is able to transfer data by emitting light source, and able to receive themby light sensor. There are several advantages in this communication method compared to other wireless communication
methods, such as radio waves and infra-red light. The visible
light is not harmful to human body, and it is able to transmit
with high power. There are not legal limitations for any
existing light source, such as room illuminations and displays
to be used. It can be used at the places where radio waves
cannot be used, for example hospitals and areas around
precision machines. Komine and Nakagawa[5] have achieved
VLC using illumination light. It is a communication between
This work was partially supported by the Ministry of Internal Affairs andCommunications, Strategic Information and Communications RD Promotion
Programme(SCOPE)
H. Chinthaka N. Premachandra, T. Yendo, M. P. Tehrani, T. Yamazato and M. Tanimoto are with the Graduate School of Engineering, Nagoya university, Chikusa-ku, Nagoya, 464-8603 JAPAN

chinthaka@tanimoto.nuee.nagoya-u.ac.jp
H. Okada is with the Graduate School of Science and Engineering,
Saitama University, Saitama, JapanT. Fujii is with the Graduate School of Science and Engineering, Tokyo
Institute of Technology, Tokyo, Japan

PCs and illumination light, and considered as an alternative
method for the wireless LAN. As a latest application of
VLC in ITS, Suzuki et al. [6] introduced a support systemfor visually impaired person by utilizing VLC technology at
signalized intersections.
High-speed cameras are getting popular and they are also
applied in ITS as well as in other scientific research, militarytest, and industry. High speed cameras can capture more
information of fast moving objects and changing objects inhigh frequency, compared to the normal cameras. Specificimage processing technique are also necessary when thesecameras are used. In the proposed VLC system, the LEDs
in the transmitter emit light in 500Hz and the images, which
include those emitting LEDs are captured by the on-vehicle
high-speed camera in 1000fps, while the vehicle is moving.
The luminance value of LEDs in consecutive frames should
be captured by processing the images from the camera
for conducting communication. For this purpose, first thetransmitter should be found, then it should be tracked inconsecutive frames. In this paper, new effective methods areproposed for finding and tracking the transmitter concerning
the above mentioned abilities of high-speed cameras. With
the our previous finding and tracking methods[4][8][9], partof the LEDs were used for tracking and part of the emission
used for finding, so not all LEDs could be used for continuous communication. With our proposed methods, continuous
emission of all LEDs can be used for communication resulting in a higher data rate. According to experiments conductedunder different conditions, the proposals were effective fordesired finding and tracking.
This paper consists of six main sections to explain our
project work. Proposed VLC system is explained in section
II. Section III and IV explain the previous methods and new
proposed methods for finding and tracking respectively. The
experimental results are described in section V. The section
VI concludes the paper.
II. P ROPOSED VLC SYSTEM
Figure 1 shows the structure of the proposed VLC systemusing LED traffic light as a transmitter and high-speed
camera as a receiver. If the LEDs in the transmitter could
be recognized individually, it is possible to use each of themas a separate transmitter communicating in parallel at thesame time. In other words, each LED transmits different
data in parallel and they are received at the same time. As aresult we can dramatically increase the communication speedby modulating each LED individually. Moreover, we can45（第4分冊）

FIT2010（第 9 回情報科学技術フォーラム）communicate with several transmitters and receive different
information in parallel. These are the main advantage ofusing a camera as a receiver(Fig. 2). Another issue is themodulation method. Since this is a unique communication
method using visible light and image, it requires particular
modulation method which considers the characteristics of thecommunication. We use hierarchical coding[7] for Visible
Light Communication, which modulates data on spatial frequency and enables long distance communication.
The transmitter used for the experiments is square in shape
as shown in Fig. 3 and it consists 16x16 LEDs. Figure 2
shows the high-speed camera which is used as the receiverin the experiments.

Fig. 4.

The main flow of finding and Tracking

III. P REVIOUS METHODS FOR TRANSMITTER FINDING
AND TRACKING

Figure 4 illustrates the main flow for finding and tracking
the transmitter. Here, first the transmitter is found, thencut out image area(Icut ) including the transmitter from theoriginal image as shown in Fig. 5. The cut out image
area is indicated using a white square in the Fig. 5. The
found transmitter is tracked detecting it only searching thecorresponding image area to this cut out image area inconsecutive frames. This tracking is conducted until it appear
on this image area. If the transmitter stayed out from thisimage area, process return to the transmitter finding step.
Figure 6 illustrates the emission patterns of the transmitterin our previous work. The LEDs in the transmitter are emitted
sequence by sequence and in the first part of the sequence, allLEDs are blinked(ON and OFF) at the same time in 500Hz.
This stage is set for finding the transmitter by processing theimages from the high-speed camera. In the second part of thesequence, LEDs in the transmitter are emitted for different
levels in 500Hz, except LEDs in the two exterior lines, andcommunication is conducted in this stage. In our previous
work, these non-emitting(keep lighting without emitting) two
exterior lines are kept to make it easy to track the foundtransmitter.

Fig. 5.

Found transmitter

A. Previous transmitter finding
While the vehicle is moving, the receiver (High-speed
camera) installed in the vehicle takes images of the roadin 1000fps. If transmitter exists on the road and it is at thefirst part of the sequence, it is expected to appear lighting
transmitter on images once in two frames, since traffic lightemits (ON and OFF) in 500Hz and high-speed camera takes
images in 1000fps. Figure 7 shows the appearance of thetransmitter in four consecutive frames at the first part of thesequence. Iwasaki et al. [4] proposed a method for finding thetransmitter using this feature, by subtracting two consecutive
frames.
B. Previous transmitter tracking
Previously, we proposed a method[8][9] for tracking thefound transmitter in consecutive frames using the edge
information. This method is very effective, if the nonemitting(keep lighting) two exterior lines are available. Figure 8 illustrates appearance of the traffic light in four frameswith non-emitting two LED lines. This method is explainedin more details in section IV.B.1.

Fig. 1.

Structure of the proposed visible light communication system

IV. N EW METHODS FOR TRANSMITTER FINDING AND
TRACKING

Fig. 2.

ReceiverFig. 3.

TransmitterIn this paper, we removed the blinking part from thesequence as shown in Fig.10 and new method was proposed to find the transmitter only using randomly emitting LEDs as detailed in the section IV.A. The previous
tracking method[8][9] was improved to be able to track thetransmitter, without keeping non-emitting(keep lighting) two
exterior LED lines. The new method can track the transmitter
when all LEDs in the transmitter emit light randomly. This

46
（第4分冊）

FIT2010（第 9 回情報科学技術フォーラム）Fig. 8. Consecutive frames of emitting all LEDs in different luminancelevels keeping non-emitting(keep lighting) two exterior LED lines
Fig. 6.

Structure of the emission sequence(before improvement)

Fig. 7.

Fig. 9. Consecutive frames of emitting all LEDs in different luminancelevels without keeping non-emitting(keep lighting) two exterior LED lines

Consecutive frames of blinking all LEDs

new tracking method is explained in section IV.B. Due toabove improvements, it is not necessary to have two parts inthe structure of the emission sequence(Fig. 6). And the allLEDs in the transmitter cannot be used for communicationand the continuous communication cannot be conductedwith our previous methods[4][8][9]. All LEDs can be used
for communication and continuous communication can be
conducted with new proposed methods. For this reason,communication speed can be increased when new methods
are applied.
A. Proposed transmitter finding method
The new transmitter finding method is detailed in thissection.
1) Generation of differences image: In this method, differences between current frame and some just previous
consecutive frames are calculated following the Equation 1.
Dtotal =m∑|Dt − Dt−n |objects with white pixels belong to the noise have sizes assimilar as transmitter in some cases. For this reason, first thetransmitter candidates are selected, and then transmitter is
extracted from the candidates.
3) Transmitter area candidate selection : The candidates
are selected considering the geometrical information of transmitter. In this study, we plan to conduct communication
distance between 70m and 20m, far from the transmitter.
When the vehicle move this distance the side length of
the transmitter has maximum length(lmax ) and minimum
length(lmin ) on the image. lmax : Vehicle is almost 20m
far from the transmitter, lmin : Vehicle is almost 70m far
from the transmitter. The transmitter candidates are selectedfollowing below conditions calculating the circumscribing
rectangle of each object.
lmin < (Ch , Cw ) < lmax
Aspectratio(Ch , Cw ) < 1.2

(1)

n=1

In the above equation Dt represent the current frame
and m represent the number of previous frames used forthe differences calculation. The total differences(Dtotal ) areprojected to the new image. In this paper, this image is called
as differences image. The transmitter area get highlighted
there, since differences are comparatively higher than otherareas due to the random emission of the transmitter. Figure
11 shows an arbitrarily example of generated differences
image.
2) Binarization and morphology operation: This differences image(Dtotal ) is processed for binarization using appropriate threshold to get the transmitter area with white
pixels as shown in Fig 12. The morphological operationis conducted to the binarized difference image to connect
the non-connected white pixels belong to transmitter area.
In this process, first image is dilated two times and eroded
it two times. There are some noise appeared in the image
after these processing as shown in Fig. 13. In addition, someCw and Ch mean the width and height of the circumscribing
rectangle belong to the searching object. Here, the object
which take square circumscribing rectangle are selected
as candidate by setting the aspect ratio since the desiredtransmitter is square in shape. In some cases, there are few
candidates appeared as shown in Fig. 13 and single candidate
appeared in many cases. So, it is necessary to confirm thetransmitter.
4) Confirmation of the transmitter: This confirmation isconducted by searching the differences density(Dden ) andthe average intensity(Lavg ) in the corresponding candidates
areas in the difference image. Figure 14 shows corresponding
differences image for Fig. 13. Dden and Lavg are calculated
following the Equation 2 and 3 respectively.

47（第4分冊）

Dpix
Tpix
∑
Lpixi
Lavg =
Tpix
Dden =(2)(3)

FIT2010（第 9 回情報科学技術フォーラム）Fig. 10.

Structure of the emission sequence(after improvement)

Fig. 12.

Fig. 11.

Binarized image

Differences image

Dpix and Tpix in the above equations represent the numberof differences pixels(pixel value>20) and total number ofpixels inside the candidate respectively. In addition, theLpixi represents the intensity value of each pixel inside thecandidate. The candidate which are under below condition
is confirmed as transmitter.
Dden × Lavg > thresholdIn the Fig. 14, the transmitter confirmation result is also indicated using cut out image area(Icut ) with white square. The
differences density(Dden ) and the average intensity(Lavg )values inside the transmitter area take stronger values thanother objects, since differences pixels takes higher intensity
value due to accumulation of m − 1 differences(differences
between current frame and just previous m consecutive
frames) belongs to the randomly emitted transmitter.
B. Proposed transmitter tracking method
After finding the transmitter, it is tracked only searching
the corresponding cut out image area(Icut ) in consecutive
frames. In this paper, at the emission sequence, all LEDs
emit light randomly(Fig. 10). But, when all the LEDs emit
light randomly, it is difficult to track transmitter, because,the bounding of the transmitter is not appear properly(seeFig. 9). We propose a new method for this tracking and it
includes two main steps as edge-based tracking and opticalflow-based tracking.
1) Edge-based tracking : In this method[8][9], first edge
detection is conducted, then the edge component whose
have circumscribing rectangles as nearly similar size as justpreviously detected transmitter are selected as transmittercandidates. After that the transmitter is extracted from thecandidates using defined transmitter likelihood conditions
and transmitter likelihood probability. According to so far

Fig. 13.

An example of candidate selection

experiment results, it was possible to detect the transmitter inconsecutive frames with at least 20% of detection rate whenall LEDs emit light randomly. The reason for this is that, theabove method detect transmitter considering circumscribing
rectangle of the edge component. In the case of that theat least four LEDs in the each bounding LED lines of
the transmitter light and those are connected each otherthrough the other lighting LEDs, it is possible to gain theappropriate circumscribing rectangle of the edge component
which belongs to the transmitter.
2) Optical flow-based tracking : If the above edge-based
method could not detect the transmitter，the movement of
the transmitter in consecutive frames are measured by using
the optical flow as a second step．In this paper, we use
Lucas-Kanade s optical flow method[10]. Here, if edgebased method does not detect the transmitter, it is tracked on

Fig. 14. Indication of the corresponding candidates areas on differences
image and transmitter confirmation result for the example shown in Fig. 13

48（第4分冊）

FIT2010（第 9 回情報科学技術フォーラム）optical flow until is detected by edge-based method again.
This optical flow-based detection is explained below. If thetransmitter could not be detected in the edge-based step,
preprocess the previous frame(called as base frame)of thecurrent frame using a Gaussian low pass filter. The deformation matrix(D) is calculated using directional derivatives at
each pixel in the base frame following Equation (4).
[
]
d
dxy
D = xx(4)
dyx dyy
where, for example, dxx represents the 2nd order derivativein the x direction
The eigenvalues of the deformation matrix(D) are calculated, and perform non-maxima suppression. In this work, we
assume that local minima exist in the 5 × 5 neighborhood.
Select the best N number of feature points using the eigenvalues in order of priority whose do not exist in approximatetransmitter area. Here, the feature points in the transmitterarea is not considered. The image area(If ea ) used for thisfeature point selection is indicated following Equation (5).
If ea = Icut − (Itra + Iof f set )(5)

In the above Equation (5), Itra is the just previously
detected transmitter area and Iof f set is set up, since transmitter can have a movement in consecutive frames. After
selecting the feature points in the base frame, gain thecorresponding set of feature points in the current frame. Eachfeature point movement between current and base frame
are determined. The movement of transmitter is determinedusing the movement of feature points. Here, the maximum
maxfrequency of the feature point movement(distance(F Pdis)max
and direction(F Pdir )) between current frame and baseframe is selected as the movement of transmitter. If thecorresponding feature points in current frame became lessthan 0.7N points, feature points are renewed. As mentioned
above, the transmitter is tracked on optical flow until it isdetected by edge-based method again. Figure 15 illustrates
the main steps of transmitter detection by optical flow.
In the above algorithm, the feature points which are inapproximate transmitter area are not used. We consider the
movement of the transmitter as same as the movement of
the N best feature points on the background since error canbe occurred in calculating feature points on the transmitterarea due to its emission. While the camera is moving ina horizontal direction and if the objects are far from thecamera, the affection on angular motion of the camera ismore than the one on horizontal direction. As a result,the movement of the transmitter take similar movement asbackground. The size of the transmitter cannot be updated
while it is tracked by optical flow and the size updatingis conducted when the transmitter is tracked by edge-based
method only. The conducted experiment to confirm theeffectiveness of proposals are explained in the next section.

30km/h∼40km/h, towards the transmitter. The moved distance of the vehicle is from 70m to 20m, from the transmitter.
Transmitter is emitted light in 500Hz and grayscale imagesof emitting transmitter are captured by high-speed camera
in 1000fps with size of 1024×512pixels. The cut out image
area was 128×128pixels.
Table 1 illustrates transmitter finding results. Each sequence in the Table 1 includes approximately 6000frames.
These sequences were captured under sunny, cloudy, andvery dark(night) conditions. In the experiments, the numberof previous frames used for the differences calculation(m)was set as five. This value was decided by conducting a subexperiment.
In this method, transmitter is found in every five frames.
The expected findings means number of maximum findings
can be happened, when transmitter is found in five framesby five frames. Success rate shows successfully found percentage out of expected findings. No false findings showedin any experiment.
Figure 16(a)v(j) show some tracking results in consecutive frames. In the Fig. 16 the red squares indicate thetracked transmitter by edge-based method and yellow squares
indicate the tracked transmitter by optical flow. The reddashes indicate the movement of the feature points. In
between Fig.16(i) and (j) it does not show any dashes,
meaning that there is not a movement between these twoframes. Figure 16(a), (e), (h), and(i) are some example fordetecting the transmitter by edge-based method. As shownin Fig. 16 (b)v(d), the transmitter is tracked in consecutive
frames on optical flow. According to the experiments sofar, the transmitter could be tracked for 100 consecutive
frames with high accuracy only on optical flow. But, it is notnecessary to track for 100 consecutive frames, transmitter istracked by edge-based method also within 100frames, sinceedge-based method can also detect it at least 20% when allLEDs emit light randomly.
VI. C ONCLUSION
The new transmitter finding and tracking methods were
proposed for a road-to-vehicle Visible Light Communication
system. All LEDs in the transmitter can be used for achieving
continuous communication with new proposals, which was
not possible before. Experimental results showed that the
new proposals are very effective in desired finding andtracking.

V. E XPERIMENTAL RESULTS
In the experiments, we fixed the high-speed camera on
a vehicle and images were captured while driving between

49（第4分冊）

Sequence
1
2
3
4
5
6

Expected
findings
1176
568
1140
1183
908
1213

Findings
1170
549
1127
1175
908
1197

Success
rate%
99.48
96.65
98.85
99.32
100.0098.68

TABLE IT RANSMITTER FINDING RESULTS

False
findings
0
0
0
0
0
0

FIT2010（第 9 回情報科学技術フォーラム）

(a)

(b)

(c)

(d)

(e)

Fig. 15.
The main steps of optical flow-based tracking (a)Arbitrarily previous frame(based frame); (b)Selected feature point on based frame; (c)Corresponding feature points on current frame; (d) Movement of the feature points between based and current frames(movement is shown by red dashes);(e) Detected transmitter on current frame

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Fig. 16.

Tracking results

ACKNOWLEDGMENT
This work was partially supported by the Ministry ofInternal Affairs and Communications, Strategic Information
and Communications RD Promotion Programme(SCOPE)
R EFERENCES[1] K. Mori, T. Takahashi, I. Ide, H. Murase, T. Miyahara, and Y. Tamatsu,
Recognition of foggy conditions by in-vehivle camera and millimeterwave radar, Proc. of IEEE Intelligent Vehicle Sym., Turkey, 2007, pp.
87-92[2] G. Y. Song, K. Y. Lee, and J. W. Lee, Vehicle detection by edgebased candidate generation and appearance-based classification, Proc.
of IEEE Intelligent Vehicle Sym., Netherlands, 2008, pp. 428-433[3] Y. Shen, U. Ozguner, K. Redmill, and J. Liu, A Robust Video based
Traffic Light Detection Algorithm for Intelligent Vehicles, Proc. ofIEEE Intelligent Vehicle Sym., China, 2009, pp. 521-526[4] S. Iwasaki, H. C. N. Premachandra, T. Yendo, T. Fujii, M. Tnimoto, Y.
Kimura, Visible Light Raod-to-Vehicle Communication Using HighSpeed Camera, Proc. of IEEE Intelligent Vehicle Sym., Netherlands,
2008, pp. 13-18[5] T. Komine and M. Nakagawa, Fundement analysis for visible-light
communication system using LED lights, IEEE Trans. of Consumer
Electronics, Vol. 150, pp.100-107, 2004
[6] K. Suzuki, M. Fujita, Y. Hayashi, K. Fukuzono, A Study on Visually
Impaired Person’s Support System Utilizing Visible Light Communication Technology at Signalized Intersections, Intern. Journ. of ITS
Research, Vol. 5, pp.37-45, 2007

[7] S. Arai, S. Mase, T. Yamazato, T. Endo, T. Fujii, M. Tanimoto, K.
Kidono, Y. Kimura and Y. Ninomiya, Experiment on HierarchicalTransmission Scheme for Visible Light Communication using LED
Traffic Light and High-Speed Camera, Proc. of IEEE Intertern. Symp.
on Wireless Vehicular Communica. (WiVeC 07), 2007[8] H. C. N. Premachandra, T. Yendo, T. Yamasato, T. Fujii, M. Tanimoto,and Y. Kimura, Detection of LED Traffic Light by Image Processingfor Visible Light Communication System, Proc. of IEEE Intelligent
Vehicle Sym., China, 2009, pp. 179-184[9] H. C. N. Premachandra, T. Yendo, M. P. Tehrani, T. Yamasato, T. Fujii,
M. Tanimoto, and Y. Kimura, Edge-Based Tracking of an LED Traffic
Light for a Road-to-Vehicle Visible Light Communication System,
Journal of Broadcast Engineering published by KoSBE, the Korean
Society for Broadcast Engineers, Vol. 14, pp.475-487, 2009[10] D. Bruce, D. Lucas, and T. Kanade, ”An Iterative Image Registration
Technique with an Application to Stereo Vision”, iProc. DARPA Image
Understanding Workshop, pp. 121-130, 1981

50
（第4分冊）