Chapter 6

Vehicle Detection by CCTV

Vehicle detection is the essential
technology to monitor traffic statuses
such as traffic volume, travel
speed, and traffic density in a city.
Traditionally, a Loop-coil detector or
ultrasonic detector has been used as
the vehicle detector to monitor traffic
states. However, the cost of installingsuch a vehicle detector is costly.
So, vehicle detection by a Closedcircuit television camera (CCTV) is
one of the practical approaches.
The performance of the camera has
improved, and even inexpensive
cameras can now shoot with highresolution. Also, identifying a vehicle
from the video image became
reliable because of the recent
development of image processing
technology. Besides, the image
data compression technology andtransfer technology have improved
dramatically. Due to the above
technological innovations, CCTV
vehicle detection technology has
become a very inexpensive anddeployable technology.
This technology has already been
introduced in many developed cities,
but some problems could be solved
when introduced in developing cities.
One is how to detect mixed traffic,
including small type vehicles such as
an auto-rickshaw, a motorcycle, etc.
Since those vehicles, it is challenging
to observe each of these vehicles’
conditions, as they often run as agroup without following the lane.
In developing countries, the traffic
is highly heterogeneous and often
densely populated. With the helpof CCTV feed from Ahmedabad,the vehicle detection datasetwith classes like motorcycle andauto-rickshaw, in addition to theclasses like car, bus, truck, etc wereannotated as part of the project.

38

This collection and annotation ofdata speciﬁc to Indian roads will help
boost the performance of the vehicle
detection model because of thesimilarity of distributions of training
data and testing data (data from
Ahmedabad).

Introduction
1: Case Study Field
The project’s case study ﬁeld isthe west side of Ahmedabad cityof Gujarat state in India, where it is
one of the cities rapidly economic
grow and heavy traffic congestion bytransportation demand.

2: CCTV
Closed-circuit television (CCTV),
also known as video surveillance,
uses video monitoring cameras
to transmit a signal to a speciﬁcplace on a limited set of monitors.
It differs from broadcast televisionin that the traffic signal is not openlytransmitted, though it may employ
point-to-point (P2P), point-tomultipoint (P2MP), or mesh wired,wireless links. Though almost all
video cameras ﬁt this deﬁnition,
the term is most often applied tothose used for surveillance in areasthat may need monitoring, such as
banks, stores, and other areas wheresecurity is needed. In this program,
we installed a 360-degree direction
local controlled movement with a
4K resolution camera as shown in
Figure 6-1.

3: Vehicle Detection
In this section, programs weredeveloped to draw reference linesand detect vehicles in each frame.
The programs allow users to draw

reference lines in the ﬁrst frame of avideo ﬁle, programs were developed
to provide threshold values to sortout potential detected objects.
The outputs of a processed image
are rectangular bounding boxes andthe score of those boxes. The scoremeans a conﬁdence level for a
detected object, i.e., how conﬁdent
YOLO is that the box contains a valid
object. This value lies in the range of
0 to 1. The higher the score value,
the higher the conﬁdence that an
object is indeed an object of interest.
The bounding boxes are processedusing a non-maximal suppression
method in YOLO. The controlling
factor in this process is a thresholdvalue (a factor to screen out
bounding boxes with low conﬁdence
levels), which means that boundingboxes having a score value higher
than threshold are accepted forfurther processing. Those thathave values below threshold are

Figure 6-1 Monitoring Camera, 360-degrees
4K Camera (photo by Zero Sum Ltd.)Chapter 6
Vehicle Detection by CCTV

eliminated. The sorted boundingboxes are sent to the tracker
program to track detected vehicles.

4: Vehicle Tracking
In this step, the sorted boundingboxes are tracked frame by frame.
For the tracking process, Kalman Box
Tracker was used. In this research,
programs were developed to drawa centerline on each boundingbox. Since a video ﬁle consists ofthousands of frames, an object
must be tracked from one frame toanother to determine its direction of
movement. This tracker compares thecurrent frame with the immediately
previous frame using the pixel
variance of the frame. When it ﬁnds
a similarity in pixel values, it updates
the object (i.e., bounding box) andmemorizes it for consideration in thenext frame. Then, it compares the
updated frame to the next frame.
The tracker titles each tracked
bounding box by a numeric value
such as 1, 2, 3, etc. In this way,
the tracker tracks an object fromframe to frame.
Rectangles are drawn around eachtracked object on the computer
screen. The tracked rectangles areused for vehicle counting.

5: Vehicle Counting
Vehicle counting plays a signiﬁcantrole in vehicle behavior analysis
and traffic density detection for
better traffic optimization. Accurate
estimation of the number of vehicles
on the road is an important endeavor
in the intelligent transportation
system (ITS). An effective measureof on-road vehicles can havea plethora of applications intransportation sciences including
traffic management, signal control
and on-street parking.
In the earlier days before the riseof machine learning, the process ofvehicle counting was done manually.
It was performed by a person
standing by the roadside; using anelectronic device to record the datausing a tally sheet. In some cases,
the person may do the counting byobserving video footage captured bycity cams or closed-circuit television(CCTV) placed above the roador highway. Although the manual
method provides high accuracy,
it requires an extensive amount of
human resources. Besides, it tends
to be error-prone, especially onsevere traffic ﬂow and multiple roadlines. Therefore, manual calculations
are usually performed with only asmall sample of data, and the results
are extrapolated for the whole year
or season for long-term forecasts.
Existing sensor methods and the
traditional image processing method
have the problems of difficulty
in installation, high cost, and low
precision, which resulted in serious
road damage, expensive information
construction, and poor vehicle
counting accuracy. Therefore, it isof great theoretical and practicalsigniﬁcance to make full use ofexisting monitoring resources andapply the methods of deep learningand computer vision to study thevideo-based vehicle counting
method for traffic monitoring andtraffic optimization.
The vehicle counting system is madeup of three main components:
a detector, tracker and counter.
The detector identiﬁes vehicles ina given frame of video and returns
a list of bounding boxes around thevehicles to the tracker. The tracker
uses the bounding boxes to track
the vehicles in subsequent frames.
The detector is also used to
update the trackers periodically toensure that they are still tracking
the vehicles correctly. The countercounts vehicles when they leave theframe or makes use of a counting
line drawn across a road.
The counting method is based on
the vehicle regional bounding box
marks and the virtual reference line.
This technique assumes that thevehicle movement is in a direction.
For counting, each detected vehicle
in the detection step is assigned witha unique label and tracked until itreaches the virtual line. In this work,
we have used ﬁve different classlabels, namely auto, car, motorcycle,
bus and truck. And all these labels
are categorized as vehicle objects
and will be used in the counting
system. After that, each vehicle
position is checked whether it has
crossed the horizontal reference line
(red line) at the y-axis as drawn inﬁgure. If it passes the line, then it will
be counted as one.

Experimental Results
Figure 6-2 is the output on the Paldi
junction in Ahmedabad for an intervalof 15 sec during the night conditions,
red line is the reference line andvehicles are counted from both thedirections and the value is updated
at the left topmost corner.
Figure 6-3 is the output on the IIT
Hyderabad main gate for an intervalof 15 sec, the two red lines are thereference lines at both the directionsand vehicles are counted frommultiple directions and the value isupdated at the left topmost corner.
Figure 6-4 shows the vehicle
distribution for 20 min at Paldi
junction during the peak intervalof the day and also the pie chart
explains the vehicle type distribution
during that interval. Motorbikes
contribute to the highest occupancyof 46% on the road followed by Auto
which contribute 29% of the roadtraffic during the peak interval.

Future Issue
Based on the results of this chapter,
we are considering the deploymentof our system to other cities as a
future task. Secondly, it is necessary
to deal with the night-time problem ofsystems using vehicle detection by
image sensors. In the following, we
ﬁrst explain the possible problems
in applying the method describedin this chapter to other cities. Next,
we present two methods that are
generally considered to be effectivein addressing this problem, based

39

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

Figure 6-2 Vehicle counting on Paldi junction data (Ahmedabad) for
reference line

Figure 6-3 Vehicle counting on IITH data for multiple directions

Figure 6-4 Vehicle count distribution with multiple vehicle types at Paldi junction(Fine Tuning), and the other is tolearn only the parameters of theoutput layer, which is newly prepared
for a new task or data.
In this veriﬁcation, we adopt M2Det [ 1 ],which has been proposed as arobust object detection model forsmall objects. M2Det consists ofa backbone network, a multi-levelfeature pyramid network (MLFPN)
and a prediction layer. M2Det ischaracterized by a multi-level feature
pyramid (hierarchical structure offeature maps), which consists of multiscale and multi-level feature maps.
on simple experimental results.
Furthermore, we present the resultsof using ToF sensors as one of thesolutions to the problem of vehicle
detection at night by image sensors.
Expansion to other cities
The performance of CNN is highly
dependent on the quality andquantity of training data. Because of
the speciﬁc appearance of the target
environment, it is more importantto prepare data that matches thetarget than to prepare generallarge-scale data. However, the costof preparing enough training data
is a major problem. There are twotechniques, called Transfer Learningand Data augmentation, they may
be useful in extending the results ofthis case study to other cities. These
are approaches that can be applied
to a variety of methods withoutmajor changes to the framework.

40

A simple experiment will show theeffectiveness of these techniques.

1: Transfer Learning
In order to achieve the best
performance in traffic image analysis
and traffic measurement, it is useful
to have a sufficient amount of training
data in the target environment, but
the cost of this task is prohibitive. To
reduce this cost, we have tested theeffectiveness of transfer learning fortraining a CNN.
Transfer learning is a method of
efficiently and effectively training
a model to ﬁt a new task or data(domain) by reusing the results oftraining on another related task
or data (domain). Typical transfer
learning in DNN has two main
approaches: one is to retrain theparameters of the whole model

By training M2Det on large-scale data,
backbone network and MLFPN of
M2Det, are expected to acquirefeature extraction methods thatare generally effective for object
detection, including vehicle detection.
Therefore, we ﬁrst train M2Det onlarge scale data for generic object
detection. Then, a vehicle detection
model tuned to the target junctionin India is built by transfer learning,where only the prediction layer,
which is the output layer of M2Det, istrained using a small-scale data setof the target junction.
The M2Det model, which is trainedon the COCO dataset [2] for genericobject detection consisting of 80
classes, is transfer learned under two
different conditions. One condition istrained using a large-scale dataset,the India driving dataset (IDD), andthe other condition is trained usingChapter 6
Vehicle Detection by CCTV

Table 6-1 Details of each data set
IDD
54,25863,12132,2809,723
15,305Car
Motorbike
Auto
Bus
Truck

Paldi
328
556
748
156
40

output by the system. The correct
results are the out results minus theincorrect counting.
The results suggest the effectivenessof using small data sets created in
the target environment rather thanlarge-scale data sets.

2: Data Augmentation
our own small-scale dataset (Paldi).
IDD [3] is a vehicle detection dataset
provided by Indian Institute ofInformation Technology, Hyderabad.
The Paldi dataset was annotated byus using 100 images from the videoof the target Paldi junction. Table 6-1
shows the number of vehicles ineach dataset.

Data Augmentation is one ofeffective methods to increase object
detection performance. It is almost
impossible to manually collectenough numbers of image data toachieve object recognition engine,
because the data scale often could
be several thousand or more.

The result of the vehicle counts
for each training condition on a
30-minute video of the Paldi junction
are shown in the Table 6-2. The out
results are the vehicle count results

In outdoor object recognition likevehicle and pedestrian recognition,
the “appearance” changes
remarkably depending on thesunshine conditions, in addition to

the movement of the object itself.
It is costly and not realistic to learn
all vehicle data corresponding tothis change. For the issue, the data
augmentation is applied. In it,
artiﬁcially adding deformationaccording to the expected change
to the seed image and inﬂating thetraining data is used as a seed ofdata acquired very coarsely for thechange. We focused Yolo V3 [4],well known object detection andclassiﬁcation method, and apply it todetect auto-rickshaw with using thedata augmentation. We compare thedetection accuracy (recall and false
positive rate) between learning
auto-rickshaw with very few 250
samples and learning it by applying
data augmentation, where rotatedand brightness changed data is
artiﬁcially generated up to 17,800
samples. Figure 6-5 show an
example of the augmentation for the
rickshaw image. Table 6-3 shows the

Table 6-2 Comparison of measurement accuracy under different learning conditions

Car
320
352
286
89.4
81.3

Ground truth (A)Out results (B)Correct results (C)Recall [%] (C/A)
Precision [%] (C/B)

IDD
Motor-bike Auto
Bus
976
575
90
1321
405
71
877
369
59
89.9
64.2
65.6
66.4
91.1
83.1

Truck
26
59
18
69.2
30.5

Total
1987
2208
1609
81.0
72.9

Car
320
319
291
90.9
91.2

Paldi
Motor-bike Auto
Bus
976
575
90
980
490
68
845
438
66
86.6
76.2
73.3
86.2
89.4
97.1

Truck
26
20
16
61.5
80.0

Total
1987
1877
1656
83.3
88.2

Figure 6-5 Data Augmentation Examples for the Rickshaw Image

Data Augmentation

Original

Brightness changed

Color intensity

RotatedTable 6-3 Comparison of Auto-Rickshaw Detection Performance
Recall Ratio(True Positive/Correct Ans.)

Error Ratio(Err./Detected)① Small Data (250)

46% (38/82)

2.5% (1/39)② Augmentation (17,800)

70% (58/82)

1.6% (1/59)

41

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

Figure 6-6 Example of Improvement for Auto-Rickshaw Detection① Small Data

② Augmentation

result. As expected, the recall ratio
increased from 46% to 70%, and thefalse positive rate decreased from
2.5% to 1.6% simultaneously. In thisexperiment, number of correct autorickshaw, should be detected, is 82.
And, detected number for small data
and augmented data are 38 and58, respectively. At the same time,false detect number for both is 1,during detect number for small data
and augmented data are 39 and 59,
respectively.

3: Adaptation to night
lighting environment

Figure 6-6 shows the comparison of
auto-rickshaw detection. In the Right
Result Augmentation is used, andthe bottom-left rickshaw is correctly
detected. Figure 6-7 shows anew
detected rickshaw examples withthe augmentation. These rickshaws
cannot be detected in small data set.

Image-type vehicle detectors areaffected by ambient light, e.g.
headlights at night. We believe thatan effective solution to this problem
is the supplementary use of laser
ranging image sensors (ToF sensors),
which are not affected by lighting.
A Time-of-ﬂight (ToF) sensor
irradiates near-infrared pulse andmeasures the distance from the time
it takes for the laser reﬂected fromthe object. Since the near-infrared
pulse uses a coaxial optical systemthat irradiates and receives light at
points, it is resistant to the effectsof ambient light and can measure
distance values day and night.
The 3D sensor has a disturbanceresistance of 200,000 lux or more
for about 130,000 lux of direct

Figure 6-8 Sensor Installation and Detection Range

42

Figure 6-7 A New Detected Rickshaw Examples

sunlight in midsummer, and canbe used outdoors at an operating
temperature of −10℃ to +50℃.
Figure 6-8 shows the installation
position and detection range
assumed in this study. The installation
position was on the shoulder, and theshooting direction was an incident
angle of 90 degrees with respect tothe vehicle running in the lane. The
maximum detection range is 15 m.
Calculations show that if the distancefrom the sensor to the vehicle is atleast 2.1 m, it is possible to detectvehicles up to 70 km/h.
The background subtraction methodof image processing can be usedfor vehicle detection. Figure 6-9
shows a captured image of the ToF
sensor and an image of backgroundsubtraction. The distance information
to the sensor is expressed in color

Chapter 6
Vehicle Detection by CCTV

Figure 6-9 Image of ToF Sensor and Background Subtraction Method
FAR

Distance
SampleNear
Background image

Object image

Extract object

Figure 6-10 Detection Result Sample of Lane 3

Camera image

for the same two-dimensional image
as the camera image. The closer the
distance, the redder the color,
and the farther the distance,
the bluer the color. From thebackground image on the left,
the moving object can be extracted
as shown in the image on the rightby taking the subtraction from theimage containing the moving object
in the middle. However, due tothe characteristics of the sensor,
the movement of the backgroundis taken as noise, so we devised
ways to eliminate noise, such as
preparing multiple back-groundimages. Here, the extracted moving
object is to be determined whether
a vehicle or not. The width of the

Detection result

moving object is calculated as thevehicle length, and those below thethreshold are removed. We also
check the ground contact conditions
at the bottom of the moving object,because the tires are contacted withground absolutely. A moving object
is detected as a vehicle only whenthe vehicle length judgment andthe ground contact condition aresatisﬁed. The vehicle judgment isperformed only in the ﬁrst lane. And
the ground contact condition cannot
be conﬁrmed in the second andthird lane due to the distance is far.
Therefore, it is deter-mined whether
or not a vehicle based on the size ofthe moving object.

The vehicle detection performanceat night (after 21:00) was tested
on a one-way three-lane road inJapan. The results of the experimentshowed that the detection accuracyof the ToF sensor was 89% on
average. An example of detection
in lane 3 is shown in Figure 6-10. In
particular, the accuracy was 100% in
the traffic measurement in the front
lane, conﬁrming the effectivenessof the sensors as vehicle detectors.
The results show that the 3D sensor
is a promising complement to theimage sensor, which has a problemof accuracy degradation due toheadlights.

References[ 1 ] Zhao, Q., T. Sheng, Y. Wang, Z. Tang, Y. Chen, L. Cai, and H. Ling (2019) M2Det: A Single-Shot Object Detector based on
Multi-Level Feature Pyramid Network, The Thirty-Third AAAI Conf. on Artiﬁcial Intelligence, pp. 9259–9266.
[ 2 ] Lin, Y.T., M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár: Microsoft
COCO: Common Objects in Context, In ECCV 2014, pp.740–755, 2014.
[ 3 ] Varma, G., A. Subramanian, A. Namboodiri, M. Chandraker, C. V. Jawahar: A Dataset for Exploring Problems ofAutonomous Navigation in Unconstrained Environments, IEEE Winter Conference. on Applications of Computer Vision,
pp. 1743–1751, 2018.
[ 4 ] Redmon, J., and A., Farhadi; “YOLOv3: An Incremental Improvement”, arXiv:1804.02767, 2018.

43