Chapter 5

Video/Image Processing with AI

Video/image processing, and inparticular digital image processing,is the processing of a digital image
through some algorithm using acomputer. Video sensors havebecome particularly importantin traffic applications mainly
due to their fast response, easy
installation, maintenance, andtheir ability to monitor wider areas.
Traffic applications like monitoring,prediction etc use AI based methods
for video image processing andanalysis. In order to develop intelligent
automated traffic applications,
it is useful to use robust AI based
methods. Traffic applications
such as vehicle detection, vehicle
segmentation, vehicle tracking andcounting and speed estimation for
careful analysis of traffic congestion
can be developed by applying object
detection, semantic segmentation,
object tracking and object counting
techniques.
Recent high-performance image
processing using AI is based on
Deep Learning (DL), which is an endto-end process of feature extraction
and recognition. Various DL
architectures have been proposed,
but the basic one is the Deep Neural
Network (DNN), which is a deep
version of the Neural Network (NN).

Especially, in the ﬁeld of image
processing, it is effective to use themodel called Convolutional Neural
Network (CNN).
A CNN (LeCun, 1998) is a NN thattakes advantage of the image
locality, where neighboring pixels
have a strong relationship with eachother, but the relationship becomes
weaker as the distance increases.
CNN is inspired by Neocognitron(Fukushima, 1982), a proposal
inspired by neurophysiologicalﬁndings on the primary visual cortex.
Both Neocognitron and CNN are
hierarchical NNs, consisting of a layer
for feature extraction and a layer for
obtaining an abstract representation
by subsampling. CNN is the basis
and probably the best choice forcurrent AI-based image processing.

Overview of analysis
process (Architecture/
Conceptual image of wholeprocess)
As shown in Figure 5-1, Video/
image processing consists ofthree main components: object
Sensing Technology (that includes
object detection, object instancesegmentation), object tracking, andobject counting. The object sensing
module identiﬁes objects in a given
frame of video and returns a list ofbounding boxes or segmented areas
around the objects to the tracker.
The tracker uses the bounding boxes
to track the objects in subsequent
frames. The detector is also used to
update the trackers periodically toensure that they are still tracking theobjects correctly. The counter counts
objects when they leave the frame or
makes use of a counting line drawn
across a road.
The counting method is based on
the object’s regional bounding box
marks and the virtual reference line.
This technique assumes that theobject movement is in a direction.
For counting, each detected object
in the detection step is assignedwith a unique label and tracked
until it reaches the virtual line. In thiswork, we have used ﬁve different
object class labels, namely auto,car, motorcycle, bus and truck. And
all these labels are categorized as
vehicle objects and will be used in
the counting system.
Detailed explanation of the individual
modules is given in the subsequent
sections.

Key Technologies
Figure 5-1 Overview of Video/Image processing

1: Object Sensing Technology

Object Sensing
Technology

1a. Object Detection

Vehicledetection/Counting
Input
Video

ObjectTracking
ObjectInstance
Segmentation

28

Vehicledetection/Counting

Traffic
Analysis

Object detection is a computer visionand image processing technology,
which deals with detecting instancesof semantic objects of certain classes(like person, car, bus, truck, etc) ina digital image/video. Currently, themost popular approach is to use a
CNN to solve both detection andclassiﬁcation tasks.

Chapter 5
Video/Image Processing with AI

There are majorly two types of object
detection: Two-stage and One-stage.
Initial works in this task were all
two-stage detectors like RCNN [5],Fast RCNN [4], Faster RCNN [6] andR-FCN [7]. These methods have highlocalization and object recognition
accuracy, but a low inference speed.
For real-time detection, one-stage
detectors like YOLO [9,10,11], SSD [12]and RetinaNet [13] are used.
These methods have high inference
speed, but comparatively low
accuracy compared to two-stage
detectors. This trade-off of detection
accuracy vs inference speed madeone-stage detectors more practicalto use for real-time applications
like autonomous vehicles, video
surveillance, traffic monitoring, etc.jj

bounding boxes. The second stage,
which is in essence Fast R-CNN,
extracts features using RoIPool fromeach candidate box and performs
classiﬁcation and bounding-box
regression. The features used byboth stages can be shared for faster
inference. We refer readers to forlatest, comprehensive comparisons
between Faster R-CNN and otherframeworks.

Mask R-CNN
YOLO

IoU is used in both detection andsegmentation tasks. In general, an
IoU > 0.5 is considered to be good.

YOLO (You Only Look Once) is
one of the single stage object
detection algorithms introduced
to achieve real-time performance.
Our applications like vehicle
detection, and counting require
getting real-time detections fromthe video feed. YOLOv4 is thestate-of-the-art object detector.
It is YOLOv3 + Bag of freebies +Bag of Specials. YOLOv5 is thefastest object detector till date. It is
YOLOv3 + Image data augmentation
techniques. Achieves more than
300 FPS (faster than YOLOv4). Threekinds of augmentations: scaling,color space adjustments, and mosaic
augmentation.

R C-NN/Faster R-CNN

1b. Instance Segmentation

R-CNN: The Region-based CNN(R-CNN) approach to boundingbox object detection is to attend
to a manageable number ofcandidate object regions andevaluate convolutional networks
independently on each RoI. R-CNN
was extended to allow attending to
RoIs on feature maps using RoIPool,
leading to faster speed and better
accuracy. Faster R-CNN advanced
this stream by learning the attention
mechanism with a Region Proposal
Network (RPN). Faster R-CNN isﬂexible and robust to many follow-up
improvements, and is the current
leading framework in several
benchmarks.

Instance Segmentation

IoU (Intersection over Union)
It is the extent of overlap between
the ground truth from the training
data and the prediction from themodel. Greater the IoU, better thepredictions.

Faster R-CNN: We begun bybrieﬂy reviewing the Faster R-CNN
detector. Faster R-CNN consistsof two stages. The ﬁrst stage,
called a Region Proposal Network(RPN), proposes candidate object

sensitive output channels fully
convolutionally. These channels
simultaneously address object
classes, boxes, and masks, making
the system fast. But FCIS exhibits
systematic errors on overlapping
instances and creates spurious
edges, showing that it is challenged
by the fundamental difficulties ofsegmenting instances.

Driven by the effectiveness of
RCNN, many approaches to instancesegmentation are based on segment
proposals. Earlier methods resorted
to bottom-up segments. DeepMask
and following works learn to propose
segment candidates, which are thenclassiﬁed by Fast R-CNN. In thesemethods, segmentation precedes
recognition, which is slow andless accurate. Likewise, Dai et al.
proposed a complex multiple-stage
cascade that predicts segment
proposals from bounding-box
proposals, followed by classiﬁcation.
Instead, our method is based onparallel prediction of masks andclass labels, which is simpler andmore ﬂexible. Most recently, Li etal. combined the segment proposal
system in an object detection system
for “fully convolutional instancesegmentation” (FCIS). The common
idea is to predict a set of position

Mask R-CNN adopts the sametwo-stage procedure, with an
identical ﬁrst stage (which is RPN).
In the second stage, in parallel topredicting the class and box offset,
Mask R-CNN also outputs a binary
mask for each RoI. This is in contrast
to most recent systems, whereclassiﬁcation depends on mask
predictions. Our approach follows
the spirit of Fast R-CNN that applies
bounding-box classiﬁcation andregression in parallel (which turned
out to largely simplify the multistage pipeline of original R-CNN.
Formally, during training, we deﬁnea multi-task loss on each sampled
RoI as L = Lcls + Lbox + Lmask. The
classiﬁcation loss Lcls and boundingbox loss Lbox are identical as thosedeﬁned in. The mask branch has a
Km2 - dimensional output for each
RoI, which encodes K binary masksof resolution m × m, one for eachof the K classes. To this we apply a
per-pixel sigmoid, and deﬁne Lmask
as the average binary cross-entropy
loss. For an RoI associated withground-truth class k, Lmask is only
deﬁned on the k-th mask (other mask
outputs do not contribute to the loss).
Our deﬁnition of Lmask allows the
network to generate masks for everyclass without competition amongclasses; we rely on the dedicatedclassiﬁcation branch to predict theclass label used to select the output
mask. This decouples mask andclass prediction. This is different from
common practice when applying
FCNs to semantic segmentation,
which typically uses a per-pixel
softmax and a multinomial crossentropy loss. In that case, masks
across classes compete; in ourcase, with a per-pixel sigmoid and a29

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

binary loss, they do not. We show by
experiments that this formulation is
key for good instance segmentation
results.

2: Object Tracking
Tracking gives information about
speed, distance and location whichhelps in solving sub-tasks like
speed estimation, vehicle tracking.
Tracking more than one object inthe video raises several challenges,
including correct mapping of identity
to each object throughout the video,
frequent occlusions, initializationand termination of tracks, and similar
appearance of different objects. To
address these difficulties/challenges,
we have used:
1. Siamese neural networks tocorrectly map identities tocorrect objects.
2. Memory module to store stateof object for a long time. It
helps in re-initiating the state ofan object that has lost due to
occlusion.
3. Segmentation mask module foran accurate representation of
objects.

Siamese Neural Networks

Methodology

It is a class of neural networkarchitectures that contain two or
more identical subnetworks withsame parameters and weights. It
is used to ﬁnd the similarity of the
inputs by comparing its feature
vectors. Euclidean distance iscalculated between these two
feature vectors to ﬁnd similarity.
Triplet loss function(L) is used to train
the network which calculates loss bytaking Anchor(A) the original image,
Positive(P) the matching image and
Negative(N) the non matching image.
Network should learn parameterssuch that the euclidean distance issmall if inputs are similar and large if
inputs are dissimilar.

Figure 5-2 represents thearchitecture to track multiple objects:
1. We use YOLO object detection
for the initial frame only to
output bounding boxes ofobjects of interest. We callthem exemplar images.
2. We use a siamese neuralnetwork fӨ (ResNet-50 base) to
extract features for each object(Feature Map X).
3. For each frame i after the ﬁrst
frame:a For each object j, we takethe region in frame i, whichcorresponds to the locationof the larger crop of thebounding box of j in frame
i-1. We call them the searchregions.
b We then send these searchregions into fӨ to obtainfeatures for each object’s
search region (Feature
Map Y).
c We use depthwise crosscorrelation (*d) between
feature maps of respective
objects to obtain an
activation map representing
the location of an object inframe i (Feature Map Z).

L(A,P,N)=max(|f(A)-f(P)|-|f(A)-f(N)|+m,0)
Where, f(A), f(P), f(N): feature vectorsof A,P, N and m is margin which
avoids zero solution.
For the task of object tracking, we
modify siamese neural networks
where instead of treating it as asimilarity learning network, we treat
it as a network which ﬁnds crosscorrelation between the feature
vectors of the two sister networks.

Figure 5-2 Proposed Architecture for Multi-object Tracking

YOLO OBJECT
DETECTORfό

Feature Map X
× 3 objects

Initial Frame Onlyfό

Feature Map Y
× 3 (For Each Larger Crop Input)

For Each Frame After 1st Frame

hƜ

Mask

bƱ

Bounding Box

sƶ

Score

Feature Map X*dFeature Map Y

30

Feature Map Z

× 3 objects

Chapter 5
Video/Image Processing with AI

d We use fully convolutional
nets (hΦ) to obtain a maskfor each object for accurate
pixel-wise tracking.
e bσ outputs a bounding box
corresponding to the highest
score sψ.
We maintain an in memory data
structure which stores the state of
all objects that are being tracked atcurrent instance. We add an object’s
state if it is new and we remove theobject’s state if it does not appear
for more than 5 seconds in video(we assume that average time ofocclusion is 5 seconds, which mayvary based on the situation).
This helps to track objects undershort term occlusions.

3: Object Counting
The widely applied methods ofobject counting mainly include object
detection and object tracking.
The earliest object detection was
to extract the moving targets fromthe image sequence and identify
the extracted targets. The object
detection method during thisperiod mainly included backgroundsubtraction method, frame difference
method, and optical ﬂow method.
However, the background subtraction
method uses the weighted averagemethod for background update,
and the effect of background
update will affect the accuracy ofobject detection. Moreover, theframe difference method is greatly
affected by the object speed andthe time interval of continuous
frames. Furthermore, the opticalﬂow method is a pixel-level density
estimation, which is not suitable
for real-time applications due to its
large computation. In recent years,
to solve the impact of complicatedscenarios towards accurate target
detection, machine learning methods
and classiﬁcation methods havebeen widely used before deeplearning becomes the mainstream ofcomputer vision. However, machinelearning methods and classiﬁcation
methods have the disadvantages ofhigh time complexity, weak region
selection, and poor robustnessof manually extracted features.
Consequently, deep learning
methods are presented for target
detection, which demonstrates thatfeatures extracted by the deep
convolutional neural networks aresuperior to hand-crafted features.
Vision-based object counting is aninteresting computer vision problem
tackled by different techniques.
As per the taxonomy accepted in,the counting approach could bebroadly classiﬁed into ﬁve maincategories: counting by frame
differencing, counting by detection,
counting by density estimation anddeep learning based counting.
The ﬁrst three counting methods
are environmentally sensitive andgenerally don’t perform very well in
occluded environments or videos
with low frame rates. While counting
by density estimation follows asupervised approach, they perform
poorly in videos that have a larger
perspective and contain oversized
objects. Density estimation based
methods are also limited in theirscope of detection and lack object
tracking capabilities. Finally, out of
all these counting approaches, deeplearning based counting techniques
have had the greatest developments
in recent years. The advancementin their built architectures havesigniﬁcantly improved the vehicle
counting performance. In this study,
we mainly focus on studying counting
methods that are founded on deeplearning based architectures.
Unlike machine learning, the existingtarget detection systems based on
deep learning can be divided into
proposal-based methods, suchas R-CNN, SPP-net, Fast R-CNN,
Faster R-CNN and Mask R-CNN,
and proposal-free methods, such asSingle Shot Multibox Detector (SSD)
and You Only Look Once (YOLO).
Unlike the proposal-based methods,
YOLO uses the method of setting
the default box and the method of
dividing the input image into a ﬁxed
grid to predict the target location andclassiﬁcation quickly, which makes
the process of training and detecting
much faster than the R-CNN series.
On the whole, the comprehensive
use of the effective deep learningmodels and dataset is of criticalimportance to improve the speed
and accuracy of object detection.

Experimental Results at
IITH testbed: Tirtl device
Tirtl device (as ground truth) is usedfor vehicle counting at IITH main gate.
As shown in Figure 5-3, our vehicle
counting algorithm complements the
Tirtl device.

Object Tracking Results
Given a video stream from asurveillance camera, our model
would process each frame in thevideo to provide positions and state
information of multiple objects.
In Figure 5-4 (a), we track andsegment general objects like bus,
person, vehicle. In Figure 5-4 (b),we track and segment only cars.
All images shown are contiguous
frames from a video.
We assign a unique identity for eachobject (which can be observed in two
contiguous frames in Figure 5-4 (a)and Figure 5-2) and are represented
using a bounding box. For moreaccurate representation of objects,
we also segment the object along
the bounding box which is evident
from ﬁgures.

Case Study
We developed an integrated systemfor other tasks like, (a) Real-Time
Detection of Motorcyclist without
Helmet using Cascade of CNNs
on Edge-device, (b) smart parking
system that helps in identifying theempty parking bays, and (c) Deﬁning
Traffic States using Spatio-temporal
Traffic Graphs using Ahmedabad
Aerial-view Dataset that helps inachieving the ﬁnal objective oftraffic congestion prediction andoptimization to reduce vehicularcarbon emission.

31

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

Figure 5-3 Tracking output of general objects. (a) & (b) are two frames in the video

(a)

(b)

Figure 5-4 Tracking output of 7 cars. (a) & (b) are two frames in the video

(b)

(a)

1: Real-Time Detectionof Motorcyclist without
Helmet using Cascade of
CNNs on Edge-device
The real-time detection of traffic ruleviolators in a city-wide surveillance
network is a highly desirable but
challenging task because it needs
to perform computationally complex
analytics on the live video streams
from a large number of cameras,
simultaneously. In this work, we
propose an efficient frameworkusing edge computing to deploy
a system for automatic detectionof bike-riders without helmets asshown in Figure 5-5 and Figure 5-6.
First, we propose a novel robust andcompact method for the detectionof motorcyclists without helmets
using convolutional neural networks(CNNs). Then, we scale it for the
real-time performance on an edgedevice by dropping redundant ﬁlters32

and quantizing the model weights.
To reduce the network latency,
we place the detector module onedge-devices in the cameras. The
edge-nodes send their detectedalerts to a central alert database
where the end users access thesealerts through a web interface. To
evaluate the proposed method, we
have collected two datasets of real
traffic videos, namely, IITH Helmet
1 which contains sparse traffic and
IITH Helmet 2 which contains dense
traffic. The experimental results showthat our method achieves a highdetection accuracy of ≈ 95% while
maintaining the real time processing
speed of ≈ 22f ps on Nvidia-TX1.
An intelligent transportation system isthe integration of various advanced
technologies such as intelligent
computing, network communications,visual representation, visual-based
analysis, efficient sensor electronics.
Since, motorcycles are an affordable

and daily mode of transport, there
has been a rapid increase inmotorcycle casualties due to thefact that most of the motorcyclists
do not wear the helmet whichmakes it an ever-present danger
every day to travel by motorcycle.
The main contributions of thispaper are, a) Design of a robust
and reliable method for detectionof moving motorcyclists in real-timeusing convolutional neural network(CNN) under the various challenging
conditions, such as viewpoint,
illumination effects, weather change,
etc., b) Acceleration of the CNN
model used to detect motorcyclists
in real-time on the limited-resource
embedded device. c) Develop
lightweight but powerful CNN modelfor efficient classiﬁcation of head(i.e. violator) and helmet, with very
limited set of training samples. d)Useof an Edge-computing framework
to overcome the communication
overhead and network latency.

Chapter 5
Video/Image Processing with AI

Figure 5-5 Block diagram of proposed framework for the detection of motorcyclists without Helmet

Figure 5-6 The architecture and user interface of the proposed edge computing-based framework for the detection(a) Framework

As described in Figure 5-7, theproposed framework for real-time
detection of motorcyclists driving
without helmets is able to perform indiverse surveillance conditions. Also,
there is a signiﬁcant reduction in
the number of false alarms becauseof the use of cascaded CNNs. The
placement of the detector modules inthe vicinity of the capturing devices
in an edge-computing framework
reduces the communication
overhead and solves the issue of

network latency. The experimental
results show the efficacy of the
proposed approach.

2: Smart Parking System
Outdoor parking lot vacancy
detection systems have attracted
a great deal of attention in thelast decade due to many practicalapplications. While parking in itself
does not seem to be a preeminentconcern, we do not realize that it(b) Interface

has signiﬁcant implications when we
consider the bigger picture of theentire driving experience and thepsychology of the driver together.
The following is a non-exhaustive
list of the impact a bad parking
experience could lead to:(i) Increased emissions of harmful
gases due to continual search for avacant spot for parking. (ii) Increased
driver frustration due to inability toﬁnd a suitable parking spot that could
subsequently lead to issues on the

Figure 5-7 Experimental results

(a)

(b)

33

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

road or elsewhere, such as running a
red signal. (iii) Cascading effect of not
ﬁnding a vacant spot, with each newentrant into the parking area, theconfusion and maneuvering spacedecreases. This creates chaos which
can take anywhere from minutes tohours to resolve. (iv) Leads to parking
on roads, which again causes
congestion on roads, which leads
to additional traffic, congestion, andfrustration, increasing both pollution
and risk of road accidents.

with modern-day processors.
• The entire parking system will be
automated, requiring bare minimum
human intervention, if any.
• Completely compatible with existing
infrastructure. Cameras already exist
in most parking lots.
• Extensible seamlessly, as long as
the new area is in the camera’s visualﬁeld, no new efforts need to be put
in it. In the case of the completely
new area, cameras may need to be
added.

The currently existing solutions tomanage parking lots are as follows:• Manual Management:
Pros:– Simple to implement. It justrequires hiring someone to work theparking area.
– No dependence on otherresources, i.e., power, internet, etc.
Cons:
It– Impossible to scale to large and(or) multi-ﬂoor parking facilities.
– Prone to human errors.
– Human limitations, memory, logic,
etc.
• Sensor Based Management:
Pros:– Automatic system, no humanmanagement required.
– Reliable system, i.e., low error rate.
Cons:– Not extensible, new parking
areas cannot be added as andwhen required without bringing
new sensors, placing them, andconnecting them to the grid,following which software changes
need to be made to make it aware of
the new spaces found.
– Not compatible with existing
infrastructure, new hardware insensors and connection options(wires, wireless, etc.) will be required.

Implemented Prototype

As shown in Figure 5-8, we propose
a computer-vision solution to guidethe drivers attempting to park theirvehicles using computer-detectedavailable parking slots. The
proposed solution solves most of the
problems mentioned in the solutions
as mentioned earlier while keeping
most of their advantages.
• Computer vision is lightning fast

34

Server SideThe server side has two main
components:• A ﬂask server.
• A deep learning model inTensorFlow.

Flask server
The ﬂask server essentially acts
as an abstraction layer between
the deep learning model and any
application that uses the information.
In our project, we have created theserver so that the application canquery the server at a predetermined
API call using a URL and a port at
which the server has exposed an
endpoint. On receiving this query,the server determines the current
status of the parking lot using thelearned model and thereby sends
the summary of this information in theform of a JSON object.

project. The application essentiallyqueries the server for the status
information of the parking lot and,with this information, presents thedriver with parking assistance in themost straightforward way. Ideally,using shortest path algorithms like
Dijkstra’s, we can let the driver know
the nearest position to park. The
prototype presented was merely
a beginning, and this project has
a vast scope for innovation andenhancement. Some immediate
measures we thought of are as
follows:• Add segmentation to the deeplearning model.
• Add real-time communication
between CCTV cameras and theserver.
• Use graph algorithms in both app
and server to make parking more
systematic and faster.
• Implement a GPS so that as soon as
the driver approaches a parking lot,the parking lot can be identiﬁed andthe server can be queried strictly forthis parking lot automatically, ﬁnallydelivering a notiﬁcation to the driver’s
device.
• Add security measures in which
sensitive information like the license
plates and faces can be auto-blurred.

3: Deﬁning Traffic Statesusing Spatio-temporal Traffic
Graphs using Ahmedabad
Aerial-view Dataset

Model
The model we have used todemonstrate is a straightforward one.
It has the following steps:• The parking lot is segmented
into bays, i.e., parking slots. This isideally done through a segmentation
process using a Mask R-CNN, forexample.
• Proceeding from the above, we doa binary classiﬁcation of whether aslot is occupied or vacant. The server
delivers the summary of the above
output to the application.

End-User App Prototype
We have also made an android
application to demonstrate how an
end-user would beneﬁt from ourAs described in Figure 5-9, Different
types of traffic states (a) clumping the vehicles bunch together rapidly,(b) neutral - the vehicles maintain thesame relative speed and gap, and (c)unclamping- the vehicles disperse
away rapidly.
Intersections are one of the main
sources of congestion and hence,
it is important to understand traffic
behavior at intersections. Particularly,
in developing countries with highvehicle density, mixed traffic type,and lane-less driving behavior, it isdifficult to distinguish between
congested and normal traffic behavior.
In this work, we propose a way toChapter 5
Video/Image Processing with AI

Figure 5-8 Proposed architecture of Smart parking system

(a)

(b)

(c)

35

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

Figure 5-9 Different types of traffic states(a) clump(b) neutral(c) unclump

understand the traffic state of smaller
spatial regions at intersections using
traffic graphs. The way these trafficgraphs evolve over time reveals
different traffic states - a) congestion
is forming (clumping), b) congestion
is dispersing (unclamping), or c) traffic
is ﬂowing normally (neutral). We train
a spatio-temporal deep network to
identify these changes. Also, we
introduce a large dataset called
EyeonTraffic (EoT) containing 3 hours
of aerial videos collected at 3 busy
intersections in Ahmedabad, India.
Our experiments on the EoT dataset
show that the traffic graphs can help
in correctly identifying congestionprone behavior in different spatial
regions of an intersection.
Spatial regions considered forannotation at each of the threeintersections (Paldi, Nehru Bridgeand APMC) in the EoT dataset. Red
denotes clumping, yellow denotes
neutral, and blue denotes unclamping.
Each way is denoted by a number.
Intersections are a major causeof congestion in urban networks
especially in the case of lane-less36

mixed traffic where a large numberof smaller vehicles bunch together
at stop signs in an irregular fashion.
Determining the level of congestion
at intersections is mainly performed inexisting works by counting the numberof vehicles [ 1 ]. While this is suitable
for lane-based traffic, these methods
are not applicable to high irregulartraffic density with varying sizes ofvehicles in developing countries like
India. Furthermore, there is a highpropensity for lateral movements andlow gap maintenance in such trafficthat may indicate congestion but may
actually denote normal traffic ﬂow.
The aforementioned issues make itimperative to deﬁne a set of traffic
states speciﬁcally for understanding
mixed lane-less traffic. Hence, inthis work, we propose a novel
characterization of traffic states using
traffic graphs. The evolution of theseinteractions can be best represented
using traffic graphs [2] that changeover time. Particularly, we observedthat the traffic graph structure of aparticular spatial region changes
when too many vehicles congregate
or disperse in a short span of time.

The temporal pattern of changing
neighbors is best represented using
the adjacency matrix of the trafficgraph. So, we propose to learn thespatio-temporal pattern of theseadjacency matrices using a network
consisting of Convolutional Neural
Network (CNN) and Gated Recurrent
Unit (GRU) units.
In this work, an approach to identify
traffic states in lane-less traffic using
temporal changes in the traffic graph
was proposed. We showed that aspatio-temporal CNN-GRU network
trained on the adjacency matrix of atraffic graph can identify clumping,
unclumping, and neutral traffic
states in various spatial regions ofan intersection. Further, we showed
the effectiveness of the proposed
approach on a large annotatedaerial dataset called EyeonTrafficthat covered 3 intersections in
Ahmedabad, India. In the future,
we would like to apply this approachto predict the onset of congestion
by observing the changes in traffic
behavior.

Chapter 5
Video/Image Processing with AI

Figure 5-10 Annotation at the three intersections in the EoT dataset(a) Paldi (P)(b) Nehru Bridge–Ashuram Road (N)(c) APMC Market (A)References[ 1 ] Y. LeCun, L. Bottou, Y., Bengio, and P. Haffner (1998) Gradient-based learning applied to document recognition, Proc. IEEE,
Vol. 86, No. 11, pp.2278–2324.
[ 2 ] K. Fukushima and S. Miyake (1982) Neocognitron: A new algorithm for pattern recognition tolerant of deformations andshifts in position, Pattern Recognit., Vol. 15, No. 6, pp. 455–469.
[ 3 ] Roy, Debaditya, K. Naveen Kumar, and C. Krishna Mohan. “Deﬁning Traffic States using Spatio-temporal Traffic Graphs.” In2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–6. IEEE, 2020.
[ 4 ] Singh, Dinesh, C. Vishnu, and C. Krishna Mohan. “Real-Time Detection of Motorcyclist without Helmet using Cascade of
CNNs on Edge-device.” In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–8.
IEEE, 2020.
[ 5 ] P. Chakraborty, Y. O. Adu-Gyamﬁ, S. Poddar, V. Ahsani, A. Sharma,and S. Sarkar, “Traffic congestion detection from camera
images using deep convolution neural networks,” Transportation Research Record, vol. 2672, no. 45, pp. 222–231, 2018.
[ 6 ] L. Zhao, Y. Song, C. Zhang, Y. Liu, P. Wang, T. Lin, M. Deng, andH. Li, “T-gcn: A temporal graph convolutional network fortraffic prediction,” IEEE Transactions on Intelligent Transportation Systems, 2019.
[ 7 ] Y.-B. Lin and C.-P. Young, “High-precision bicycle detection on single side-view image based on the geometric
relationship,” Pattern Recognition, vol. 63, pp. 334–354, 2017.
[ 8 ] Wang, Qiang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. “Fast online object tracking and segmentation:
A unifying approach.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 1328–1338, 2019.
[ 9 ] Li, Bo, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. “High performance visual tracking with siamese region proposal
network.” In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 8971–8980, 2018.
[10] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: uniﬁed, real-time target detection,” in Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1804–1812, 201637