Chapter 11

Big Data Analytics

Introduction
Lot of traffic related data of different
modalities gets generated intransportation setup, especially inurban regions and smart cities. The
traffic cameras installed at different
junctions capture traffic videos
and send them to central servers.
There are cameras with additional
processing capabilities that cancount the number of vehicles atdifferent time instances, and alsosystems that can detect speed,
headway etc. Often users reporttraffic-related incidents or scenarios
through social media. All these data
can be helpful to generate several
insights about current and future
traffic scenarios including but notlimited to data-driven identiﬁcation of
traffic bottleneck, seasonality in traffic
volume, route planning, and also
identifying and alerting about current
or future traffic incidents.

1: Collecting Traffic Related
Data for Processing
The traffic related big data can beof various modalities such as video(feeds from traffic cameras), streamthat the number of larger vehicles(possibly heavy vehicles) increaseslightly around that time. Thesevehicles are generally slower due
to the load. This mixed traffic leads
to a slowdown in the speed. The
plots also corroborate the fact thatvehicle speeds are generally higherof numbers (such as counts reported
by traffic counters), text and image(traffic-related posts by users andcity governance). Such data can be
stored in appropriate storage forfurther analysis.

2: Analyzing Traffic Datafor Better InsightAs part of the project, traffic cameras
were set up at different parts of thecity. The locations of the cameras areshown in Figure 11-1. The cameras
additionally gave information about
the number of vehicles that crossed
the junctions at speciﬁc intervals,
vehicle speed, length of the vehicles
etc. This information was extracted
and analyzed.

Figure 11-1 The region considered for the
study related to traffic counts

For example, Figure 11-2 shows thevariation of speed and the variationof vehicle length at two junctions,as captured by the cameras. We
see that the average speed drops
down after 10 am, which is generally
true due to work schedules ofoffices and businesses. The speed
decreases further around 3 pm,
which is generally not a busy hour.
However, we see from Figure 11-3

Figure 11-2 Average Speed Heatmap

Figure 11-3 Length of vehicles

Daily Variation in speed (All Days)

Daily Variation in length (All Days)

Monday38

Monday7.0

Tuesday36

Tuesday

6.5

Wednesday

6.0

Wednesday

34

ThursdayThursday

5.5

32

FridayFriday30

Saturday28

Sunday5.0

Saturday
4.5

Sunday

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

4.0

Hour of day (24 Hr format)

Hour of day (24 Hr format)

63

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

Table 11-1 Traffic Count Prediction
Model
LSTM
Conv LSTM
Social LSTM
Deep Transformer

RMSE
16.44
15.73
15.45
15.43

MAPE
33.38%
23.60%
20.97%
20.98%

Figure 11-4 Root Mean Squared Error vs actual counts

Time series modeling canbe performed using different
techniques. One can use the ARIMA
based statistical models for thesame or use the recent modelingtechniques involving LSTMs andstacked-LSTMs. These techniques
only consider the past counts forthe future predictions. However, thetraffic volume is also impacted by thespatial layout of the junctions, andtraffic from one junction is inﬂuencedby (and itself inﬂuences) traffic atother junctions. Hence advanced
methods, that consider spatial
information also while making thefuture traffic predictions can give
better predictions.

Figure 11-5 Percentage Error vs Actual Counton weekends due to lesser traffic
on the road. This may also indicate
the necessity of deploying traffic
personnel on road to regulate thetraffic at these hours.

3: Predicting Future Traffic
Appropriate forecasting of future
traffic can help in predicting
congestions, which can in turn help
in making various decisions. For
example, based on this information
civic bodies can decide on
deployment of traffic personnel,
adjust signal timings, and individuals

64

timestamps (say 7 am, 7:15 am,
7:30 am etc.) and the output is the
expected traffic volume at thatjunction at some future timestamp(e.g. 9 am). These traffic counts canbe obtained from the traffic counters(either from the traffic cameras
directly) or by analyzing the traffic
videos captured from different
junctions.

can make appropriate routing
decisions. Such actions reduce
congestion and the waiting/plying
time of vehicles on the road, thereby
reducing the amount of emission.
However, traffic forecasting, which
deals with the prediction of traffic
at different future timesteps basedon past traffic information, is anextremely challenging task due tothe highly complex traffic patterns.
The traffic forecasting problem
has often been posed as a timeseries prediction problem, where
the input is the traffic volume (e.g.
vehicle count) at the different past

To verify the possibility of making
good predictions for future traffic
volume, a case study was performed
based on traffic data collectedfrom the Ahmedabad city. The
performance of different models
for this prediction task are shownin Table 11-1. Here, RMSE stands for
Root Mean Square Error and MAPE
stands for Mean Average Percentage
Error. Since both are error metrics,
lower values indicate better
performance.
From the above comparison, we seethat the social LSTM that considers
the spatial information, and the deeptransformer model that considers
higher order interaction between
counts can predict the traffic volume
with more accuracy. The column plots
shown in Figure 11-4 and Figure 11-5
indicate that as the count of actual
vehicles increases, the misprediction
count (RMSE) also increases slightly,
however the percentage error comes
down considerably.

Chapter 11
Big Data Analytics

Figure 11-6 Traffic Count Prediction

Figure 11-7 Estimated MFD indicators

Production:
averageweighted flow
ﬂow
Production:
averageweighted

5000
5000
4000
4000
3000
3000
2000
2000
1000
1000

00
00

50
50

100
100

150
150

200
200

Accumulation:
averageweighteddensity
Accumulation:
averageweighteddensity

These observations mean that by
employing appropriate methods,
it is possible to predict the volume
of future traffic on the roads. Thiscan help in better preparednessof intelligent decision making forpredicted high congestion levels at
future timestamps.

VÚ¬æ¬Ä¤ÚcÚĆ
Condition
Macroscopic Fundamental Diagramor MFD is one of key performanceindicator to express area traffic
states. MFD expresses the interaction
between area traffic states whichconnecting the total number of carson the road at any given time (theaccumulation) with the rate at whichtrips reach their destinations (theoutput). The area is set as group of
detectors in Figure 11-6. Paldi area
(Yellow boundary) is a targeted areaof estimation form other three area.
Both MFD indicators were estimatedwith high accuracy in comparison
with observed data form traffic
detectors in Figure 11-7.
MFD is available for local
government and its traffic agency
to manage and control traffic in the

viewpoint of area. Dynamic Inﬂow
restrictions can be applied theareas where traffic is concentrated
and it can be said that traffic canbe dispersed spatially, leading toefficient use of the road network as
stock.

Using Social Media
Data for IdentifyingcÚĆ.Ä¬ÄæÞ
Often road traffic congestion canbe induced by incidents or events
like road accidents, infrastructuredamages, rallies, protests, adverse
weather events, disabled vehicles,
roadway debris, daily rush hours etc.
Detection of these incidents on timeor ahead of time wherever possible
will help the traffic authorities toalleviate road congestion problem,and can help the commuters as they
can pre-plan their trip accordingly.
As mentioned earlier, several
advanced devices such as loop
detectors, GPS probe vehicles,
cameras etc. installed ontransportation network can help
detect traffic congestion. But, due
to rapid growth of transportation
networks the cost of procurement,

installation and maintenance of thesesensors also increases. On the other
hand, with more and more people
joining social media and posting
information about real informationfrom the ground, obtaining traffic
related information from social media
has become a possibility. Suchinformation would complement the
information which can be obtained
from the hardware devices in the
traffic network.
It is observed that civic authorities
as well as general people or groups
often publish traffic related data insocial media sites like Twitter.
The Twitter handles of Traffic Policeof several major Indian cities arehighly active and give periodic
updates and alerts about traffic
ﬂow. Users also report about
congestions, road blockades,
diversions. Interestingly, it has also
been observed that often reports
or news about future incidents likerallies, processions, marches etc.
that can affect traffic are posted insocial media well ahead in time.
The impact of such events on
the traffic can be detected by thecameras, loop detectors and otherhardware sensors only at the time of
the impact. Active consumption and

65

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

Figure 11-8 Sample tweets about traffic scenario or incidents

ﬁltering of social media data can thus
complement the information givenby the hardware sensors.
As it can be seen from the tweets
shown in Figure 11-8, many posts
contain information about current
situation, situation due to recent
past incident and also about future

Figure 11-9 Workﬂow of a system forutilizing social media posts for traffic
information extraction

Post extraction

Post classiﬁcation

Fine grained
information extraction(Location, time, type)

Notiﬁcation/Broadcast

66

events that can impact traffic. Oftenthese posts contain speciﬁc location
information to get an idea of which
region will be impacted. Timely
extraction of such posts, and thenidentiﬁcation of the nature of impact,time, and location can be helpful for
traffic planning.
As part of the project, a system wasdeveloped to consume social media
posts and ﬁrstly determine whether
they are about traffic incidents.
Important ﬁne-grained informationfrom the posts were extracted.
The same information can then bebroadcast of notiﬁed to users. A brief
workﬂow of the system is shown in
Figure 11-9.
For example, consider the following
tweets (real tweets extracted fromsocial media):
1. “All are invited to attend therally for ﬁling Nomination
Papers of Dr Bhagwanth Rao
BJP candidate of Hyderabad
parliament Constituency
on 25/ 3/ 2019 at 9amfrom Bhagyalaxmi temple
Charminar”
2. “#Pedestrians account for 40%
of all #road #traffic deaths in#Bengaluru. It is criticalto design our #streets for#pedestrians, save lives -@akhilasuri talks to@ChristinMP_TOI”
3. “Surgery of Tumor 23.5 + kg
at Gujarat Cancer ResearchInstitute:GCRI Ahmedabad#oncology #surgery#oncosurgery #tumor #Cancer#India #Gujarat #Ahmedabad@GCRI_1972 @civilhospamd@PariseemaD @bonestumor@MoHFW_GUJARAT”
Among the above tweets, the ﬁrst
tweet is related to traffic incidents.
The third one is not related to traffic
incident. Although the second
tweet belongs to traffic policy, it is
not related to traffic incident. A welltrained machine learning system
is able to classify such contents.
Also from the ﬁst tweet, speciﬁc
information like event type=“rally”,
date=“25-Mar-2019”, Time=“9 am”,
and location=“Bhagyalaxmi temple
Charminar” can be extracted.

Mobility Performance
Report
The mobility performance is able tobe provided by Big Data Annalistic.
The department, related of traffic
management within existing
resources, collect, analyze, andsummarize highway congestion dataChapter 11
Big Data Analytics

and make it available upon request
to regional transportation planning
agencies, congestion management
agencies, and transit agencies.
The mobility performance by thedepartment with big data shall be
opened not only these management
agencies but also to the public. The
citizen is ability to know their current
traffic state and environmental
impact so that they have an
opportunity to change their mobility
for better mobility of themselves
and sustainable choice for better
environmental impact.
A lot of countries and region has
been trying to make a mobility
performance report. As a example
of California department ofTransportation, United State ofAmerica, data is collected in realtime from nearly 40,000 individual
detectors spanning the freeway
system across all major metropolitan
areas of California. These big data
opened the two ways:Website
and reports. The website, called
Performance Measurement System(PeMS), is available for related publicagencies, private-traffic information
suppliers, researchers/engineers
and citizen. The accessibility to thebig data is enhanced. On the other
hand, annual mobility performance
report is legislatively mandated
by Government Code in thecase of U.S.A. The department oftransportation, CA, issues annual andquarterly reports, bottle neck mappingand et al through the website.
The big data and mobility
performance should be recognized
as shared wisdom toward smart
transport and city in the viewpoint ofboth public agency and citizen.

1: Motivating more people
to use public transport byproviding timely information
Smart systems that can help people
to take transport related decisions
can be helpful for various aspects.
For example, on-demand information
regarding public transport scheduleor current running status, information

about congestion along different
routes, accurate estimate of travel
time along junctions, information
about planned or unplanned eventsor incidents along traffic route etc.
can be helpful for people to taketransport/commute related decisions.
Although this information can beobtained from multiple sources, thisinformation is scattered over different
special purpose applications andwebsites. People tend to use
multiple mobile based applications
or websites to get transit schedules
for different modes of transport.
Online schedules in many instances
are clunky, involving multi-step input
attributes to gain a simple answer.
Many have pdf-based schedules
or separate links to each metro/railline. In some instances, there are
separate sites for bus and rail.
Going to each individual source
according to the exact informationneed, and forming an appropriate
instruction by ﬁlling out input ﬁelds
in a web-based interface or app(e.g. for ﬁnding train schedule or
running status), or searching in amap (e.g. ﬁnding nearby points ofinterest), or going through large text
to ﬁgure out (e.g. prior information
about traffic diversion in a route,
pdf-based schedule) requires lotsof effort from the user. It is desirable
to streamline these different typesof activities and bring them under
one common interface where users
can seek information in a natural
way through conversations. Dueto the ease of interactions, such
conversational systems will make thetask of information seeking easierfor everyone, and especially for nontech-savvy users. Moreover, it willsave the user from taking multiple
steps (opening the app/website,
entering speciﬁc search query in
the form expected by the serviceetc.). Getting timely information
about public transport services
can help towards better adoptionof public transport. Efficient routeﬁnding and alerts regarding traffic
related incidents can help people
from taking appropriate routes
thereby reducing congestion andtraffic delays. Hence, developmentof such AI-powered conversational
systems for smart transportation canbe beneﬁcial for implementation ofsmart cities and urban transportation
systems.
Towards this goal of developing a
AI-powered conversation system fortransport queries, suitable dataset
is required. Unfortunately, there are
no datasets for this purpose thatﬁts the necessity of a multilingual
country like India, where people
prefer communicating in regional
languages. Often in a single
utterance there are both regional
language and English language
words. Handling such code-mixed
utterances in a conversation requires
special attention.
As a part of the project, we havecreated a dataset and also anexperimental system for a transportdomain conversational system. We
call the dataset as mTransDial. The
system can address user queries
belonging to multiple categories
such as place-search, train-search,
distance-search, traffic etc. A brief
description of the query intents areprovided below:
Place_search: It contains queries forsearching nearby Points of Interest
like petrol pumps, toll plazas, shops,
restaurants etc.
Train_search: It contains queries toﬁnd bus/train/metro from a source todestination place.
Distance_search: It consists ofqueries to ﬁnd distance from asource to a destination, and timerequired to reach destination.
Traffic: Queries about traffic
information and travel alert on aparticular place/city are added under
this intent.
Greeting: This intent is added towake up the dialog assistant and has
queries like “hi”, “hello” etc.
Thank you: This intent is for the
dialog system to understand thatthe user has got the answer and the
dialog system can check the needfor further assistance.
Goodbye: Queries in this intent are
to end the conversation.

67

Handbook of Multimodal Transport for Smart City
Application of Sensing, Networking and Big Data Analysis

Table 11-2 A few sample queries from dataset and the corresponding English translations
IntentTrain_search
Place_searchDistance_search
Traffic
OOS
Greeting
Thank you
Goodbye

Query from the dataset
please mujhe kal mehadeepattanam se
somajeeguda tak kee sabhee buses dikhaen
mujhe batao ki sabase kareebee coffee shop
kahaan hai
Begamapet jaane mein mujhe kitana time lagane
vaala hai
kya husain saagar ke raaste mein traffic halka
hogabank kab tak khula hai
hellooodhanyawad
theek hai byee

OOS: These are the out-of-scopequeries and are added in the dataset
so that our domain speciﬁc dialog
system is fault tolerant to out of
domain queries.
A few sample queries from thedataset, from the above intents areshown in Table 11-2. We expect thesystem to receive queries where
non-standard spellings are used (likeplz for please, tym for time etc.).
To assist in the development of roadtransport domain dialog system,
we created a prototype that shows
the usability of mTransDial in thisdomain. To develop this prototype
we use Rasa framework, an opensource machine learning framework
to develop conversational chatbots.
This framework has two components(a) Rasa NLU responsible forunderstanding the user intent andextracting entities and (b) Rasa Core
which decides the set of actions tobe performed based on the previous
user inputs.
We used a subset of the English
data to build this prototype. Basedon Rasa NLU, we annotated thisdata subset with the following entity
information: Source_loc, Destination_
loc, Near_loc, point_of_interest. The
Source_loc and Destination_loc
entities are to be extracted if thequery is of Train search or Distance_
search intent class. For Place
search intent we extract Near_loc,

68

Query in English
Please show me all the buses frommehadeepattanam to somajeeguda for tomorrowtell me where is the nearest coffee shop
how long is it going to take me to get to Begumpet
will traffic be light on the way to hussain sagar
how long is the bank open until
hellothank youokay byeFigure 11-10 Sample response for train
search query

Figure 11-11 Sample response for placesearch queryChapter 11
Big Data Analytics

point_of_interest entities to search
for the nearby point of interests
around the location provided by theuser. From Rasa NLU pipeline weused CRF entity extractor to extract
the relevant entities from the intents.
After intent classiﬁcation andextraction of entities we performed
relevant sets of action based on
the intents. If the classiﬁed intent isPlace_search or Distance_search
we call an external map-based API(here, MapmyIndia [https://www.
mapmyindia.com] ) to ﬁnd the points
of interest near to the Near_loc
extracted or to ﬁnd distance between
Source_loc and Distance_loc. If theclassiﬁed intent is Train_search we

use GTFS (General Transit Feed
Speciﬁcation, a common format for
public transportation schedules andassociated geographic information)
to fetch the relevant bus/metro/train schedule. Figures 11-10 and
Figure 11-11 show the responses ofthe system for train search and placesearch queries. It can be seen thatthe system can handle user queries
posted in natural language andprovide appropriate responses.

Conclusions
In this chapter we assess the use of
big data analysis for understanding
traffic ﬂows and demonstrate how
this can be used for providing

appropriate intervention suggestions
for decision making at multiple
levels. It is understood that lowering
the congestions will lead to lesser
carbon emission. Making people
better informed about the current
and future traffic can make them plan
accordingly. Being properly informed
about public transportation will also
increase their trust on the transportsystem and may motivate them toadopt public transportation. At ahigher level, the city authorities canuse insights from big data analytics
for civic and traffic planning which
can lead to both short- and long-term
beneﬁts to the traffic infrastructure,and congestion handling.

69