8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

Session : XIV - D5/04
Paper No. : 1021 (revised version)

Constructing a Benchmark Data Set for Validity Evaluation of Road
Network Simulation Models

Ryota HORIGUCHI*
Toshio YOSHII**
Hirokazu AKAHANE***
Masao KUWAHARA**
Masahiko KATAKURA****
Haruo OZAKI*****
Takashi OGUCHI****
*)

**)

***)

****)

*****)

Kumagaigumi Co., Ltd.
2-1 Tsukudo-cho, Shinjuku-ku, Tokyo 162, Japan
Phone: +81-3-52615526, Fax.: +81-3-52619350
Institute of Industrial Science, University of Tokyo
7-22-1 Roppongi, Minato-ku, Tokyo 106, Japan
Phone: +81-3-34026231, Fax: +81-3-34016286
Chiba Institute of Technology
2-17-1 Tsudanuma, Narashino, Chiba 275, Japan
Phone: +81-474-780444, Fax.: +81-474-780474
Tokyo Metropolitan University
1-1 Minamiosawa, Hachioji, Tokyo 19203, Japan
Phone: +81-426-772781, Fax.: +81-426-772772
Toyo University
2100 Kujirai, Kawagoe, Saitama 3500815, Japan
Phone: +81-492-391393, Fax.: +81-492-314482

1

8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

1. INTRODUCTION
This research aims to construct a precise data set for validity evaluations of road network
simulation models. To increase the reliability of the data set, the data was collected throughthe precise survey to read number plates of all vehicles at every input/output roadside ofmajor intersections, and carefully processed. Such data set will be used for benchmark ofsimulation models.
Needs to estimate the effects of planning and/or improvements on traffic management has
encouraged activities of developing traffic simulation models. Especially, the simulation
models to deal with the traffic on road networks are strongly required for ATIS/DRGS
applications and/or traffic management evaluation. In fact, dozens of simulation models havebeen developed and used for the last two decade.
However, they often have a problem in the model validation stage using real world data toevaluate their ability to reproduce traffic conditions, nevertheless the most of developers andusers consider that the validity check is important (e.g. see SMARTEST, 1997).
The major difficulty to prevent developers from model validations is that such simulation
models generally require the data which is hard to directly observe. For example, most ofthem require time-varying origin-to-destination (O-D) traffic demand as the input data. If the
network size become larger, it is easily conceived how we have to pay a big effort toobserve the origin and destination of each vehicle.
Even though several simulation models are validated, the developers use their own data set
which may be in different conditions. Thus, it is difficult to directly compare their abilities ofreproduction. This may be another difficulty for the users to select an appropriate simulation
model to serve their purposes.
For the last five years, we have been developing and validating a traffic simulator 'AVENUE'
which deals with urban road networks (Horiguchi et al., 1996). Through the experience ofthe development, we have spent substantial effort to collect precise data set including timevarying O-D demand and individual vehicle trajectories besed on number plate survey.
The target area is Kichijoji-Mitaka area, the western part of central Tokyo, which spreads
about 2 km from east to west and 1 km from north to south. This area contains 70 roadsidepoints where the observers read the number plates of all passing vehicles. The survey was
carried out during morning peak period from 7:00 a.m. to 10:00 a.m. by more than 200
observers.
The collected data was processed by the heuristic matching technique presented in this paper
to extract vehicle trajectories. The vehicle trajectories have been aggregated into timevarying O-D traffic demand. Link flows and link/path travel times to be compared with thesimulation results are figured out as well.
In the following chapters, the contents of benchmark data set required for the validation ofroad network simulation models are indicated at first. Secondly, the outline of the number
plate survey are described. Then, the extraction of individual vehicle trajectories from thesurvey data is explained. The extraction process interpolates misreading data by observers

2

8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

with heuristics rules. The extracted vehicle trajectories are subsequently aggregated to timevarying O-D demand. Finally, further analysis currently on-going and future topics aroundbenchmark data set is presented.

2. REQUIRED ITEMS OF BENCHMARK DATA SET
According to the purpose of traffic simulation, there must be several types of benchmark
data set which have different contents. For example, the benchmark data set for thesimulations of weaving sections has different contents from that of network simulations.
This chapter discusses the required items of the benchmark data set for network simulation
models regarding simulation inputs and traffic conditions to be compared with simulation
results.
2.1. Simulation inputs
Simulation inputs consist of four categories, i.e. 1) network geometry, 2) time-dependent
O-D traffic demand, 3) traffic control and management schemes and 4) notable events which
may affect traffic flow conditions.
Network geometry -- such as node positions, link lengths, link-node adjacency, etc., isnecessary for all simulation models. Link connectivity considering prohibited turn must be
included. The curvature and the lane configuration of each link is optional but preferable
especially in the case of microscopic simulation models. In this study, not only the lane
configuration but also the turning movement at each lane are collected.
O-D traffic demand -- is the most important input, since it substantially affects thesimulation result. Therefore, it should be accurately observed. O-D traffic demand should betime-depending and layered by vehicle types and any drivers' attributes.
Traffic control and management schemes -- are mainly the history of signal control plans,
ramp metering, traffic regulations, i.e. one-way treatment, turning prohibition, route
guidance, travel information provided, etc. They may be incorporated into simulation models.
Related facilities, e.g. signals, detectors, beacons, variable message signboards, should belisted.
Notable events -- which may affect traffic flow conditions should be recorded. Incidents,
blocking by roadside parked cars, interaction with pedestrians, queue spill back from theoutside of the area, etc. may be useful to define network configuration as well as model
parameters.
2.2. Traffic conditions
Observed traffic conditions should be compared with the simulation result. As most of thesimulation models provide indicators to measure flow condition and travel efficiency,
following values should be observed.
Flow condition -- is represented by link flow and queue length. Saturation flow rates atbottleneck intersections should be precisely measured, because it is dominant in flow3

8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

reproduction and useful in model calibration.
Travel efficiencies -- are mainly referred to speed and travel time. These are measured alonglinks and, if possible, routes. Other indicators of efficiency may be delay and number of stops
at each signalized intersection.
Other indicators -- are sometimes used for model validation. For example, when the model
objective includes the evaluation of adaptive signal control, detector data should be recorded
during the survey. Other indicators may be those about public transport regularity, safety,
environment, etc., depending on model objectives.

3. PRECISE SURVEY IN KICHIJOJI-MITAKA AREA
In this chapter, the outline of the survey that we have carried out in Kichijoji-Mitaka area on
30th Oct. 1996 is presented.
3.1. Target area
The target area is Kichijoji-Mitaka area, the western part of central Tokyo, which spreads
about 2 km from east to west and 1 km from north to south (Fig. 1). We selected this area
based on two reasons; there were several alternative routes for major O-D pairs and changes
in traffic condition were seen during morning peak period.

Kichijoji-doriSeikei-dori

Mitaka-dori

Itsukaichi-kaido

Kichijoji Sta.

Inokashira-doriRoadside point

Fig. 1: Kichijoji-Mitaka area and roadside points where observers were located.
The network in the target area consists of four major north-south streets and two major
east-west streets. Most of the links are two-lane roads and partially four-lane roads.
Mitaka-dori Avenue has a reversible center lane.
Traffic congestion during morning peak period is constantly found on Itsukaichi-kaido
Street, on Inokashira-dori Street, both are caused by the traffic from west to east, and on
Mitaka-dori Avenue from north to south.
3.2. Number plate survey

4

8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

Seventy roadside points where observers read the number plates of passing vehicles werearranged in the target area. Each input and output link of major intersections has a roadsidepoint near the intersection (Fig. 2).
2

Itsukaichi-kaido

1

22

21
19

3

50

52

54

56

58

60

4

51
30

53

55

57

59

61

31

62
41

40

47

32

43

45
35

34

5

72
70

7

42

44

74

75
10

73

71

6

Kichijoji-dori

33

Seikei-dori

Mitaka-dori

36

9

46

91

18

63

17

90
49

48

93

92

95

94

15

76

78

77

79

16
14

80
81

13

11 12

Inokashira-dori
Kichijoji Sta.

8

Roadside Point No.
45

Direction of Travel

44

72

74

73

75
9

10

Fig. 2: Roadside points and the detail of major intersections.
Observers recorded vehicle type and the four large digits in the number plate of all passing
vehicles. Vehicles were divided into tree types, i.e. buses, taxies and other miscellaneousvehicles. If the observer could not catch the number plate of a vehicle, 'unknown' statement
was recorded. Passing time of vehicles were simulteniously recorded in minutes. In order toincrease the reliability of the data, two observers for one lane were assigned to each roadsidepoint. One recorded type and numbers with a cassette tape recorder and another wrote
down them. These double data source were used to detect contradictions like mistypes in thedigitized data.
The survey was carried out during morning peak period from 7:00 a.m. to 10:00 a.m.
Totally 71,837 data were collected during the valid time period, 7:50 - 10:00.
3.3. Other observed data
Signal control parameters, i.e. cycle, split and offset, of each signalized intersection wererecorded every fifteen minutes. Video recorders were placed in two roadside points tomeasure the percentage of large vehicles. Furthermore, the saturation flow rates atdistinguished bottleneck intersections were observed.

4. EXTRACTION OF VEHICLE TRAJECTORIES
Vehicle trajectories can be extracted from the result of the survey by matching number plates58TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997and vehicle types between two neighboring roadside point (Hanabusa et al., 1997). Travel
times are also calculated by subtraction of the passing times. The extraction process has six
steps; 1) resolving inconsistencies in the data, 2) number plate matching between tworoadside points, 3) pruning irregular matches, 4) disentangling twisted vehicle trajectories,5) retouching irregular trajectories and 6) reuse of unused data elements. The extracted
trajectories are aggregated to time dependent O-D traffic demand.
4.1. Resolving inconsistencies
Inconsistencies in the observed data are divided into two types. One is coming from clock
synchronization error. If an observer's clock is faster than the other synchronized clocks,
each vehicle's travel time from where the observer stands to the next roadside point isshortened, and sometimes may be minus. This inconsistencies can be detected by preliminary
matching between two neighboring roadside points and checking too many minus travel
times.
Another is duplicated number plates at the roadside points on multi-lane links. As theobservers are assigned to each lane, they sometimes read same vehicle at the same time
when the vehicle shows untidy movement over two lanes (Fig.3). Thus, one of the data
which have the same number plate, the same vehicle type and the same time stamp at thesame roadside point should be eliminated. In this step, 838 double counting were detected
and the total number of the data became 70,999.
Observers for Lane 1
Observers for Lane 2
Vehicle5
Lane 1
Lane 2

Fig. 3: Double count of a vehicle with untidy movement.

4.2. Number plate matching
In this step, matching between two adjacent roadside points are checked at first. Fig. 4
illustrates an example of adjacent matching. Matching pairs of which time difference are too
longer than the ordinary link travel time are out of consideration. In this case, the link travel
time is limited within five minutes.
However, if the number plate of a vehicle was misread at one roadside point, the actual
vehicle trajectory would extract as two divided trajectories. Therefore, a data element whichhave no matching pairs in the adjacent roadside point skips them and may be matched withthe data at one more farther adjacent roadside point. Fig. 5 shows an example of skip
matching.

68TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997....
8:00, No. 2345
8:00, No. 1234
8:01, No. 3456....

Matching!!

....
8:00, No. 5678
8:01, No. 4321
8:01, No. 1234....

Observed data
74

76

Actual vehicle trajectory

Plate No. : 1234

Fig. 4: Number plate matching between two neighboring roadside points.
....
8:00, No. 2345
8:00, No. 1234
8:01, No. 3456....

Skip Matching!!

....
8:02, No. 5678
8:02, No. 4321
8:03, No. 1234....

Misread!!
74

76

78

Actual vehicle trajectory

Plate No. : 1234

Fig. 5: Skip matching
4.3. Pruning irregular matches
A vehicle trajectory must be extracted as a direct road without merging and diverging. There
are, however, irregular matches which form merged or diverged trajectories. Those matches
are should be eliminated.
There are two types of irregular matches, loops and shortcuts. Byroads which have noroadside point at either end cause irregular matches. Fig. 6 shows the examples of a loop anda shortcut. In order to resolve this problem, loops can be detected by depth-first-search fromthe root of a vehicle trajectory, and shortcuts can be done by width-first-search.

7

8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

Irregular match
8:00, No. 1234
95
8:00, No. 1234

80

78

Regular match

8:00, No. 1234a) a 'loop' trajectory
8:00, No. 1234
92

Irregular match

94
8:00, No. 1234
8:00, No. 1234

81
12

8:00, No. 1234

Regular matchb) a 'shortcut' trajectory
Fig. 6: Irregular matches in 'loop' and 'shortcut' trajectories.
4.4. Disentangling twisted trajectories
When two vehicles of which the number plate and the vehicle type are same as each otherclosely travel in the area, their trajectories will be entangled with unnecessary matches. In
order to disentangle those trajectories, we give preferences to ambiguous matches using
following heuristic rules and take higher ones; i.e.
Rule-1: To give higher preference to the matches that the disentanbled trajectories are as
possoble as long when those matches are removed. -- In the example in Fig. 7, matches {A,F} divides the trajectories into two long ones and should be removed..

8

8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

A
B

C

D

E
F

Fig. 7: Prefernce to longer trajectories.
Rule-2: To give higher preference to the matches that the disentangled trajectories are as
possible as straight when those matches are removed. -- In the case in Fig. 8, A and B are to
be removed to take two straightforward trajectories.

B

A

Fig. 8: Preference to straight movement.
Rule-3: To give higher preference to the matches that the disentangled trajectorires keep
first-in-first-out principal when those matches are removed. -- The example in Fig. 9, A andB should be removed.
....

8:00, No. 1234....
8:01, No. 1234

B

....

A

8:01, No. 1234
....

....

8:02, No. 1234

Fig. 9: Preference to FIFO trajectories.
98TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

4.5. Retouching irregular trajectories
There are irregular trajectories that have either end inside an intersection. Such trajectories
should be retouched as to move their irregular ends to the outside of the intersections. For
that purpose, those irregular ends are connected to 'similar' data elements at neighborroadside points (Fig. 10) with the following rules.
Rule-4: If four digits are same but in different order, those two data elements are similar.
Rule-5: If three digits out of four are same, those two data elements are similar.
Rule-6: If one of two data elements has no information for number plate, those two aresimilar.
If there are no similar data elements, new data elements will be imposed within the data of
the nearest roadside point.
Same four digits

8:00, No. 1243.

Retouch trajectory

8:00, No. 1234.

Irregular trajectory starts here.

Fig. 10: Retouching a irregular trajectory.
4.6. Reuse of unused data elements
There are still remaining data elements which are 'unused' for extracted trajectories.
According to the previous rules, detected similar pair of unused elements between two
adjacent roadside points across an intersection are connected to be a trajectory. If there areno similar data elements to be connected, new data elements will be imposed within the dataof the nearest roadside point and create a trajectory with them.
4.7. Aggregation to O-D demand
The number of extracted trajectories and the number of used data (percentage out of
70,999) at each step are listed below.
Step# of trajectories # of used data
1) resolving inconsistencies
2) number plate matching
13,93662,713 (88.3)
3) pruning irregular matches
14,20864,038 (90.2)
4) disentangling twisted trajectories
14,953
68,334 (96.2)
5) retouching irregular trajectories
14,953
69,123 (97.4)
6) reuse of unused data elements
16,043
71,265 (100.4)
10

8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

As the step proceeds, the number of trajectories and the number of used data increases. The
total number of the data finally increases up to 71265 because of the newly created data
elements. The number of routes used by vehicles are 930.
Those trajectories are aggregated into time dependent O-D traffic demand sliced in every 10
minutes by their departure time. The number of O-D pairs are summed up to 747. Not only
the nodes at the network end but also every links have flows and sinks of the demand,
therefore links can be origins and/or destinations.

5. SUMMARY AND FUTURE TOPICS
In this paper, the required contents of benchmark data set for the validation of network
simulation models have indicated. In order to build such data set, we coordinated precise
survey and data processing based on number plate matching. We have obtained the extracted
16,043 vehicle trajectories and time dependent O-D traffic demand which are very close toreal ones.
Further analysis is required to complete the benchmark data set. Path-based analysis are
currently on-going to identify the drivers' route choice behavior. Traffic conditions to becompared with simulation results can be obtained from similar analysis. Traffic conditions
includes not only link flows but also path flows.
Using this benchmark data set, the following items are recommended to be addressed in themodel validation stage, i.e. 1) correlation of time-varying link flows as well as link travel
times and 2) correlation of time-varying path flows as well as path travel times.
This benchmark data set will be distributed through World Wide Web site. Any developers
and users of network simulation models can access the data. The data of the number platesurvey and the result of the analysis will be included with documents describing theprocedure of the analysis. Validation results of simulation models using this data will beappended if possible.
This benchmark data set is useful for the researches such as estimation of time-varying O-D
matrix from observed vehicle counts on roadside. It is expected that this data set will be used
not only for validation but for other applications.

REFERENCES
Hanabusa H., Shiraishi T., Akahane H., Yoshii T. and Horiguchi R. (1997) Construction ofa benchmark data set for traffic simulation systems. Proceedings of Infrastructure
Planing in Civil Enginnering, Japan Society of Civil Engineering, Vol. 20 (1), 497-499,(in Japanese)
Horiguchi R., Kuwahara M., Katakura M., Akahane H. and Ozaki H. (1996) A network
simulation model for impact studies of traffic management 'AVENUE Ver. 2'.
Proceedings of the Third Annual World Congress on Intelligent Transport Systems,
Orlando, CD-ROM

11

8TH WORLD CONFERENCE ON TRANSPORT RESEARCH, Antwerp, July 29, 1997

SMARTEST Project (1997) Smartest : review of micro-simulation models. Distributed
through WWW site, http://www.its.leeds.ac.uk/smartest

12