The views presented in this paper are those of the author(s) and do not represent the official views of any Federal Government agency/department or Westat.

Using Computer Vision to Process 

Vehicle Dashboard Displays in 

Transportation Safety Research

Kristin Jiating Chen, Alexander Cates, Rick Huey, Marcelo Simas, 

James Jenness, Gonzalo Rivero
2

Outline 

• Introduction

• Nature of data

 Videos

 Icons

• Methodology – Machine learning pipeline with OpenCV

• Results

 Model performance 

 Future research
Introduction – Background of the Overall Study

• Understand driver behavior in the context of driver-assist systems in 

Toyota Safety Sense system (TSS)

 Driver-assist systems

Adaptive Cruise Control Lane Departure Alert Pre-Collision System

• when the systems are triggered, related driver-assist icons appear on the 

vehicle dashboard displays

3

When 

driver-assist 

icons were 

presented

When 

driver-assist 

systems 

were used 

How drivers 

interpret 

and respond 

to the 

presence of 

the icons

How drivers 

understand 

systems 

over time
Introduction – Research Problem

• Problem

 Identify the presence of icons on the vehicle dashboard displays

• Pilot study

 Collect data by recording the central dashboard displays while driving 

instrumented Toyota vehicles

4
Nature of Data - Videos

• Video recordings of central dashboard display

 Pilot study data 

• 200+ 1-min videos

 Study data (estimated)

• 10-20 instrumented Toyota vehicles

• 12 weeks participation per vehicle

– 500+ 1-min videos per week

5
Nature of Data - Icons

• 10 icons 

6

Adaptive Cruise Control Lane Departure Alert Headway Bar Indication

Lane Line Indication

Leading Car

Brake
Methodology

• Technique

 OpenCV: open source computer vision library

• Machine learning Pipeline utilized OpenCV Python API and R

7

Frame 

extraction from 

videos

Feature 

matching 

between icons 

and extracted 

frames (image 

preprocessing)

Visualization
Methodology –

Frame Extraction

• Extract frames from videos per 

half-second 

8
Methodology – Feature Matching

9

• Feature

• Good feature – corner feature

Flat features (A, B)

Edge features (C, D)

Corner features (E, F)
Methodology –

Feature Matching

10

1. Detect features in frames

1. Detect features in icons

2. Match same 

features between two 

images
Methodology –

Feature Matching

• Matching Criteria 

 3 matched features between 

two images

11
12

Methodology –

Feature Matching

• Scale invariant

• Rotation invariant

Leading Car
Methodology –

Feature Matching

• Mismatching

 Local features detected in the 

frames are noise

• Strategy

 Lower noise and narrow target 

areas for feature matching

13
Methodology – Image Preprocessing

• Crop the dashboards from the frames - focus on region-of-interest

• Deskew the dashboards

• Denoise

14
Methodology – Image Preprocessing

• Crop into 3 parts

 Narrow down the target area

15

Adaptive Cruise Control

Lane Departure Alert
Methodology – Feature Matching

16
Methodology – Visualization (R Shiny)

• Interactive time series plots 

17

speed

time
Results – Modeling Performance

• Sample data: 10 icons, 29 selected frames, 290 combinations

• Confusion Matrix

TPR: 0.66 TNR: 0.98 FPR: 0.018 FNR: 0.342

• Runtime (4 threads multiprocessing)

 Week 1 subject 1 real study data: 10 icons, 57,000 frames, 570,000 

combinations (500+ 1-min videos)

 Computation time

Frame extraction: 10 mins Image preprocessing: 2 hours+ Feature matching: 2 hours+ 

18

n = 290 Predicted Negative Predicted Positive

Actual Negative TN = 213 FP = 4 217

Actual Positive FN = 25 TP = 48 73

238 52
Results – Conclusion & Future Research

• Conclusion: Works well!

 Leads to 66% true positive rate and 98% true negative rate

 Meets computation requirement when data scales up 

 Processes with reasonable computation time

• Current research 

 Customizes matched features criteria

for each icon

• Balance FP/FN error

• Future research

 Train customized model

19
Thank you for listening!

• Contact information

Kristin Chen KristinChen@Westat.com

Alexander Cates AlexanderCates@Westat.com

• Citation

ANN ARBOR. Mich, (2017, November 15). Toyota's Collaborative Safety Research Center 

to Study Societal Acceptance of Connected and Automated Vehicle Technologies. 

Retrieved from 

https://corporatenews.pressroom.toyota.com/releases/toyota+collaborative+safety+research

+center+study+societal+acceptance+connected+automated+vehicle.htm

https://www.toyota.com/safety-sense/featurevideo/DtoqSOUSR0A

https://docs.opencv.org/3.4/db/d27/tutorial_py_table_of_contents_feature2d.html

20