Autonomous Driving Dataset Visualization with Python and VizViewer
By: Jose Rojas
Tags: Python, Visualization, Autonomous Vehicles, Autonomous Cars, Jupyter Notebook

Introduction

As part of a recently published paper and Kaggle competition, Lyft has made public a dataset for building autonomous driving path prediction algorithms. The dataset includes a semantic map, ego vehicle data, and dynamic observational data for moving objects in the vehicle's vicinity. The challenge presented by Lyft with this dataset is to use this data to build a model that can predict the paths of moving objects and the path an autonomous vehicle (“AV”) should take based on the observations made by the AV sensors and perception stack.

In more specific terms, the challenge is, given a set of information about the current vehicle state and its surroundings, to predict the best plan — a set of actions and behaviors — for the vehicle to safely navigate autonomously.

Lyft provides a large volume of training data in the L5 Prediction dataset; tens of thousands of 25-second sequences of data are available in over 100GB. Along with the data, Lyft has also offered a set of tools for parsing and visualizing the data.

This article will explore the details of the L5 Prediction dataset with these tools and a novel data visualization platform called VizViewer (or “VV” for short). Utilizing the VizViewer platform, we’ll uncover insights about the data while discussing the benefits of the visualization techniques for dataset tuning and feature engineering. To wrap up, we’ll preview a lane prediction visualization that could be used to solve the general path planning problem.

Dataset Structure

Within the context of autonomous driving, there are two general subsets of data to consider: the static environment and the dynamic environment. The former would include data that mostly remain relatively fixed over time, such as the road network paths, the number of lanes in the current road, traffic signs and traffic lights, etc. The latter includes data about the varying driving conditions such as the location and speeds of pedestrians or vehicles nearby, or the color of an upcoming traffic light.

The L5 Dataset provides data for both of these data types. One form is a semantic map, sometimes referred to as an HD map [1], which encodes details about the static driving environment. The second is a voluminous “scene” database for dynamic time-series data. The L5 Kit provided by Lyft includes tools for extracting data from both these sources.

Semantic Map Overview

Within the dataset, the static environment is defined by the semantic map. This can be thought of as an environmental 2D map that has been densely annotated with information appropriate for the driving context. The semantic map offers a predefinition of the expected driving environment; without a semantic map, this static information would need to be continuously perceived and interpreted by the vehicle's sensors and CPUs. Thus, a semantic map is a powerful tool for pre-computing and offloading much of the work involved in AV planning and prediction problems.

Animation by author, Semantic Map Visualization — roads/lane lines (gray), parking zones (red), crosswalks (yellow), stop signs (red cylinders), traffic lights (gray cylinders)

The semantic map itself contains these various attributes:

a directed graph of roads and their lanes

the physical position of the lane lines of a road, down to the centimeter

the physical position of the stop lines, stop signs, traffic lights, crosswalks, and other traffic control elements

speed limits

the possible states for a given traffic light (e.g., red, green, yellow) and the lanes for which they control traffic (e.g., left turns, through lanes, right turns)

parking zones (specifically if they share a lane)

Scene Database Overview

The dynamic features encoded in the dataset include the spatial information of the “ego” vehicle (the AV collecting the data), the “agents” (freely moving observed objects), and the traffic light states (“red”, “green”, “yellow”). Each agent also has a “class” label, describing it as a set of probabilities of common object types such as cars, pedestrians, cyclists, etc. These three data sources are encoded and indexed separately into tabular form.

The spatial features of the ego and agents contain the “pose” of the objects (their x, y, z cartesian coordinates, and orientation) and for agents, their “extent” (size of the object). Each data sample has a timestamp, and all observations with a common timestamp represent a “frame” of data. A “scene” consists of a contiguous sequence of observation frames with respect to time. The scene links frames from each of the other three data tables using a list of indexes to each record in the table.

Image by author, Structural Diagram of the L5 Prediction Dataset

The motivation behind this scene-centric structure is important to note. In many machine learning problems, each time-based data sample is used independently as examples to train a model. However, in this dataset, the entire scene — as a collection of data samples — are the atomic units of data used to train a machine-learned model.

The reasons for this should be fairly intuitive; to make predictions about the paths a set of objects can take, the samples must be coherent and causally linked across time to build an accurate description of motion. If an amalgam of data samples from different scenes is used to generate a path, the resulting path would most likely be completely inconsistent; the objects would move in impossible ways from the ground truth. Given the need for coherency across time for accuracy in path planning, scenes are the building blocks of data that we will examine holistically using VizViewer.

So what is VizViewer?

VizViewer is a web application and platform for collaboration and visualization of complex, multi-modal datasets. It consists of a suite of communication, data processing, and visualization components bundled into an accessible and easy to use dashboard UI. VV provides tools for interpreting data and accelerating productivity in data analysis workflows. It achieves these goals through a cohesive, configurable, interactive, and versatile toolset for analyzing datasets of different modalities while interoperating with Python and the Jupyter Notebooks.

In short, VizViewer is a helpful extension of coding tools for data exploration and insights into dense datasets with different types of content. In the next sections, we’ll explore the characteristics of the L5 Prediction dataset using VizViewer to understand the data better, build improved training sets, and utilize it to debug and evaluate models.