LatentDrive for Autoware (latent_drive)

Camera-only, Mapless end-to-end planning models for autonomous driving, run with TensorRT by autoware_latent_drive (a C++ library, an offline sample and a ROS 2 node). LatentDrive turns the last 2 s of front-camera video, six points along the route and the ego motion into a 4 s ego trajectory, at 10 Hz.

LatentDrive is built on a latent world model. Its video encoder is a V-JEPA ViT-L, a self-supervised world model that learns how driving scenes look and evolve by predicting masked parts of the video in its own latent space rather than in pixels; it was pretrained on driving video. The planner plans in that latent space: a small planning head reads the world model's latent features of the camera frames together with the route and the ego state, with no object detection or tracking in between.

Two models are published here, sharing the encoder, the route input and the output:

Model	Cameras	File
Three cameras	front, front-left, front-right	camfront3-rptw-lane2-e1.onnx
One camera	front	camfront-rptw-lane2-e1.onnx

Model overview

Task	Ego trajectory planning: 40 poses at 0.1 s steps, covering 0.1 s … 4.0 s
Inputs	4 frames per camera, 0.5 s apart (t − 1.5 s … t); six route points 5, 10, 15, 20, 30 and 50 m ahead along the route; ego velocity and acceleration
Architecture	Latent world model: V-JEPA ViT-L video encoder (304M parameters), run once per camera view with shared weights → planning head (6M parameters); 310M parameters in all
Runtime	TensorRT through autoware_latent_drive: the encoder in fp16, the planning head pinned to fp32 (the library's validated "mixed" recipe)
Latency	Three cameras: about 14 ms per plan on an NVIDIA GeForce RTX 4090. On the test vehicle's computer both models hold 10 Hz with no skipped ticks (about 35 ms per tick with three cameras, 11–12.5 ms with one)
Format	ONNX (opset 18), fp32, self-contained (no external data), dynamic batch axis. No TensorRT plugin is needed. TensorRT engines are built locally
License	Apache-2.0

Files

File	Description
camfront3-rptw-lane2-e1.onnx	The three-camera model (1.25 GB)
camfront-rptw-lane2-e1.onnx	The one-camera model (1.24 GB)
media/	The scenes on this page
replay.mp4	Open-loop replays of six scenes of two recorded drives, the three-camera model's plan drawn on the front camera; this page's video preview
deploy_metadata.yaml	Deployment metadata recording the artifact version of this repository
SHA256SUMS	Checksums of the files above

TensorRT engines are not distributed here. An engine is specific to the GPU architecture and TensorRT version it is built on. scripts/build.sh in autoware_latent_drive builds one per ONNX file (models/engine/<name>-mixed.engine): about 7 minutes for the three-camera model and 1.5 minutes for the one-camera model.

Inputs and outputs

Tensor	dtype / shape	Content
video (three cameras)	float32 [B, 3, 3, 4, 256, 512]	3 views × 4 frames × RGB. View order: front, front-left, front-right
video (one camera)	float32 [B, 3, 4, 256, 512]	4 frames × RGB of the front camera
status	float32 [B, 16]	[p1x, p1y, …, p6x, p6y, v_fwd, v_left, a_fwd, a_left]
plan (out)	float32 [B, 40, 3]	x forward, y left [m], dyaw [rad] in the current ego frame, 0.1 s apart

Frames. Four frames per view, 0.5 s apart, oldest first, each the frame nearest in time to its slot. Each frame is centre-cropped to 2:1 over the full width, resized to 512 × 256 and normalised with the ImageNet mean and standard deviation. The side views are the frames nearest in time to the front frame; in training they were within 50 ms. A missing side view was seen in training only as an all-zero image before normalisation.

Route points. The route is resampled every 1 m; the ego position is projected onto it, and point k is the route point 5, 10, 15, 20, 30 or 50 m of arc length past that projection, in the ego frame, divided by 10, clamped at the end of the route. On a vehicle the route is the centerlines of the mission route's preferred lanelets (autoware_planning_msgs/msg/LaneletRoute). The rosbag replay of autoware_latent_drive takes the route points the vehicle recorded; its offline sample can also build them from the mission route and the Lanelet2 map (scripts/convert_map.py).

Ego motion. Velocity and acceleration in the ego frame (forward, left) [m/s, m/s²], from poses 0.1 s apart: the velocity is the chord over the last two intervals, the acceleration its change over one.

Cameras on the TIER IV test vehicle (for reference; any cameras with a similar layout):

View	Topic	Azimuth	HFOV
front	/sensing/camera/camera0	1.2°	57°
front-left	/sensing/camera/camera3	+49.5°	57°
front-right	/sensing/camera/camera4	−49.6°	57°

Usage with autoware_latent_drive

autoware_latent_drive runs these models with TensorRT. Its README has the full quick start; in short, with an NVIDIA GPU, Docker and the NVIDIA Container Toolkit, everything runs in the public Autoware development image (ROS 2 humble, CUDA 12, TensorRT 10.8):

git clone https://github.com/autowarefoundation/autoware_latent_drive.git
cd autoware_latent_drive
hf download AutowareFoundation/latent_drive --revision v1.0.0 --local-dir models/onnx/latent_drive \
  --exclude "media/*" --exclude replay.mp4
IMAGE=ghcr.io/autowarefoundation/autoware:universe-devel-cuda BAG=/path/to/rosbag ./docker/run.sh

In the container, build the library, the ROS 2 node and the engines, then replay a rosbag of a LatentDrive run with the plan drawn on the front camera, as in the video on this page:

scripts/build.sh                                                        # the three-camera model (default)
scripts/build.sh models/onnx/latent_drive/camfront-rptw-lane2-e1.onnx   # the one-camera model
samples/rosbag/prepare_bag.sh /bag bags/my_bag
samples/rosbag/run_replay.sh bags/my_bag models/engine/camfront3-rptw-lane2-e1-mixed.engine \
  start_offset:=60 out_video:=bags/my_bag/replay.mp4

Pin --revision to a tag rather than relying on main, and add --include "camfront3-*" or --include "camfront-*" to download only one model.

samples/sample_data runs the same models offline on a clip cut out of a bag, through the library API alone.

Training

The models were trained by TIER IV on camera data recorded in Japan; the data is not public. The V-JEPA ViT-L latent world model was pretrained self-supervised on that driving video (masked prediction in latent space) and then fully fine-tuned with the planner. Training samples on tight turns (mean turn radius 15 m or less) were repeated four times. Accuracy figures are measured on internal validation data and are not published.

Limitations

• Research model. It is not a certified driving system and must not control a vehicle on public roads. Run it in simulation, on closed courses, or with a trained safety driver and independent safeguards downstream of the planner.
• Camera only. The planner reads no tracked or predicted objects, no LiDAR and no traffic-light recognition: other road users and traffic lights reach it only through the camera frames.
• Sensor setup and domain. Trained on the vehicles, cameras and roads of its training data in Japan (left-hand traffic, Japanese traffic lights). Performance on other platforms, camera placements and operational domains is not characterized.
• Known behaviours. Where the route doubles back (a junction entry whose centerline turns back), near route points can land behind or beside the car and the plan cuts in; autoware_latent_drive drops route vertices that turn by more than 120°. The models creep slightly while the car is stopped.
• Evaluation. The video preview and the scenes on this page are open-loop replays of two recorded drives (Tokyo, Ariake, 2026-10-23), made with samples/rosbag/run_replay.sh: the plan is drawn on the camera frames, not executed. The ribbon runs green (now) to red (+4 s).

Provenance

Source	Exported to ONNX by the LatentDrive training code (PyTorch 2.5.1), checked against PyTorch
This repository	AutowareFoundation/latent_drive, tag v1.0.0

SHA256SUMS lists the files' checksums.

References

• autoware_latent_drive: https://github.com/autowarefoundation/autoware_latent_drive
• V-JEPA: https://github.com/facebookresearch/jepa
• Autoware: https://github.com/autowarefoundation/autoware