OpenExplorer/e2e_planner

E2E Diffusion Planner + HENet-tiny

e2e_planner is an end-to-end autonomous-driving model on NAVSIM: an 8-camera BEV multitask perception front-end (HENet-tiny + single-frame BevFormer view transformer; dynamic/static 3D detection, occupancy, discrete map, online MapTR) feeding a BEV-feature-conditioned Diffusion Planner that outputs future ego trajectories. It covers the full pipeline from 5-stage float training → quantization calibration → HBIR export → HBM compilation → NAVSIM closed-loop metric evaluation.

---

Deployment Metrics

Model Parameters

Model	Model Input	Backbone	Neck	Model Output
E2EPlanner (BEVFusionMultitask + Diffusion_Planner)	8-camera multi-view images (B,8,3,512,960) + ego state + route lanes	HENet-tiny	FPN + SingleBevFormerViewTransformer	future ego trajectory (1,8,4) via diffusion

Accuracy Metrics

March	Metric	float	calibration	hbm
J6M	PDMS	0.8223	0.8142	—

Measured 2026-10-17 on NAVSIM navtest with HAT_USE_NEIGHBORS_FUTURE=0 and dequantize_bev_embed=False. No QAT stage. HBM accuracy not yet measured.

HEAL version: heal 0.6.0 / hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5 (board UCP 3.16.2).

Performance Metrics

Performance benchmark: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage. Data source: tools/perf.py (--perf-time 1; DDR sampled by hrt_ucp_monitor -d 200). Measured 2026-10-19 (dequantize_bev_embed=False). Toolchain: hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5. No native:: CPU ops.

March	latency (ms)	fps	Memory Usage (MB)
J6M	99.14 (BPU 99.07)	10.11	306.30
J6P	37.84 (BPU 37.78)	88.23	242.20

---

Model Overview

Core Design

The model has two halves joined at the BEV feature: an 8-camera multitask perception front-end producing a 128×192 BEV grid, and a BEV-conditioned Diffusion Planner. HENet-tiny extracts multi-view camera features; a single-frame SingleBevFormerViewTransformer lifts them to BEV; five task heads (DenseBEVSparseHead dynamic/static, BEVOCCHead2D occupancy, BEVFormerDet2DDecoder discrete map, MaptrDetDecoder online map) share the BEV feature. The planner encodes the pooled BEV feature into 384 tokens, conditions a 3-layer DiT via a route-lane RouteEncoder, and denoises future ego waypoints over 5 reverse diffusion steps.

• Task type: End-to-end planning (perception + planning joint).
• Perception backbone: HENet-tiny (block_nums=[4,3,8,6], embed_dims=[64,128,192,384]), initialized from the ImageNet-pretrained henet_tinym float checkpoint.
• Perception neck: FPN + SingleBevFormerViewTransformer (128×192 BEV, [-51.2m, 102.4m] × [-51.2m, 51.2m]).
• Dynamic det head: DenseBEVSparseHead (3 classes; num_anchor=384, MemoryBank 128 temporal instances).
• Static det head: DenseBEVSparseHead (4 classes; num_anchor=200).
• Occupancy head: BevformerOccDetDecoder + BEVOCCHead2D (Dz=16, 9 classes).
• Discrete map head: BEVFormerDet2DDecoder (2 classes: stop_line/crosswalk).
• Online map head: MaptrDetDecoder + MapInstanceDetectorHead (1 class lanes, SD-map fusion, 20 points per polyline).
• Planner encoder: Diffusion_Planner_Encoder (BEVFusionEncoder ×8 downsample → 384 tokens, 3-layer FusionEncoder transformer).
• Planner decoder: 3-layer DiT (hidden_dim=256, 8 heads, RouteEncoder), model_type="x_start", diffusion_pred_steps=5, linear VP noise schedule.
• Loss functions: perception — FocalLoss + L1Loss + CrossEntropyLoss + GaussianFocalLoss / CrossEntropyLoss / MapTRCriterion / BevFormer2DCriterion per head; planner — DiffusionLoss (ego 2.0 + collision 1.0).
• Model input: 8-camera multi-view images (B,8,3,512,960) (YUV444 + Normalize(128,128); camera-only) + ego state + route lanes.
• Model output: 8-step future ego trajectory (1,8,4).

Official Repo and Paper

NAVSIM benchmark: https://github.com/autonomousvision/navsim Paper: https://arxiv.org/abs/2406.15349 SparseBEV-style det head repo: https://github.com/MCG-NJU/SparseBEV Paper: https://arxiv.org/abs/2308.09244 MapTR repo: https://github.com/hustvl/MapTR Paper: https://arxiv.org/abs/2208.14437

Note: The camera backbone HENet and the E2E assembly are HEAL in-house implementations; the official repos use different backbones and pipelines.