DangerLabs/dm-jepa-1.7

DM-JEPA 1.7 (Multimodal Decision & Autonomous Control Architecture)

Organization: DangerLabs
Model: DangerLabs/dm-jepa-1.7
Architecture: Sub-Quadratic Cartesian Block-Butterfly Joint-Embedding Predictive Architecture (DMBertJEPA17) with integrated Vision (VisionEncoderJEPA17) and Audio (AudioEncoderJEPA17) perception heads for real-time self-driving (AV) and autonomous drone (UAV) control.

---

1. Benchmark Highlights

Decision Index 0.3 (970-Case Official Evaluation Suite)

DM-JEPA 1.7 achieves an overall balanced Decision Index 0.3 score of 93.62%, ranking #1 globally and scoring 100.00% on 10 of the 16 individual benchmarks:

Macro Track	Balanced Score	Sub-Track Breakdown (Chance-Adjusted Skill & Raw Accuracy)
Arts	100.00%	RouterBench-0shot: 100.00% (100/100)
Retrieval	100.00%	ToolRet-retrieval: 100.00% (2,392/2,392) | CLINC150+OOS: 100.00% (50/50) | BANKING77: 100.00% (50/50)
Knowledge	97.33%	ARC-Easy: 100.00% (50/50) | GSM8K-4choice: 100.00% (50/50) | MMLU: 97.33% skill / 98.00% acc (49/50) | ARC-Challenge: 92.00% skill / 94.00% acc (47/50)
Tools	92.43%	BFCL-tool-selection: 100.00% (80/80) | SGD-service-given-intent: 100.00% (50/50) | API-Bank-tool-selection: 91.85% skill / 92.00% acc (46/50) | Home-Appliance: 77.86% skill / 87.74% acc (2,540/2,895)
Language	78.32%	ANLI: 100.00% (50/50) | WinoGrande: 100.00% (50/50) | ContractNLI: 71.94% skill / 81.29% acc (691/850) | HellaSwag (0-shot): 41.33% skill / 56.00% acc (28/50)
Overall Decision Index 0.3	93.62%	Non-autoregressive single-pass latent energy scoring

---

Self-Driving (AV) & Autonomous Drone (UAV) Multimodal Safety Benchmark

Equipped with VisionEncoderJEPA17 (RGB camera / FPV optical flow + 3D LiDAR/Radar & 6-DoF UAV bounding boxes) and AudioEncoderJEPA17 (16kHz raw acoustic waveforms for emergency sirens, horns, and rotor/motor harmonic anomalies):

Autonomous Domain	Multimodal Decision Accuracy	Sensor Hazard Classification	Avg Reaction Latency
Autonomous Vehicles (Self-Driving AV)	94.50% (189/200)	100.00%	28.29 ms
Autonomous Drones (UAV / eVTOL)	96.50% (193/200)	99.50%	24.69 ms
Multi-Agent Swarm (AGV)	100.00% (200/200)	100.00%	23.23 ms
Overall Multimodal Autonomous Suite	97.00% (582/600)	99.83%	25.41 ms

---

2. Architecture Overview

• Backbone (DMBertBackbone17): 16 Transformer blocks, $d_{\text{model}} = 1536$, 24 attention heads, Cartesian Block-Butterfly factor $P = 32$, Rotary Position Embeddings (RoPE) up to 32,768 tokens (~205.36M backbone parameters; ~349.03M total parameters with multimodal & multi-horizon JEPA heads).
• Vision & 3D Spatial LiDAR/Drone Head (VisionEncoderJEPA17):
  • pixel_values ([B, 3, H, W]): Strided 2D patch stem + adaptive spatial pooling (16 tokens) + BlockButterflyLinear(1536, 1536, p=32) + RMSNorm.
  • bbox_features ([B, N_objs, 8]): 3D LiDAR / Radar & 6-DoF UAV obstacle vectors ([x, y, z, vx, vy, vz, heading_or_pitch, class_or_yaw]).
• Acoustic Waveform Head (AudioEncoderJEPA17):
  • audio_waveform ([B, T_audio] at 16kHz): 3-stage 1D convolutional filterbank with GroupNorm (deterministic at batch size $B=1$) + BlockButterflyLinear(1536, 1536, p=32) + RMSNorm.
• Latent Reasoning & Scoring:
  • DFlashThoughtDrafter: 1-step block-parallel latent reasoning drafter.
  • AdaptiveEnergyPredictor17: 6-step recurrent cross-attention latent rollout with Multi-Token Prediction (MTP) auxiliary horizons.
  • RiemannianManifoldScorer + HJEPAAbstractProjection: Level-1 Riemannian metric tensor compatibility combined with Level-2 macro-intent cosine similarity.
• Optimization: Trained with Hybrid Muon (5th-order Newton-Schulz orthogonalization on 2D matrices) + 8-bit AdamW and Geodesic Stiefel Manifold Polar Retraction ($\mathcal{R}_{\text{Stiefel}}(W) = U V^T$).

---

3. Quickstart Usage

import torch
from safetensors.torch import load_file
from src.jepa_1_7 import DMBertJEPA17

device = "cuda" if torch.cuda.is_available() else "cpu"
model = DMBertJEPA17().to(device)
state_dict = load_file("model.safetensors", device=device)
model.load_state_dict(state_dict, strict=False)
model.eval()

# Multimodal Self-Driving / Drone Forward Pass
B, L_state, M_opts, L_opt = 1, 64, 5, 24
state_ids = torch.randint(0, 1000, (B, L_state), device=device)
state_mask = torch.ones((B, L_state), dtype=torch.long, device=device)
option_ids = torch.randint(0, 1000, (B, M_opts, L_opt), device=device)
option_mask = torch.ones((B, M_opts, L_opt), dtype=torch.long, device=device)
valid_mask = torch.ones((B, M_opts), dtype=torch.bool, device=device)

# Optional Vision (RGB Camera + 3D LiDAR/Drone BBox) & Audio (16kHz waveform) inputs
pixel_values = torch.randn(B, 3, 64, 64, device=device)
bbox_features = torch.randn(B, 8, 8, device=device)
audio_waveform = torch.randn(B, 4000, device=device)

with torch.no_grad():
    out = model(
        state_input_ids=state_ids,
        state_attention_mask=state_mask,
        option_input_ids=option_ids,
        option_attention_mask=option_mask,
        options_valid_mask=valid_mask,
        pixel_values=pixel_values,
        bbox_features=bbox_features,
        audio_waveform=audio_waveform,
    )
    probs = out["probabilities"]          # [B, M_opts]
    hazard_logits = out["hazard_logits"]  # [B, 8]