Brain: Spiking Visuomotor Policy

Model Description

Brain is a recurrent Spiking Neural Network (SNN) visuomotor policy trained end-to-end for robot manipulation through imitation learning.

The model processes:

• Dual RGB camera observations
• 7-dimensional proprioceptive joint state

and predicts a 7-dimensional action delta (6 joints + gripper) at every timestep.

The policy is implemented entirely with spiking neural network components using SpikingJelly, employing Parametric Leaky Integrate-and-Fire (PLIF) neurons, recurrent temporal processing, and surrogate-gradient backpropagation.

The released checkpoint is the best-performing model trained using delta-action prediction, which significantly outperformed an equivalent absolute-action formulation.

---

Model Details

Property	Value
Architecture	Recurrent Spiking Neural Network
Parameters	1.65M
Framework	PyTorch + SpikingJelly
Neuron Type	Parametric LIF (PLIF)
Inputs	Dual RGB images + 7-DoF joint state
Output	7-DoF action delta
Training Episodes	407 teleoperated demonstrations
Best Checkpoint	full_run_v1_delta (step 3600)

---

Architecture

Global Camera ───────────┐
                          │
                          ▼
                  Spiking CNN Encoder
                          │
Gripper Camera ───────────┤
                          ▼
                  Spiking CNN Encoder
                          │
Joint State ──────────────┐
                          ▼
                Spiking MLP Encoder
                          │
                          ▼
            Recurrent PLIF Fusion Network
                          │
                          ▼
                    Linear Projection
                          │
                          ▼
             Non-Spiking LIF Readout
                          │
                          ▼
                7-DoF Action Delta

The network consists of independent spiking vision encoders for each camera, a spiking proprioceptive encoder, recurrent PLIF fusion layers, and a non-spiking LIF readout that produces continuous control outputs.

---

Training Data

The model was trained using 407 teleoperated robot manipulation demonstrations containing:

• Dual synchronized RGB camera streams
• 7-dimensional robot joint state
• Demonstrated robot actions

Training images were resized to 128×128, and trajectories were divided into overlapping temporal windows of length 32 with stride 16.

---

Training Objective

Rather than predicting the next absolute joint configuration, the model predicts the difference between the demonstrated action and the current joint position.

This formulation substantially improved optimization by matching the scale of real per-step robot motion, resulting in significantly lower prediction error than direct absolute-action regression.

---

Evaluation

Evaluation was performed offline on a held-out validation split using open-loop trajectory prediction.

Best Checkpoint (full_run_v1_delta)

Metric	Value
MAE	0.00989 rad
RMSE	0.01873 rad
R²	0.99917
Cosine Similarity	0.99960
Mean Episode Correlation	0.99135
Final Pose Error	0.05417 rad

Compared to the absolute-action baseline:

Metric	Absolute	Delta
MAE	0.0834	0.0099
RMSE	0.1278	0.0161
Mean Correlation	0.812	0.991

---

Intended Use

This model is intended for research in:

• Spiking neural networks
• Robot imitation learning
• Neuromorphic computing
• Multimodal visuomotor policies
• Temporal sequence modeling for robot control

---

Limitations

The model has been evaluated only in offline open-loop settings against held-out demonstrations.

Reported metrics measure trajectory prediction accuracy rather than real-world task completion. Physical deployment or closed-loop simulation is required to assess manipulation performance.

---

Citation

If you use this model in your research, please cite this repository.

@misc{brain2026,
  title={Brain: A Recurrent Spiking Neural Network for Visuomotor Control},
  author=Adith,
  year={2026},
  howpublished={Hugging Face}
}