SpikeVLA catcher for the reBot B601-DM
SpikeVLA policy that catches an 8 m/s ball thrown from 4 m with a reBot Arm B601-DM and a five-finger hand, from two
event cameras only. Code, simulator and real-robot steps: taucatch.
| Same 300 tosses, 8 m/s, 4 m (simulation, 20 ms latency, 24 V motor limits) |
Caught |
| SpikeVLA, actions from the action head |
84.7% |
| SpikeVLA ball track + expert planner |
91.3% |
| Fastest balls, 4 m, first 100 tosses |
8 m/s |
9 m/s |
10 m/s |
| Expert with the true ball state |
98% |
95% |
73% |
| SpikeVLA ball track + expert planner |
87% |
45% |
13% |
| SpikeVLA, actions from the action head |
84% |
49% |
22% |
Model
- Input: two 224 x 224 trace frames (per pixel, the time since its latest event, decayed with 10 / 50 / 250 ms as
the three channels) and the arm state.
- SDT-V3 19M spiking backbone; an in-model ball locator on its tokens, a ballistic track over the calls;
spiking vision-language fusion; spiking action head with Spatially Grounded Spike Attention (SGSA).
- Output: a 12 x 20 ms chunk of end-effector targets and the hand command, every 4 ms.
- 39.2 M parameters. The text model is not included: the instruction "catch the ball" is fixed and its
frozen SmoothSpike-BERT states are stored in the weights.
- Trained in MuJoCo on expert demonstrations and one DAgger round on its own tosses (saim_r1_b601/ema_3000.pt).
Files
| File |
|
config.json |
model configuration, policy flags, state / action normalisation |
model.safetensors |
weights (vla.* SpikeVLA, locator.* ball locator) |
Use
git clone https://github.com/wxqnl/taucatch.git && cd taucatch
pip install -r requirements/vla_catch.txt
python tools/fetch_spikevla_deps.py
tools/fetch_b601.sh
python -m taucatch.vla_catch.evaluate --ckpt Minko030/taucatch ...
from taucatch.vla_catch.policy import load_release
policy, config = load_release("Minko030/taucatch")
