fusion-embedding-2-tremor

Tremor is the inertial (IMU) sensor pack for Eximius Labs' fusion-embedding stack. It maps a short window of body-worn motion (a 3-axis accelerometer stream) into the Qwen3-VL-Embedding-2B text embedding space, so a robot's or wearable's motion history becomes searchable in plain language: "find when it was walking", "find when it picked something up".

Tremor is part of the fusion-embedding family and plugs into the same shared space as fusion-embedding-2. It targets the frozen Qwen3-VL-Embedding-2B text embeddings that the family is built on, and because fusion-embedding-2's text, image, and video paths are byte-identical to that base, Tremor's motion embeddings are directly comparable to that model's audio, text, image, and video in one 2048-d space. Motion becomes a first-class, language-addressable modality alongside the rest.

GitHub | fusion-embedding-2 | Technical report: in preparation

Model Overview

Tremor is a frozen UniMTS motion encoder plus a small trained projector. A single accelerometer window is placed at one joint of UniMTS's orientation-invariant skeleton-graph encoder (ST-GCN), which produces a 512-d motion feature; a projector of roughly 2.6M parameters maps that into the frozen Qwen base's 2048-d text space. Nothing is trained except the projector. Motion and text embeddings are L2-normalized and compared by cosine similarity.

The design choice that matters is the encoder: rather than train an IMU encoder from scratch, Tremor reuses a frozen, orientation-invariant pretrained one and learns only the projection into the language space. The pretraining prior, not our training, is what carries cross-subject and cross-mounting generalization.

Feature	Value
Parameters	~2.06B frozen Qwen base + frozen UniMTS encoder; 2.6M trained
Modality	inertial motion (3-axis accelerometer)
Supported tasks	zero-shot activity retrieval, zero-shot activity classification
Input	one accelerometer window [3, T] (any length/rate; resampled to 200 samples @ 20 Hz)
Sensor placement	single IMU, mapped to one skeleton joint (robots/wearables have a known, fixed mount)
Embedding dimension	2048
Pooling strategy	last-token pooling (text side)
Base model	Qwen/Qwen3-VL-Embedding-2B (frozen)
Motion encoder	UniMTS ST-GCN, accelerometer-only (frozen)
Trained components	projector 2.6M LayerNorm(512) -> Linear(512,1024) -> GELU -> Linear(1024,2048)
Distribution	~10 MB trained projector; the frozen encoder and base download from their own repos

See it in action

A full activity sequence, read live. A Unitree G1 performs each activity in simulation — sitting, walking, running, climbing — while a window slides across a real, held-out body-worn accelerometer stream (RealWorld, out of training) and Tremor's scores update in real time, tracking the motion as it changes. The robot shows each activity full-body; the signal and the predictions are real, frame-synced data.

On the robot's own IMU and camera. On a real Unitree G1 humanoid (Humanoid-Everyday), Tremor reads the robot's own IMU as language, live and frame-synced to its egocentric camera — zero simulation. As it switches from adjusting a phone stand to walking to a door, its IMU signature changes from flat to dynamic and Tremor's read tracks the activity in real time.

Any body-worn sensor. No robot is required — the same works on a wristband or wearable. Here the raw accelerometer stream slides by directly and Tremor's scores track sitting → walking → running → climbing on the held-out RealWorld set.

Text → motion retrieval. Query an activity in plain language and Tremor returns the nearest motion windows. Below, on the RealWorld set held out of training entirely (zero-shot), each thumbnail is a retrieved window's 3-axis accelerometer trace; correct retrievals (the window's true activity matches the query) are outlined in green.

Training and Evaluation

Tremor is trained with InfoNCE against the frozen base's native chat-template text embeddings, on a diverse 16-source pool of real inertial data: 13 standard human-activity-recognition datasets, Ego4D (egocentric human), Humanoid-Everyday (Unitree robot), and Capture24 (151 subjects of free-living wrist accelerometry). Only the projector learns.

The single most important lesson of the project is the evaluation metric. Instance retrieval (finding the exact paired window) is misleading for this task and makes a working activity recognizer look dead. The right metric is k-way activity classification on held-out subjects and held-out datasets, and all numbers below use it (chance = 1/k).

Diversity is what makes the base general

Trained on one source (Ego4D alone), an IMU encoder generalizes poorly to sensor streams that look nothing like it. Tremor's headline result is zero-shot transfer to datasets held out of training entirely. On five held-out datasets, a single-source baseline (the identical architecture trained on Ego4D only) sits at or below chance, while Tremor generalizes.

Held-out dataset (5-way, zero-shot)	Single-source baseline	Tremor v0.1
RealWorld	0.16	0.68
USC-HAD	0.13	0.59
DSADS	0.29	0.58
UTD-MHAD	0.21	0.33
held-out mean	0.20 (chance)	0.545

On the two in-domain holds (held-out subjects/episodes of sources seen in training), Tremor reaches 0.40 5-way on both Ego4D and the Humanoid-Everyday robot set. The held-out datasets above are excluded from training entirely, so those numbers are genuine cross-dataset generalization.

Usage

Requirements

• torch (CUDA recommended), numpy, transformers>=4.46, huggingface_hub
• The frozen UniMTS encoder code and weights (Apache-2.0 code):

  git clone https://github.com/xiyuanzh/UniMTS      # provides model.py::ST_GCN_18
  

  Point UNIMTS_REPO (or unimts_repo=) at the clone. The UniMTS weights download automatically from the hub.
• The frozen Qwen3-VL-Embedding-2B base downloads from its original repository.

via inference.py (this repository)

import numpy as np
from inference import TremorEmbedder

tr = TremorEmbedder.from_pretrained(
    "EximiusLabs/fusion-embedding-2-tremor",
    revision="v0.1-preview",
    unimts_repo="UniMTS",          # path to your git clone of xiyuanzh/UniMTS
)

# accel: a 3-axis accelerometer window, shape [3, T] (any length / sample rate)
accel = np.load("window.npy")      # e.g. [3, 300]

# rank candidate activities by similarity to the motion
for text, score in tr.rank(accel, ["walking", "sitting", "running", "climbing stairs"]):
    print(f"{score:+.3f}  {text}")

# or embed motion and text into the shared space directly
m = tr.embed_motion(accel)                       # 2048-d, L2-normalized
t = tr.embed_text(["walking", "sitting"])        # 2048-d each

Embedding quality is sensitive to the base's chat-template formatting; inference.py applies it for you, so match candidate activities against motion through this API rather than embedding text yourself with a different template.

Related models

Tremor comes in two tiers, both built on fusion-embedding-2:

Model	Best for	Accuracy
This model — general base	any sensor / platform, unseen datasets (zero-shot)	0.545 held-out 5-way (unseen datasets)
fusion-embedding-2-tremor-g1	deployment on the Unitree humanoid (G1/H1)	0.740 in-domain 5-way

Start with this general base — it is the tier that generalizes to sensors it has never seen. Move to the per-fleet Tremor-G1 head only when you deploy on that platform and want maximum in-domain accuracy. Specializing a head raises in-domain accuracy but does not improve cross-dataset transfer; the two do different jobs. Deployment guidance (sensor contract, streaming, on-device inference, per-fleet fine-tuning) is in the Tremor-G1 card.

License

The trained projector weights in this repository are released under CC-BY-NC-4.0 (non-commercial).

This posture is deliberate and reflects the model's lineage. Tremor's frozen encoder is UniMTS, which was pretrained on synthetic IMU derived from the AMASS / HumanML3D motion corpora. AMASS is released under a non-commercial research license, so we ship Tremor non-commercial to respect that lineage. A commercially-clean encoder (re-pretrained on permissively licensed motion) is future work; a commercial license may follow.

Limitations

• Accelerometer only. The released UniMTS encoder is accelerometer-only; gyroscope is not used. Rotational-rate information is not currently modeled.
• Cross-dataset ceiling. Zero-shot cross-dataset accuracy plateaus around 0.55 (5-way). This is a research preview, not a production activity classifier. The right use is language-addressable motion search, not high-stakes classification.
• Single-sensor, single-joint. One IMU mapped to one skeleton joint, matching a fixed-mount deployment (a robot torso, a wristband). Multi-sensor fusion is not modeled.
• English text only, and the base's chat-template formatting must be used.
• Specialization vs. generalization. Fine-tuning on a specific platform's data raises accuracy on that platform substantially but does not improve cross-dataset transfer. Tremor is the general base; per-fleet fine-tuning is a separate recipe.

Citation

If you use Tremor, please cite this repository and acknowledge the frozen components it builds on:

@misc{tremor2026,
  title  = {Tremor: an inertial-motion sensor pack for the fusion-embedding space},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  note   = {Eximius Labs. Model weights CC-BY-NC-4.0.},
  url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor}
}

Tremor's frozen motion encoder is UniMTS (Zhang et al., UniMTS: Unified Pre-training for Motion Time Series, NeurIPS 2024), pretrained on synthetic IMU derived from AMASS (Mahmood et al., ICCV 2019). The text space is Qwen3-VL-Embedding-2B. Please cite those works as well.