ShaCoZha/HiT-v1

HiT v1: body-worn IMUs → full-body pose and motion language

HiT maps 1–6 body-worn IMUs (accelerometer + gyroscope) to per-frame full-body pose, to natural-language descriptions of the motion, and to text-aligned motion embeddings. It needs no calibration pose: each sensor may be rotated arbitrarily about its skin normal, and the world heading is irrelevant. It was trained on Nymeria only.

This repository holds the first version (v1) of HiT. The code that loads and runs the checkpoints is the imu2language package of SPICExLAB/IMU2Language at commit 156f014 (branch updates-2026-10-22).

Files

File	Training stage	Contents
stage1.pt (516 MB)	stage 1, step 25000	motion encoder, pose head, Fine captioner (LoRA for Qwen3-0.6B)
stage2_nc.pt (611 MB)	stage 2, step 15000	stage 1, frozen, + atomic readout and Atomic captioner (LoRA); trained without user context, targets are the original Nymeria atomic narrations
stage2_ctx.pt (611 MB)	stage 2, step 15000	the same architecture, trained to accept an optional user-supplied context string (location / people / objects)
imuposer_fullft.pt (475 MB)	pose only, fine-tuned on IMUPoser	the same encoder and pose head, pre-trained on Nymeria with the pose loss only (step 25000), then fine-tuned end to end on IMUPoser participants P1–P8 for consumer phones / watches / earbuds; no captioner. See IMUPoser pose model
imuposer_wisdm.pt (530 MB)	imuposer_fullft.pt + WISDM classifier	the same encoder and pose head (frozen, identical pose output) + a stage-2 readout and an 18-class smartwatch activity classifier fine-tuned on WISDM. See WISDM activity classifier
wisdm/WISDM/ (148 MB)	data	the WISDM smartwatch windows used to fine-tune and test imuposer_wisdm.pt
wisdm_example.py	code	runs imuposer_wisdm.pt on the WISDM test windows

Each file is a plain dictionary of tensors and configuration (torch.load(path, weights_only=True) works). The stage-2 files contain the frozen stage-1 encoder and pose head, so pose estimation works from any of the three files. The base LLM (Qwen/Qwen3-0.6B) is not included; it is downloaded from the Hub when a captioner is attached. Context strings and text queries are embedded with BAAI/bge-large-en-v1.5.

Which file to use:

• Pose: any file.
• Fine captions (detailed, body-part-level narration of the motion): stage1.pt.
• Atomic captions and motion–text embeddings: stage2_nc.pt by default; it scores best on all benchmarks below. Its training targets are the original narrations, which name rooms and objects, so its captions also guess them from the motion alone, and these guesses can be wrong.
• Pose from phones in pockets, smartwatches and earbuds (IMUPoser-style devices): imuposer_fullft.pt. Pose only.
• Smartwatch activity recognition (WISDM's 18 activities): imuposer_wisdm.pt. It also gives the same pose as imuposer_fullft.pt.
• Use stage2_ctx.pt when you know the context (for example the room) and want the caption to use it. Without context it describes the motion only and names no rooms or objects, but scores lower than stage2_nc.pt (see Results).

Model

• Motion encoder (~75M parameters in its 24 blocks). Each sensor's signal is rotated about its skin normal so that the in-plane component of the window's mean gravity points along +x, then cut into 0.25 s patches (15 frames at 60 Hz) and embedded with a per-patch MLP. A learned embedding marks the body site of each sensor. Twelve pairs of transformer blocks follow, alternating local attention (within a 0.25 s patch, across sensors) with global attention (across the whole window, with rotary time embeddings).
• Pose head: five body-part queries read the encoder's tokens; a DPT-style decoder fuses 1 / 2 / 4 Hz features and the 60 Hz input into per-frame outputs at 60 Hz.
• Fine captioner: motion Q-Former (2 queries per second) → Qwen3-0.6B with LoRA (rank 16).
• Stage 2 readout: a Q-Former with 4 action queries per 2.5 s chunk and 2 window-level context queries, reading the frozen encoder. Its output is projected (a) into the bge-large-en-v1.5 text space (retrieval, zero-shot activity recognition), and (b) into Qwen3-0.6B with LoRA (atomic captions).

The model is offline: within a window, attention is bidirectional, and the gravity reference is the window mean.

Setup

Tested with Python 3.11, PyTorch 2.13, transformers 5.17, sentence-transformers 6.1, numpy 2.3 and scipy 1.17.

git clone https://github.com/SPICExLAB/IMU2Language.git
cd IMU2Language
git checkout 156f014
pip install torch transformers sentence-transformers numpy scipy
hf download ShaCoZha/HiT-v1 --local-dir HiT-v1

Input format

Item	Requirement
Body sites	1–6 sensors, each at one of SLOTS = ('head', 'lwrist', 'rwrist', 'pelvis', 'lthigh', 'rthigh', 'lshank', 'rshank'); at most one sensor per site
Sensor frame	z axis = the sensor's outward skin normal: head out of the forehead (forward), wrists out of the back of the wrist, pelvis / thighs / shanks out of the front of the segment. Any rotation about z is fine, and so is a device worn face-in (z pointing into the body). No T-pose or other calibration is needed
Signals	accelerometer reading incl. gravity (specific force, m/s²; about +9.81 along "up" at rest) and angular velocity (rad/s), both in that sensor frame
Rate and length	60 Hz (resample first). Training windows were 5–30 s; we evaluate on 10–30 s windows. For long recordings, use 30 s windows with 15 s overlap and blend them (imu2language.eval.bench.predict does this for pose)
Missing gyroscope	set its gyroscope to 0 and its gyro_ok to False

The model reads acc / 9.81 and gyro / 3.0. The helper below builds a batch from raw signals.

Quick start

The repository runs everything below on one recording, downloading the checkpoints itself (imu2language.hub.checkpoint):

python -m imu2language.hit_quickstart --npz rec.npz --sites head lwrist rwrist [--context 'location: in the kitchen']

rec.npz holds acc and gyro, each (T, S, 3), as described above. Without --npz a random signal is used.

The same steps in Python, from the repository root with the checkpoints in HiT-v1/. A GPU is assumed.

import torch
from imu2language.data.windows import PATCH, SLOTS
from imu2language.models.hit import load_hit
from imu2language.utils.geometry import sixd_to_matrix


def make_batch(acc, gyro, sites, device='cuda'):
    """acc (T, S, 3): accelerometer incl. gravity in m/s^2; gyro (T, S, 3): rad/s; both at 60 Hz, each sensor in a
    frame whose z axis is its outward skin normal. sites: one name of SLOTS per sensor."""
    T, S = acc.shape[0] // PATCH * PATCH, len(sites)
    imu = torch.cat([acc[:T] / 9.81, gyro[:T] / 3.0], -1)[None].float()          # (1, T, S, 6)
    ok = torch.ones(1, T, S, dtype=torch.bool)
    b = {'imu': imu, 'frame_ok': ok, 'slot': torch.tensor([[SLOTS.index(s) for s in sites]]),
         'sens_pad': torch.zeros(1, S, dtype=torch.bool), 'gyro_ok': torch.ones(1, S, dtype=torch.bool),
         'token_ok': ok.view(1, T // PATCH, PATCH, S).float().mean(2) >= 0.5,
         'pad': torch.zeros(1, T // PATCH, dtype=torch.bool)}
    return {k: v.to(device) for k, v in b.items()}


@torch.no_grad()
def pose(model, b):
    """Per-frame local joint rotations (1, T, 24, 3, 3) of the 24-joint body."""
    with torch.autocast('cuda', torch.bfloat16):
        taps, _, pad, _ = model.encode(b['imu'], b['frame_ok'], b['slot'], b['sens_pad'], b['gyro_ok'],
                                       b['token_ok'], b['pad'])
        out = model.pose(taps, pad)
    return sixd_to_matrix(out['rot6d'].float())


@torch.no_grad()
def caption(model, b, sites, task, context=''):
    with torch.autocast('cuda', torch.bfloat16):
        return model.generate({**b, 'task': [task], 'positions': [sites], 'context': [context]})[0]


@torch.no_grad()
def embed(model, b):
    """Stage 2: one embedding per 2.5 s chunk (1, K, 1024) in the BAAI/bge-large-en-v1.5 text space."""
    with torch.autocast('cuda', torch.bfloat16):
        _, _, pad, ctx = model.encode(b['imu'], b['frame_ok'], b['slot'], b['sens_pad'], b['gyro_ok'],
                                      b['token_ok'], b['pad'])
        h, _ = model.atomic_readout(ctx)
        z = model.semantic(model.pool_actions(h, ctx, model.chunk_spans(ctx)))
    return torch.nn.functional.normalize(z.float(), dim=-1)


sites = ['head', 'lwrist', 'rwrist']
acc, gyro = torch.randn(600, 3, 3), torch.randn(600, 3, 3) * 0.3        # replace with your 10 s of IMU
acc[..., 2] += 9.81
b = make_batch(acc, gyro, sites)

stage1, _ = load_hit('HiT-v1/stage1.pt', 'cuda', with_llm=True)
print('pose', tuple(pose(stage1, b).shape))
print('fine:', caption(stage1, b, sites, 'fine'))

nc, _ = load_hit('HiT-v1/stage2_nc.pt', 'cuda', with_llm=True)
print('atomic (nc):', caption(nc, b, sites, 'atomic'))
print('embedding', tuple(embed(nc, b).shape))

ctx, _ = load_hit('HiT-v1/stage2_ctx.pt', 'cuda', with_llm=True)
print('atomic (ctx):', caption(ctx, b, sites, 'atomic', 'location: in the kitchen; objects: a knife, the cutting board'))

For pose alone, use load_hit(path, 'cuda') (no LLM is loaded).

Outputs

• Pose: model.pose(...) returns, per frame at 60 Hz:
  • rot6d (24 joints × 6D). Joint 0 is the root's tilt relative to gravity (the heading is not estimated); joints 1–23 are local rotations in SMPL joint order.
  • vel: root velocity (m/s) in the heading frame.
  • yaw_rate: rad/s.
  • height: pelvis height (m).
  • contact: left and right foot-contact logits.
  • joints: joint positions, and joint_logb: per-joint uncertainty.
  • imu2language.body.skeleton.fk_torch(R, model.offsets) gives joint positions on the template skeleton.
• Captions: Nymeria narration style, in which "C" is the person wearing the sensors. Fine captions narrate the body parts over the window. Atomic captions are short action sentences with time spans.
• Context strings (stage2_ctx.pt only): free text in the training format, e.g. location: in the kitchen; people: talking to a friend; objects: the cutting board. An empty string means no context.
• Embeddings: compare with SentenceTransformer('BAAI/bge-large-en-v1.5').encode(texts, normalize_embeddings=True) by cosine similarity, e.g. for retrieval or zero-shot activity recognition with class names as texts.

IMUPoser pose model (imuposer_fullft.pt)

A pose-only model for consumer devices. It has the encoder and pose head architecture of stage1.pt, but its own weights: its Nymeria pre-training used the pose loss only (no Fine captions; 25,000 steps, otherwise the same configuration). The whole encoder and pose head were then fine-tuned on IMUPoser participants P1–P8 with a recipe fixed in advance: 1,875 steps of 32 random 10 s windows (the sample volume of MobilePoser's own IMUPoser fine-tuning), AdamW, learning rate 5e-5, pose loss only, and a random one of IMUPoser's 24 device combinations per window. The last step was kept; P9–P10 were not used for any choice.

IMUPoser's five devices map to the sites lwrist, rwrist (watches), lthigh, rthigh (phones in the left / right trouser pocket) and head (earbud); any non-empty subset can be used. The code below reads an IMUPoser recording (.pkl, 25 Hz: free acceleration and device orientation in a y-up world), converts it to the model's input (specific force and angular velocity in each device's skin-normal frame, 60 Hz), and returns local SMPL joint rotations at 25 Hz. For other devices, give the signals as described in Input format and call imu2language.eval.bench.predict directly.

import pickle

import numpy as np
import torch
from imu2language.eval import bench as B
from imu2language.hub import checkpoint
from imu2language.models.hit import load_hit

FPS = 25                                                    # IMUPoser rate
SITES = ('lwrist', 'rwrist', 'lthigh', 'rthigh', 'head')   # IMUPoser devices: watches, phones in the pockets, earbud


def load_recording(path):
    """IMUPoser .pkl -> free acceleration (T, 5, 3) in m/s^2 and device orientations (T, 5, 3, 3), y-up world."""
    d = pickle.load(open(path, 'rb'), encoding='latin1')
    imu = np.asarray(d['imu'], np.float64)
    return imu[:, :15].reshape(-1, 5, 3), imu[:, 15:].reshape(-1, 5, 3, 3)


def predict(model, acc, ori, devices, device=torch.device('cuda')):
    """devices: indices into SITES (any non-empty subset) -> local SMPL joint rotations (T, 24, 3, 3) at 25 Hz, root
    without heading (the evaluation, like MobilePoser's, ignores the root)."""
    x = B.resample(B.sensor_from_ori(ori, acc, 'y', FPS), FPS)            # 60 Hz specific force + angular velocity
    sites = [B.IMUPOSER_SITES[k] for k in devices]
    xs = B.to_normal(x[:, devices], sites)                                # each device in its skin-normal frame
    R = B.predict(model, xs, [SITES[k] for k in devices], device)        # (T60, 24, 3, 3)
    idx = np.clip(np.round(np.arange(len(acc)) * 60 / FPS).astype(int), 0, len(R) - 1)
    return R[idx]


model, ck = load_hit(checkpoint('hf://ShaCoZha/HiT-v1/imuposer_fullft.pt'), 'cuda')    # pose only: no LLM
acc, ori = load_recording('imuposer_dataset/P9/5. Walking.pkl')
R = predict(model, acc, ori, devices=[0, 4])                                           # left watch + earbud

ck['finetune'] records the fine-tuning recipe and the P9–P10 results below. Run through this code with the released commit, the 12 MobilePoser combinations on P9–P10 give the MPJVE in the table below.

WISDM activity classifier (imuposer_wisdm.pt)

imuposer_wisdm.pt adds an activity classifier for a smartwatch to imuposer_fullft.pt. It contains that file's encoder and pose head unchanged (frozen during this fine-tune, so the pose output is identical), a stage-2 readout with the architecture of stage2_nc.pt (4 action queries per 2.5 s and 2 window-level queries, projected into the bge-large-en-v1.5 space), and ck['wisdm_classifier']: the 18 class names, one vector per class and a logit scale. The readout, its projection, the class vectors (initialized from bge-large-en-v1.5 embeddings of short descriptions of each activity) and the logit scale were fine-tuned on WISDM's training windows: Adam, learning rate 1e-4, batch 64, 20 epochs, keeping the epoch with the lowest training loss. The test windows were not used for any choice.

Data (wisdm/WISDM/). The WISDM Smartphone and Smartwatch Activity and Biometrics Dataset (Weiss, 2019; 51 participants, 18 activities, 20 Hz), smartwatch only, in the windowing and train / test split of UniMTS:

File	Contents
X_train.npy, X_test.npy	float64 (N, 200, 6): 10 s windows at 20 Hz; accelerometer including gravity (m/s²) and gyroscope (rad/s), in the watch's own frame
y_train.npy, y_test.npy	class index per window; 12,406 training and 3,045 test windows
WISDM.json	class names, e.g. walking, typing, eating pasta, folding clothes

Run from the IMU2Language checkout (commit 156f014) with this repository downloaded to HiT-v1/ (see Setup):

PYTHONPATH=. python HiT-v1/wisdm_example.py      # WISDM test: 3045 windows, accuracy 82.1%

wisdm_example.py reads the windows with imu2language.eval.har_datasets.load (trailing zero padding removed, resampled from 20 to 60 Hz, the watch on the rwrist site), runs the model with each window as one span, and scores the classes:

z = model.span_embed(1, taps, g, span, pad)[:, 0]            # (B, 1024), L2-normalized
logits = clf['logit_scale'] * z @ clf['class_vectors'].T      # clf = ck['wisdm_classifier']

For your own watch recordings, save them in the same format (windows of accelerometer including gravity and gyroscope, the watch's z axis out of its face) and load them the same way; for another sampling rate, change the WISDM entry of har_datasets.DATASETS or resample to 20 Hz first. Windows longer than 10 s are cut to 10 s.

Results

All results are zero-shot unless noted: no part of the model was trained on the evaluation datasets. The exception is imuposer_fullft.pt on IMUPoser, which was fine-tuned on other participants (P1–P8) of that dataset.

Zero-shot activity recognition: 10 datasets, the AnyMo protocol; mean accuracy / macro-F1 (%); class names are matched in the embedding space. AnyMo and UniMTS rows are the published numbers listed by AnyMo.

Model	Accuracy	Macro-F1
stage2_nc.pt	54.5	48.2
stage2_ctx.pt (no context)	53.7	47.7
AnyMo	43.9	36.6
UniMTS	40.3	33.7

Motion–text retrieval on the AnyMo-Bench held-out Nymeria split (3,908 windows); MRR (%), IMU→text / text→IMU.

Model	100 candidates	all 3,908
stage2_nc.pt	55.6 / 57.9	15.3 / 14.9
stage2_ctx.pt (no context)	42.0 / 48.4	10.3 / 11.6
AnyMo (paper)	44.6 / 46.7	7.0 / 7.5

Atomic captions on the same split; BLEU-1 / BLEU-4 / ROUGE-L / METEOR / BERTScore-F1.

Model	Score
stage2_nc.pt	36.1 / 11.1 / 36.7 / 45.3 / 71.7
stage2_ctx.pt, no context	22.1 / 3.9 / 29.2 / 30.5 / 68.0
stage2_ctx.pt, context taken from the reference narration*	46.1 / 20.8 / 44.7 / 55.0 / 74.8
AnyMo (paper)	25.0 / 6.5 / 31.1 / 33.5 / 69.7

* This row gives the model information taken from the reference text. It shows how well the model uses context and is not comparable with the other rows.

Activity recognition on WISDM (smartwatch, 18 classes): the test windows of UniMTS's split; imuposer_wisdm.pt was fine-tuned on the training windows (above). UniMTS rows are the fine-tuned results reported in the UniMTS paper.

Model	Accuracy	Macro-F1
imuposer_wisdm.pt (encoder frozen)	82.1	82.1
UniMTS, fine-tuned	81.3	80.9
BioBankSSL, fine-tuned (best baseline in UniMTS's table)	83.3	82.9

The hardest classes are the eating activities (sandwich, chips, soup, pasta), which differ in the food rather than in the wrist motion.

Pose on IMUPoser (consumer phones / watches / earbuds): participants P9–P10, MobilePoser's 12 device combinations, MobilePoser's evaluator, every model on the same frames; MPJVE (cm), mean and by number of devices. Uncalibrated: each device of a recording gets a fixed random mounting rotation (up to 180° about the skin normal and 15° of tilt) and a random world heading, i.e. no calibration at all.

Model	Mean	1 device	2 devices	3 devices	Uncalibrated
imuposer_fullft.pt (fine-tuned on P1–P8)	8.2	10.5	8.1	7.2	8.2
MobilePoser, released weights (trained on AMASS, fine-tuned on IMUPoser P1–P8)	10.4	13.0	10.7	8.7	28.6
IMU4D, IMUPoser fine-tuned release	10.1
HiT v1 (stage1.pt), zero-shot	14.6				14.8
Pose-only pre-training of imuposer_fullft.pt, zero-shot	14.4				14.5

Frame-to-frame jitter is higher than MobilePoser's (1.17 vs 0.10 × 100 m/s³ in its evaluator, where the ground truth has about 0.5). Part of it comes from reading our 60 Hz output at the 25 Hz frames; a 3 Hz zero-phase low-pass filter on the output removes most of it without changing MPJVE.

Limitations

• Offline only: the model uses future frames within its window, so it is not causal.
• Fixed sites: only the eight body sites listed above. Sensors on the chest, feet or upper arms are not supported.
• Frame convention: each sensor's z axis must be its skin normal. A device whose normal is another axis must be rotated first; a wrong normal axis mainly degrades static postures (sitting vs standing).
• Training data: Nymeria only. The head and wrist sensors are real Project Aria IMUs. The pelvis, thigh and shank signals were synthesized from Nymeria's XSens motion capture, so real sensors at those sites face a domain gap.
• imuposer_fullft.pt is pose only (its Fine / Atomic captioners are untrained; do not attach an LLM) and was fine-tuned on IMUPoser, so its IMUPoser numbers are in-domain, not zero-shot.
• imuposer_wisdm.pt: its readout was retrained for WISDM classification, so its embeddings are tuned to WISDM's 18 activities, and it has no captioner. Its accuracy is in-domain (fine-tuned on other WISDM windows).
• Captions: they follow Nymeria's narration style and vocabulary. Rooms and objects named by stage2_nc.pt are guesses from motion alone. The context-variant targets used to train stage2_ctx.pt were derived from the Nymeria narrations with Qwen3-30B-A3B-Instruct-2507.

License and attribution

The weights are released under CC BY-NC 4.0 (non-commercial). They are derived from the Nymeria dataset (Meta, CC BY-NC 4.0); see the Nymeria project page for its terms and citation. The LoRA adapters apply to Qwen3-0.6B (Apache 2.0), which is downloaded separately.

The WISDM data in wisdm/WISDM/ and the classifier fine-tuned on it derive from the WISDM Smartphone and Smartwatch Activity and Biometrics Dataset (CC BY 4.0): Weiss, G. (2019). WISDM Smartphone and Smartwatch Activity and Biometrics Dataset. UCI Machine Learning Repository. https://doi.org/10.24432/C5HK59. The windows and train / test split are those prepared by UniMTS (Zhang et al., NeurIPS 2024).