DaoyuanZhu/wb_pointed_chair_pull_push_posture

Chair-task sit-down recogniser (Unitree G1)

A 224 K-parameter CNN that tells whether the person in front of the robot is standing, starting to sit down, or seated, from the robot's own chest camera. It exists to trigger one action: in the task "pull out the chair indicated by the human gesture, then push it back when the human starts sitting down", the robot needs to know when the sit-down begins.

No pose estimator and no motion-capture rig at inference — one small CNN on raw greyscale pixels. Runs at 680 FPS on a single CPU thread.

Usage

from modeling_posture import PostureRecognizer

rec = PostureRecognizer.from_pretrained('DaoyuanZhu/wb_pointed_chair_pull_push_posture', device='cpu')
for bgr in camera_stream():                 # 640x480 BGR frames, in order, 30 FPS
    out = rec.push(bgr)
    if out['warmup']:                       # first ~1.1 s fills the ring buffer
        continue
    print(out['posture'], out['confidence'], out['language'])
    if out['posture'] == 'sitting_down' and out['confidence'] > 0.5:
        robot.push_chair_back()

push() is causal — only the current and past frames are used — so it is safe to run online. It keeps its own 33-frame ring buffer and samples 9 frames at stride 4 (a 1.07 s window).

Inputs and outputs

input	cam_chest RGB, any size (resized to 160x120 greyscale internally)
window	9 frames at stride 4 = 1.07 s, causal
output	standing / sitting_down / seated + probabilities + a language phrase

Results

Held-out episodes (10 of 80; no episode appears in both train and test):

	accuracy	macro-F1	standing F1	sitting_down F1	seated F1
this model	97.6 %	0.916	0.991	0.815	0.941
keypoint MLP baseline	95.2 %	0.845	0.976	0.730	0.829

Sit-onset detection (probability of sitting_down + seated >= 0.5 held for 3 frames): 10/10 events detected, 0 missed, 0 false alarms, median 0.25 s before the labelled onset.

Per-episode frame agreement: mean 97.7 %, worst 93.8 %, best 99.5 %.

Speed

Batch size 1, one forward pass per camera frame.

device	ms/frame	FPS
RTX 5080	0.15	6 600
CPU, 32 threads	0.80	1 250
CPU, 1 thread	1.47	680
preprocessing (640x480 BGR -> 160x120 grey)	0.18	—

At 30 FPS the per-frame budget is 33.3 ms, so a single CPU thread uses about 5 % of it. No GPU is needed.

How the labels were made

Training labels were not hand-annotated. They come from a synchronised markerless mocap recording of the same sessions, which is already resampled 1:1 onto the LeRobot episode timeline (episode e frame i is row i), so no timestamp alignment was needed.

From the 40-point 3D skeleton, normalised hip height h = (z_hip - z_floor) / (z_hip_standing - z_floor) is thresholded with hysteresis: h >= 0.95 standing, h <= 0.68 seated, and the sit-down window runs from the last standing-height frame to the start of the seated run. Using the hysteresis crossing rather than a velocity walk-back keeps the onset stable when the subject pauses part-way down. 79 of 80 episodes yield exactly one clean sit event (median 1.80 s); the remaining episode has none — the human genuinely stays standing.

label_posture.py in this repo reproduces the labels; train_rgb.py reproduces the model.

Two things worth knowing about the source data

The mocap pipeline's "primary" tracklet is the wrong person in 27 of 80 episodes — it locks onto the teleoperator or a bystander rather than the person who sits. The per-episode identity letters are not stable across episodes either. The subject is re-selected per episode by taking the tallest well-covered tracklet; the separation is unambiguous (seated subject 1.65-1.72 m, teleoperator ~1.49 m hunched in the VR rig, bystanders ~1.25 m). Using the primary tracklet directly would mislabel a third of the data.

The 2D pose landmarks shipped with the dataset are not usable for this. During labelled sitting_down frames cam_chest.valid averages 0.46, and 40 of 79 episodes are below 50 % coverage with several at zero; cam_head is 0.04 during the sit. The person is plainly visible in the video throughout — it is an identity-lock failure in the pose pass, not occlusion. That is why this model reads pixels instead, and why the keypoint baseline above misses 1 of 10 events.

Onset timing

The onset was tuned against the demonstrator's own reaction, measured as the robot's wrist-motion burst when pushing the chair:

H_STANDING	push vs. labelled onset
0.88 (hips already descending)	median -0.43 s — the label is late
0.95 (this model)	median +0.33 s — the label leads slightly

End to end the trigger lands about 0.58 s before the demonstrator started pushing, which gives a controller headroom rather than chasing the event.

Limitations

• Deep forward lean can trigger it. If the person bends far over the table without lowering their hips, the model may call it sitting_down. In the worst held-out episode this fired 1.9 s early for that reason. The label calls it standing (hip height barely moves); the image genuinely looks like a sit beginning. If false triggers matter more than latency, require the probability to hold for ~10 frames instead of 3 (costs about 0.23 s).
• One scene, one subject. All 80 episodes are the same room, camera pose and person. Expect to fine-tune for a new environment.
• standing_up is not supported. Every episode ends while seated, so the class has no training examples. The label code emits it if the data contains it.
• Labels are machine-generated from mocap geometry, not human judgement. Boundary frames between classes are intrinsically fuzzy.

Files

file
modeling_posture.py	self-contained model + ring-buffer inference wrapper
model.safetensors	weights (0.9 MB)
config.json	window, stride, input geometry
example.py	run over a video file
label_posture.py	reproduces the labels from the markerless mocap
train_rgb.py, common.py	reproduces the model
demo.mp4	held-out episode with predictions and probability bars overlaid

Citation

Derived from DaoyuanZhu/wb_pointed_chair_pull_push_rgb.