YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Motion Caption Augmentation

This project augments low-frequency motion_type captions from:

/cpfs01/videogen/buwendong/motion_example_0714_demo/metadata.jsonl

The source dataset is treated as read-only. All reports, generated captions, embeddings, and run summaries are written under:

/cpfs01/users/yetinghong/code/rewrite_motion/dataset

What The Pipeline Does

  1. Counts the long-tail distribution of motion_type.
  2. Extracts original action references from each motion_caption/*.json file using only whole_body fields.
  3. Uses an OpenAI-compatible LLM to generate short English text-to-motion captions for low-frequency subclasses.
  4. Embeds original and generated captions with the same downstream text backbone by default: Qwen2.5-Omni-3B thinker.
  5. Automatically calibrates a cosine-similarity threshold from same-class positives and same-major hard negatives.
  6. Keeps only generated captions that are close enough to their original motion_type cluster and not closer to sibling classes.
  7. Optionally visualizes video_caption or extracted whole_body embeddings with PCA + t-SNE.

Default target is 100 captions per low-frequency motion_type.

Environment

Recommended environment, matching the training scripts:

cd /cpfs01/users/yetinghong/code/rewrite_motion
source /cpfs01/users/yetinghong/code/Psi0-yth-v15/.venv-psi/bin/activate
pip install -r requirements.txt

For LLM generation:

export OPENAI_API_KEY=...
export OPENAI_MODEL=...
# Optional for OpenAI-compatible endpoints:
export OPENAI_BASE_URL=...

The default embedding model is:

/cpfs01/videogen/public_ckpt/Qwen/Qwen2.5-Omni-3B

One-Command Runs

Small end-to-end test:

python script/motion_caption_augment.py pipeline --mode test

Full run:

python script/motion_caption_augment.py pipeline --mode full

If you only want a structural smoke test without loading Omni, use the dev-only hash encoder:

python script/motion_caption_augment.py pipeline --mode test --encoder hash

--encoder hash is only for debugging the pipeline mechanics. Use the default Omni encoder for real filtering.

Debuggable Steps

Distribution stats:

python script/motion_caption_augment.py stats

Build reference captions:

python script/motion_caption_augment.py build-references

Calibrate threshold:

python script/motion_caption_augment.py calibrate-threshold

Generate raw captions:

python script/motion_caption_augment.py generate --mode test

Filter a run:

python script/motion_caption_augment.py filter --run-dir /path/to/run_dir

Validate a run:

python script/motion_caption_augment.py validate --run-dir /path/to/run_dir

Visualize Caption Embeddings With t-SNE

The visualization command reuses the same Omni thinker embedding implementation as the semantic filter. By default it reads video_caption from the 0630 demo metadata, keeps at least three examples from every motion_type, samples 5,000 records in total, reduces the embeddings to 50 dimensions with PCA, and then runs Barnes-Hut t-SNE to 2D:

CUDA_VISIBLE_DEVICES=0 python script/motion_caption_augment.py visualize-embeddings \
  --text-source video_caption \
  --encoder omni \
  --max-samples 5000 \
  --color-by major

The output directory is printed at the end of the run and is created below dataset/visualizations/. It contains:

tsne_interactive.html   # interactive plot with caption text on hover
tsne_static.png         # publication/debug-friendly static plot
tsne_points.csv         # IDs, labels, captions, and 2D coordinates
tsne_points.jsonl       # the same data in JSONL form
coordinates.npy         # raw [N, 2] coordinates
visualization_report.json

Useful variants:

# Color all 131 subclasses separately.
python script/motion_caption_augment.py visualize-embeddings \
  --encoder omni --color-by motion_type

# Compare against the motion-only whole_body references.
python script/motion_caption_augment.py visualize-embeddings \
  --text-source whole_body --encoder omni

# Use every caption. This is substantially slower and more memory-intensive.
python script/motion_caption_augment.py visualize-embeddings \
  --text-source video_caption --encoder omni --max-samples 0 \
  --compact-output --discard-embedding-cache \
  --output-dir dataset/video_caption_tsne_full

# Structural smoke test only; hash embeddings are not semantically meaningful.
python script/motion_caption_augment.py visualize-embeddings \
  --encoder hash --max-samples 300 --tsne-max-iter 500

The step can also be coupled to the existing end-to-end pipeline:

python script/motion_caption_augment.py pipeline --mode test \
  --visualize-embeddings --text-source video_caption --max-samples 5000

Embedding and t-SNE coordinates are cached. Repeating the same command reuses them; use --overwrite-embeddings or --overwrite-tsne to force recomputation. For a clean final delivery, --compact-output keeps only HTML, PNG, CSV, and the report, while --discard-embedding-cache removes the large intermediate embedding cache after all final artifacts are written successfully. The report includes 10-nearest-neighbor label agreement for major and motion_type. Remember that t-SNE is intended for local-neighborhood inspection: global distances and apparent cluster sizes should not be interpreted literally.

Rewrite Rare Video Captions With Local vLLM

Start the project-local Qwen3.5-27B OpenAI-compatible service:

bash vllm/3_qwen3_5_27b_vllm.sh

The service listens on http://127.0.0.1:8301/v1. The launcher was copied from compose_motion_example/3_qwen3_5_27b_vllm.sh and adjusted to manage only its own PID instead of killing unrelated API servers.

Generate and select ten diverse rewrites for low-frequency motion types:

CUDA_VISIBLE_DEVICES=2 \
/cpfs01/users/yetinghong/code/Psi0-yth-v15/.venv-psi/bin/python \
  script/video_caption_rewrite.py \
  --dataset-root /cpfs01/videogen/buwendong/motion_example_0714_demo \
  --output-dir dataset/caption_rewrite_0714_motion_only_10 \
  --exclude-majors dance \
  --num-target-types 24 \
  --rewrites-per-source 4 \
  --api-concurrency 8 \
  --num-results 10 \
  --device cuda \
  --distance-quantile 0.75

The workflow first uses Qwen3.5 to convert every source video_caption into a short BoneSeed-style action description, removing clothing, scene, camera, and filming effects. It then creates genuinely different motions while treating the exact fine-grained motion_type as the semantic boundary, rather than merely paraphrasing the old motion. All original and rewritten captions are jointly embedded with the Qwen2.5-Omni thinker and mapped through PCA + t-SNE. The novelty threshold is the 75th percentile of original-caption nearest-neighbor distances. No cosine-similarity acceptance constraint is applied; 2D novelty, non-copying, length, and motion-only context guards are used. Temporary candidates and embeddings are removed after a successful run.

Main Outputs

Each run directory contains:

raw_generated.jsonl
accepted_captions.jsonl
rejected_captions.jsonl
threshold_report.json
summary.md
validation_report.json

Accepted records follow this shape:

{
  "augment_id": "aug_sports_swimming_000001",
  "motion_type": "sports_swimming",
  "major": "sports",
  "minor": "swimming",
  "caption": "A swimmer performs a smooth backstroke with alternating arm pulls and steady flutter kicks.",
  "source_ids": ["00030660", "00030348"],
  "embedding_filter": {
    "encoder": "Qwen2.5-Omni-3B-thinker",
    "max_same_type_similarity": 0.82,
    "threshold": 0.76,
    "nearest_source_ids": ["00030660"]
  }
}

Safety Notes

  • The original dataset is never modified.
  • video_caption is not used as the similarity anchor because it contains background and appearance details.
  • Generated captions are capped so accepted samples do not exceed the missing count needed to reach the target.
  • The default filter uses Omni thinker embeddings; the hash encoder is not semantically meaningful and should not be used for final data.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support