YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Motion Caption Augmentation
This project augments low-frequency motion_type captions from:
/cpfs01/videogen/buwendong/motion_example_0714_demo/metadata.jsonl
The source dataset is treated as read-only. All reports, generated captions, embeddings, and run summaries are written under:
/cpfs01/users/yetinghong/code/rewrite_motion/dataset
What The Pipeline Does
- Counts the long-tail distribution of
motion_type. - Extracts original action references from each
motion_caption/*.jsonfile using onlywhole_bodyfields. - Uses an OpenAI-compatible LLM to generate short English text-to-motion captions for low-frequency subclasses.
- Embeds original and generated captions with the same downstream text backbone by default:
Qwen2.5-Omni-3B thinker. - Automatically calibrates a cosine-similarity threshold from same-class positives and same-major hard negatives.
- Keeps only generated captions that are close enough to their original
motion_typecluster and not closer to sibling classes. - Optionally visualizes
video_captionor extractedwhole_bodyembeddings with PCA + t-SNE.
Default target is 100 captions per low-frequency motion_type.
Environment
Recommended environment, matching the training scripts:
cd /cpfs01/users/yetinghong/code/rewrite_motion
source /cpfs01/users/yetinghong/code/Psi0-yth-v15/.venv-psi/bin/activate
pip install -r requirements.txt
For LLM generation:
export OPENAI_API_KEY=...
export OPENAI_MODEL=...
# Optional for OpenAI-compatible endpoints:
export OPENAI_BASE_URL=...
The default embedding model is:
/cpfs01/videogen/public_ckpt/Qwen/Qwen2.5-Omni-3B
One-Command Runs
Small end-to-end test:
python script/motion_caption_augment.py pipeline --mode test
Full run:
python script/motion_caption_augment.py pipeline --mode full
If you only want a structural smoke test without loading Omni, use the dev-only hash encoder:
python script/motion_caption_augment.py pipeline --mode test --encoder hash
--encoder hash is only for debugging the pipeline mechanics. Use the default Omni encoder for real filtering.
Debuggable Steps
Distribution stats:
python script/motion_caption_augment.py stats
Build reference captions:
python script/motion_caption_augment.py build-references
Calibrate threshold:
python script/motion_caption_augment.py calibrate-threshold
Generate raw captions:
python script/motion_caption_augment.py generate --mode test
Filter a run:
python script/motion_caption_augment.py filter --run-dir /path/to/run_dir
Validate a run:
python script/motion_caption_augment.py validate --run-dir /path/to/run_dir
Visualize Caption Embeddings With t-SNE
The visualization command reuses the same Omni thinker embedding implementation
as the semantic filter. By default it reads video_caption from the 0630 demo
metadata, keeps at least three examples from every motion_type, samples 5,000
records in total, reduces the embeddings to 50 dimensions with PCA, and then
runs Barnes-Hut t-SNE to 2D:
CUDA_VISIBLE_DEVICES=0 python script/motion_caption_augment.py visualize-embeddings \
--text-source video_caption \
--encoder omni \
--max-samples 5000 \
--color-by major
The output directory is printed at the end of the run and is created below
dataset/visualizations/. It contains:
tsne_interactive.html # interactive plot with caption text on hover
tsne_static.png # publication/debug-friendly static plot
tsne_points.csv # IDs, labels, captions, and 2D coordinates
tsne_points.jsonl # the same data in JSONL form
coordinates.npy # raw [N, 2] coordinates
visualization_report.json
Useful variants:
# Color all 131 subclasses separately.
python script/motion_caption_augment.py visualize-embeddings \
--encoder omni --color-by motion_type
# Compare against the motion-only whole_body references.
python script/motion_caption_augment.py visualize-embeddings \
--text-source whole_body --encoder omni
# Use every caption. This is substantially slower and more memory-intensive.
python script/motion_caption_augment.py visualize-embeddings \
--text-source video_caption --encoder omni --max-samples 0 \
--compact-output --discard-embedding-cache \
--output-dir dataset/video_caption_tsne_full
# Structural smoke test only; hash embeddings are not semantically meaningful.
python script/motion_caption_augment.py visualize-embeddings \
--encoder hash --max-samples 300 --tsne-max-iter 500
The step can also be coupled to the existing end-to-end pipeline:
python script/motion_caption_augment.py pipeline --mode test \
--visualize-embeddings --text-source video_caption --max-samples 5000
Embedding and t-SNE coordinates are cached. Repeating the same command reuses
them; use --overwrite-embeddings or --overwrite-tsne to force recomputation.
For a clean final delivery, --compact-output keeps only HTML, PNG, CSV, and
the report, while --discard-embedding-cache removes the large intermediate
embedding cache after all final artifacts are written successfully.
The report includes 10-nearest-neighbor label agreement for major and
motion_type. Remember that t-SNE is intended for local-neighborhood inspection:
global distances and apparent cluster sizes should not be interpreted literally.
Rewrite Rare Video Captions With Local vLLM
Start the project-local Qwen3.5-27B OpenAI-compatible service:
bash vllm/3_qwen3_5_27b_vllm.sh
The service listens on http://127.0.0.1:8301/v1. The launcher was copied from
compose_motion_example/3_qwen3_5_27b_vllm.sh and adjusted to manage only its
own PID instead of killing unrelated API servers.
Generate and select ten diverse rewrites for low-frequency motion types:
CUDA_VISIBLE_DEVICES=2 \
/cpfs01/users/yetinghong/code/Psi0-yth-v15/.venv-psi/bin/python \
script/video_caption_rewrite.py \
--dataset-root /cpfs01/videogen/buwendong/motion_example_0714_demo \
--output-dir dataset/caption_rewrite_0714_motion_only_10 \
--exclude-majors dance \
--num-target-types 24 \
--rewrites-per-source 4 \
--api-concurrency 8 \
--num-results 10 \
--device cuda \
--distance-quantile 0.75
The workflow first uses Qwen3.5 to convert every source video_caption into a
short BoneSeed-style action description, removing clothing, scene, camera, and
filming effects. It then creates genuinely different motions while treating the
exact fine-grained motion_type as the semantic boundary, rather than merely
paraphrasing the old motion. All original and rewritten captions are jointly
embedded with the Qwen2.5-Omni thinker and mapped through PCA + t-SNE. The
novelty threshold is the 75th percentile of original-caption nearest-neighbor
distances. No cosine-similarity acceptance constraint is applied; 2D novelty,
non-copying, length, and motion-only context guards are used. Temporary
candidates and embeddings are removed after a successful run.
Main Outputs
Each run directory contains:
raw_generated.jsonl
accepted_captions.jsonl
rejected_captions.jsonl
threshold_report.json
summary.md
validation_report.json
Accepted records follow this shape:
{
"augment_id": "aug_sports_swimming_000001",
"motion_type": "sports_swimming",
"major": "sports",
"minor": "swimming",
"caption": "A swimmer performs a smooth backstroke with alternating arm pulls and steady flutter kicks.",
"source_ids": ["00030660", "00030348"],
"embedding_filter": {
"encoder": "Qwen2.5-Omni-3B-thinker",
"max_same_type_similarity": 0.82,
"threshold": 0.76,
"nearest_source_ids": ["00030660"]
}
}
Safety Notes
- The original dataset is never modified.
video_captionis not used as the similarity anchor because it contains background and appearance details.- Generated captions are capped so accepted samples do not exceed the missing count needed to reach the target.
- The default filter uses Omni thinker embeddings; the hash encoder is not semantically meaningful and should not be used for final data.