AmirHossein-razlighi/SoundSparksMotion

Sound Sparks Motion

Audio and Text Tuning for Video Editing

This repository is the official checkpoint and result release for Sound Sparks Motion: Audio and Text Tuning for Video Editing.

Sound Sparks Motion is a training-free framework for motion-centric video editing. Given a source video and an edit instruction, it keeps the LTX-2.3 generator frozen and optimizes two lightweight conditioning variables at test time:

1. an audio-conditioning latent initialized from the source video; and
2. a residual perturbation added to the text-conditioning representation.

A frozen Qwen2.5-VL critic supplies semantic feedback about whether the requested action or state change appears in the generated video. Latent, perceptual, and temporal regularization help preserve the source content and stabilize motion.

This is not a standalone Transformers or Diffusers model and does not include the LTX-2.3 base weights. The .pt files are optimized per-edit conditioning tensors used by the official Sound Sparks Motion code. Use them with the matching codebase and LTX-2.3 Retake pipeline.

Links

Resource	Link
Paper	arXiv:2605.15307
Hugging Face paper page	huggingface.co/papers/2605.15307
Project page	amirhossein-razlighi.github.io/Sound_Sparks_Motion
Official code	github.com/amirhossein-razlighi/Sound_Sparks_Motion
Benchmark dataset	AmirHossein-razlighi/SoundSparksMotionBenchmark
Frozen base model	Lightricks/LTX-2.3

What is in this repository?

The release contains optimized conditioning checkpoints, paper-example outputs, prompts, and sanitized run records. It is intended for:

• inspecting baseline and optimized videos used to support the paper's claims;
• reproducing the released examples with the official implementation;
• studying the learned audio and text conditioning variables; and
• transferring a learned motion direction to a new video with a related prompt.

Most standard experiment directories have this structure:

<motion_edit>/<variant>/
├── run_config.json
├── prompt.txt
├── static_prompt.txt
├── negative_prompt.txt
├── qwen_motion_question.txt
├── baseline_video.mp4
└── mode_both/
    ├── best_audio_latent_both.pt
    ├── best_text_delta_both.pt
    ├── best_params_both.pt
    └── best_optimized_video_both.mp4

Artifact meanings:

File	Description
best_audio_latent_both.pt	Optimized audio-conditioning latent, denoted by $\alpha^*$ in the paper.
best_text_delta_both.pt	Optimized residual text-conditioning vector, denoted by $\Delta v^*$.
best_params_both.pt	Best-run state and optimization metadata saved by the official optimizer.
best_optimized_video_both.mp4	Final render from the selected best latent checkpoint.
baseline_video.mp4	Prompt-only LTX Retake result for comparison.
run_config.json	Sanitized record of the run arguments. User-specific model paths are replaced with [FILL THIS].
prompt.txt / static_prompt.txt	Edit instruction and source-content description.
qwen_motion_question.txt	Binary motion question used by the frozen VLM critic.

The repository also contains:

• real_vids/: edits of real source videos, including audio-bootstrap artifacts for clips without source audio;
• transfer/: transferred motion directions, target-video baselines, transferred outputs, configs, and scores; and
• multiple variants that study frame sampling, gradient accumulation, LPIPS/temporal regularization, and related settings.

There are 30 standard optimized experiment directories, 7 released transfer runs, and 4 real-video examples in this snapshot.

Quick start

1. Download the checkpoints and benchmark inputs

Install the current Hugging Face CLI, then download this repository:

hf download AmirHossein-razlighi/SoundSparksMotion \
  --local-dir ./SoundSparksMotion-checkpoints

Download the processed benchmark videos separately:

hf download AmirHossein-razlighi/SoundSparksMotionBenchmark \
  --repo-type dataset \
  --include 'processed_videos/*.mp4' \
  --local-dir ./SoundSparksMotionBenchmark

The benchmark currently exposes 26 validation cases. The paper reports experiments on 25 motion-centric tasks (20 generated and 5 real videos); the paper's full baseline comparison uses a representative 12-scenario subset.

2. Install the official implementation

git clone https://github.com/amirhossein-razlighi/Sound_Sparks_Motion.git
cd Sound_Sparks_Motion
uv sync --frozen
source .venv/bin/activate
pip install pyyaml

Follow the code repository's setup scripts to obtain the required dependencies:

• LTX-2.3 22B development checkpoint;
• Qwen2.5-VL-7B-Instruct; and
• Gemma-3-12B text encoder.

Set their local locations:

export CKPT_ROOT=/path/to/dir/containing/ltx-2.3-22b-dev.safetensors
export QWEN_ROOT=/path/to/Qwen2.5-VL-7B-Instruct
export GEMMA_ROOT=/path/to/gemma-3-12b-it-qat-q4_0-unquantized

The LTX-2.3 weights are distributed separately and remain subject to the LTX-2 Community License Agreement.

Validate the released examples

Each standard experiment places the prompt-only result and optimized result side by side in the directory tree. For example:

SoundSparksMotion-checkpoints/a_dog_yawning/normal/baseline_video.mp4
SoundSparksMotion-checkpoints/a_dog_yawning/normal/mode_both/best_optimized_video_both.mp4

Inspect the associated prompt and sanitized run record:

cat SoundSparksMotion-checkpoints/a_dog_yawning/normal/prompt.txt
jq . SoundSparksMotion-checkpoints/a_dog_yawning/normal/run_config.json

The released outputs make it possible to audit qualitative claims and experiment settings without rerunning the approximately 22B-parameter base model. Recomputing paper-level aggregate metrics additionally requires the evaluation protocol described in the paper and supplementary material.

Transfer a learned motion direction to your own video

The official transfer script reads best_audio_latent_<mode>.pt and best_text_delta_<mode>.pt from an optimized experiment directory. A mode_both directory in this repository can therefore be used directly as opt_dir.

Start from the official transfer template:

cp editing/configs/transfer/yours.yaml editing/configs/transfer/my_transfer.yaml

Edit the important fields:

target_video: "/path/to/my_video.mp4"
opt_dir: "/path/to/SoundSparksMotion-checkpoints/a_dog_yawning/normal/mode_both"
mode: "both"

edit_prompt: "A cat yawning."
static_prompt: "A cat sitting on a chair."
experiment_name: "dog_to_cat_yawning"

Then run:

bash editing/scripts/transfer.sh editing/configs/transfer/my_transfer.yaml

mode must match the suffix of the saved files. For the checkpoints in this release, use both with mode_both/.

Transfer works best when the new prompt describes a motion related to the source optimization—for example, transferring yawning between animals or wing opening between birds. It is not guaranteed to work for arbitrary subjects, clip lengths, or unrelated actions. The implementation interpolates the audio latent along time when necessary, but visual success still depends on the target video and prompt.

Released transfer examples are available under transfer/, including:

• dog yawning $\rightarrow$ cat yawning;
• learned shouting/surprise directions applied to a different person;
• bird wing opening $\rightarrow$ red bird wing opening; and
• a learned laughing direction applied to a cartoon-style target.

Reproduce an optimization run

The official codebase contains annotated YAML configurations and a shell entry point:

bash editing/scripts/run.sh editing/configs/dog_yawning.yaml

For a custom source, copy editing/configs/yours.yaml and set at least:

src_video: "/path/to/source.mp4"
edit_prompt: "A dog yawning."
static_prompt: "A dog sitting on a chair."
opt_mode: "both"
experiment_name: "dog_yawning"

The run_config.json files in this repository are provenance records, not drop-in launch configurations. Values marked [FILL THIS] must be replaced with local model paths, or supplied through the environment variables expected by the official code.

Paper configuration

The paper reports the following default setup; individual released variants may differ, so consult each experiment's run_config.json.

Component	Paper setting
Generator/editor	Frozen LTX-2.3 22B Retake
VLM critic	Frozen Qwen2.5-VL-7B-Instruct
Text encoder	Gemma-3-12B
Optimized variables	Audio latent $\alpha$ and residual text conditioning $\Delta v$
Resolution	$512 \times 320$
Processed video length	89 frames at 25 FPS
Retake denoising steps	30
Gradient propagation	Final 8 denoising steps
Optimizer	Adam
Learning rate	$5 \times 10^{-3}$ with cosine decay
Tuning budget	Up to 30 iterations; early-stopping patience 15
Latent regularization	$\lambda_\alpha=0.01$, $\lambda_v=0.001$
Quantization	FP8-cast
Reported hardware	One NVIDIA H100 80GB
Representative runtime	Approximately 589.6 seconds for a 15-iteration run

Some configs request 95 frames; LTX preprocessing adjusts the usable frame count to satisfy its temporal constraint, $(T-1) \bmod 8 = 0$, yielding 89 frames in the reported experiments.

Results reported in the paper

The following results are reported on the representative 12-scenario comparison subset. They are paper-reported results, not independent Hugging Face evaluations.

Human evaluation (21 participants)

Method	First-place win rate $\uparrow$	Top-3 rate $\uparrow$	Average score $\uparrow$	Task achieved $\uparrow$
LTX-2.3 Retake	10.3%	63.1%	1.08	39.2%
Sound Sparks Motion	56.3%	95.2%	2.37	79.1%

GPT-5.5-Thinking evaluation

Method	Task success $\uparrow$	VLM score $\uparrow$	Edit alignment $\uparrow$	Motion quality $\uparrow$	Rank $\downarrow$
LTX-2.3 Retake	50.0%	5.77	4.67	4.29	3.00
Sound Sparks Motion	83.3%	7.16	7.42	6.48	1.58

See the paper for all baselines, ablations, evaluation prompts, and per-scenario analysis.

Security and file integrity

The .pt files are PyTorch-serialized artifacts. Load only files obtained from this trusted repository and review the official loading code before use. The transfer implementation loads these files with torch.load.

No executable model code is stored in this repository. Use the linked official implementation, and verify that you are on the intended revision before loading checkpoints.

Limitations

• Test-time tuning adds substantial computation and may limit interactive use.
• The requested motion is not always fully realized, especially for complex or ambiguous actions.
• The binary VLM critic provides coarse supervision and may be imperfect or exploitable.
• Transferability is empirical and works best for related actions; it is not universal.
• The released tensors are tied to the Sound Sparks Motion implementation and the LTX-2.3 conditioning structure.
• Generated videos may inherit failures, biases, unsafe outputs, or prompt-following limitations from LTX-2.3 and the auxiliary models.

Responsible use

Use this release only for lawful and responsible research or creative applications. Obtain permission before editing identifiable people, clearly disclose synthetic or edited media where appropriate, and do not use the system for harmful impersonation or deception. Users must follow the LTX-2 Community License Agreement and its acceptable-use restrictions.

License

These optimized conditioning checkpoints are derived from and designed for LTX-2.3. They are released under the LTX-2 Community License Agreement, included in LICENSE. The LTX-2.3 base model is not redistributed here.

The associated benchmark dataset is hosted separately and has its own dataset-card terms. Review the licenses and provenance of source videos before redistribution or downstream use.

Citation

If you use these checkpoints, results, or the benchmark, please cite:

@article{razlighi2026sound,
  title={Sound Sparks Motion: Audio and Text Tuning for Video Editing},
  author={Razlighi, AmirHossein Naghi and Mikaeili, Aryan and Mahdavi-Amiri, Ali and Cohen-Or, Daniel and Chrysanthou, Yiorgos},
  journal={arXiv preprint arXiv:2605.15307},
  year={2026}
}

Acknowledgements

Sound Sparks Motion builds on LTX-2.3 by Lightricks and uses Qwen2.5-VL as the motion critic. Please cite and follow the licenses of all upstream projects used in your workflow.