VitronEdit: Universal Visual Editing across Images, Videos, and 3D Scenes

This repository contains the model presented in VitronEdit: Universal Visual Editing across Images, Videos, and 3D Scenes.

VitronEdit is a universal framework for image, video, and 3D scene editing. It adopts a shared sequence representation and progressive multimodal training to transfer editing knowledge across modalities, and introduces geometry-aware post-training using a pretrained geometry reconstruction model.

• Project page: https://vitron-llm.github.io/VitronEdit
• Code: https://github.com/Vitron-LLM/VitronEdit-Code

The model supports five editing tasks:

Task	Input	Output	Modality token
i2i	text + image	edited image	<T_IMG>
ti_i2i	text + image + reference image	edited image	<T_IMG>
v2v	text + video	edited video	<T_VID>
tiv2v	text + video + reference image	edited video	<T_VID>
3d23d	text + multi-view video	edited multi-view video (optionally 3D Gaussians)	<T_3D>

Sample usage

The following inference command is taken from the GitHub README:

MODEL_PATH=ckpts/omniweaving CHECKPOINT_PATH=ckpts/univedit/geo3d \
IMAGE_METADATA=examples/inference/image_metadata.jsonl \
VIDEO_METADATA=examples/inference/video_metadata.jsonl \
THREED_METADATA=examples/inference/threed_metadata.jsonl \
OUTPUT_DIR=outputs/univedit_batch \
bash scripts/infer/batch.sh

The results are written to OUTPUT_DIR/<task>/.

License

The code and the released weights are derived from Tencent HY-OmniWeaving and are distributed under the Tencent HY Community License Agreement. See the License.txt and Notice.txt files in the code repository for details.