LIME: Learning Intent-aware Camera Motion from Egocentric Video

Paper · Project page · Code

CoRL 2026. Given one RGB image and a free-form intent ("Find my bike"), LIME predicts where the camera should move next: a relative target camera pose, and a description of what the new view should reveal.

These are the lime_v1 weights: a fine-tuned Qwen3-VL-4B-Instruct backbone and a flow-matching pose head of about 20M parameters. Running them needs the code in the GitHub repository.

Usage

git clone https://github.com/cvg/LIME && cd LIME
bash scripts/setup_env.sh        # creates the conda environment and downloads these weights
conda activate lime
python scripts/predict.py --image examples/living_room.jpg --intent "Find my bike." --seed 0

from lime_camera import LIME

model = LIME.from_pretrained("weights/lime_v1")
prediction = model.predict("room.jpg", "Find my bike.", n_samples=5, seed=0)
prediction.observation_gain     # what the new view should reveal
prediction.T_mean               # 4x4 target pose, the mean of the 5 samples

Poses are the end camera in the start camera's frame, in metres, with OpenCV axes (x right, y down, z forward).

Files

File	What it is
config.json, model-*.safetensors, tokenizer and processor files	the fine-tuned Qwen3-VL-4B backbone
flow_head.safetensors, flow_head_config.json	the flow-matching pose head
pose_norm_stats.json	per-axis mean and std the head's poses are standardized by

License

The weights are released under CC BY-NC 4.0, for non-commercial use. They are a modified version of Qwen3-VL-4B-Instruct, which is licensed under Apache-2.0; its license and our notice are in LICENSE-QWEN.

Citation

@inproceedings{sun2026lime,
  title={LIME: Learning Intent-aware Camera Motion from Egocentric Video},
  author={Sun, Boyang and Li, Jiajie and Yang, Yung-Hsu and Zhang, Chenyangguang
          and Engelbracht, Tim and Hong, Sunghwan and Cadena, Cesar
          and Pollefeys, Marc and Blum, Hermann},
  booktitle={Conference on Robot Learning (CoRL)},
  year={2026}
}