Naming something that is not in the frame makes the model bring it into the frame
Title: I2V: naming a person or object that is not in the frame makes the model bring it in (with numbers and a workaround)
Setup: LTX-2.5 distilled bf16, ComfyUI two-stage I2V, 720p, 121 frames @ 24 fps, one seed per clip; a few hundred clips across three episodes of a film series.
An off-screen listener walks into the frame. In 10 prompts that mentioned someone "out of frame" / "outside the frame" (e.g. "speaks to a man who kneels out of frame"), that person entered the shot in 6 clips: a head at the edge, a shoulder, or a whole person, often in modern clothing that has nothing to do with the start frame. "Someone just below and in front of the camera" did the same. The line "He is alone in the frame and nobody else enters it." was in the same prompts and did not prevent it. A different seed does not fix it: on seed 11 only 1 of 7 such shots came out clean.
A light source named in the setting pulls the camera away from the face. Close-ups of a woman indoors had the setting line "soft daylight from a smoke-hole in the roof". In 4 of 7 close-ups the camera left her face at second 2–3 and showed an empty room with a hole in the roof. A part of the same scene whose setting line named a lamp that was in the frame had no such drift.
Workaround that held for us: never name what is not in the frame. Describe gaze by direction ("looks to the left of the frame"), not by addressee; in close-ups give light only ("soft dim daylight indoors, warm brown shadows"), with no objects or sources. After this change, none of the next 188 clips drifted away from the subject.
- Smaller sample, for what it's worth: in a two-person shot, a gesture written for one character (holding up a torch and looking at the sky) was performed by the other one in 2 of 2 clips (same start frame and seed), while the line of dialogue was spoken by the right character.
Our reading is that the text encoder treats every noun as something to show. If that is expected behaviour, a line about it in the prompting guide would help; if not, we hope these numbers are useful. Happy to share frame strips if the team wants them.
Thanks for documenting this with examples and numbers. Describing gaze direction and visible lighting instead of naming off-screen people or objects sounds like a useful approach.
I’m working on a film-restoration project where preserving composition, actor identity, expressions, and movement matters. A few questions that could help explore this further:
Controlled comparisons: Have you rerun the original failing shots using the revised wording while keeping the starting image, seed, and settings identical? A before-and-after prompt pair with frame strips would be especially helpful.
Prompt enhancement: Was any automatic prompt enhancement enabled? If so, could you share the final text passed to the model? It would be interesting to see whether the enhancement introduced or emphasized off-screen details.
Character actions: For the gesture assigned to the wrong person, have you tried identifying each character by screen position and visible clothing—for example, “the person on the left wearing a brown robe raises the torch”? Does that improve action assignment?
Performance fidelity: After removing off-screen references, did you also see better preservation of gaze, expressions, and intended movement, or was the improvement mainly in keeping the camera on the subject?
Your results seem worth including in the prompting guide. I’d also be interested in the team’s interpretation of whether this comes from text encoding, the model’s learned response to those descriptions, or another part of the pipeline.