Title: Frame In-N-Out: Unbounded Controllable Image-to-Video Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Image Generation

Diffusion Model
Abstract: Controllability, temporal coherence, and detail synthesis remain the most critical challenges in video generation. In this paper, we focus on a commonly used yet underexplored cinematic technique known as Frame In and Frame Out. Specifically, starting from image-to-video generation, users can control the objects in the image to naturally leave the scene or provide breaking new identity references to enter the scene, guided by a user-specified motion trajectory. To support this task, we introduce a new dataset that is curated semi-automatically, an efficient identity-preserving motion-controllable video Diffusion Transformer architecture, and a comprehensive evaluation protocol targeting this task. Our evaluation shows that our proposed approach significantly outperforms existing baselines.


Peer reviews:
comments: SUMMARY:
This paper introduces Frame In-N-Out, a novel task in video generation. It adapts a video diffusion transformer to incorporate conditional inputs and curate a dedicated dataset along with evaluation benchmarks. Experimental results demonstrate that the proposed method outperforms existing baselines.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well written and easy to follow.
2. The proposed dataset and evaluation benchmark can be valuable for the community.

Drawbacks:
1.  The novelty is somewhat limited in the architecture, as the method primarily adapts an existing Diffusion Transformer to accommodate conditional inputs. Given this, the core contribution appears to be the proposed dataset and evaluation benchmark—making the work potentially better suited for a dataset/benchmark track rather than the main conference.
2. There is a resolution mismatch between training (384 × 480) and inference (480 × 720), which may negatively impact model performance. To address this, I recommend adding a final training stage using either high-resolution or mixed-resolution data.
3. The curated dataset lacks videos with significant camera motion, limiting the model's ability to handle Frame In/Out and camera control simultaneously. Can the current pipeline be extended to support camera motion?

QUESTIONS:
Why is random noise added to the ID reference during training? Given that inference only uses either zero noise or full noise, wouldn’t it be more consistent to align the training scheme with these two discrete noise levels?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel video generation problem called \"Frame In\" and \"Frame out\", which an desired object can enter the scene or leave the scene, with fully controllable trajectory. A dataset and a evaluation protocol is designed for this task. The authors also proposed an initial attempt towards this problem. Leveraging an efficient identity-preserving motion-controllable video Diffusion Transformer architecture, The proposed method outperform baselines in the designed evaluation protocol.\n\nSTRENGTHS AND WEAKNESSES:\nS1) This paper is well-written and easy to follow, with a clear motivation and problem formulation. It serves as a strong example of work that introduces a novel and meaningful problem setting.\n\nS2)  The technical design is sound and informative, with clear architectural choices and efficient integration of motion, identity, and unbounded spatial control. Although no empirical ablation is provided, the proposed framework appears adaptable to a wide range of current I2V architectures, which speaks to its generality.\n\nW1) While the Frame In-N-Out scenario is novel, it’s a narrow subset of controllable video generation, building upon prior tasks like identity -referenced video generation and trajectory / motion-controlled video generation.\n\nW2) The designed automated metrics looks valid to me. But given the subjective nature of video realism and controllability, a small-scale user study would strengthen the evaluation.\n\nQUESTIONS:\nLine 287 - 289 \"Vision Language Model evaluation (VLM) utilizes a modern large vision language model, Qwen 2.5 VL 32B [65], with full video sequences and an instruction prompt to justify if the object gets out of the first frame or enter the first frame. We evaluate the ratio of correctness compared to the returns with GT video inputs. This metric is intended to align the overall subjective success rate analysis.\"\n\nVLMs can be unreliable and sensitive to prompt phrasing, raising concerns about consistency and reproducibility. Why not use more interpretable alternatives like object tracking or mask-based analysis? Would a hybrid evaluation be more robust?\n\nLIMITATIONS:\nThe authors have addressed technical limitations reasonably well, but a more thorough discussion of potential societal risks and ethical considerations would strengthen the paper. Particularly given the realism and identity control aspects of the generated videos. E.g. Deepfake-like misuse (inserting people into scenes realistically)?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFrame In-N-Out introduces a novel paradigm for controllable image-to-video generation that enables objects to move beyond the spatial boundaries of the initial frame. The approach supports two key cinematic techniques:\n- Frame Out: Existing objects in the first frame move completely outside the visible bounds\n- Frame In: New identity objects enter the scene from outside the frame boundaries\nThe method extends the conditioning canvas beyond the first frame region, uses a two-stage training approach, and introduces specialized evaluation metrics for this new task. The authors build upon CogVideoX-I2V architecture with modifications for unbounded motion control and identity preservation.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- Novel Problem Formulation\n This work is the first to explicitly tackle “Frame In–Out” patterns in video generation, addressing a gap in the literature and offering a new paradigm for cinematic editing tasks.\n- Comprehensive System Design\n The authors present an end-to-end pipeline—from semi-automatic data curation through two-stage model training to thorough evaluation—ensuring that each component is well integrated and motivated by practical production needs.\n- Rigorous Data Curation\n Dataset creation combines automated methods with human quality control; a cycle-tracking pipeline verifies annotation accuracy, and intelligent pattern recognition reliably isolates Frame In and Frame Out scenarios.\n- Thoughtful Architectural Choices\n Key design elements (an unbounded canvas, full-field loss, and efficient channel concatenation) are carefully chosen to support flexible spatial generation and maintain consistency across frames.\n- Strong Experimental Validation\n The method is benchmarked against both motion-controllable and elements-to-video baselines using diverse metrics—including novel task-specific measures like VSeg MAE and Trajectory Error—demonstrating clear quantitative gains.\n\n\nWeakness\n- Limited Technical Novelty\n Many architectural ideas (e.g., canvas expansion) are incremental adaptations of existing Diffusion Transformer (DiT) frameworks and resemble prior outpainting or “Follow-Your-Canvas” methods.\n- Evaluation Constraints\n The test set is small (183 Frame In and 189 Frame Out examples), lacks human subjective studies for cinematic quality, and omits statistical significance testing to bolster confidence in reported gains.\n- Scalability and Practical Concerns\n A fixed low training resolution (384×480) limits real-world applicability at higher resolutions; the two-stage training pipeline adds complexity and computational overhead, and memory requirements for varying canvas sizes are unspecified.\n- Methodological Gaps\n The canvas expansion strategy is under-motivated from a theoretical standpoint, identity preservation relies on older techniques rather than recent advances (e.g., ConsisID), and there are few ablation studies on critical design choices like full-field loss or training stages.\n\nQUESTIONS:\nSee Strength & Weakness\n\nLIMITATIONS:\nSee Strength & Weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel video generation task dubbed “Frame In-N-Out,” which breaks through the limitations of first-frame boundaries to enable natural entry and exit of objects, flexible identity switching, and fine-grained control of motion trajectories within scenes. To support this, the authors present a new semi-automated dataset, a comprehensive evaluation protocol, and an efficient video diffusion Transformer architecture that ensures both identity preservation and controllable motion. Experimental results show that the proposed method substantially outperforms existing approaches in generation quality, controllability, and boundary-crossing capability, offering promising prospects for applications such as film production and advertising.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- This paper is the first to propose and formalize the novel video generation task of “Frame In-N-Out,” which effectively overcomes the limitation of traditional video generation being confined to in-frame objects, thereby expanding the application boundaries of video generation models.\n- This paper proposes a novel video diffusion Transformer architecture that conditions on motion trajectories, reference subjects, and a boundary-free canvas, effectively enhancing the controllability of the generation process and the extensibility of the model.\n- A high-quality Frame In-N-Out dataset with semi-automatic annotation support has been released, along with a dedicated evaluation system for the \"in and out of frame\" scenario to provide a reliable basis for model assessment.\n- A high-quality Frame In-N-Out dataset was constructed, and a dedicated evaluation set tailored to this task was introduced, providing a solid foundation for model evaluation.\n- The experiments are comprehensive, and the proposed method demonstrates advantages across multiple quantitative and subjective metrics, validating its effectiveness and superiority.\n- Some physically consistent cases are impressive, such as picking up the cup shown in Appendix Fig. 2E; however, the hand movement appears somewhat stiff.\n\n### Weakness\n- The video scenes presented in the paper are relatively simple in terms of background, and there is a lack of scenarios involving crowded environments, frequent occlusions, significant background changes, or non-target motion interference.\n- In the supplementary materials, some task scenes in the videos exhibit facial distortion (such as at 1 minute 40 seconds), which may reflect a general limitation of current video-generative models.\n- The current generation paradigm is still primarily focused on 2D video, and has not yet addressed more complex scene composition issues in 3D space, such as motion in three-dimensional environments or virtual camera transitions.\n- As mentioned in the limitations, this method is still unable to effectively understand the proportion between the given reference image and the background, and therefore cannot generate natural videos.\n\nQUESTIONS:\n- Is it possible to achieve some complex interactions? For example, can two people dance along specified trajectories and gradually exit the scene?\n- If Frame In-N-Out is extended to real-world videos, especially those with complex lighting and physical constraints, what challenges might arise?\n- How does the proposed model perform in terms of computational requirements and inference speed? How many frames can it support for video inference, and are there any optimization strategies?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}