Title: Dynamic Shadow Unveils Invisible Semantics for Video Outpainting
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Shadow Processing

Video Generation
Abstract: Conventional video outpainting methods primarily focus on maintaining coherent textures and visual consistency across frames.
However, they often fail at handling dynamic scenes due to the complex motion of objects or camera movement, leading to temporal incoherence and visible flickering artifacts across frames. This is primarily because they lack instance-aware modeling to accurately separate and track individual object motions throughout the video. In this paper, we propose a novel video outpainting framework that explicitly takes shadow-object pairs into consideration to enhance the temporal and spatial consistency of instances, even when they are temporarily invisible. Specifically, we first track the shadow-object pairs across frames and predict the instances in the scene to unveil the spatial regions of invisible instances. Then, these prediction results are fed to guide the instance-aware optical flow completion to unveil the temporal motion of invisible instances. Next, these spatiotemporal guidances of instances are used to guide the video outpainting process. Finally, a video-aware discriminator is implemented to enhance alignment among dynamic shadows and the extended semantics in the scene. Comprehensive experiments underscore the superiority of our approach, outperforming existing state-of-the-art methods in widely recognized benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the first shadow-aware video outpainting framework that uses the motion of dynamic shadows to infer and synthesize content outside the camera’s original field of view. \nShadow-aware Instance Prediction tracks object–shadow pairs across frames and predicts the spatial extent of objects that are currently outside the frame. Instance-aware Optical Flow Completion fills missing flow fields in the padded regions using cross-attention to the predicted masks, preserving object-specific motion. \nA latent-diffusion outpainting network is augmented with two adapters, plus a video-aware shadow-alignment discriminator that jointly enforces spatial detail and temporal coherence. \nOn the DAVIS and YouTube-VOS benchmarks the method improves PSNR, SSIM and FVD over prior GAN- and diffusion-based outpainting systems, and a user study shows clear preference for visual quality and temporal consistency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. It is interesting to treat shadows as informative, temporally consistent cues, enabling the model to “unveil” otherwise invisible objects and motions.\n2. extensive experiments, including quantitative metrics, qualitative comparisons, ablations, and a 40-participant user study.\n\nWeaknesses:\n1. The author should further clarify the difference between the proposed method and MOTIA[29]. MOTIA also used input-adapters in the outpainting network. It seems that the only difference is that the author introduces shadow as input prior to improving the performance of outpainting. However, observed from Tab.1, on DAVIS, the proposed method only outperforms MOTIA in PSNR and FVD (2 out of 4), and in many metrics, the two methods produce close results. The author should explain the inherent reasons.\n2. The paper only presents the cases of the static camera. If the camera is moving, the relative position of the light source would change unexpectedly, leading to dramatic changes in shadow. Can the method still perform well in such cases?\n3. From Tab.2, the model performs inferior to MOTIA when w/o Video-aware Shadow Alignment Discriminator. This makes the effectiveness of the shadow prior doubtful.\n4. The novelty of Video-aware Shadow Alignment Discriminator is limited. As described in the manuscript, the module source from [23][10][35], making the module incremental.\n5. The author should elaborate details of the user study.\n\nQUESTIONS:\nHow does the system resolve cases where multiple objects cast overlapping shadows?\n\nWhat is the performance of training only the adapters (freezing the diffusion backbone) vs. fine-tuning the entire U-Net?\n\nHow sensitive is optical-flow completion when the camera pans quickly and shadows smear?\n\nLIMITATIONS:\nWhen the method comes across some special scenarios, such as camera motion, low-light, which may severely affect the visibility of the shadow, would the method still perform well? This is not validated in this manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This work improve the visual quality and temporal consistency by dynamic shadow information and robust object generation during occlusion. I think the motivation is reasonable but lacks further relevant verification.

STRENGTHS AND WEAKNESSES:
Strengths：
1. The ablation study shows that the proposed method outperforms the baselines in terms of both visual quality (e.g., clarity, color fidelity, texture detail) and temporal consistency (e.g., motion smoothness, object continuity, temporal coherence).
2. The paper demonstrates that incorporating dynamic shadow information improves the accuracy of both object and shadow mask predictions across multiple datasets.
3. The visual analysis indicates that the proposed method can generate temporally consistent objects with high fidelity in appearance and motion, even as the duration of object invisibility increases, outperforming the baselines.

Weaknesses:
Please see Questions.

QUESTIONS:
1. My biggest concern about this work is that it does not clearly explain the relationship between shadows and video outpainting.
2. I suspect that the most effectiveness for this work is still in the GT in the dataset, rather than the detected shadow.
3. For the visualizations in Figures 3 and 4, it is suggested that the input image should be added to make it clearer.
4. In fact, this work does not involve semantically-related content. Instances and shadow are visual features, not semantic features. Therefore, I think the hidden semantics of the above work is unconvincing.

LIMITATIONS:
Although it is novel to mine temporal continuity from shadows, if the shadow part in the video is difficult to find, then this method meaning may reduce. In my opinion, generality may be the most important.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel video outpainting method that enhances the temporal and spatial consistency of generated content by incorporating the prior relationship between instances and their dynamic shadows into an adapter. Conventional video outpainting methods often suffer from issues such as temporal inconsistency, motion mismatch, and a lack of instance-level modeling when processing dynamic scenes. The authors innovatively propose an auxiliary feature generation pipeline that effectively reveals the spatial regions and temporal motion of unseen instances in the input video by explicitly tracking and modeling shadow-object pairs. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods across multiple evaluation metrics on the DAVIS and YouTube-VOS datasets, exhibiting particularly strong performance in maintaining the temporal consistency and spatial coherence of objects.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The method explicitly incorporates prior knowledge about instances and shadows, which offers good interpretability.\n2. The method is well-designed, with four modules that have distinct functions and collaborate effectively to form a complete solution. The authors validate the effectiveness of each component through comprehensive ablation and user studies.\n3. Video outpainting is an important problem in the field of computer vision with broad application potential.\n4. The experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art approaches across multiple metrics.\n5. The paper is the first to explicitly propose the use of dynamic shadow information for video outpainting, presenting a novel research perspective.\n\n**Weaknesses**\n1. The method's applicability is constrained by the availability of shadow information. Its performance may be limited in scenarios where distinct shadows are not present.\n2. The notation for superscripts and subscripts is somewhat confusing due to inconsistent use and omissions without clear explanation.\n\nQUESTIONS:\n1. The proposed method introduces several modules to generate auxiliary features, which are then injected into the diffusion model via an adapter. How does the computational overhead (both for training and inference) of this approach compare to the methods it is benchmarked against?\n2. When multiple instances are detected, how is the global feature $h^t$ derived from the individual instance features $h^{t,n}$? For instance, are they concatenated, summed, or aggregated in some other manner?\n3. Could the authors elaborate on the design of $z^0$? Is a unique latent code $z^{0,n}$ assigned to each object class? If not, how is $h^{0,n}$ derived from a single $z^0$, and how is the Inter-Pair Fusion subsequently computed across different instances?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new framework for video outpainting that makes innovative use of dynamic shadows to infer and reconstruct semantically consistent content in scenes where objects temporarily disappear from view. The authors design a multi-stage architecture that includes modules for predicting object-shadow masks, completing instance-aware optical flow, and guiding a latent diffusion model with spatio-temporal priors. The system also introduces a video-aware discriminator aimed at aligning generated content with the motion and structure implied by dynamic shadows. Experiments on widely used benchmarks show competitive improvements over several recent baselines, both in terms of visual quality and temporal coherence.\n\nSTRENGTHS AND WEAKNESSES:\nOne of the most compelling aspects of this work lies in its central idea: treating dynamic shadows not as noise but as informative signals that reflect underlying instance motion. This perspective departs meaningfully from prior approaches, which tend to overlook or discard such visual phenomena. The technical formulation is coherent and thoughtfully designed, especially the integration of shadow-based instance priors into a diffusion-based generation pipeline. The empirical results are strong, and the ablation studies offer reasonable support for the importance of each module in the system.\n\nOn the other hand, the method’s reliance on accurate shadow detection raises questions about its robustness in varied lighting conditions or scenes where shadows are faint or occluded. The proposed approach also appears to require relatively heavy computational resources, which may limit its scalability to longer or higher-resolution videos. In terms of evaluation, while the reported metrics show clear improvements, the absence of statistical uncertainty estimates (e.g., variance or confidence intervals) makes it difficult to assess how consistent these gains are across different runs or datasets. Lastly, while the architectural design is detailed, it is not always easy to distinguish which parts are trained jointly and which are fixed, which could hinder reproducibility for less experienced practitioners.\n\nQUESTIONS:\n(1) How well does the method generalize to real-world videos where shadows are less prominent or heavily distorted (e.g., indoor lighting, artificial light sources)? It would be helpful to see some quantitative or qualitative evaluation under such conditions.\n(2) Have you considered how sensitive the overall pipeline is to the quality of the initial object-shadow detection? If this module fails or produces imprecise masks, what are the typical failure modes of the system?\n(3) In your architecture, different components (e.g., instance adapter, flow adapter) are introduced in a modular way. Are these modules jointly trained end-to-end, or is there a staged training process?\n\nLIMITATIONS:\nWhile the method demonstrates impressive performance on benchmark datasets, there are notable limitations that deserve further attention. First, the reliance on high-quality dynamic shadows makes the system less generalizable to scenes where such shadows are weak, inconsistent, or altogether missing. This dependency may reduce the method’s effectiveness in many indoor environments or scenarios involving multiple light sources. Second, the approach assumes that shadow-object associations can be reliably extracted, which might not hold in crowded scenes, strong occlusion, or when shadows are cast on complex textured backgrounds. These assumptions could limit the model's robustness in unconstrained or in-the-wild video content.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-04"}