Dynamic Shadow Unveils Invisible Semantics for Video Outpainting
Authors: Ruilin Li
Hang Yu
Jiayan Qiu
Venue: NeurIPS 2025 poster ()
Keywords: Video Shadow Processing
Video Generation
Primary area: deep_learning

Abstract: Conventional video outpainting methods primarily focus on maintaining coherent textures and visual consistency across frames.
However, they often fail at handling dynamic scenes due to the complex motion of objects or camera movement, leading to temporal incoherence and visible flickering artifacts across frames. This is primarily because they lack instance-aware modeling to accurately separate and track individual object motions throughout the video. In this paper, we propose a novel video outpainting framework that explicitly takes shadow-object pairs into consideration to enhance the temporal and spatial consistency of instances, even when they are temporarily invisible. Specifically, we first track the shadow-object pairs across frames and predict the instances in the scene to unveil the spatial regions of invisible instances. Then, these prediction results are fed to guide the instance-aware optical flow completion to unveil the temporal motion of invisible instances. Next, these spatiotemporal guidances of instances are used to guide the video outpainting process. Finally, a video-aware discriminator is implemented to enhance alignment among dynamic shadows and the extended semantics in the scene. Comprehensive experiments underscore the superiority of our approach, outperforming existing state-of-the-art methods in widely recognized benchmarks.

Dynamic Shadow Unveils Invisible Semantics for Video Outpainting
Ruilin Li
Hang Yu!" Jiayan Qiu²
Schooo of Computer Engineering and Science, Shanghai University, China Schooo of Computing and Mathematical Sciences, University of Leicester, United Kingdom fruilinli, yuhang Gaatnnge@hhu...e edu. cn, jq46dleicesterr ac.uk
Abstract
Conventional video outpainting methods primarily focus maintaining coherent textures and visual consistency across frames. However, they often fail at handling dynamic scenes due to the complex motion of objects or camera movement, leading to temporal incoherence and visible flickering artifacts across frames. This is primarily because they lack instance-awan modeling to accurately separate and track individual object motions throughout the video. In this paper, we propose a novel video outpainting framework that explicitly takes shadow-object pairs into consideration to enhance the temporal and spatial consistency of instances, even when they are temporarily invisible. Specifically, we first track the shadow-object pairs across frames and predict the instances in the scene to unveil the spatial regions of invisible instances. Then, these prediction results are fed guide the instance-aware optical flow completion to unveil the temporal motion of invisible instances. Next, these spatiotemporal guidances of instances are used to guide the video outpainting process. Finally, video-aware discriminator is implemented to enhance alignment among dynamic shadows and the extended semantics in the scene. Comprehensive experiments underscore the superiority of our approach, outperforming existing state-of-the-ee methods in widely recognized benchmarks.
Introduction
Shadows are an integral part of many natural scenes, providing rich but often ignored cues about both invisible semantics. While shadows in static images can help infer approximate categories and coarse shapes of invisible objects [44), dynamic shadows in videos offer even more, they contain spatiotemporal information of invisible instances like unveiling the motion and actions of objects and even dynamic light sources in the scene.
making significant progress in generating contextually matching content with coherent textures and scene structure. However, this consistency often applies to broader backgrounds rather than specific instances: objects may lack detail, exhibit distortions, or fail to preserve identity across frames, or become temporally misaligned with dynamic shadows, leading inconsistencies with the motion and action of extended semantics.
Despite recent video outpainting methods
Specifically, given an input video, conventional video outpainting methods often suffer from temporal inconsistency and action misalignment when instance temporarily outside the visible scene. As shown in Fig. [a), while the shadow moves, conventional methods tend to generate instances that remain static. In contrast, our method aligns the generated instance with the motion of the dynamic shadow, thereby preserving consistency across frames. Additionally, conventional approaches often lead to action misalignment, shown in Fig. [b), where the actions of the generated instance do
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

not correspond to the dynamic shadow in the scene. In contrast, our method captures the variations of the dynamic shadow and generates appropriate actions for the instance. Beyond addressing the aforementioned temporal inconsistency and action misalignment with dynamic shadows, our method also capable of generating a harmonious light source, even when it is not explicitly visible the ground truth, shown in Fig. [c).
Temporal Inconsistency
(b) Action Misalignment
(c) Light Source Generation
M3DDM Ground Truth
M3D0M Ground
Ground
Ours
Figure I: Failure example of previous methods. Many previous methods, including intensively trained models for video outpainting, often generate semanties that are temporally inconsistent with dynamic shadows and misaligned with them within the scene, as they primarily focus on maintaining coherent textures and visual consistency. In contrast, our method extends consistent semantics with dynamic shadow alignment and even generates harmonious light across the frames.
To perform video outpainting by leveraging the hidden information dynamic shadows, we propose novel framework that explicitly models the associations between dynamic shadows and their corresponding instances, enabling instance-level spatiotemporal consistency even when the instances temporarily invisible.
Specifically, we first track the shadow-instance pairs across frames and predict the instances outside the scene to unveil the spatial regions of invisible instances. Then, the prediction results are fed to guide the instance-aware optical flow completion to unveil the temporal motion of invisible instances. Next, the prediction results and the completed optical flow are used to guide the video outpainting process. Finally, a video-aware discriminator is implemented to enhance spatiotemporal alignment among dynamic shadows and the extended semantics in the scene.
Our main contributions are as follows:
We propose shadow-awan module that tracks and predicts the spatial region of instances, and completes optical flow in invisible areas to generate spatiotemporal guidance for outpainting.
We introduce an instance-aware spatiotemporal guidance module that employs separate adapters for spatial predictions and completed flows to guide video outpainting, enhancing temporal consistency and semantic coherence.
We present the first framework that explicitly leverages dynamic shadows for instance-aware video outpainting, by modeling the associations between shadows and their corresponding instances in the visible scene to unveil invisible semantics based on the cues of dynamic shadows.
Related Work
Video Shadow Processing, Conventional video shadow processing methods seek track and seg-
ment dynamic shadows in complex scenes (50) 6 2 [5). Unlike static image shadow processing, which only focuses on individual frames, dynamic shadow processing enables the capture of spa- tiotemporal information in objects and light sources over time Then, a sequence of methods [I]
18 35
developed to enhance the tracking and segmentation of moving objects. ViShadow (38]
not only detects shadows and their corresponding objects in video frames, but also continuously tracks each shadow, object, and their relationships throughout the entire video sequence. Additionally,
several methods in video shadow processing, such as 38 52 not only track shadows and objects but

---

also enable video editing by removing instances, all while preserving temporal coherence. However, none of the existing methods deeply look into the spatiotemporal hidden information of dynamic shadows to generate invisible semantics.
Video Outpainting. Conventional video outpainting methods
[
42
30 seek extend the con-
tent beyond its original borders based on the dynamic scene, maintaining inter-frame and intra-frame consistency videos. Although image outpainting (4,
14
44
24
19 has been extensively studied,
video outpainting still needs to be fully researched. Dehan
5 proposes modified Generative
Adversarial Network (GAN) 44 for video outpainting, performing flow estimation and background estimation before integrating them into complete result. Recently, with the powerful generative some diffusion-basee approaches have been intro- duced. M3DDM [8] presents global frame- -umdeeddded training with coarse-to to-fine inference pipeline to handle the artifact accumulation issue. Meanwhile, MOTIA (36) proposes test sample-specific fine-tuning strategy to learn the patterns of each sample. However, none of these methods can take advantage of the dynamic shadows in the scene to reasonably extend the invisible areas in the video.
capabilities of diffusion models (29 22 23 25].
3 Method
In this section, detail the working scheme of the proposed video outpainting framework, which comprises four sequential stages, as illustrated in Fig. 2
Optical Flow
padded How
conpllated BlAs
Inbral per mosts
Video Shadow
Shadow Instance Prediction
prediced mark
Instence-uuare Optical Completion
Input
Add
Noise
Discriminator
noise latent
Decoder
outpainted video
masked latent
Video-aware
Shadow Alignmant
Latent Diffusion Model
LDM Layer
Mation Layer
padded video
mask
Video Outpainting with Multi-adapters
Flow Adapter
Adapter
Figure 2: Illustration of the proposed framework. We first extract the shadow-object pair mask from the video and perform shadow-aware instance prediction to obtain spatial information of instances beyond the scene. This then fed into instance-aware optical flow completion module generate temporal guidance for video outpainting. With the predicted spatiotemporal guidance, we perform video outpainting with types of adapters. Finally, video-aware discriminator is employed to enhance temporal consistency between the extended semantics and dynamic shadows.
3.1 Shadow-aware Instance Prediction
Tracking the mask of foreground instances in video sequence is easy while using existing video object/instance segmentation methods [38] 27 3). However, predicting the masks of invisible instances that are temporarily outside the scene, either before they appear after they leave, remains challenging due to their complex motions and actions.
To address this problem, we propose novel shadow-aware instance prediction module that utilizes dynamic shadow information in the scene to recover the masks of instances while they are temporarily outside the visible scene.

---

Specifically, given the input video Vn {10, .'), where each frame I' E R "xx x3 and T is the total number of frames, we extend the video borders according to user-defined target aspect ratio to
obtain the padded video V = { /1 .!..... }² where each padded frame 1 E R(#+pA) (x(p))x((+++)))) +p results from adding Ph pixels horizontally and Pie pixels vertically to the original frame. And we define Miis E {0. 1) Hx vv
as the visible region mask of the video, where 1 indicates the visible area and O indicates the non-visible area, which is defined as:
V = V @ Muss ViIvis = V o (1i)
(1)
where @ denotes the Hadamard product, Vinvis denotes the region of V that needs to be outpainted, representing the content beyond the original field of view.
We first adopt a pretrained video shadow instance detection model (88] to obtain the initial visible shadow-object pairs from the input video, which is defined as:
{ (Mo M, ) Video Instance Detection(Val)
1,2, N
(2)
where we assume that each frame contains N shadow-object pairs, and each pair consists of an object mask M and its corresponding shadow mask M vis- vis? indicating the visible region of the object
and its shadow time 4, respectively.
After obtaining the initial masks of visible shadow-object pairs in the scene, we aim to predict the shadow-object masks for instances that are temporarily outside the visible scene while simultancously refining the initial tracking results, which may be inaccurate due to shadow overlap and discontinuity in complex dynamic scenes, as discussed in [44]. For simplicity, we denote the initial masks of single shadow- object pair as (Mr n_vis> M [ at each time step.
To predict the instance masks outside the visible scene, we treat each shadow-object pair as token in spatiotemporal transformer (33), which is widely used in video understanding [12] 45 ST] for capturing long-range spatiotemporal coherence.
Specifically, we first apply shared image encoder to extract high-level semantic features F/ of each padded frame fr, And the visible object mask and the shadow mask are fed it into a mask encoder to obtain their feature embeddings Fg and 17, respectively, which are defined as:
= Image__aaaaa Encoder(P))
F- Mask (ncoodr(MM
Fd Enceedernnr,,,,
(4)
where Enngger_cccde(( (-) and Mask Encoder(-) are convolutional networks (CNNs) designed to extract features from the input image and the respective masks. Note that a standalone image encoder helps reduce redundant computation over V when predicting multiple shadow-objec pairs.
Next, we feed the extracted features of single pair into the transformer for spatiotemporal fusion across frames. Specifically, for each shadow-object pair, we construct sequence of length 7 + 1, where the 0-th input token is a learnable vector zd used to aggregate global information of this shadow-object pair, and the remaining T tokens correspond the local instance features at each time step. For each time f, we concatenate F; Ft. and F3 along channels and feed the result into a feature encoder, obtaining the instance feature 2, of time f used for inter-frame fusion, which is defined as:
Feature Encoder F Ft, F.),, t= 1,,,,...,,
(5)
Ao "",,,...., h² = Irter-Frame Fusion (2 22.....z?)
(6)
where Peature_Fnnoddee EEccder(() is convolutional network designed encode instance features of single frame at time 1, and frame information of instances. The 0-th output feature ha is the aggregated global embedding of the shadow-object pair, serving as shadow-object memory in the subsequent outpainting module, while the 1st to T-th output features h' are inter-frame fused features capturing spatiotemporal correlations.
( is a transformer-hasee module to fuse inter-
When multiple shadow objjctt pairs are detected, their predictions often influence each other, making it necessary to fuse spatial information across pairs. For object mask prediction, each pixel can belong to only one object. In contrast, for shadow mask prediction, a pixel may belong to several shadow regions, since shadows from different pairs may overlap.
To achieve inter-pair feature fusion within a single frame across pairs, we perform spatial fusion on the features he for all pairs at each time step t. The temporal fused features hi are then passed

---

through another transformer for spatial fusion across shadow-object pairs, which is defined as:
ho h-2 ..... N Lnter-PPair_Pusion(hh h.. 3.... N t 1.1,2.....
where each h is spatiotemporal fused feature across frames and Shadow-objer pairs. The spatiotemporal fused feature is passed through a lightweight mask prediction head to predict the object mask Mo. and the shadow mask M.." of each shadow-object pair at each time step:
M M = Mask Mask_ Decokeeeeeddr----
t= 1 1........7
.., N
(8)
where Ylask-Deodderfe are convolutional networks designed to decode from each mask logit for the object and its corresponding shadow masks at time
Finally, the predicted outputs Mr.. and M are used to compute the training loss for supervising the object and shadow mask prediction in the completed scene. Additionally, the aggregated global feature to serves the memory embedding of shadow-object pair and is passed the video outpainting module to guide the recovery of the appearance of the object. Besides, the mask logits of each pair he are passed to the next module for guiding optical flow completion, facilitating instance-level temporal consistency outside the visible scene.
3.2 Instance-aware Optical Flow Completion
Pretrained flow completion modules are commonly used in video generation networks 47 46
as they provide a simpler solution for handling invisible flow compared to directly extending complex RGB content [4)). This approach leverages the inherent simplicity of flow completion to enhance video generation efficiency by using the completed flow to propagate pixels, thereby reducing the burden of video outpainting and better preserving temporal coherence.
To address this problem, we propose a novel instance-aware optical flow completion module, which recovers the instance-level optical flow outside the visible scene and provides temporal consistency guidance for the video outpainting process.
Specifically, we first use the pretrained optical flow estimation model RAFT 31 and the visible region mask Mvi obtain the optical flow of padded video V and initialize these flows with Laplacian filling for the initialized optical flow Om. which is defined as
Oin = BaaaciianFFlliingRRATTO @ Mvis)
Then, the initialized optical flow is fed into pseudo-3D) U-Net (26) encoder-decoder structure with skip connections from the encoder to the corresponding layers in the decoder, which takes the initialized optical flow On and invisible region mask (1 Mvix) and output the completed optical flow O Outrs which is defined as:
Filow
Flow Encoder(Oun)
Fmask = Mask Encoder(11 - MMis))
(10)
Ouut U-Net(comcat( Fmask), "n*)
(11)
where Flow Encoder(.) is a convolutional network that extracts features from the initialized optical flow and the invisible region mask. The U-Net architecture is conditioned on the spatiotemporal features ht, which enables the network incorporate instance-level mask features, enhancing the prediction of the completed optical flow outside the scene.
Specifically, to enable optical flow completion of the padded areas with instance-awarenes. we introduce a cross-attention mechanism (33) after the temporal 2D convolutions in U-Net. This mechanism allows the network to effectively incorporate conditions from the dynamic foreground, enhancing its ability to complete the optical flow, which is defined as:
Q =We(z), K = Wx(connat (i,,, ,i??....,h"] (T).,,T))
(12)
V = Wy-cconctt ..,,,......,
(r)))
(13)
QQTT
Attention (Q, K,V) = softmax
(14)
Vdk
5

---

where z denotes the latent feature from the U-Net, and ht represents the conditional mask features of time t. We W K. and Wy are learned linear projection layers that map inputs into the query, key, and value spaces of dimension dy.
Finally, we utilize the instance logits generated by the previous module to achieve instance-aware optical flow completion, particularly in regions with sharp motion boundaries in previously invisible areas. In regions around the padded boundaries, the module promotes smoothness where there is no moving instance, while preserving sharp motion boundaries when moving instances are present.
3.3 Video Outpainting with Multi-adapters
To enhance the temporal coherence and spatial consistency the instance level in video outpainting, we incorporate spatiotemporal instance conditioning and flow conditioning into a video diffusion model via two-branch architecture, complemented by multi-scale adapter proposed by (43), which provides different types of guidance across the U-Net blocks of latent diffusion model (LDM).
For the spatial consistency of video outpainting, we first combine the mask logit he of the t-th frame and the memory embedding hü for each pair at time t. Then, the combined features of all objects are concatenated and projected to obtain the condition for the k-th block and t-th frame in up-blocks:
f = h + hu... n= 1.2...... Cmask Mask Projection(concatt fr,1 fr.2.... ...... fr.0
(15)
(16)
T = TRit + 71 x Cross Cross_Attn(Jw., Ozask). t 1,2,.....
(17)
where the parameter ? controls the influence of the instance condition the generated video And Cross_Aie (ooss_AAtt()) denotes the cross cross- attention mechanism applied at the t-th frame and the k-th block, where Tk,t is used to project the query, and C,,t is used to project the key and value.
For the temporal conherence of video outpainting, We inject them into the U-Net with the multi-scale optical flow features extracted from Oout, We combine the latent features IyK of the k-th block of the U-Net with the condition from flow features C& through element-wise addition. The combined feature is then processed by a linear layer, and its output is fed directly into the cross attention layer before each motion module in the up-blocks:
C Hflow Flow_ Projeerrection
(18)
x'k .h + 7 x Cross_Attn(zx: Ci Cifiawy
(19)
where the parameter 7/2 controls the influence of the flow condition on the generated video, and zy is used to project the query, and C₂ C.flow is used to project the key and value.
Finally, after the diffusion process, the denoised latents are obtained by integrating the spatiotemporal conditioning signals through the U-Net. The instance and flow conditioning help guide the denoising process, ensuring both spatial consistency and temporal coherence in the generated video.
3.4 Video-aware Shadow Alignment Discriminator
After obtaining the denoised video latent, the latents are fed to the decoder to transform them from the latent space back to pixel space, producing the outpainted video.
Our outpainting module builds a pre-trained image LDM 29 whose autoencoder, trained on individual images, causes flickering when applied to temporally coherent sequences, leading to shadow-object inconsistency in videos.
To address this problem, we incorporate dedicated temporal layers into the decoder, proposed in (28] to capture spatio-temporal dependencies across video frames. These layers are subsequently fine- tuned on video data, allowing the decoder to reconstruct temporally coherent textures and structures of instances within the outpainted regions.
we propose a video-aware shadow alignment discriminator that focuses on the dynamic shadow-object pairs in the scene. After obtaining the output video Vout of decoder, extract masks of shadow-object pairs M and M For each shadow-object pair, we take the object mask and the shadow mask to extract mask features. For each frame, we also extract image features and then concatenate them

---

along the channel dimension. The resulting feature dimension is [T, w. h, C2 + Co + czl, where T' is the number of frames, w and h are the width and height of the image feature, c, is the number of channels in the shadow mask feature, Co is the number of channels in the object mask feature, and e; the number of channels in the image feature. Then, the result fed into a local shadow-instance alignment video discriminator to ensure that each pair of extended semantics is visually coherent, which is defined as
min (DD) 3 Ee pa (=) -b?) E. Be My. MMM.M M²))),
D
2
(20)
min V(G) 2 (x) [D(Vout, Mr. My) c)
where D denotes the alignment discriminator. a and b denote the ground truth real and fake labels, respectively. e denotes the value that G wants to believe for the fake data.
Moreover, to ensure photorealistic reconstructions, a patch-wise temporal discriminator 100] built with 3D convolutions is implemented. Additionally, the image-level shadow-instane' alignment discriminators proposed by (44) are also adapted to enhance alignment between the extended seman- ties and their shadows. By integrating these discriminators during finetuning, the decoder learns to generate outpainted content that not only matches the spatial quality of individual frames but also preserves temporal coherence, reducing flickering, shadow misalignment, and even light source inconsistencies in the final video output.
4 Implementation Details
Video Shadow Instance Detector We directly use ViShadow 38 as our video shadow instance detector, which provides robust paired tracking of objects and their corresponding shadows, even in cases where one of them is temporarily invisible or occluded. The initial tracking results provide object-shadow relationships that serve as heuristic to guide the tracking and prediction mask of instances outside the scene.
Our method is built upon Stable Diffusion v1-5. The temporal modules initialized with the weights from pretrained motion modules (9] to obtain additional motion priors. Each spatial 2D convolutional layer is followed by temporal 1D convolutional layer, which is pretrained in text-to-video tasks on the WebVid dataset (
Diffusion Module
Multi-instance Pair Prediction Since the video shadow instance detector is limited to identifying shadow-objed pairs with visible shadows, we employ YOLO (34) and SAM2 [27] to track and segment objects without visible shadows, particularly those near boundary regions. To maintain computational efficiency, we limit the number of instance pairs to a maximum of 10.
Outpainting Strategy For long video outpainting, M3DDM 8 adopts hybrid coarse-to saare--toffine toneine pipeline, performing coarse outpainting over larger time intervals followed by fine outpainting over shorter intervals. In contrast, MOTIA (36) sequentially splits the long video into short clips with temporal overlap for outpainting. Our approach, however, takes the lifespan of instances into account. Specifically, we dynamically split the long video based on the predicted instance masks to ensure spatial consistency of instances across frames. In scenarios with multiple instances, we prioritize splitting and outpainting at segments with the highest instance overlap.
Training Details Our method is implemented using PyTorch (20) and trained on eight NVIDIA RTX A6000 GPUs During training, we employ the AdamW optimizer [17] with fixed learning rate of 1 x 10-4 and the warm-up learning rate step is 1k. Following the training scheme of [8], the model is trained on the WebVid dataset [1] for 6 epochs with a batch size of 16, utilizing gradient accumulation to stabilize optimization under memory constraints. During the inference process, we use the PNDMScheduler [16] to guide the denoising steps in the reverse diffusion process. Also, we use 50 inference steps and a scaled linear 33 schedule that starts at 0.00085 and ends at 0.012.
5 Experiments
In this section, we describe our experimental setup and present quantitative and qualitative compar- isons, ablation studies, and user study to validate the effectiveness of our method.
7

---

5.1 Datasets
We evaluate our proposed approach on two common datasets, DAVIS 21 and YouTube- VOS 39 which are widely used benchmarks for video outpainting.
DAVIS [21] dataset collects 150 videos in total (60 for training, 30 for validation, and 60 for testing), with each video annotated with multiple foreground instances per frame. It originally introduced as a benchmark for Video Object Segmentation (VOS) to segment and track object instances with precise and temporally consistent annotations across video frames.
YouTube- VOS [39] dataset collects 4,000 high-resolution YouTube videos, totaling over 340 minutes, providing rich diversity in object appearance, motion patters, and scene complexity. Each vidco is densely annotated with high-quality pixel-level masks for multiple objects in selected frames.
Additionally, since these two datasets only provide foreground object segmentation annotations and lack instance-level shadow annotations, we introduce dataset with shadow annotations further validate the effectiveness of our approach.
SOBA-VID 38 dataset collects 292 videos with total of 7045 frames, capturing shadow-object interactions across diverse scenarios. It emphasizes varied shadow patterns, dynamic backgrounds, occlusions, and lighting conditions. The dataset is split into 232 training videos with 5863 sparsely annotated frames and 60 testing videos with I 182 densely annotated frames, resulting in 637 shadow- object pairs total. On average, each video contains 24.1 frames and 2.2 shadow-object pairs.
5.2 Comparisons to Baseline Methods
Qualitative Comparisons. As shown in Fig. we show qualitative results comparing our method to baseline approaches. M3DDM and MOTIA fail align extended semantics with dynamic shadows. In contrast, our method generates visual content consistent with shadows, improving temporal coherence and shadow alignment. This is due to our shadow-ohject prediction and flow completion modules, which provide effective spatiotemporal guidance for video outpainting. Additionally, the video-aware shadow alignment discriminator enhances spatiotemporal consistency across frames.
Ground Truth
Ground Truth
M3DDM
M3DOM
MOTIA
MOTIA
Ours
Figure Qualitative comparison of video outpainting results with a mask ratio of 0.666 on the DAVIS 21
and YouTube- 39
datasets.
Quantitative Comparisons. We compare our method with several state-or -at---f-tfhhheeee art approaches on the widely used video outpainting benchmarks DAVIS [2I] and YouTube- VOS [39], using four common evaluation metries: PSNR, SSIM 37), LPIPS (88), and FVD 32] As shown Tab. [
our approach achieves significant improvements over the state-of-the-art in terms of PSNR, SSIM, and FVD. However, the L.IPS score slightly higher compared MOTIA. Compared our efficient shro approach, MOTIA achieves better perceptual alignment and lower LPIPS scores

---

by fine-tuning the model on each test sample. Note that we follow the test setting of 36 using mask ratios 0.25 and 0.666 the horizontal direction.
Table I: Quantitative Comparisons on DAVIS 21 and YouTube-VOS 39 t means "better when higher", and indicates "better when lower". +denotes based on test sample-specifie fine-tuning.
DAVIS
21
YouTube-VOL
39
Method
PSNRT
SSIM
LPIPS
FVDJ
PSNR
SSIM
LPIPS|
FVD!
Dehan
17.96
0.6272
0.2331
363.1
18.25
0.7195
0.2278
149.7
SDM 28
20.02
0.7078
0.2165
334.6
19.91
0.7277
0.2001
94.81
M3DDM
20.26
0.7082
0.2026
300.0
20.20
0.7312
0.1854
66.62
MOTIAf
20.36
0.7578
0.1595
286.3
20.25
0.7636
0,1727
$8.99
Ours
20.81
0.7254
0.1842
234.7
20.32
0.7719
0.1793
40.78
5.3
Ablation Studies
Ablation Study on Multi-adaption. We further conduct visual analysis to evaluate the effects of different adaptations on the quality of generated results, as shown Fig. can be seen that, although all methods can generate plausible results shortly after the object becomes invisible, as the invisibility duration increases, notable differences emerge. Specifically, without the instance adapter, objects tend to exhibit artifacts causes significant changes the appearance of the object, while omitting the flow adapter leads to blurred object motion. In contrast, our method leverages dynamic shadow to generate temporally consistent objects, maintaining both appearance and motion fidelity over longer periods of invisibility.
disappear entirely. The absence of object memory embedding
instance adapter
w/o
object mem.
w/o
now adaptal
w/o
Ours
Figure 4: Visual results of multi-adaption ablation study on the DAVIS 21 dataset. The region to the right of the red line is masked out and needs to be generated.
Ablation Study on Outpainting. We conduct an ablation study on the YouTube- VOS [3] dataset to analyse the contribution of each module, shown in Tab. The results demonstrate that both the instance and flow adapters significantly improve the video generation quality, with noticeable enhancements in PSNR. SSIM, and LPIPS. Additionally, the video-aware discriminator significantly helps reduce the FVD, further improving the overall performance of the model.
Ablation Study on Shadow-aware Prediction. We conduct an ablation study to assess the impact of dynamic shadow information on object mask predictions across multiple datasets, as shown Tab. 3 Our results show that incorporating dynamic shadow improves both object and shadow mask accuracy.

---

Note that we introduce the SOBA-VID (38 dataset to further validate the impact of shadows, which provides annotations for both objects and shadows. In this dataset, without shadow information incorporated, objects and shadows are treated as independent instances for prediction.
Table 2: Ablation study on the YouTube- NOS
39
dataset.
Method
PSNR
SSIM
LPIPSL
FVD.,
w/ instance adapter
19.89
0.7583
0.1881
64.51
flow adapter
19.94
0.7602
0.1865
61.34
w/ instance & flow adapter
20.15
0.7657
0.1813
51,92
video-aware discriminator
20.11
0.7628
0.1827
42.66
Ours
20.32
0.7719
0.1793
40.78
5.4
User study
We conducted a user study comparing our method against M3DDM [8] and MOTIA (36) using the YouTube- VOS (39] dataset with a horizontal mask ratio of 0.66 as source videos. We collected preferences from 40 volunteers, each evaluating 20 randomly selected result sets. The evaluation focused on two aspects: visual quality (e.g.. clarity, color fidelity, and texture detail) and temporal consistency (e.g., motion smoothness, object continuity, and temporal coherence). As shown in Tab, 4 our method is preferred over both baselines in terms of both visual quality and temporal consistency.
Table 3: Ablation study of mask IoU with and
Table 4: User preference distribution (%) across three methods.
without shadow incorporated across datasets.
DAVIS YouTube-VOS
SOBA-VID
Method
Visual Quality Temporal Consistency
Method
Object
Object
Object/ / Shadow
M3DDM
18.0%
22.3%
w/o shadow
0.75
0.78
0.65 / 0.55
MOTIA
25.6%
15.7%
shadow
0.82
0.84
0.76 0.65
Ours
56.4%
62.0%
6 Conclusion
In this paper, we propose a novel instance-aware video outpainting framework that leverages shadow- object associations to enhance the spatial and temporal consistency of generated content. By explicitly tracking and modeling shadow-object pairs, our method effectively unveils both the spatial region and temporal motion of instances that are invisible in the input video. The integration of spatiotemporal instance guidance with optical flow completion, along with a video-aware discriminator, ensures temporal consistency and coherent scene extension even in complex dynamic scenarios. Extensive experiments on several benchmarks demonstrate that our approach significantly outperforms existing state-of-the-arr methods.
Limitations. Our method exploits shadow cues to improve instance generation beyond the visible scene. The informativeness of shadows depends on the angle of the light source. For example, lower angles usually cast longer shadows, which carry more semantic hints of invisible instances, such as their motion, actions, or the location of the light source. In scenes without visible shadows, these cues are absent, and our approach falls back to standard instance-aware video outpainting.
Societal Impacts. Our approach can extrapolate scenes beyond the original camera view, supporting applications film production, surveillance, and autonomous driving. However, the generated regions are synthetic and may not reflect reality, raising concerns about authenticity and trust. Following (7). we apply watermarking to label generated content and reduce the risk of misuse.
Acknowledgements
This work is supported by National Natural Science Foundation of China (Grant No.62302287) and projects of the Shanghai Committee of Science and Technology, China (Grant 23.R 1423500).
10

---

References
[1] Max Bain al. "Frozen time: A joint video and image encoder for end-to-end retrieval", In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 1728- 1738.
[2] Zhihao Chen et al. "Triple-cooperative video shadow detection". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition. 2021, pp. 2715-2724. [3] Ho Kei Cheng et al. "Putting the object back into video object segmentation", In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 3151-
3161.
[4] Yen-Chi Cheng al. "Inout: Diverse image outpainting via gan inversion". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 11431- 11440.
[5]
Loië Dehan et al. "Complete and temporally consistent video outpainting" In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 687-695.
[6] Xinpeng Ding et al. "Learning shadow correspondence for video shadow detection". In:
European Conference on Computer Vision. Springer. 2022, pp. 705-722.
[7] Patrick Esser et al. "Structure and Content-Guided Video Synthesis with Diffusion Models". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023,
pp. 7346-7356.
[8] Fanda Fan "Hierarchical masked 3d diffusion model for video outpainting". In: Proceed- ings of the st ACM International Conference on Multimedia. 2023, pp. 7890-7900.
[9] Yuwei Guo et al. "Animatediff: Animate your personalized text-to-image diffusion models without specific tuning". In: arXiv preprint urXiv:2307. 004750 (2023).
[10] Phillip Isola et al. "Imagg-to-image translation with conditional adversarial networks". In: Pro- ceedings of the IEEE conference computer vision and pattern recognition. 2017, pp. 1125- 1134.
[11] Hao Jiang and Mark S Drew. Conference Multimedia and Expo. (CME'03. Proceedings (Cat. No. 03TH8698). Vol. 3. IEEE. 2003, pp. 111-77.
tracking in video". In: 2003 International
[12] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. "Occlusion-aware video object inpainting". In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 14468- 14478.
[13] Zhen Li et al. "Towards an end-to-end framework for flow-guided video inpainting". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. pp. 17562-17571.
[14] Han Lin, Maurice Pagnucco, and Yang Song. "Edge guided progressively generative image outpainting". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 806-815.
[15] Lihao Liu al. "Scotch and soda: A transformer video shadow detection framework". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 10449-10458.
[16] Luping Liu al. "Pseudo Numerical Methods for Diffusion Models Manifolds". In: International Conference on Learning Representations. 2022. URL: URL: http::tttts://opereeeii Bet/forummiddPP1Ke PWWVd2yyykKKYykkY
[17] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. 2019. arXiv:
1711.05101 [cs.] LG] URL: lhttps///arxi:
[18] Sohail Nadimi and Bir Bhanu. "Physical models for moving shadow and object detection in video". In: IEEE transactions on pattern analysis and machine intelligence 26.8 (2004),
pp. 1079-1087.
[19] Mahdi Naseri, Jiayan Qiu, and Zhou Wang. "Generated Contents Enrichment" In: arXiv
preprint arXiv:2405..336 03650 (2024).
[20] Adam Paszke et al. "Pytorch: An imperative style, high-performance deep learning library".
In: Advances in neural information processing systems 32 (2019).
[21] Federico Perazzi et al. "A benchmark dataset and evaluation methodology for video object segmentation". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 724-732.
11

---

[22]
Jiayan Qiu et al. "Hallucinating visual instances in total absentia". In: European Conference on Computer Vision. Springer. 2020. pp. 264-282.
[23]
Jiayan Qiu et al. "Matching seqlets: An unsupervised approach for locality preserving sequence matching". In: IEEE transactions on pattern analysis and machine intelligence 43.2 (2019),
pp. 745-752.
[24] Jiayan Qiu et al. "Scene essence". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021. pp. 8322-8333.
[25] Jiayan Qiu et al. "World from blur". In: Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition. 2019, pp. 8493-8504.
[26] Zhaofan Qiu, Ting Yao, and Tao Mei. "Learning spatio-temporal representation with pseudo-3d residual networks". In: proceedings of the IEEE International Conference on Computer Vision.
2017, pp. 5533-5541.
[27] Nikhila Ravi et al. "SAM 2: Segment Anything in Images and Videos". In: arXiv preprint arXiv:2408. 0222400.001444 14 (2024). URI.: htttps://axxiiiii org/ abs abs/2408. 00714.
[28] Robin Rombach et al.
Image Synthesis With Latent Diffusion Models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022, pp. 10684-10695.
[29] Robin Rombach et al. "High-resolution image synthesis with latent diffusion models". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, pp. 10684-10695.
[30] Sheyang Tang et al. "Controllable Data Generation with Hicrarchical Neural Representations". In: Forty-second International Conference on Machine Learning.
[31] Zachary Teed and Jia Deng. "Raft: Recurrent all-pairs field transforms for optical flow". In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. Springer. 2020, pp. 402-419.
[32] Thomas Unterthiner et al. "Towards accurate generative models of video: new metric & challenges". In: arXiv preprint
(2018).
[33] A Vaswani. "Attention is all you need". In: Advances in Neural Information Processing Systems (2017).
[34] Ao Wang et al. "Yolov 10: Real-time end-to-end object detection". In: Advances in Neural Information Processing Systems 37 (2024). pp. 107984-108011
[35] Yang Wang. "Real-tima moving vehicle detection with cast shadow removal in video based on conditional random field"'. In: IEEE transactions circuits and systems for video technology 19.3 (2009), pp. 437-441.
[36] Yun Wang et al. "Be-your-outtaainte:: Mastering video outpainting through input-specific adaptation". In: European Conference on Computer Vision. Springer. 2025, pp. 153-168.
[37] Zhou Wang et al. "Image quality assessment: from error visibility to structural similarity", In:
IEEE transactions on image processing 13.4 (2004), pp. 600-612.
[38] Zhenghao Xing et al. "Video Instance Shadow Detection Under the Sun and Sky". In: IEEE Transactions on Image Processing (2024).
[39] Ning Xu et al. "Youtube-vos: A large-scale video object segmentation benchmark". In: arXiv preprint 18Xiv:180..03327 (2018).
[40] Rui Xu et al. "Deep flw-guidedd video inpainting". In: Proceedings of the IEEE/CVF confer- ence computer vision and pattern recognition. 2019, pp. 3723-3732.
[41] Xiaoyu Xu et al. "Relationship spatialization for depth estimation". In: European Conference on Computer Vision. Springer. 2022, pp. 615-637.
[42] Xiaoyu Xu et al. "Visual Relationship Transformation" In: European Conference on Computer
Vision. Springer. 2024, pp. 251-272.
[43] Hu Ye et al. "Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion (2023).
models". In: arXiv preprint arXiv:2308.067
[44] Hang Yu al. "Shadow-Enlle Enlightnend Image Outpainting". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2024, pp. 7850-7860.
[45] Yanhong Zeng, Jianlong Fu, and Hongyang Chao. "Learning joint spatial-temporal transfor- mations for video inpainting". In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part XVI 16. Springer. 2020, pp. 528-543.
12

---

[46] Kaidong Zhang, Jingjing Fu, and Dong Liu. "Flow "Flow-guided transformer for video inpainting". In: European conference on computer vision. Springer. 2022. pp. 74-90.
[47] Kaidong Zhang, Jingjing Fu, and Dong Liu. "Inertia-guided flow completion and style fusion for video inpainting" In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 2022, pp. 5982-5991.
[48] Richard Zhang et al. "The unreasonable effectiveness of deep features as a perceptual metric". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 586-595.
[49] Shengyu Zhao et al. "Large Scale Image Completion via Co-Modulated Generative Adversarial Networks". In: International Conference on Learning Representations (ICLR). 2021. [50] Haipeng Zhou et al. "Timeline and boundary guided diffusion network for video shadow detection", In: Proceedings of the 32nd ACM International Conference Multimedia. 2024, pp. 166-175.
[51] Qianyu Zhou et al. "TransVOD: End-to-end video object detection with spatial-temporal transformers". In: IEEE Transactions on Pattern Analysis and Machine Intelligence 45.6 (2022), pp. 7853-7869.
[52] Shangchen Zhou al. "Propainter: Improving propagation and transformer for video inpaint- ing". In: Proceedings of the JEEF/CVF International Conference on Computer Vision. 2023, pp. 10477-10486.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of the paper have been truthfully described throughout the abstract and the introduction, and concisely summarized the end of the introduction. Each of the listed contributions is backed up by experimental results or literature review throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
The paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We included detailed descriptions of our model architecture in Section 3 references publicly available pre-trained weights, data processing procedures, and other training details in Section 4 and further in the supplementary material. We will also opensource our code and model checkpoints upon paper acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: As stated in the response above, provided detailed instructions on how replicate our experiment results in the main paper and further in the supplementary material. We will release our code and models upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/
CommeSumeeesa /otii///u ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided detailed instructions on replicating the training and evaluation procedures in Section 4 Section 5 and further in the supplementary material. We did not perform delicate tuning for the hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not compute error bars due to the lack of adequate computational resources. However, we tried our best to provide faithful and reliable quantitative evaluation results, including user study. All experiments are conducted with the same and commonly used random seed.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We described our compute setup as well as the training time and inference runtime in Section and further in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: https::/neul h/teuris////uusss.... https///cpll///hcccccciieeeeee
Answer: [Yes]
Justification: Authors carefully read the NeurlPS Code of Ethics and preserved anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The socictal impacts of the work are discussed in Section 6
17

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We mentioned in Section that the generated content from our model can be watermarked prevent potential misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package or dataset. All the pre-trained models and tools used are specified in Section 4 and further in the supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
18

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy hbte::er//errrrsecc////offeerrre for what should or should not be described.
20


Reviews:
[{"review_id": "1fKFW9o8mE", "reviewer": "Reviewer_G7tk", "summary": "This paper proposes a new framework for video outpainting that makes innovative use of dynamic shadows to infer and reconstruct semantically consistent content in scenes where objects temporarily disappear from view. The authors design a multi-stage architecture that includes modules for predicting object-shadow masks, completing instance-aware optical flow, and guiding a latent diffusion model with spatio-temporal priors. The system also introduces a video-aware discriminator aimed at aligning generated content with the motion and structure implied by dynamic shadows. Experiments on widely used benchmarks show competitive improvements over several recent baselines, both in terms of visual quality and temporal coherence.", "questions": "(1) How well does the method generalize to real-world videos where shadows are less prominent or heavily distorted (e.g., indoor lighting, artificial light sources)? It would be helpful to see some quantitative or qualitative evaluation under such conditions.\n(2) Have you considered how sensitive the overall pipeline is to the quality of the initial object-shadow detection? If this module fails or produces imprecise masks, what are the typical failure modes of the system?\n(3) In your architecture, different components (e.g., instance adapter, flow adapter) are introduced in a modular way. Are these modules jointly trained end-to-end, or is there a staged training process?", "limitations": "While the method demonstrates impressive performance on benchmark datasets, there are notable limitations that deserve further attention. First, the reliance on high-quality dynamic shadows makes the system less generalizable to scenes where such shadows are weak, inconsistent, or altogether missing. This dependency may reduce the method’s effectiveness in many indoor environments or scenarios involving multiple light sources. Second, the approach assumes that shadow-object associations can be reliably extracted, which might not hold in crowded scenes, strong occlusion, or when shadows are cast on complex textured backgrounds. These assumptions could limit the model's robustness in unconstrained or in-the-wild video content.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "One of the most compelling aspects of this work lies in its central idea: treating dynamic shadows not as noise but as informative signals that reflect underlying instance motion. This perspective departs meaningfully from prior approaches, which tend to overlook or discard such visual phenomena. The technical formulation is coherent and thoughtfully designed, especially the integration of shadow-based instance priors into a diffusion-based generation pipeline. The empirical results are strong, and the ablation studies offer reasonable support for the importance of each module in the system.\n\nOn the other hand, the method’s reliance on accurate shadow detection raises questions about its robustness in varied lighting conditions or scenes where shadows are faint or occluded. The proposed approach also appears to require relatively heavy computational resources, which may limit its scalability to longer or higher-resolution videos. In terms of evaluation, while the reported metrics show clear improvements, the absence of statistical uncertainty estimates (e.g., variance or confidence intervals) makes it difficult to assess how consistent these gains are across different runs or datasets. Lastly, while the architectural design is detailed, it is not always easy to distinguish which parts are trained jointly and which are fixed, which could hinder reproducibility for less experienced practitioners.", "was_revised": true, "final_justification": "The authors' response partially addressed my questions. There are still some minor issues not fully answered, but they do not change my rating.", "rebuttal": "**Q1:** The generalizability of the method to real-world videos.\n\n**R1:** Our method performs well on real-world videos because shadows commonly exist in natural scenes.\nAs can be seen in the supplementary material, videos from real-world datasets like DAVIS and YouTube-VOS have an average of 0.9 and 1.2 shadow-object pairs per video, with each pair lasting around 38.5 frames in DAVIS and 24.2 frames in YouTube-VOS.\nMoreover, for 76\\% of videos with shadow-object pairs, the object becomes invisible while its shadow is still visible.\n\n\nIt can be seen from R-Tab 1 and R-Tab 2 that our method performs robustly on the cases with less prominent shadows (R-Tab 1) and heavily distorted shadows (R-Tab 2). This is because our proposed spatio-temporal fusion strategy alleviates the effect of low-quality shadow detection in single frames by considering the entire video. \n\n\n*R-Tab 1: Sensitivity analysis of our method to different confidence thresholds used in the shadow detector.*\n\n| Threshold | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |\n|:-----------:|:--------:|:--------:|:---------:|:-------:|\n|    0.2    | 19.85  | 0.7632 | 0.1865  | 42.30 |\n|    0.4    | 20.12  | 0.7685 | 0.1817  | 41.32 |\n|    0.6    | 20.32  | 0.7719 | 0.1793  | 40.78 |\n|    0.8    | 20.10  | 0.7701 | 0.1806  | 41.05 |\n\n\n\nIn R-Tab 1, we change the confidence threshold in the shadow detector, where a low threshold means detections with low confidence will be preserved and used in the following outpainting process.\nUsually, the less prominent shadows have low confidence, and thus can be filtered out by setting a high confidence threshold.\nEven though some of the less prominent shadows cannot be filtered out by setting a high confidence threshold, our method still performs robustly on them.\nAs can be seen from R-Tab 1, our method is not sensitive to the quality of the shadow detection, and performs well even when setting the confidence threshold to be 0.2, in which situation most of the low-quality shadow detections are preserved while the outpainting quality decreases slightly.\n\n*R-Tab 2: Quantitative comparison on video in different camera motion settings from the Youtube-VOS dataset.*\n| Camera Motion | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |\n|:---------------:|:--------:|:--------:|:--------:|:---------:|:-------:|\n|      Slow     | M3DDM  | 20.31  | 0.7351 | 0.1823  | 65.47 |\n|               | MOTIA  | 20.39  | 0.7682 | 0.1701  | 57.94 |\n|               | Ours   | 20.43  | 0.7752 | 0.1772  | 39.18 |\n|      Fast     | M3DDM  | 19.89  | 0.7197 | 0.1943  | 70.06 |\n|               | MOTIA  | 19.98  | 0.7518 | 0.1804  | 62.21 |\n|               | Ours   | 20.13  | 0.7648 | 0.1855  | 44.86 |\n\n\nThe scale of the shadow distortion is hard to quantify directly since there are no ground-truth shadow annotations in our adopted datasets. \nWe therefore measure the camera motions instead, in which large camera motions between two frames are more likely with largely distorted shadows.\nThe inter-frame camera motion is first measured by Structure-from-Motion (SfM).\nThen, for a single video, if its average per-frame translation is less than 5 Euclidean distance and average per-frame rotation angle is less than 5 degrees, we label it as a slow-motion video, otherwise a fast-motion one.\nAs can be seen from R-Tab 2, although the fast-motion videos should be with largely distorted shadows, our method performs robustly on them.\n\n\n---------------------------------------------------\n**Q2:** The impact of the initial object-shadow detection.\n\n**R2:** As can be seen from  R-Tab 1, our method can handle the impact of initial object-shadow detection even when the confidence threshold in detection is set to be low, where lots of detected shadows with low confidence are used for outpainting.\nThis is because our spatio-temporal fusion module is specifically designed to refine the initial detection results, thus enhancing the system’s robustness to errors such as imprecise masks or overlapping shadows.\n\n---------------------------------------------------\n**Q3:** The training strategy of the modular components.\n\n**R3:** The shadow-aware instance prediction module and the instance-aware optical flow completion module are pre-trained separately. \nThen, the entire framework is trained together in an end-to-end manner."}, {"review_id": "5UYZgIiGoX", "reviewer": "Reviewer_uqHS", "summary": "This paper proposes a novel video outpainting method that enhances the temporal and spatial consistency of generated content by incorporating the prior relationship between instances and their dynamic shadows into an adapter. Conventional video outpainting methods often suffer from issues such as temporal inconsistency, motion mismatch, and a lack of instance-level modeling when processing dynamic scenes. The authors innovatively propose an auxiliary feature generation pipeline that effectively reveals the spatial regions and temporal motion of unseen instances in the input video by explicitly tracking and modeling shadow-object pairs. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods across multiple evaluation metrics on the DAVIS and YouTube-VOS datasets, exhibiting particularly strong performance in maintaining the temporal consistency and spatial coherence of objects.", "questions": "1. The proposed method introduces several modules to generate auxiliary features, which are then injected into the diffusion model via an adapter. How does the computational overhead (both for training and inference) of this approach compare to the methods it is benchmarked against?\n2. When multiple instances are detected, how is the global feature $h^t$ derived from the individual instance features $h^{t,n}$? For instance, are they concatenated, summed, or aggregated in some other manner?\n3. Could the authors elaborate on the design of $z^0$? Is a unique latent code $z^{0,n}$ assigned to each object class? If not, how is $h^{0,n}$ derived from a single $z^0$, and how is the Inter-Pair Fusion subsequently computed across different instances?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The method explicitly incorporates prior knowledge about instances and shadows, which offers good interpretability.\n2. The method is well-designed, with four modules that have distinct functions and collaborate effectively to form a complete solution. The authors validate the effectiveness of each component through comprehensive ablation and user studies.\n3. Video outpainting is an important problem in the field of computer vision with broad application potential.\n4. The experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art approaches across multiple metrics.\n5. The paper is the first to explicitly propose the use of dynamic shadow information for video outpainting, presenting a novel research perspective.\n\n**Weaknesses**\n1. The method's applicability is constrained by the availability of shadow information. Its performance may be limited in scenarios where distinct shadows are not present.\n2. The notation for superscripts and subscripts is somewhat confusing due to inconsistent use and omissions without clear explanation.", "was_revised": true, "final_justification": "The authors' partially resolved my concerns. I will raise my rating to 4.", "rebuttal": "**Q1:** The generalizability of the method to videos with or without shadows.\n\n**R1:** Our method performs well on videos both with and without shadows, as can be seen from R-Tab 3.\n\n*R-Tab 3: Quantitative comparison on videos with and without shadows from the Youtube-VOS dataset.*\n| Shadow Presence | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |\n|:-----------------:|:--------:|:--------:|:--------:|:---------:|:-------:|\n| With Shadows    | M3DDM  | 20.19  | 0.7330 | 0.1844  | 66.54 |\n|                 | MOTIA  | 20.24  | 0.7639 | 0.1732  | 58.76 |\n|                 | Ours   | 20.35  | 0.7756 | 0.1762  | 39.62 | \n| Without Shadows | M3DDM  | 20.22  | 0.7315 | 0.1857  | 66.75 |\n|                 | MOTIA  | 20.26  | 0.7634 | 0.1725  | 59.20 |\n|                 | Ours   | 20.28  | 0.7661 | 0.1833  | 42.30 |\n\nNote that, for a single video, if the shadow area is less than 10\\% of the single frame size and lasts for less than 20\\% of the video duration, we label it as a video without shadows, otherwise a video with shadows.\n\nWhen shadows are absent (i.e., the shadow mask is all zeros), our method still performs conventional outpainting, which indicates that our model is more general, rather than specific to scenarios with shadows.\n\n-----------------------------\n\n**Q2:** The computational overhead of the proposed method relative to baselines in training and inference.\n\n**R2:** Our method offers a good trade-off between speed and quality during inference, as can be seen from R-Tab 4.\n\n*R-Tab 4: Comparison of model size, trainable parameters, and inference time across different methods on Youtube-VOS.*\n| Method | Inf. Time/Video | Params | Trainable Params | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |\n|:--------:|:-----------------:|:--------:|:------------------:|:--------:|:--------:|:---------:|:-------:|\n| M3DDM  |       48 s      | 1382 M |       1299 M     | 20.20  | 0.7312 | 0.1854  | 66.62 |\n| MOTIA  |       480 s     | 2254 M |       12 M       | 20.25  | 0.7636 | 0.1727  | 58.99 |\n| Ours   |       55 s      | 1593 M |       92 M       | 20.32  | 0.7719 | 0.1793  | 40.78 |\n\nBecause the training code for M3DDM has not been publicly released, we have re-implemented it for testing.\nAnd MOTIA requires training a LoRA for each test-time sample, which incurs significant per-instance computational cost during inference preparation.\nAs mentioned in the supplementary material, we randomly selected 10 videos for outpainting at a resolution of 256×256. \nOn a single RTX A6000 GPU, M3DDM required approximately 48 seconds per video, MOTIA took about 8 minutes, and our method completed the task in approximately 55 seconds per video.\n\n-----------------------------\n\n**Q3:** The way to derive the global feature $h^t$ of a single frame.\n\n**R3:** The global feature $h^t$ is obtained by concatenating individual instance features $h^{t,n}$, as can be seen from Eq 16.\nIn addition, the number of instance pairs is capped at a fixed maximum of 10 to ensure consistent input dimensions, as detailed in the supplementary material.\n\n-----------------------------\n**Q4:** The design of the learnable vector $z^0$ in the inter-frame fusion module. \n\n\n**R4**: This learnable vector $z^0$ is shared by each pair to extract their global embedding $h^0$ individually.\nSuch an aggregation structure is similar to that used in Vision Transformer (ViT)[1], which has been demonstrated to be effective.\nThis learned unique latent code $z^{0,n}$ is assigned to each shadow-object pair to derive the global embedding $h^{0,n}$.\nThen, to avoid confusion across different pairs, the global embedding $h^0$ of each pair is not involved in the inter-pair fusion process, as can be seen from Eq 7. \n\n[1] \"An image is worth 16x16 words: Transformers for image recognition at scale.\" In the International Conference on Learning Representations (ICLR) 2021."}, {"review_id": "22S8XGxfhU", "reviewer": "Reviewer_ak5T", "summary": "This work improve the visual quality and temporal consistency by dynamic shadow information and robust object generation during occlusion. I think the motivation is reasonable but lacks further relevant verification.", "questions": "1. My biggest concern about this work is that it does not clearly explain the relationship between shadows and video outpainting.\n2. I suspect that the most effectiveness for this work is still in the GT in the dataset, rather than the detected shadow.\n3. For the visualizations in Figures 3 and 4, it is suggested that the input image should be added to make it clearer.\n4. In fact, this work does not involve semantically-related content. Instances and shadow are visual features, not semantic features. Therefore, I think the hidden semantics of the above work is unconvincing.", "limitations": "Although it is novel to mine temporal continuity from shadows, if the shadow part in the video is difficult to find, then this method meaning may reduce. In my opinion, generality may be the most important.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The ablation study shows that the proposed method outperforms the baselines in terms of both visual quality (e.g., clarity, color fidelity, texture detail) and temporal consistency (e.g., motion smoothness, object continuity, temporal coherence).\n2. The paper demonstrates that incorporating dynamic shadow information improves the accuracy of both object and shadow mask predictions across multiple datasets.\n3. The visual analysis indicates that the proposed method can generate temporally consistent objects with high fidelity in appearance and motion, even as the duration of object invisibility increases, outperforming the baselines.\n\nWeaknesses:\nPlease see Questions.", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The relationship between shadows and video outpainting.\n\n**R1:** Even when an instance (e.g., an object) goes outside the frame, its shadow often stays in the video and with useful cues such as motion and action.\nTraditional video outpainting methods often overlook shadows in videos or treat them as noise, which is unreasonable. \nIn contrast, our method explicitly models the dynamic shadows and incorporates this information as a spatiotemporal cue to enhance the spatiotemporal consistency of the generated video.\n\n----------------------------------------------------------------\n**Q2:** The ground truth or the detected shadow.\n\n**R2:** We did not use ground truth shadow annotations, and adopted DAVIS and YouTube-VOS datasets even do not include the ground truth of shadow.\nBesides, the quality of detected shadows can impact performance, as shown in R-Tab 5.\n\n\n*R-Tab 5: Sensitivity analysis of our method to different confidence thresholds used in the shadow detector.*\n\n| Threshold | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |\n|:-----------:|:--------:|:--------:|:---------:|:-------:|\n|    0.2    | 19.85  | 0.7632 | 0.1865  | 42.30 |\n|    0.4    | 20.12  | 0.7685 | 0.1817  | 41.32 |\n|    0.6    | 20.32  | 0.7719 | 0.1793  | 40.78 |\n|    0.8    | 20.10  | 0.7701 | 0.1806  | 41.05 |\n\nWe conducted experiments using multiple threshold settings of the video instance shadow detector.\nOur method remains robust across different thresholds, which shows our method can handle the impact of initial object-shadow detection from the video instance shadow detector.\n\n----------------------------------------------------------------\n**Q3:** The better visualizations for the visual result.\n\n**R3:** Thank you for your valuable suggestion to improve the visualizations in Fig 3 and 4. \nWe will incorporate the input images in the final version along with the current captions that explicitly indicate the masked regions to provide better visual reference and enhance clarity.\n\n----------------------------------------------------------------\n**Q4:** The hidden semantics of dynamic shadow in the video.\n\n**R4:**  Instances such as objects and shadows are not merely visual features but also carry rich semantic information.\nSpecifically, shadows are 2D projections of objects and often implicitly convey rich semantic information about objects.\n(1) In static images, shadows can unveil the category and spatial location of objects.  \n(2) In videos, they further unveil the actions and motion of the objects over time.  \nTherefore, modeling shadows is not just a visual consideration but also a way to capture the hidden semantics of objects in the scenes."}, {"review_id": "YP35kAWFKm", "reviewer": "Reviewer_wJGo", "summary": "The paper proposes the first shadow-aware video outpainting framework that uses the motion of dynamic shadows to infer and synthesize content outside the camera’s original field of view. \nShadow-aware Instance Prediction tracks object–shadow pairs across frames and predicts the spatial extent of objects that are currently outside the frame. Instance-aware Optical Flow Completion fills missing flow fields in the padded regions using cross-attention to the predicted masks, preserving object-specific motion. \nA latent-diffusion outpainting network is augmented with two adapters, plus a video-aware shadow-alignment discriminator that jointly enforces spatial detail and temporal coherence. \nOn the DAVIS and YouTube-VOS benchmarks the method improves PSNR, SSIM and FVD over prior GAN- and diffusion-based outpainting systems, and a user study shows clear preference for visual quality and temporal consistency.", "questions": "How does the system resolve cases where multiple objects cast overlapping shadows?\n\nWhat is the performance of training only the adapters (freezing the diffusion backbone) vs. fine-tuning the entire U-Net?\n\nHow sensitive is optical-flow completion when the camera pans quickly and shadows smear?", "limitations": "When the method comes across some special scenarios, such as camera motion, low-light, which may severely affect the visibility of the shadow, would the method still perform well? This is not validated in this manuscript.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. It is interesting to treat shadows as informative, temporally consistent cues, enabling the model to “unveil” otherwise invisible objects and motions.\n2. extensive experiments, including quantitative metrics, qualitative comparisons, ablations, and a 40-participant user study.\n\nWeaknesses:\n1. The author should further clarify the difference between the proposed method and MOTIA[29]. MOTIA also used input-adapters in the outpainting network. It seems that the only difference is that the author introduces shadow as input prior to improving the performance of outpainting. However, observed from Tab.1, on DAVIS, the proposed method only outperforms MOTIA in PSNR and FVD (2 out of 4), and in many metrics, the two methods produce close results. The author should explain the inherent reasons.\n2. The paper only presents the cases of the static camera. If the camera is moving, the relative position of the light source would change unexpectedly, leading to dramatic changes in shadow. Can the method still perform well in such cases?\n3. From Tab.2, the model performs inferior to MOTIA when w/o Video-aware Shadow Alignment Discriminator. This makes the effectiveness of the shadow prior doubtful.\n4. The novelty of Video-aware Shadow Alignment Discriminator is limited. As described in the manuscript, the module source from [23][10][35], making the module incremental.\n5. The author should elaborate details of the user study.", "was_revised": true, "final_justification": "At this stage, there remains some confusion and questions about the method and performance of shadow priors. I think it is a borderline paper.", "rebuttal": "**Q1:** Difference between MOTIA and our method.\n\n**R1:** The differences between MOTIA and our method are significant.\nSpecifically, MOTIA relies on test-time adaptation by training a LoRA module for each test sample, which means they need to spend considerable time training on every video during the inference stage. \nIn contrast, our method does not require retraining on each video, making a comparison unfair. \nNevertheless, the performance comparison on DAVIS remains comparable.\n\n--------------------------------------------------------------------\n\n**Q2:** The performance in scenarios with a moving camera.\n\n**R2:** As can be seen from Fig 1, 3 and 4, these several samples are from moving camera scenarios. \nAnd the issue of dramatic changes in shadow caused by the relative movement of the light source can be addressed by our method.\nBecause our module observes the shadow-object pairs over time to infer the changing relative position of the light source. \nIn addition, the spatiotemporal fusion strategy can implicitly incorporate the understanding of the changing relative position of the light source into the refinement of each shadow-object pair.\n\n\nNote that in both the DAVIS and YouTube-VOS datasets, most videos have a moving camera, and only a few are shot from a fixed position.\nTo further address your concern about handling dramatic changes in shadows, we conducted an additional experiment using videos from the YouTube-VOS dataset, as can be seen from R-Tab 6.\n\n*R-Tab 6: Quantitative comparison on video in different camera motion settings from the Youtube-VOS dataset.*\n| Camera Motion | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |\n|:---------------:|:--------:|:--------:|:--------:|:---------:|:-------:|\n|      Slow     | M3DDM  | 20.31  | 0.7351 | 0.1823  | 65.47 |\n|               | MOTIA  | 20.39  | 0.7682 | 0.1701  | 57.94 |\n|               | Ours   | 20.43  | 0.7752 | 0.1772  | 39.18 |\n|      Fast     | M3DDM  | 19.89  | 0.7197 | 0.1943  | 70.06 |\n|               | MOTIA  | 19.98  | 0.7518 | 0.1804  | 62.21 |\n|               | Ours   | 20.13  | 0.7648 | 0.1855  | 44.86 |\n\n\nThe relative position of the light source and the scale of the shadow distortion are hard to quantify directly since there are no ground-truth shadow annotations in our adopted datasets. \nWe therefore measure the camera motions instead, in which large camera motions between two frames are more likely with largely distorted shadows.\nThe inter-frame camera motion is first measured by Structure-from-Motion (SfM).\nThen, for a single video, if its average per-frame translation is less than 5 Euclidean distance and average per-frame rotation angle is less than 5 degrees, we label it as a slow-motion video, otherwise a fast-motion one.\nAs can be seen from R-Tab 2, although the fast-motion videos should be with largely distorted shadows, our method performs robustly on them. \n\n------------------------------------------\n\n**Q3:** The effectiveness of the shadow prior in our pipeline.\n\n**R3:** Our method shows the effectiveness of the shadow prior, as can be seen from Fig 3 and Tab 1 through comparisons with M3DDM.\nAs MOTIA performs test-time adaptation by training a LoRA for each test sample, which incurs large inference-time computation. \nOur method achieves better performance than M3DDM, which also does not require any test-time retraining. \nFurthermore, when our proposed components are integrated, our model outperforms MOTIA, demonstrating that the shadow prior in our method is effective.\n\n------------------------------------------\n\n**Q4:** The novelty of video-aware shadow alignment discriminator.\n\n**R4:** It not only enhances the spatiotemporal consistency of dynamic shadow-object pairs but also enhances scene-level generation, even generating a reasonable light source, as can be seen from Fig 1(c).\nModules like the instance adapter and flow adapter focus on capturing instance-level motion and scene-level coherent motion individually, to enhance video consistency.\nHowever, video generation goes beyond just handling moving instances; it also requires generating a realistic and semantic-related background, including accurate lighting and scene semantics.\nDuring the decoding process from latent space to video space, our discriminator contributes to generating more realistic shadows and background illumination compared to the pre-trained decoder of Stable Diffusion.\n\n---------------------\n\n**Q5:** The details of the user study.\n\n**R5:** We conducted a user study to compare our method with M3DDM and MOTIA on the YouTube-VOS dataset with a horizontal mask ratio of 0.66. \nA total of 40 volunteers participated in the study, and each volunteer evaluated 20 randomly selected result sets.\nParticipants were asked to choose the best result under each of the two criteria: visual quality (including clarity, color fidelity, and texture detail) and temporal consistency (including motion smoothness, object continuity, and temporal coherence).\n\n------------------------------------------\n\n**Q6:** The method for resolving the overlapping shadows cast by multiple objects.\n\n**R6:** We introduce an inter-pair fusion module and a relaxed shadow mask supervision strategy to address the issue of overlapping shadows cast by multiple objects.\nSpecifically, the inter-pair fusion module enables spatial information sharing across different object-shadow pairs.\nAdditionally, a relaxed shadow mask supervision strategy does not penalize predictions containing connected regions of other shadows, encouraging the model to learn to complete the shadow regions of each object in complex multi-object scenarios.\n\n---------------------\n\n**Q7:** The performance comparison of adapter-based training versus full U-Net fine-tuning in diffusion models.\n\n**R7:** Full U-Net fine-tuning outperforms adapter-based training, as can be seen from R-Tab 7.\n\n*R-Tab 7: Comparison of different fine-tuning strategies on the shrunk Stable Diffusion on Youtube-VOS.*\n| Method                 | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |\n|:------------------------:|:--------:|:--------:|:---------:|:-------:|\n| Only Training Adapter  | 16.74  | 0.6421 | 0.2486  | 85.32 |\n| Full U-Net Fine-tuning | 17.89  | 0.6654 | 0.2317  | 76.45 |\n\n\nHowever, due to computational resource constraints, we are unable to fine-tune the full U-Net of Stable Diffusion. \nInstead, we choose to freeze the backbone and train only the adapters, which still achieves competitive performance.\n\n-----------------------\n\n**Q8:** The sensitivity of optical-flow completion when the camera pans quickly and shadow smear.\n\n**R8:** Our module is not sensitive to such cases as fast camera panning and shadow smearing, as can be seen from R-Tab 8.\n\n*R-Tab 8: The sensitivity of optical-flow completion under different camera motions.*\n| Camera Motion | End-Point Error(EPE) |\n|:---------------:|:----------------------:|\n|      Slow     |          0.57        |\n|      Fast     |          0.62        |\n\nNote that, unlike video deblurring, the task of video outpainting involves clear input frames.\nTherefore, shadow smear caused by fast motion is not an issue in our task setting. \nEven under fast camera motion, our optical-flow completion still performs well due to the object-shadow mask providing instance-level spatial guidance."}]

Author rebuttal:

Meta-review: This paper introduces a shadow-aware video outpainting framework that explicitly models object–shadow pairs to reveal the spatial extent and motion of temporarily invisible instances, coupling instance-aware optical-flow completion with a diffusion outpainting backbone and a video-aware shadow-alignment discriminator. Reviewers find the core idea novel and well-motivated, with coherent modular design and strong empirical gains (PSNR/SSIM/FVD) plus a user study; after rebuttal, all reviewers are positive (one accept, three borderline accept). The rebuttal further strengthens the case with robustness analyses (insensitivity to shadow-detector thresholds), camera-motion splits showing stability under fast motion, and an efficiency comparison indicating competitive inference without per-video test-time retraining (unlike MOTIA). Thus, the AC decides to accept the submission.
Decision: Accept (poster)