Video Decomposition Prior: Editing Videos Layer by Layer
Authors: Gaurav Shrivastava
Ser-Nam Lim
Abhinav Shrivastava
Venue: ICLR 2024 poster (2024)
Keywords: Video decomposition
Test-time optimization technique
Video Relighting
Video Dehazing
Unsupervised Video Object Segmentation
Edits Propagation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce a novel inference-time optimization framework that performs three primary vision task of video relighting, unsupervised object segmentation and video dehazing leveraging the compositionality inherent in the videos.

Abstract: In the evolving landscape of video editing methodologies, a majority of  deep learning techniques are often reliant on extensive datasets of observed input and ground truth sequence pairs for optimal performance. Such reliance often falters when acquiring data becomes challenging, especially in tasks like video dehazing and relighting, where replicating identical motions and camera angles in both corrupted and ground truth sequences is complicated. Moreover, these conventional methodologies perform best when the test distribution closely mirrors the training distribution. Recognizing these challenges, this paper introduces a novel video decomposition prior `VDP' framework which derives inspiration from professional video editing practices. Our methodology does not mandate task-specific external data corpus collection, instead pivots to utilizing the motion and appearance of the input video. VDP framework decomposes a video sequence into a set of multiple RGB layers and associated opacity levels. These set of layers are then manipulated individually to obtain the desired results. We addresses tasks such as video object segmentation, dehazing, and relighting. Moreover, we introduce a novel logarithmic video decomposition formulation for video relighting tasks, setting a new benchmark over the existing methodologies. We evaluate our approach on standard video datasets like DAVIS, REVIDE, & SDSD and show qualitative results on a diverse array of internet videos.

Published as a conference paper at ICLR 2024
VIDEO DECOMPOSITION PRIOR: EDITING VIDEOS LAYER BY LAYER
Gaurav Shrivastava
Ser-Nam Lim University of Central Florida sernamêucf edu
University of Maryland, College Park gauravshe edu
Abhinav Shrivastava
University of Maryland, College Park abhinavêcs. umd. edu
ABSTRACT
In the evolving landscape of video editing methodologies, majority of deep learning techniques are often reliant on extensive datasets of observed input and ground truth sequence pairs for optimal performance. Such reliance often falters when acquiring data becomes challenging, especially in tasks like video dehazing and relighting, where replicating identical motions and camera angles in both cor- rupted and ground truth sequences is complicated. Moreover, these conventional methodologies perform best when the test distribution closely mirrors the training distribution. Recognizing these challenges, this paper introduces novel video de- composition prior 'VDP' framework which derives inspiration from professional video editing practices. Our methodology does not mandate task-specific exter- nal data corpus collection, instead pivots to utilizing the motion and appearance of the input video. VDP framework decomposes video sequence into multiple RGB layers and associated opacity levels. These set of layers are then manipulated individually to obtain the desired results. We addresses tasks such as video object segmentation, dehazing, and relighting. Moreover, we introduce a novel logarithmic video decomposition formulation for video relighting tasks, set- ting new benchmark over the existing methodologies. We evaluate our approach on standard video datasets like DAVIS, REVIDE, & SDSD and show qualitative results on a diverse array of internet videos
of
1 INTRODUCTION
Deep learning techniques. prominently convolutional networks (Krizhevsky et al. 2012) and vi- sion transformers (Dosovitskiy et al. (020). have emerged as the preferred choice for video-related tasks (Shrivastava & Shrivastava 2024; Bodla et al. 2021 Shrivastava & Shrivastava 2021 Saini et al. 2022, Shrivastava 2021) due to their superior results over the classical techniques. Typically, these frameworks are trained using a standardized method: curate a dataset of input/corrupted and ground truth/clean sequences, design the model, and minimize the 1.2 loss between predictions and actual values. While this supervised approach sets state-of-the-an benchmarks, it grapples with two significant challenges. First, collecting extensive datasets with annotated pairs for every task is im- practical and expensive. Second, as demonstrated by prior works like Shrivastava et al. (202)), the performance of pretrained models deteriorates when the test data slightly deviates from the training data.
To cireumvent these challenges, emerging video editing methods (Lu et al., 2020a b 2021 Kasten et al., [02T]: Shrivastava et al. (023) were developed and they offer three key advantages: they operate directly on test sequences (also known as inference-time optimization), thereby obviating the need for collecting input and ground truth pairs; they are immune to dataset biases and generalize well to unseen examples (Shrivastava et al. 2023);; and they are resource-efficien as they do not have to optimize over external training corpus instead they optimize only on the given test input video.
Navigate to the webpage for video results.

---

Published as a conference paper at ICLR 2024
Input Video Flow
Video Decomposition
Decomposition Guidance
Final Edits Performed
Object Segmentation
Mask
FG Manipulation
Dehazing
Aidggt (lood)
Tmap
Haze- free frame
Relighting
Tmap
Relit frame
Visual representation of video edits obtained using VDP. The first row demonstrates foreground manipulation example that leverages an object mask as the decomposition guide. This object mask obtained by performing Unsupervised Video Object Segmentation (UVOS), which is a downstream task of video de- composition and is achieved using our proposed framework. Our approach effectively separates the foreground objects and background in the video sequence, enabling to perform object manipulation. The second row shows the result of approach for video dehazing, where our method effectively removes the haze from scene. Finally, the third row showcases the effectiveness of our approach for video relighting, where our method effectively changes the lighting of the scene. Our proposed approach outperforms the state-of-the-as methods for the latter two tasks, highlighting the efficacy of our framework for video decomposition.
Figure 1:
However, these techniques are not without limitations. They are often designed for very specific tasks, and they often perform inferior to their data-trained counterpart.
In this paper, propose inference-ting cireenece-iieeeeee optimization framework VDP, that does not rely collection task-specific video input and ground truth sequence pairs. We took inspiration from professional video editing techniques where sequences are broken down into multiple RGB layers. Each RGB layer is associated with a distinct opacity layer, and by manipulating these RGB layers and their associated opacity levels, desired outcomes achieved. In this work, emphasize that proper decomposition formulation of video leads to visually realistic solutions for the tasks. Fig, illustrates our approach for the 3 different tasks discussed. Our framework relies only on the input video and its derived optical flow to solve wide variety of video-specific tasks.
From a technical standpoint, VDP consists of integral modules: 'RGB-Net' that operates on the appearance aspect of the input video and 'a-Net' that operates on the associated optical flow RGB of the input video. Both of these modules are designed using the convolutional U-Net architec- ture. Based on the tasks, we use the appropriate video decomposition formulation and appropriate regularization terms. We then jointly optimize these convolutional U-Nets for the same.
Our work makes three main contributions. 1) To the best of our knowledge we are the first approach utilize FlowRGBs in an inference-time optimization technique to exploit the motion cues from query video. 2) We introduce
formulation of the logarithmic decomposition of a video to perform a relighting task. This new formulation leads to significant improvement in performance when compared the current baseline methods. 3) We achieve state-of-the-an -art performances relevant downstream applications of video decomposition, such as video dehazing and relighting. VDP outperforms the existing inference-time optimization techniques for video object segmentation.
RELATED WORKS
Traditionally, priors played a pivotal role in transforming an 'ill-posed problem' (one with multiple plausible solutions) to well-posed problem. Past proposed priors (Wang & Adelson 1994 Shade
et al. 1998 Shechtman et al. 2002 2005; He et al. 2010) were very limited in their capability of learning good representation of videos to generalize well various scenes. All of this changed with the recent developments in deep learning techniques that considerably improve the representa- tion of videos and deliver high-performance gains on video editing tasks.
2

---

Published as a conference paper at ICLR 2024
Established works such (Assaf Shocher 2018 U yanoov et al. 2018: Bell-Kligler et al. 2019 Pollak Zuckerman et al. 2020; Shrivastava et al. 2023) are test time optimization techniques, i.c., they directly work the corrupted imput image/ video sequences and do not require any ground truth for training. These approaches have highlighted the importance of formulating a loss func- tion, combined with the optimization of neural network parameters, achieve significant results for both image and video restoration problems. However, these insights predominantly catered to limited vision tasks such as denoising, super-resolution, and inpainting. Using our approach, we are able to address video relighting. dehazing, and unsupervised video object segmentation tasks. We demonstrate this breadth of applications by leveraging the compositionality properties inherent in videos.
DoubleDIP (Gandelsman et al. 2019) employs a similar strategy, but their convolutional framework relies heavily on the appearance of the video when optimizing a task objective. This becomes limitation when working with videos, as motion plays critical role alongside appearance. Our method tackles this bottleneck by harnessing optical flow motion cue. This inclusion of optical flow not only captures motion effectively but also permits warping operations vital for downstream tasks such as edit propagation, ability absent in DoubleDIP's decomposition framework.
Other inference-time optimization methodologies, such as (Kasten et al. 2021, Lu et al. 2021 2020a Bar-Tal et al., 2022) are very task-specific and utilize singular form of video composition formulation, i.e., a--leendigg Techniques like these are heavily reliant on user-generated masks or image-centric baselines. In contrast these methods, our proposed method tackle wide vari- ctt of tasks. In addition, our framework does not rely on pretrained image segmentation networks, thereby enhancing its robustness, especially when confronted with unfamiliar objects in videos. Method (Ye et al. (022) is another inference-time optimization technique that avoids the use of pretrained image segmentation but encounters issues with objects whose appearances fluctuate over time, perhaps due to shifts in lighting. Such a limitation arises from their approach of encapsulating an object's 's global appearance in videos into a single canonical texture layer. contrast, our pro- posed methodology takes advantage of frame-by-frane appearance and motion cues, making our model significantly more resilient to variations in lighting.
3 VIDEO DECOMPOSITION PRIOR FRAMEWORK
Given video sequence denoted as {X,)T. this section delves into framework's methodology to decompose the video into multiple layers. To facilitate this decomposition, our framework requires an associated forward optical flow of the input video, represented {Fr-e+1]2 11 The method
for obtaining this optical flow is elaborated upon in the supplementary material (Supp. Sec. H). Subsequently, we will detail the design of the modules incorporated within VDP and outline the generic structure of our loss function.
3.1 MODULES OVERVIEW
VDP framework consists of the following two modules.
(I) RGB layer predictor module (RGBnet): RGBnet (functional notation fRCB) is a fully convolu- tional bottleneck network (Ronneberger et al., 2013). Given that we only optimize the weights over a single video, a shallow convolutional U-Net is sufficient for the task. RGBnet takes image frame X, of the video sequence as input and outputs a RGB layer (a channel layer same resolution as the input frame Xz).
(II) a layer predictor module (a-net): a-net (functional notation fa) architecture similar to RGBnet, i.e., fully convolutional bottleneck network. We utilize shallow convolutional U-Net for predicting the Tmaps or opacity layer. a-net takes flow-RGB (RGB representation of forward optical flow denoted as FRGB f feature maps of the video sequence input and outputs the Tmap/opacity layer (a channel layer of same resolution as the input frame X.).
More details about the convolutional U-Net architecture is discussed in Supp. Sec.
3

---

Published as a conference paper at ICLR 2024
1/v
RGB
RGB
Frame
Relit frame
*
(1/A,)
Reconstructed Frame
X (1/A)0(RGB) JM
Flow; I+1
Tmap
VDP for relighting a video sequence. In this pipeline, the input video frame is fed into shallow U-Net denoted by f!l) While the flow-RGB is given as input to separate shallow U-Net network denoted by fa The intermediate output of JRGB is re-lit version of the input frame While output of f(() transmission maps or Tmap denoted by 1/A,. It is important to note that > is also treated as trainable parameter in the pipeline. After obtaining the reconstructed frame we apply reconstruction and warping losses respectively to optimize f(I) f(l), y-
Figure 2:
3.2 VDP: OPTIMIZATION OBJECTIVE FOR GENERIC VIDEO DECOMPOSITION
VDP framework decomposes the input vidco into multiple layers and then composite these layers again reconstructs the same video input. In this process, the intermediate outputs generated by VDP are of particular interest. In this section, we will discuss a generic structure of the loss function that will help ensure temporal consistency and faithful reconstruction of the original input video.
Loss Objective: The loss objective used to optimize our framework for different tasks is given by the following Eqn]]
Cinal AreoCrec + Awarp Cwarp-
(1)
Reconstruction Loss: We use combination of L1 and perceptual loss (Zhang al. 2018) to facilitate faithful reconstruction of the original input video, which is given by,
Lee Σ1IX, - - (l((x) o(8.)1l1
where, o(.) denotes embeddings obtained from the pre-trained VGG network trained on Ima- genet (Deng et al. 2009).
Optical Flow Warp Loss: The intermediate output of interest X should follow the same temporal
coherence as the input video X, any given timestamp. This ensured by the optical-flow warp loss,
Cwarp = Σ (X) - X
(3)
where, Fi-1-+1 denotes the forward optical flow between the frames X and X,. Refer to Supp. Sec. [orr details about the optical-flow based operations.
The subsequent sections will discuss various applications of our VDP framework.
VIDEO RELIGHTING
During the nighttime, less amount of luminescence leads darker capture of scene. Traditionally, gamma correction is used to relight such captured scene gamma is what translates well between our eye's 's light sensitivity and that of the camera. Mathematically, gamma correction can defined given by the Eqn.
X A, (X
Vt (1,
where, X is the output of gamma correction at time step t. X is the input to which the gamma correction applied. A, and the hyperparameters of gamma correction. An image frame is

---

Published as a conference paper at ICLR 2024
Soabee
Figure 3: Qualitative evaluation on Video Relighting benchmark: We compare the re-lit result of method with the baselines the SDSD (52) dataset. We compare method against both the image and video baselines. ZeroDCE+ (24) method is a image-based baseline while SDSD (52) and Stablellve (59) are video baselines. Please note that we have utilized the pretrained models of all the baselines to obtain qualitative results. Methods SDSD and Stablellve require training on the external dataset, while our approach operates directly on the low-light video sequence.
Table I: Quantitative evaluation on Video Relighting benchmarks: For the evaluation of our method, we calculate the Avg PSNR and SSIM scores between the re-lit video and ground truth video. We use the pretrain model for the baselines that are publicly made available for calculating the PSNR and SSIM scores. For all baselines and our method, the score calculated on the SDSD dataset (52). Also, blue (bold) denotes the best score.
Metrics
7eroDCE++ (24)
SDSD (52)
StableI.LVE (59)
Ours
PSNR
26.46
24.92
26.01
27.92
SSIM
0.85
0.73
0.84
0.90
either darkened or lightened up based on the value of in Eqn. Eqn. 4 as follows,
For our model, we change the
1 (m = (log(1 @ A,) + (og(X."t)
4t E E1,7).
(5)
Where Ø operation denotes element-wise division.
The goal of the video relighting task to recover a video ((X²")) with good lighting conditions from its poorly lit counter-part video {X]")T We treat the relighting problem video decom- position problem, where one layer is the well-lit video. In addition, we model the transmission maps 1 A, with a-net model. Further, model value between (0,1). In summary, to find the well-lit video from the given input video, we initialize one RGBnet, one a-net, one trainable parameter 7-1, and the Eqn. 5 for reconstruction loss. This whole pipeline is given by Fig. 2 One advantage of utilizing the convolution nets to model RGBnet(FRRR) that convolutional nets provide a high impedance 2018) when reconstructing natural image. This property proves beneficial as merely applying gamma correction to dark image can introduce noise into the relit image.
trainable parameter that take
the noise (Ulyanov al.,
Our final loss objective to perform the relighting task is given by Eqn. I
We evaluate our method for the video relighting task the SDSD dataset WWng al. 2021 and compare our model against the existing state-of- of-the-art baselines. It is important to note that the SDSD dataset consists of low-light captured RGB videos as the input. We follow the same evaluation strategy for this dataset as the paper (Wang et al. (021). We choose the ZeroDCE++ (Li et al.) 202T), SDSD(Wang et al. (022), ) and Stablei /hann et al. 2021a) approaches as baselines. SDSD and StableLLVE approaches are trained using the RGB video dataset; hence, these methods act as perfect baselines.
A qualitative evaluation can be seen in Fig. 3 Note that ours and the ZeroDCE++ approach do not require any training data. Our approach is optimized directly on the inference-time video sequence instead of other approaches that have to be trained using external data. Table. [ represents the quantitative evaluation of our approach on the SDSD dataset. It can be observed from the table that method outperforms the existing baselines by significant margin in the video relighting task.
5

---

Published as a conference paper at ICLR 2024
Flow Simibrtty Lose Hlustrttion
Pramet
Laye1
LAVIE
Osgeclarer
Resonstruction Layer soss Ilustration
Recotsmacted Frame
M.Graza MaOncal
O. RaD
BIRE
Obeeclayer
Figure 4: Our Framework: In this figure, present the pipeline for the task of decomposing the video into two different components. For the above configuration, the input video frame is fed e(1) into two different shallow U-Nets denoted by JRGB and fRGB respectively. We obtain two different intermediate components the input frame t in the form of layer and layer 2. Additionally, we
process forward flow-RGBs using separate shallow U-Net network denoted by fa This network outputs a compositing maps, which are then used for blending both layers respectively to reconstruct the original input frame t. After the compositing, regularization loss) depending on the task at hand. (Right) Illustration of Flow Similarity loss: we define this loss over flowRGGB image (FRGB) This loss ensures that pixels with similar motions grouped together. Illustration of Reconstruction layer loss: We define this loss over the masked object layers and masked input frame X This loss ensures that the masked object layer such as McJRGB ((1) RGB has the same appearance as the masked input frame M,X,.
apply various losses (reconstruction loss and
5
UNSUPERVISED VIDEO OBJECT SEGMENTATION
Given video sequence, the aim of unsupervised video object segmentation (UVOS) is to coherently track and segment the most salient object in a video without any human annotations. Several past works such these (5] 7 13 20 29
30 34 44) have made attempts to segment the primary object in video without the use of any annotations. Unlike these works, our goal is not to detect semantic object categories but to find decomposition that explains the scene motion and its appearance.
We start by leveraging a-blending equation to write the reconstruction of the input video,
X, = Σ Mj o fiGB (X,) Vt E (1,
Here, Mf denotes the alpha map for the 7th object layer and tth frame. Further, Mj is obtained using the following,
Mj = (F EE E n),
where,FRCG RGB - is the RGB image of flow estimate from frame t 1 to Additionally, the following condition about the alpha maps should also satisfy
I.
Jn
ΣM,
Mi,
(8)
where, Jn,w denotes the all-ones matrix of size [h, w].
Since the problem of video decomposition is severely ill-posed, the loss objective given by Eqn. alone not enough to find visually plausible layered decomposition an input video. Addi- tionally, to find the right decomposition we need the opacity layers to be binary masks. Hence, we introduce a few other regularization losses to mitigate such challenges.
This loss ensures the motion of the layer is uncorrelated with the rest of the video layers. We define the loss as cosine similarity between the VGG embeddings of the masked flow-RGB of layer with the rest of the layers. The illustration for this loss is depicted in Fig. We can mathematically write this loss as follows.
Flow Similarity Loss:
((M O ERGB) .(((1 M;) O FRGB)
Craim
( FRGB II [ Mf) 4 PRGB
Here, (6-) in the Eqn. denotes the embeddings obtained by the pre-trained VGG network trained on ImageNet. FRGB is the RGB image of flow estimate from frame to

---

Published as a conference paper at ICLR 2024
Ground Truth
DoubleDIP
Defomable- sprites
DyStab
Ows
Figure 5: Qualitative evaluation on VOS benchmark: We compare the mask from our method with the baselines DAVIS-16 (32) dataset. We compare our method against the baselines DS (58) and DyStab (66). Please note the above masks (exception: DoubleDIP) for baselines are pre- computed masks provided by the authors the papers.
Table 2: Quantitative evaluation VOS benchmarks: For the evaluation of our method, calculate the IOU score (T) between the generated masks and the ground truth mask. Also, blue (bold) denotes the best score. Please note that the DyStab is trained using a video dataset, while all the other methods require only test time video sequence to segment the object from the video.
Model
Flow model
DAVIS (32
FBMS (47
DoubleDIP 15
24.6
ARP 21
CPMFlow 18
76.2
59.8
CIS 555
PWCNet 46
71.5
63.5
MG (54
RAFT 48
68.3
53.1
DS (58
RAFT 48
79.1
71.8
DyStab (56)
PWCNet (46)
80.0
73.2
Ours
RAFT 48
81.1
73.2
Mask Loss:
This loss ensures binarization in the generated layer mask. We utilized the mask loss
put forth by Gandelsman al. (2019). We can mathematically write this loss follows,
L
Σ (IM) 0.5l) -1 L,T
(10)
Reconstruction layer Loss: This loss ensures the appearance of the layer is a segment of the original input video. We define the loss as loss between the masked RGB image prediction of layer and the masked RGB image prediction of the original video. We can mathematically write this loss as follows,
Luyer
|| @ X, M; @ &.ll.
(11)
Here, X, the RGB frame in the video sequence time-step Mi is the alpha map for the 7th object layer and h frame.
Final Modified Loss: We combine all the losses together into a final loss term as follows,
Luvos =rrechree + Arsim Crsim + AtyerrSuaer
(12)
+ Xwarp + AMask Mass
We leverage our final modified loss objective, given Eqn. 12 to perform unsupervised video object segmentation. For this task, we want to split an input video into foreground and background layers. Hence, initialize 2 RGBnets (JRGR) and onet (Ja). Weights for all these modules are optimized using the test sequence only. The illustration of this shown in the Fig. 4 Because are dealing with decomposition of the input video into two layers, i.e., T. 2, modify Eqn. a follows,
X, = M, C fian (X)) + (1 M,) O fean (X,) Vt E (1,
(13)
All the loss functions in the Eqn. 12 are modified with the value of L 2. We fit our model parameters using loss function given by Eqn. 12 We then use the learned a-net to generate a segmentation mask for the input video. The predicted segmentation masks for each timestamp can be obtained by M, fa (FRGR FRG (-1 the RGB image of flow estimate from frame t to l.
FRGB

---

Published as a conference paper at ICLR 2024
(a) Input Frame
Edited Frame
Ours
Deformble Sprites
Naive Flow
t U
(b) Input Frame
Edited Frame
Propagated Frames
Figure 6: Edits propagation: First column the figure represents the input frame to be edited. Second column represents the edits performed. Next columns in Fig. (a) represent the qualitative comparison propagated edits with baselines, while Fig. (b) represents propagated edits throughout the video. From Row 1, 2- it can be observed that the quality of our framework's propagated edited decals is much better as compared to the baselines. The legs of the camel(Row 1) and parachute bag (Rop 2) are preserved in our method's edit propagation. Row 3 It can be observed that consistent edits in particular layer can be performed throughout the video. In this case, we show an example of style transfer being performed on the swan. Row 4 we can selectively replace the background layer and propagate the edits through the video.
We evaluate our method for unsupervised video object detection on the DAVIS-16 dataset (Perazzi et al. 2016) and compare our model against the existing baselines. A qualitative comparison be- tween our approach and the baselines can be seen in Fig- 5 It can be observed from Fig. that our method performs better than the existing methods in terms of segmenting the most salient object from the scene. Table. presents a quantitative evaluation of our approach. Our method outperforms the existing inference-tines optimized baselines in the unsupervised video object segmentation set- ting. DoubleDIP is the worst-performing baseline our evaluation.
We also perform an ablation study of our framework and refer our readers to the supplementary materials.
5.1 COHERENT EDITS PROPOGATION
able perform downstream applications like coherently editing the foreground and background object layers in a scene. can be observed from Fig. the type of edits we can perform on one layer in some keyframe, which can then be propagated throughout the video. The distinct advantage of our method over other such methods is that the user can perform the edit on any keyframe which has a better view of the object. This makes it easier for the user make the edit instead of making it in one global atlas for the object layer with distorted single view. In Fig. 6 we perform different edits such as add decals or modify keyframe with style transfer Park et al. (2200) or swap the background. These edits are then consistently propagated using combination of flow warping operation and composition using the obtained masks. We refer readers to our webpage in the supplementary material for video results.
With approach,
6 VIDEO DEHAZING
Videos captured in outdoor settings are often degraded by a scattering medium (c.g., haze, fog, underwater scattering). The degradation is such that it grows with the depth of the scene. Typically, we can write hazy video ( (X,)T as put forth by papers (He et al. 2010: Gandelsman et al. 2019 Fattal 2008 Narasimhan & Nayar 2000):
X, = a O Clr(X,) + (1 A
(14)
where A, is the Airlight map (A-map), Clr(X,) is the haze-free image, and here is the transmission (t-map).The goal of video dehazing is to recover the clear underlying video (C))( from a hazy input video (X,) iie., recover the input video that would have been captured on clear day with
8

---

Published as a conference paper at ICLR 2024
Iepa
FIVE
MSSDN
CG-ID
DuiNeDuP
Our
Ground Truh
Figure 7: Qualitative evaluation on Video Dehazing benchmark: We compare the dehazed result of our method with the baselines on the REVIDE (61) dataset. We compare our method against both the image and video baselines. MSBDN (10) method is an image-based baseline while EDVR (53), DoubleDIP (15) and CG-IDN (6I) are video baselines. Please note that both the EDVR and CG- IDN methods are trained using the training data of the REVIDE dataset. In contrast, our approach operates directly on the hazy video sequence. As can be seen from the figure, our approach to dehazing produces color palette the dehazed frame that closely resembles the color palette in the ground truth frame.
Table 3: Quantitative evaluation on Video Dehazing benchmarks: To evaluate our method, we calculate the Avg PSNR and SSIM scores between the generated haze-free video and ground truth video. For all baselines and our method, the score is calculated on the REVIDE dataset (61).
Metries
EDVR (53
DoubleDIP [55]
MSBDN [10]
CG-IDN (61)
Ours
PSNR
21.22 0.8707
17.32
22.01
23.21
24.83
SSIM
0.7942
0.8759
0.8836
0.9164
good visibility conditions. We treat the dehazing problem video decomposition problem, where one layer is the haze-free video and the other layer is the Airlight map. This Airlight map is obtained using the standard methodology proposed in (Bahat & Irani (2016) and kept as fixed (non-trainable) layer. Further, we model the transmission t-maps with our a-net model. In summary, to find the dehazed video from the given input video, we initialize one RGBnet, one a-net, and use the Eqn. [4 for reconstruction loss.
To optimize the model parameters, we optimize the objective function given by Equ. We evaluate our method for the video dehazing task on the REVIDE dataset (Zhang et al., 2021b) and compare our model against the existing state-of-the-an baselines. It is important to note that the REVIDE dataset consists of real-world captured hazy RGB video and clean video pairs. REVIDE dataset has total of 48 sequence pairs (hazy seq, gt seq). We choose both image-based and video-based approaches as baselines. MSBDN (Dong al. 2020) is image based dehazing baseline, while DoubleDIP (Gandelsman et al. 2019), EDVR (Wang et al. 2019) and CG-IDN (Zhang et al. (221bb) the video based baseline. EDVR and CG-IDN approaches trained using the videos of the REVIDE dataset; hence, these methods act as perfect baselines.
A qualitative evaluation can be seen in Fig. 7 Note that only and DoubleDIP approaches do not require any task-specific training data. Both approaches are optimized directly on the inference- time video sequence instead of other approaches that have to be trained using external data. Table. 3 represents the guantitative evaluation of our approach. It can be observed from the table that our method outperforms the existing baselines by significant margin the video dehazing task. Dou- bleDIP performs worst among the baselines in our quantitative evaluation. We also refer the readers to an important ablation study detailed Sec. F where we examine the importance of FlowRGB utilization for relighting and dehazing tasks.
7 CONCLUSION
We have demonstrated in the paper how the layered decomposition could be applied to a wide variety of downstream tasks. It is important to note that our model does not require any explicit training, i.e., we optimize the parameters using the test sequence itself. Additionally, in this paper, we achieve state-of-the-art performances in downstream tasks such as UVOS (among the other test time optimization techniques), dehazing, and relighting. We also demonstrated that the targeted edits propagation can also be performed downstream task of UVOS. With the help of semantic or perception cues, we believe our framework can also be extended for monocular depth estimation, semantic segmentation, and other high-level vision tasks.

---

Published as a conference paper at ICLR 2024
Acknowledgements. This project was partially funded by NSF CAREER Award (2238769) to AS. We also thank Harsh Shrivastava, Saurabh Singh, Matthew Walmer, Soumik Mukhopadhyay, Shirley Huang and Bo He for providing feedback on the manuscript.
REFERENCES
Michal Irani Assaf Shocher, Nadav Cohen. "zero-shot" super-resolution using deep internal learn- ing. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2018.
Yuval Bahat and Michal Irani. Blind dehazing using internal patch recurrence. In 2016 IEEE International Conference on Computational Photography (ICCP), pp. 1-9. IEEE, 2016.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In European Conference on Computer Vision, pp. 707- 723. Springer, 2022.
Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan, Advances in Neural Information Processing Systems, 32, 2019.
Pia Bideau and Erik Learned-Miled It's moving! a probabilistic model for causal motion seg- mentation in moving camera videos. In European Conference on Computer Vision, pp. 433-449. Springer, 2016.
Navaneeth Bodla, Gaurav Shrivastava, Rama Chellappa, and Abhinav Shrivastava. Hierarchical video prediction using relational layouts for human-object interactions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12146-12155, 2021.
Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In European conference on computer vision, 282-295, Springer, 2010.
Ioana Croitoru, Simion-Vii Bogolin, and Marius Leordeanu. Unsupervised learning from video to detect foreground objects in single images. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4335-4343, 2017.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L., Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang- Multi-scale boosted dehazing network with dense feature fusion. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2157-2167, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020.
Muhammad Faisal, Ijaz Akhter, Mohsen Ali, and Richard Hartley. Exploiting geometric constraints on dense trajectories for motion saliency. arXiv preprint arXiv:1909.1 13258, 3(4), 2019.
Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In BMVC, number pp. 8, 2014.
Raanan Fattal. Single image dehazing. ACM transactions on graphics (TOG), 27(3) 2008.
Yosef Gandelsman, Assaf Shocher, and Michal Irani. " "ouble-dip":: Unsupervised image decompo- sition via coupled deep-image-priors. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11026-11035, 2019.
Kaiming He, Jian Sun, and Xiaoou Tang. Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 33(12))3341-2353 -2353, 2010.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
10

---

Published as a conference paper at ICLR 2024
Yinlin Hu, Yunsong Li, and Rui Song. Robust interpolation of correspondences for large displace- ment optical flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 481 1499, 2017.
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing, ACM Transactions on Graphics (TOG), 40(6):1-12, 2021.
Margret Keuper, Bjoem Andres, and Thomas Brox. Motion trajectory segmentation via minimum cost multicuts. In Proceedings of the IEEE international conference on computer vision, pp. 3271-3279, 2015.
Yeong Jun Koh and Chang-Su Kim. Primary object segmentation in videos based on region aug- mentation and reduction. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7417-7425. IEEE, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances neural information processing systems, 25, 2012.
Hala Lamdouar, Charig Yang, Weidi Xie, and Andrew Zisserman. Betrayed by motion: Camou- flaged object discovery via motion segmentation. Asiun Conference on Computer Vision, 2020.
Chongyi Li, Chunle Guo, and Chen Change Loy. Learning to enhance low-light image via zero- reference deep curve estimation. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2021. doi: 10.109//PAMM.202133300
Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William T Freeman, and Michael Rubinstein. Layered neural rendering for retiming people in video. arXiv preprint arXivv2009.0753 2020a.
Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William Freeman, and Michael Rubin- stein. Omnimatte: Associating objects and their effects in video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4507-451 2021.
Xiankai Lu, Wenguan Wang, Jianbing Shen, David Crandall, and Jiebo Luo. Zero-shot video object segmentation with co-attention siamese networks. IEEE transactions pattern analysis and machine intelligence, 4444):2228-22) 2242, 2020b.
S.G. Narasimhan and S.K. Nayar. Chromatic framework for vision in bad weather. In Proceedings IEEE Conference Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. No.RR006)2). volume 1, pp. 598-605 vol.1, 2000. doi: 10.1109/CVPR.2200885588
Peter Ochs and Thomas Brox Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions. In 2011 international conference computer vision, pp. 1583-1590. IEEE, 2011.
Anestis Papazoglou and Vittorio Ferrari. Fast object segmentation unconstrained video. In Pro- ceedings of the IEEE international conference on computer vision, pp. 1777-1784, 2013.
Taesung Park, Jun- Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 33:7198 13:7198-7211, 2020.
Jun-Yan
Perazzi, Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and Sorkine-Hornume A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vi- sion and Pattern Recognition, 2016.
Liad Pollak Zuckerman, Eyal Naor, George Pisha, Shai Bagon, and Michal Irani. Across scales and across dimensions: Temporal super-resolution using deep internal learning. In European Conference on Computer Vision (ECCV). Springer, 2020.
Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12240-12249, 2019.

---

Published as a conference paper at ICLR 2024
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 134-241.32 Springer. 2015.
Nirat Saini, Bo He, Gaurav Shrivastava, Sai Saketh Rambhatla, and Abhinav Shrivastava. Recogniz- ing actions using object states. In /CLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In Pro- ceedings of the 25th annual conference on Computer graphics and interactive techniques, pp- 231-242, 1998.
Eli Shechtman, Yaron Caspi, and Michal Irani. Increasing space-time resolution in video. In Com- puter Vision Vision-FCCC ECCV 2002: 7th European Conference on Computer Vision Copenhagen, Den- mark, May 28-31, 2002 Proceedings, Part I pp. 753-768. Springer, 2002.
Eli Shechtman, Yaron Caspi, and Michal Irani. Space-time super-resolution. IEEE Transactions on Puttern Analysis and Machine Intelligence, 27(4):531-545 2005.
Gaurav Shrivastava. Diverse Video Generation. PhD thesis, University of Maryland, College Park, 2021.
Gaurav Shrivastava and Abhinay Shrivastava. Diverse video generation using a gaussian process trigger. arXiv preprint arXiv:2107.04119, 2021.
Gaurav Shrivastava and Abhinav Shrivastava. Video prediction by modeling videos continuous multi-dimensional processes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2024.
Gaurav Shrivastava, Ser-Nam Lim, and Abhinav Shrivastava. Video dynamics prior: An internal learning approach for robust video enhancements. In Thirty-seventh Conference on Neural Infor- mation Processing Systems, 2023.
Josef Sivic, Frederik Schaffalitzky, and Andrew Zisserman. Object level grouping for video shots. International Journal of Computer Vision, 67(2):189-218 2006.
Hongmei Song, Wenguan Wang, Sanyuan Zhao, Jianbing Shen, and Kin-Man Lam. Pyramid dilated deeper convistm for video salient object detection. In Proceedings of the European conference computer vision (ECCV), pp. 715-731, 2018
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8934-8943, 2018
T.Brox and M.Mliik, Object segmentation by long term analysis of point trajectories. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science Springer, Sept. 2010. URL http:://lbb informatik.. uni -tteeiborrgeeeee.iicit.oooooooo. 0c
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 419. Springer, 2020.
Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid. Learning motion patterns in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3386-3394, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446-9454, 2018.
John YA Wang and Edward H Adelson, Representing moving images with layers. IEEE transactions image processing, 3(5):625-633) 1994.
Ruixing Wang, Xiaogang Xu, Chi-Ve thinWing Fu, Jiangbo Lu, Bei Yu, and Jiaya Jia. Seeing dynamic scene in the dark: A high-quality video dataset with mechatronic alignment. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 9700-9709, 2021.
12

---

Published as a conference paper at ICLR 2024
Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 7177-7188, 2021a.
Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 879-888, 2019a.
Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2826-2836, 2021b.
Zhao Yang, Qiang Wang, Luca Bertinetto, Weiming Hu, Song Bai, and Philip HS Torr. Anchor dif- fusion for unsupervised video object segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 931-940, 2019b.
Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsupervised video decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2657-2666 2022.
Fan Zhang, Yu Li, Shaodi You, and Ying Fu. Learning temporal consistency for low light video enhancement from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4967-4976, 2021a.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Xinyi Zhang, Hang Dong, Jinshan Pan, Chao Zhu, Ying Tai, Chengjie Wang. Jilin Li, Feiyue Huang, and Fei Wang. Learning restore hazy video: A real-world dataset and method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 9239-9248, 202Ib..
Tianfei Zhou, Shunzhou Wang, Yi Zhou, Yazhou Yao, Jianwu Li, and Ling Shao. Motion-attentive transition for zero-shot video object segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13066-13077 2020.
13

---

Published as a conference paper at ICLR 2024
Each convolution followed by batch normalization and leaky rectified linear unit (LeakyReLU; negative slope = 0.2), except for the last layer of the encoder and decoder. In the last layer of the encoder and decoder, we use the sigmoid activation function. We utilize Pytorch for our implementation.
Table RGBnet:
Module
Type
Kernel Size
Num Inputs
Num Outputs
Stride
Padding
Conv2D
3
3
64
Conv2D
3 X 3
64
64
Maxpoo12D
2
64
64
Conv2D
64
96
Conv2D
3 x
96
128
Encoder
Maxpoo12D
x
128
128
Conv2D
3 x
128
128
Conv2D
3
128
128
Conv2D
3x3
128
128
Maxpoo12D
2 x
128
128
Conv2D
4
128
96
Conv' ConvTranspose2
4 x 4
96
128
Bicubic upsample
(scale =2)
Conv2D
x
128
128
Conv2D
3 x
128
128
Conv2D
3
128
128
Decoder
Bicubic upsample
(scale =2)
Conv2D
x 3
128
96
Conv2D
x
96
64
Bicubic upsample
(scale =2)
Conv2D
x 3
64
64
Conv Transpose2D
x
64
3
A NETWORK ARCHITECTURE FOR RGBNET AND Q-NET
RGBnet is a fully convolutional network. It is designed in such way as to handle frames of all resolutions. The input to this network is RGB frame at time t. We utilize U-net architecture for this network without the skip connections. More details about the architecture can be found in Table. 4 Similarly, more details about a-net can be found Table.
B OPTIMIZATION DETAILS
To optimize model, video at time of resolution 856x480. We optimize the module (fRGB(-) and f-)- weights using the entire test sequence with the Adam optimizer Because our approach is non-recurrent, can be scaled up by parallelizing over multiple GPUs, which speeds up the process for longer sequences. We achieve processing rate of 60 frames/min for dehazing and relighting and 40 frames/min for VOS with one salient object. However, we can speed up the VOS task by reducing the dimension from 856x480 224x128 and then running the VOS task pipeline. This results in 16x speed-up of the pipeline, increasing the processing rate 640 frames/min without affecting the efficacy our method. We use 100 epochs for 60- frame sequence in VOS and 60 epochs for dehazing and relighting. Additionally, we conducted our experiments using multiple random initializations of fRCB (-) and fa(.) modules and observed consistent quantitative results across all runs.
single Nvidia A6000 GPU with 48G memory process single
learning rate in the range of [0.00002, 0.002].
B. HYPER PARAMETER SETTING
Unsupervised Video Object Segmentation: For getting good performance the UVOS task
we utilize the following weights for the different losses; Ares = 1, Arsim = 0.001, Atayer = 1, Awarp 0.01 and AMak 0.01.
14

---

Published as a conference paper at ICLR 2024
Table 5: a-net: Each convolution layer is followed by batch normalization and leaky rectified linear unit (LeakyReLU; negative slope = 0.2). except for the last layer of the encoder and decoder. In the last layer of the encoder and decoder, we use the sigmoid activation function. We utilize Pytorch for our implementation.
Module
Type
Kernel Size
Num Inputs
Num Outputs
Stride
Padding
Conv2D
3
3
64
Conv2D
3 x 3
64
64
Maxpoo12D
x 2
64
64
Conv2D
3
64
96
Conv2D
3 x 3
96
128
Encoder
Maxpoo12D
x
128
128
Conv2D
3
128
128
Conv2D
3
128
128
Conv2D
3x3
128
128
Maxpool2D
128
128
Conv2D
4 4
128
96
Conv SonvTranspose2
4 x 4
96
128
Bicubic upsample
(scale =2)
Conv2D
3
128
128
Conv2D
3 xx3
128
128
Conv2D
x
128
128
Decoder
Bicubic upsample
(scale =2)
Conv2D
x 3
128
96
Conv2D
x 3
96
64
Bicubic upsample
(scale =2)
Conv2D
xx3 3
64
64
ConyTranspaa Transpose2D
x
64
1
Input Sequence
Mask RCNN
Ours
Figure 8: Comparison with pretrained image segmentation model: The examples presented in the figure above are samples from the MoCA dataset (Lamdouar et al. 2020). In comparison MaskRCNN (He et al. (017). pretrained image segmentation model trained on the COCO dataset, our model is more robust in handling unfamiliar objects in the scene. The figure shows some examples where MaskRCNN fails segment the unfamiliar object correctly, while our approach produces much better segmentation results.
Unsupervised Video Dehazing: For getting good performance on the Dehazing task we utilize the following weights for the different losses; \ and Awarp 0.02.
Unsupervised Video Relighting: For getting a good performance on the Relighting task we utilize the loss weights similar to the Dehazing task, i.e., Arec 1 and \warp 0.02.
15

---

Published as a conference paper at ICLR 2024
Input Sequence
MG
Reformable-sprites
Ours
Input Video
Layers =
Layers =3
bb)
obj
obi
Figure 9: (a) Segmentation in variable lighting conditions throughout the scene. We compare the mask from our method with the baselines Deformable sprites (Ye et al.) 2022) and MG (Yang al. 202la) on frames of three video sequences. It can be observed from the above figure methods such as Deformable-sprites and MG are more prone to lighting changes as opposed to our method. (b) Decomposition into multiple layers We demonstrate an example where the number of layers L is set to both L = 2 and L = 3. L = results in foreground and background layers separation, while results the separation of individual person different object layers.
C UVOS
C.1 UVOS SCENE WITH IRREGULAR OBJECTS
we demonstrate that our method doesn't mandate the object to belong to common cat- egory to achieve the correct decomposition. This is in stark contrast to other methods like that of Kasten et al. 2021), which heavily relies on MaskRCNN or user-input mask for video decomposi- tion. This fiexibility provides our approach with an advantage, allowing for precise video foreground (FG) and background (BG) edits and ensuring scamless propagation throughout the sequence.
In Fig. 8
C.2 UVOS. LIGHTING CHANGES
It's notable that the efficacy of deformable sprites (Ye et al., 2002) tends to deteriorate in video sequences where the appearance of an object undergoes alterations due to shifts in lighting. This decrement in performance emerges primarily because deformable sprites attempt to encapsulate an framework
object's universal appearance using singular canonical RGB texture. In contrast, operates by modeling video decomposition at an individual frame level. Such an approach amplifies the resilience of the object segmentation process against lighting modifications, fact that is evident from Fig. (a).
C.3 UVOS DECOMPOSITION INTO MULTIPLE LAYERS
Our model's capabilities not confined to binary layer decomposition (Foreground layer and Background layer). As depicted in Fig. (b), upon configuring the layer parameter to 2, our model precisely segregates the foreground from the background. Intriguingly, when the layer parameter is adjusted to 3, the model successfully isolates each person into their distinct layer while maintaining separate layer for the background.
16

---

coupled version of U-nets in framework to Published as a conference paper at ICLR 2024
Groun Truth
w/o Rec
w/o Fsim
w/o RLayer
w/o warp
All Losses
Figure 10: Ablation Study: We study the effect of using only a subset of the loss function for the
video object segmentation task.
Low-lit lit Input
Our w/ Flow
Ours w/ Gaussian Noise
Ours w/ RGB Frame
(a)
Hazy Input
Ours w/ Flow
Ours w/ Gaussian Noise
Ours w/ RGB Frame
Figure 11: Ablation Study:This figure presents the results of an ablation study where analyzed the impact of using FlowRGB, Gaussian Noise, and RGB frame as input to the a-net. Row (a) shows a qualitative comparison for relighting tasks, where we evaluated three different settings for the a-net. Row (b) presents comparison for the dehazing task.
C.4 UVOS ABLATION STUDY
We performed ablation experiments to study the effect of losses for unsupervised video object segmentation method. We report the quantitative results on DAVIS-16 16 dataset Table. 6 In Fig. T0. we show a qualitative comparison between architectures optimized with different subsets of losses given by Equ. 12. can observed from Fig. 10 that removing the reconstruction loss drastically reduces the ability of our method segment the object of interest in the scene. This also reflected in the quantitative evaluation given by Table. 6 Dropping the frsim reduces our model's ability to separate the motion of the foreground object from the background scene. Opti- mizing Eqn. 12 without the Llayer results in performance drop as the obtained object mask lacks the appearance grouping provided by the addition of the layer loss. Lastly, dropping the Lwarp reduces the temporal consistency of the masks throughout the sequence.
C.5 COMPLETE VOS EVALUATION FOR DAVIS-16
We include the supervised VOS approaches along with the unsupervised approaches for the com- pleteness of the evaluation. It is important note that unsupervised in the context of this paper implies that no human annotation is required either for the training or testing process of the ap- proach. Table. rreflects the evaluation of the baselines the DAVIS-16 dataset for VOS task.
D INTUITION FOR DEHAZING/RELIGHTIN TASKS
Dehazing: Previous studies, such as Bahat Irani (016), have shown that small image patches are recurrent in natural images, but this recurrence is disrupted in non-ideal conditions such as haze/noise. Gandelsman et al. (2019) established this patch recurrence property could be exploited using coupled DIP (Ulyanov al. (2018). When combination of multiple DIPs is used to recon- struct a frame, those DIPs tend to "split" a frame such that the patch distribution each DIP output follows high patch recurrence. Hence, when
17

---

Published as a conference paper at ICLR 2024
Table 6: Ablation Study: We utilize the DAVIS-16 dataset for performing the ablation study, We
calculate the IOU score (I) between the generated mask and the ground truth mask.
w/o Lree
w/o LFsim
w/o Llayer
w/o Cwarp
All Losses
Davis
52.6
67.3
72.8
78.5
81.1
Table 7: Quantitative evaluation on Dehazing/Relighting ablations: This Table presents the sults of quantitative evaluations conducted on dehazing and relighting tasks using three different settings our method. To assess the performance of our approach, we measured the average PSNR and SSIM scores between the processed videos and ground truth videos. For dehazing evaluations, we used the REVIDE (Zhang et al. 2021b) dataset, and for relighting evaluations, we used the
SDSD (Wang et al. 2021) dataset.
Metrics
Task
Ours w/ FlowRGB
Ours w/ Gaussian Noise
Ours w/ RGB
PSNR
Dehazing
24.83
16.79
16.84
SSIM
Dehazing
0.9164
0.7768
0.7721
PSNR
Relighting
27.92
23.74
24.52
SSIM
Relighting
0.9046
0.7221
0.7342
reconstruct the haze frame, it always converges into natural image with repetitive patches (termed as haze-free image) and non-linear airlight map.
For the relighting task, we begin by defining a manifold for the relit frame, which is based on the gammaa correction formulation delineated Eqn. 5. Our architectural choice incorporates convolutional U-Nets, leveraging their inherent property to enhance self-similarity in the produced outputs. Notably, natural images, particularly those captured in optimal conditions, consistently ex- hibit the patch recurrence property. When we employ the U-Net for optimization, its output (termed as 'relit' frame) reliably gravitates towards a natural image marked by repetitive patches. Coneur- rently, the system fine-tunes and selects the appropriate parameters for 7 and Tmaps (1/A,). A salient feature of the U-Net's output is its pronounced patch recurrence, which, in turn, translates to substantial reduction in noise levels for the relit frame. This inherent noise reduction alleviates the need for a subsequent, explicit denoising process.
E ABLATION FOR DEHAZING AND RELIGHTING TASKS
We perform ablation studies of our framework for dehazing and relighting tasks. We evaluate our method utilizing three different settings, namely
First setting, where we provide flow-RGBs as an input to a-net (ours VDP prior setting),
Second setting, here, we provide Gaussian Noise as an input to a-net.
Third setting, in which we provide an RGB frame as an input to a-net.
We report the quantitative evaluation REVIDE dataset for dehazing and SDSD dataset for re- lighting tasks. Please refer to Table. and Fig. [T for the findings. can be observed that using flow-RGBs input a-net provides significant gain over the other settings. We attribute this be- havior to information obtained from neighboring frames that help the model to refine the Tmaps further and produce results with higher fidelity.
important to note that in our experiments, we initially attempted to employ raw (x.y) offsets feature maps for input to the a-Net. However, this approach exhibited suboptimal normalization performance. Conversely, the colorization of optical flow effectively normalizes flow-rgb values within the range [0,1], proving to be more suitable input for the a-Net.
18

---

Published as a conference paper at ICLR 2024
Table Quantitative evaluation on VOS benchmarks: This Table presents the results of quan- titative evaluation of our method on VOS benchmarks. To assess the performance of our approach, we calculated the IOU score (J) between the generated masks and the ground truth mask on the DAVIS dataset (Perazzi et al. 2016). The scores were computed for all baselines and our method. It important to note that the methods listed above the line require no annotated data for performing the VOS task, while the methods mentioned below the line require annotated data for training. The best score in each category is highlighted blue (bold).
Model
Amnotations
Annotations
DAVIS PPaazi
2016
HBMS
2010
DoubleDiPP (iandeloman of al. 2019
24.6
ARP
Koh Kim 2017
76.2
59.8
MG Yang 0 M74
68.1
53.1
DS
79.1
71.8
DyStan Yangg
73.2
Ours
81.1
73.2
COSNet ct
AnDiff
sag el
20196
2,000
81.7
EPON
Finsal er
2019
2,0000
80.6
Sont
LVO
Tokamaoo 01
2
2,000+
75.9
63.1
MATNer ZZwi Br 2020
14,000
82.4
76.1
Yang
Table 9: Quantitative evaluation of Video Edits Propagation: For the evaluation of our method, calculate the Avg PSNR and SSIM scores between the reconstructed videos and ground truth video sampled from the DAVIS-16 dataset 32 We use the code for the baselines that are publicly available calculate the PSNR and SSIM scores. For all baselines and our method, the score is calculated by averaging the PSNR and SSIM value over sequence length of 15 frames (frames:
2-16).
Metrics
Naive Flow
Deformable Sprites Ye et al. 2022)
Ours
PSNR
22.54
27.50
29.73
SSIM
0.4527
0.9543
0.9726
F EDIT PROPAGATION
In the case of Salient Object Eoylization/Backrroundd swapping, we utilize X, as the edited frame. The forward optical flow Fi-1++1 is obtained using the original input video. Given a frame X, at time and the forward flow Fr-1+1 from frame to frame + we can define the warped frame X/+1 using the following relation:
(4, Xi(u + Au, Av)
(15)
Where Au and Av are the horizontal and vertical displacements, respectivlyy,, from the forward flow Fe-se+1 at pixel location (u, v).
Furthermore, let M4+1 be binary mask time 1 where its values are either 0 or 1. The
prediction X..i for pixel at location (u, then obtained by:
(u,(u, v) (4 o (xti(u, + M,))(u, o (
(16)
This formulation suggests that the prediction X.,1 weighted combination of the actual frame X and the warped frame X/41, where the weighting is determined by the mask M
F.1 EDITS EVALUATION
To quantitatively evaluate the effectiveness of our approach for propagating coherent edits, we con- ducted proxy experiment. Rather than editing the frame, applied the warping operations without
19

---

Published as a conference paper at ICLR 2024
modifying the original frame. We then evaluated the quality of the reconstructed warped frames and compared our results against the baselines. The findings of this experiment are presented Table. which shows that our method outperformed all of the other baselines.
G LIMITATIONS OF OUR APPROACH
Our method has four limitations that must be considered. First, the accuracy of the decomposition is dependent on the effectiveness the optical flow estimator network (RAFT). This means that any inaccuracies limitations of the network will directly impact the quality of the results. Second, the number of separable layers is a hyperparameter that must determined prior to implementation. Third, If the number of separable layers is set to N, then 2N independent fRGB and fa need to be initialized, which can significantly increase computational costs. Finally, it's worth noting that method is test-time optimization technique and may be slower than its trained counterparts. However, as mentioned earlier, our model can be sped up by resizing the frame size for the VOS task or by utilizing multiple GPUs for optimization. These limitations must be taken into consideration when implementing our method in practice.
H OPTICAL FLOW
Optical flow pertains to the pattern visible motion of objects, surfaces, and edges within a visual scene. It provides estimation of the motion of objects between consecutive frames, which results from the relative movement between the object and the camera. Unless mentioned otherwise pretrained RAFT (Teed & Deng. 2020) model to predict the optical-ftow between the two consecutive frames.
Given two successive frames Xz Xe(u, v) and X v), the goal of optical flow is to identify displacement vector (Au, Ar) for each pixel (u, v) such that:
Xz(u, v) X Au, v Ao)
(17)
Here, Au and Au represent the horizontal and vertical displacements, respectively.
I OPTICAL FLOW WARPING OPERATION
Optical flow warping denotes the application of the computed optical flow vectors to "warp" or "remap" one image in order to predict the subsequent image in the sequence. For each pixel in the present frame, its position in the next frame is forecasted via the optical flow, transferring the intensity to this new location.
Given frame X, and its optical flow Fi-1441 (Au, Au), the warped image X for the next time frame + can be expressed as:
Xi(u + Au, + Av) (u)
(18)
J OPTICAL FLOW RGB REPRESENTATION
To visualize optical flow vectors, one can represent them in an RGB format. The 2D optical flow vectors (Au, Av) are first converted into polar coordinates (r, (r,0). The hue in the HSV color space corresponds to the motion direction, while the intensity or value indicates the motion magnitude.
To transition from (Au, Av) (r, (r: V Au² A
arctan 2(Av, Au)
(19)
(20)
Subsequently, 9 and r are proportionally scaled and mapped to their respective HSV values. After- ward, this HSV image visualization, there's s often an imposed limit on the maximum value r. Thus, any motion exceed- ing this threshold is typically visualized maximum brightness.
be converted into RGB for effective visualization. Note that to enhance
20


Reviews:
[{"review_id": "t3VxCUNR8b", "reviewer": "Reviewer_u6Cn", "summary": "This paper attempts to propose a general framework for video editing. It starts by predicting the video into several individual layers and then utilizes task-specific knowledge to reconstruct the video. A reconstruction loss and a warping loss are utilized to train the network. This is an inference-time optimization framework that does not rely on external training tools. However, the framework is rather standard, and the choice of a few loss functions for the decomposition tasks is straightforward. Overall, the novelty is limited. I am inclined to reject this article. Please check the details in other sections.", "questions": "See weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The motivation to propose such a general framework is good.", "weaknesses": "-: The reconstruction loss and warp loss used are very common loss functions and do not offer any novelty.\n\n-: The method is still not sufficiently general. For example, for the task of video segmentation, additional loss functions need to be designed as constraints. The three task-specific losses mentioned in Section 3.2 are the only ones provided. From this perspective, it is difficult to see what this framework proposed in the paper brings to this community.\n\n-: The comparison results of the experiments are not that fair. For instance, in Table 3, the comparison is made with the latest algorithm, CG-IDN (a 2021 algorithm). There are many dehazing algorithms that could be compared, including single-image hazing algorithms with stability processing (references [1][2]).\n\n-: Additionally, Table 2 lacks many baselines. Looking at the official website of DAVIS2016, the best baseline achieves an IOU score of over 82. Why wasn't this paper compared against that?\n\n-: Sometimes it is necessary to introduce additional prior knowledge, such as for the task of dehazing, where the effectiveness is actually limited.\n\n[1] Learning Blind Video Temporal Consistency\n\n[2] Blind Video Temporal Consistency via Deep Video Prior", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yN9lSoPMjV", "reviewer": "Reviewer_8QGm", "summary": "This paper introduces the Video Decomposition Prior (VDP) framework, a new approach for video editing tasks, including object segmentation, dehazing, and relighting. VDP decomposes videos into multiple layers and optimizes parameters without explicit training. The proposed logarithmic video decomposition enhances video relighting, resulting in state-of-the-art performance in downstream tasks: unsupervised video object segmentation, dehazing, and relighting.", "questions": "Please see the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Sound approach: The VDP framework presents an innovative approach to video editing, offering practicality and cost-effectiveness by not relying on extensive datasets or ground truth annotations. The ability to optimize parameters using the test sequence itself distinguishes VDP from traditional deep learning methods, which often require extensive training data.\n\n2. State-of-the-Art Performance: VDP demonstrates top-tier performance in key downstream tasks, including unsupervised video object segmentation, dehazing, and relighting.\n\n3. Good writing and representation.", "weaknesses": "I am generally positive about this paper. my main concern lies in the lack of comprehensive comparison: The paper does not provide a comprehensive comparison with other video editing techniques, making it difficult to assess the VDP framework's performance against other state-of-the-art methods. Furthermore, video editing is a broad field, including tasks such as adding or removing objects. It seems that the proposed method may not be suitable for handling these scenarios. Therefore, I suggest that the authors consider refining the title and corresponding claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cgPJtQurJw", "reviewer": "Reviewer_XPSu", "summary": "This paper introduces a new approach called the Video Decomposition Prior (VDP) framework. Unlike conventional methods, VDP leverages the motion and appearance of the input video, decomposing it into multiple RGB layers with associated opacity levels. These layers are then manipulated individually to achieve the desired results, addressing tasks like video object segmentation, dehazing, and relighting. The paper also introduces a logarithmic video decomposition formulation for relighting tasks. The approach is evaluated on standard video datasets, including DAVIS, REVIDE, and SDSD, demonstrating qualitative results on a diverse range of internet videos.", "questions": "Please refer to the questions in the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed video decomposition prior leverages the motion and appearance of the input video, decomposing it into multiple RGB layers with associated opacity levels.\n+ The proposed VDP is employed in different video-based tasks, including video object segmentation, video dehazing and video relighting.", "weaknesses": "- In the paper, the limitations of the VDP is not discussed in the paper, and all the results are good cases. \n- In the introduction of Flow similarity loss, the VGG embeddings of masked flow-RGB and those of other layers are used to calculate the cosine similarity. It is unclear how to generate VGG embeddings from masked feature maps for cosine similarity calculation. \n- In equation (12), the behavior of the reconstruction layer loss resembles that of the L1 loss in the reconstruction loss. To validate the rationale behind the design, it is essential for the paper to elucidate the distinctions between them and elucidate the impact of the reconstruction layer loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRJRDJcAxW", "reviewer": "Reviewer_6Tks", "summary": "The paper proposes a new framework that decomposes a video into multiple multiple RGB layers and associated opacity levels for various video editing tasks.\nThe proposed framework uses two neural network modules, RGB-net and α-net, to predict RGB layers and opacity/transmission layers for each video frame. \nThese layers are then composited to reconstruct the input video or achieve the desired effects during optimization.\nThe paper conducted experiments on video relighting, dehazing, and VOS tasks and achieves superior results compared to some existing baselines.", "questions": "- While the proposed method does not require pre-training, it needs to optimization on each input sequence which takes time. How long does it take for optimizing a video sequence?\n\n- Is there a common insight or general guidance on applying this framework to downstream tasks? For example, what if applying this framework to tasks like denoising or super resolution?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Writing is overall clear.\n- Various tasks are conducted and better performance w.r.t to some baseline is reported.\n- No pre-training is required (test-time optimization).", "weaknesses": "- This paper proposes to unify various downstream tasks by video decomposition. However, the formulation of the video relighting task in this paper uses one RGB layer to represent the enhanced video and an alpha-layer to represent gamma correction. This formulation does not have any meaning of “decomposition”. On the other hand, this paper carefully designs different decomposition definitions and corresponding constraints for different downstream tasks, but these constraints and decomposition definitions seem to lack a unified and general formulation. Therefore, I felt such kind of “unification” seems too artificial and forced.\n\n- The proposed formulation is not new (or at least has very few novelty).\nOverall, the proposed formulation can be summarized as a \"data-term + prior\" approach, which is a rather common formulation in optimization-based approach for image / video synthesis tasks. \nUsing neural networks as an implicit prior for data term is not new (as also mentioned in the paper - \"These approaches have highlighted the importance of formulating a loss function,\ncombined with the optimization of neural network parameters\") and the proposed paper extends this idea into other tasks.\nYet, the exact formulation designed for each task is dedicated but common (e.g., gamma-correction curves [1] for relighting; alpha-blending [2,3] for video-segmentation, dark-channel prior for dehazing).\n\n- Missing baselines for UVOS tasks: The paper compares with both pre-trained methods and test-optimization methods. For pre-trained methods, there are massive VOS methods [4,5,6] that has far better performance than compared methods in this paper; For test-optimization methods, there are also other video decomposition methods that targets this task [2,3].\n\n[1] Zhang, Mohan, et al. \"RT-VENet: a convolutional network for real-time video enhancement.\" Proceedings of the 28th ACM International Conference on Multimedia. 2020.\n\n[2] Kasten, Yoni, et al. \"Layered neural atlases for consistent video editing.\" ACM Transactions on Graphics (TOG) 40.6 (2021): 1-12. \n\n[3] Gu, Zeqi, et al. \"Factormatte: Redefining video matting for re-composition tasks.\" ACM Transactions on Graphics (TOG) 42.4 (2023): 1-14.\n\n[4] Cheng, Ho Kei, Yu-Wing Tai, and Chi-Keung Tang. \"Rethinking space-time networks with improved memory coverage for efficient video object segmentation.\" Advances in Neural Information Processing Systems 34 (2021): 11781-11794.\n\n[5] Yan, Kun, et al. \"Two-shot Video Object Segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[6] Cheng, Ho Kei, and Alexander G. Schwing. \"Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)