Associating Objects and Their Effects in Video through Coordination Games
Authors: Erika Lu
Forrester Cole
Weidi Xie
Tali Dekel
William T. Freeman
Andrew Zisserman
Michael Rubinstein
Venue: NeurIPS 2022 Accept (2022)

Abstract: We explore a feed-forward approach for decomposing a video into layers, where each layer contains an object of interest along with its associated shadows, reflections, and other visual effects. This problem is challenging since associated effects vary widely with the 3D geometry and lighting conditions in the scene, and ground-truth labels for visual effects are difficult (and in some cases impractical) to collect. 
We take a self-supervised approach and train a neural network to produce a foreground image and alpha matte from a rough object segmentation mask under a reconstruction and sparsity loss. Under reconstruction loss, the layer decomposition problem is underdetermined: many combinations of layers may reconstruct the input video.
Inspired by the game theory concept of focal points---or \emph{Schelling points}---we pose the problem as a coordination game, where each player (network) predicts the effects for a single object without knowledge of the other players' choices. The players learn to converge on the ``natural'' layer decomposition in order to maximize the likelihood of their choices aligning with the other players'. We train the network to play this game with itself, and show how to design the rules of this game so that the focal point lies at the correct layer decomposition. We demonstrate feed-forward results on a challenging synthetic dataset, then show that pretraining on this dataset significantly reduces optimization time for real videos.

Associating Objects and Their Effects in Video through Coordination Games
Erika Lu!
Forrester Cole!
Weidi Xie
Tali Dekel
William Freeman'
Andrew Zisserman*
Michael Rubinstein
Google Research
Shanghai Jiao Tong University
Weizmann Institute
University of Oxford
Abstract
We explore feed-forward approach for decomposing a video into layers, where each layer contains an object of interest along with its associated shadows, re- flections, and other visual effects. This problem challenging since associated effects vary widely with the 3D geometry and lighting conditions in the scene, and ground-truth -ruut labels for visual effects are difficult (and some cases impractical) collect. We take self-supervised approach and train a neural network to produce foreground image and alpha matte from a rough object segmentation mask under a reconstruction and sparsity loss. Under reconstruction loss, the layer decomposition problem is underdetermined: many combinations of layers may reconstruct the input video. Inspired by the game theory concept of focal points- Schelling points-we we pose the problem as a coordination game, where each player (network) predicts the effects for single object without knowledge of the other players' choices. The players learn to converge on the "natural" layer decomposition order to maximize the likelihood of their choices aligning with the other players'. We train the network to play this game with itself, and show how to design the rules of this game that the focal point lies the correct layer decomposition. We demonstrate feed-forward results on a challenging synthetic dataset, then show that pretraining this dataset significantly reduces optimization time for real videos.
Introduction
Identifying and associating effects such as shadows and reflections with the objects that produce them is fundamental and difficult task for visual understanding. A variety of cues such motion, appearance, and proximity may be used to match an object and an effect, but these cues may vary widely depending on the 3D geometry and lighting conditions of the scene, and nearby objects may produce similar effects (Fig [ There is no simple heuristic to reliably connect an object and the visual effects it generates.
The recent Onnimatte approach CNN to decompose the video into layers, where each layer contains the object well as the effects associated with that object. The CNN is given a binary segmentation mask of an object, omitting any associated effects, and learns output foreground color and alpha matte for that object and its effects. Notably, no explicit loss is provided for the association of objects and effects: the correct association emerges through the structure of the optimization and the inductive bias of the CNN. The optimization requires hours of processing for each video, however, and is vulnerable to errors when multiple objects have closely correlated motion (Fig
learns to associate an object with its effects by training a
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Input RGB Video
Separated Layer
Separated Layer
Background
(shadows and rellections marked)
Example videos and our decomposition result with shadows and reflections. Matching objects and their effects is challenging: the true object may be further from its shadow (a) reflection (b) than another object, and effects may be subtle (reflection (c), soft shadow (d)).
Figure 1:
This paper aims to solve the layer decomposition problem directly by training fred-forward neural network to predict the layers from the input video and masks. Since ground-trute labels for effects are not available for real videos, we take a sulf--uper approach and train the network only with a reconstruction and sparsity loss. Unfortunately, reconstruction and sparsity losses do not uniquely determine a layer decomposition: effects may be shuffled between layers while producing the same loss (Fig 3 For example, valid but incorrect solution puts the entire video in the front layer, leaving the remaining layers blank.
To solve this problem, we take inspiration from the game theory concept of focal points, also known as Schelling points [16].
coordination game, the players must cooperate to achieve a goal without communicating with each other ahead of time. A classic example is giving several people the instruction: "Meet in New York City on Tuesday", without giving any details about the exact location and time when they should meet. Schelling found that the most common strategy, based on the players' own prior knowledge of the city, was to go to Grand Central station at noon of that day. Players converge such "natural" solution, or focal point, in order maximize the likelihood of their choices aligning with those of other players. In our case, the network plays a coordination game against itself: it is given object mask and is asked to predict the appearance of that object and its visual effects, without knowledge of the other network instances' choices. We show that by training the network predict masked video frames, can engineer the focal point of this game lie the correct layer decomposition.
We demonstrate the success of the approach on challenging synthetic dataset of moving objects with realistic shadows and reflections. The learned prior helps the network overcome ambiguous inputs, leading to improved quality over single-video optimization [12]. The prior learned from synthetic data also provides good starting point for optimization on real videos. Fine-tuning the pretrained network
video reduces optimization time by 10x over single-video optimization, while producing similar layer decomposition quality.
2 Related Work
Video layer decomposition. Separating videos into layers is fundamental computer vision problem that has been studied for decades [8]] It important for both basic understanding of scenes (e.g..
for estimating depth (24] and occlusion boundaries 22.. as well as for supporting various operations on videos such as editing (e.g.. (I) and view synthesis (c.g.. (17). Recently, several works besides the previously mentioned Omnimatte [ [2) have applied deep learning to make progress on this
problem. Zhang al. (23] generate editable free-viewpoint videos of dynamic scenes from sequences captured by multiple cameras through a learned, layered neural representation that disentangles location, deformation, and appearance of dynamic objects. Kasten al. [9] propose decomposition method that represents a video as a set of layered 2D atlases that are global to the video, along with associated (per-frame) alpha maps. Like our work, these methods are self-supervised. However, they involve slow optimizations, overfitting neural networks' weights to input video. In contrast, this work aims to solve the decomposition problem directly by training a feed-forward model predict the layers from the input video and masks.

---

Specialized methods and supervised approaches. Several methods exist that learn to decompose videos or detect objects' effects (such as reflections and shadows) in a supervised manner. "Visual Centrifuge" []] separates videos with transparency and reflections into layers, by training neural network on a large and diverse set of synthetically blended videos. This method learns to separate difficult effects like shadows and reflections, but does not address the problem of associating those effects with the objects that produce them. Other works such as [0]] 7 [2] train a network to detect and remove shadows specifically by generating synthetic Seadow/no-ssadow image pairs and using them as training data. These methods focus on directly removing shadows, not decomposing the video into layers as in approach. Like our approach, instance shadow detection [20] [9] aims to associate objects and their shadows. They take a supervised approach, however, and manually label a large dataset of photographs. Hu, et al. [6] collect an even larger dataset of manually labeled shadows. Manual labeling is only practical for hard, clearly visible shadows, however, so models trained on these datasets are limited to those effects. In contrast, our method makes no assumptions about the appearance of shadows, reflections, or other effects, and requires no data labeling step.
Structured representation for images and video. Our approach is most similar to recent methods for image and video understanding that aim to produce an interpretable, structured representation through self-supervised learning. Methods such as 13 4 learn keypoint-based representations for predicting future video frames by training a decoder to map keypoints to RGB through a pixel reconstruction loss. The learned keypoints are useful for variety of downstream applications, such as trajectory prediction and action recognition. Like these methods, we learn a structured representation using future frame prediction, but where keypoint representations have fewer variables than the original video, our layer decomposition has more variables, making the optimization problem underdetermined. Image de-rendering methods 21 3 aim construct detailed scenegraph description of an image that can be both re-rendered using conventional computer graphics and used for alternative tasks like image captioning. Like these methods, our reconstruction stage is fixed-function operation (alpha compositing), but we operate on video, and our representation is specifically targeted at matting and associating objects with their visual effects.
3 Method
The goal in this paper is to train a feed-forward network that can decompose an RGB video into omnimattes, layers that capture an object and all the scene effects related to that object. We give formal definition of this problem in Section 3.T] We choose a network architecture that combines CNNs and transformer decoder, which allows capturing long-range effects, and binding the objects with their effects (Section (32).. The proposed architecture is trained purely with self-supervised learning (Section 3.3). which enables further improvement of the results via test-time training on real video sequences (Section 3.4
3.1 Problem Definition
For given video with T frames, aim to train feed-forward neural network that decompose the frame containing N dynamic objects into omnimatles; without loss of generality, we present the problem definition based on frame l:
, - (( Ir. M,
Mº; +)
(1)
where (I1...., } denotes the input video sequence, [M,,... :.... M.N denotes N binary segmen-
tation masks, each corresponding to one object of interest, similar to [2]. L; refers to the output i-th RGBA (color and opacity) layer, representing one of the objects and all the scene effects that related to it.
We assume known (possibly time-varying) compositing order o,, and known background C). which can be either given by the synthetic training data or obtained using a simple background estimation process in the case of static background (e.g. median pixel value). The input images 4 may be reconstructed by compositing Lj using standard alpha compositing 15). 4 R Comp(L.,,.) ot).
Unfortunately, the layer decomposition problem is ill-defined, with multiple sets of L; that reconstruct It. For example, one obvious solution would be to place all objects and effects in one layer and leave the others empty, i.e. Ci I, C = 0, vj E [1, NJ. j # i. Thus, order to train a ford-forward network for associating objects and effects, the model must be trained to exploit the correlations in
3

---

MASKED
ONN
E
ggbrecon
L
TRANSFORMER DECODER
ONN
CNN
SHARED WEIGHTS
M
TRANSFORMER
CNN
t-2
Figure 2: Overview the training pipeline. The input a short RGB video clip and masks M'. The target frame 4 masked from the input of the network. The transformer decoder is given features extracted from the RGB input and a single mask layer, and produces single output layer. The output layers Lj are composited over the background Lo form the predicted frame L/ which is compared to the target frame 4. No direct supervision is provided for Li.
video data and avoid learning trivial solutions. Our solution run the network separately on each layer, and task it with predicting unseen frames of the video from object masks and nearby frames. We give intuition for this approach coordination game with designed focal point in Section 3.5
3.2 Architecture
Our model combines CNNs and Transformer modules; an overview is shown in Figure 2 For an input video sequence with T frames, parameterize the image encoder with shared CNN, F {fa.....fr
enc (II).... Penc (Ir)) where I; "xx'3, and f, R f W x where H, W. D are the spatial height, width and channel dimensions. Similarly, the objects in frame are represented by their rough binary segmentation mask, and fed another CNN, M (mi....,my
ma (Mi)..... (M)). 2 mask (M,, )}, with each m, E IR 16
W x
As input to the transformer decoder, we treat the image features as keys and values, with the positional information injected by learnable peatio-temporal embeddings. The encoded masks are treated as queries to the transformer decoder independently, i.e. only one of the object's masks are used as the query time. The output query maintains its spatial resolution, and we further upsample the features with CNN decoder to produce the final RGBA layers corresponding to the input masks.
first, transformer decoder for associating objects with their effects, this architecture is naturally suited our task, as self-attention allows the queries learn across multiple frames, determine object motion and deformation, and thus identify the effects that are correlated with the object of interest in motion and/or shape; Second, each object's layer is predicted without knowledge of the other objects, which enables to correctly assign scene effects to the layer of the object that causes them. Additionally, running the network separately on each layer also has the benefit of generalizing to an arbitrary number of layers.
Discussion. Here, we have made critical choices the architecture design:
3.3 Self-supervised Learning
To train the proposed architecture, define the proxy task as frame reconstruction. Specifically, randomly mask a subset of frames from the input sequence, and task the model to only reconstruct the unseen frames based on the input object masks and other frames in the sequence.
In detail, follow the same loss formulations as [12], with minor modifications. The primary loss for training is a reconstruction loss Engb-recun- with sparsity regularization losses Eng to encourage sparse alpha mattes, and a mask initialization loss Emask to bootstrap training. The reconstruction loss is defined as:
1 Σ IW @ (I, Com)(Crr,o)))la.
Engb recon
(2)
T

---

where @ is element wise product and W; = - Aw E, Mi is a spatial weight to emphasize reconstruction of shadows and reflection effects over object regions.
The alpha sparsity term, aimed to prevent all layers from trivially reconstructing the entire frame, is:
Eveg Σ r ||Comp(ae, ox]l,, + to $o(Comp(as, 0z));
(3)
where (o(x) = 2 Sigmoid - 1 smoothly penalizes non-zero values of the alpha map, and controls the relative weight between the terms.
The bootstrapping, or mask initialization, loss is:
SEEEE (Ma-a) TN
-
Emask
(4)
where by is spatial weight that balances the loss in positive and negative mask regions. This bootstrapping loss ensures the optimization starts from roughly accurate layer decomposition, where the predicted alpha maps of match the input masks Mj.
The complete loss term is:
Ergi-sccon + A A,Er + Emask
(5)
where X, and Am are weighting coefficients. The weight Am is gradually decreased during training to reduce the effect of the bootstrapping loss as the optimization converges (see supplemental material).
3.4 Training Procedure
Training proceeds in two stages, namely, pre-training and test-time training.
Pretraining. At this stage, we pre-train our model on a large corpus video data, specifically, we sample sequence of T' video frames and their corresponding object masks, we then select subset of frames to mask, or hide from the network, and tasked the model to reconstruct these the unseen frames based on their input object masks.
Test-time finetuning. As our proposed model is trained from purely self-supervised learning, is possible to adapt to any test video sequence in the same form of optimisation- based approach, supporting the model to generalise to different data domain, e.g. sim2real, For this test-time training stage, we use the same losses and training setup for pretraining, with the exception of lower weight for the bootstrapping loss.
3.5 Layer decomposition as a coordination game
The reconstruction task can be seen as a co- ordination game between multiple players: N instances of the network e Each is respon- sible for producing a single layer Cj and each instance is blind to the layer produced by the oth- ers. The (negative) total loss Eq. is the reward shared by all players. Even with the sparsity regularization term Ereg and mask term Emask. the reconstruction is underdetermined: for any given decomposition Li an equally good recon- struction and sparsity score is obtained by re- moving effects from one layer and adding them to another (Fig. 3). Any decomposition that minimizes Eq. may be thought of as a Nash equilibrium 44 of this game.
Input
Correct decompositio
Incorrect decomposttion, same
Figure 3: Ambiguity of the decomposition prob- lem. Under a reconstruction loss, effects may be moved between layers without cost. The task and architecture must be designed produce the cor- rect (top) instead of incorrect (bottom) solution.
In coordination game with multiple equilibria, the players tend to choose the most prominent equilibrium, otherwise know as focal point or Schelling point, based on some shared knowledge the players bring from outside the game [16]. Importantly for our case, the most prominent equilibrium need not be explicitly defined (i.e., super- vised). Instead, may be implicitly defined by the shared knowledge of the players. Over many rounds of this game, the players learn rules to find focal point, without the rules the focal point
5

---

being explicitly specified. In other words, this approach allows the players (multiple instances of the network 4e) to be trained with sulf-vuupi The exact focal point that found through this method depends on the structure of the task. By asking each instance to produce single layer, providing a single object mask, and bootstrapping the network to match that mask, we bias the network towards explaining a single object with each instance. As shown by Lu, et al. [1], effects that are correlated with the object mask are easier to predict than uncorrelated effects, so a single object's s associated effects tend to be predicted in the object's 's layer. Forcing the network to predict the layers an unseen frame, rather than providing the entire video clip as input, helps train the network to find and exploit these correlations (Section 5.4
4 Dataset Description
For our experiments, we use a mixture of synthetic and real data. Specifically, we use the Kubric codebase [5] to generate large-scale synthetic data for pretraining, while our model is self-supervised, and can thus be pretrained on real data, we choose to use synthetic data for this stage due to the costly procedure for curating video datasets with interesting effects, such as shadows and reflections, and which also meet our constraints (a background that can be well-estimated, and objects undergoing significant motion). An additional benefit of a synthetic dataset is that the objects can be rendered individually, providing a ground truth decomposition that can be used for numerical evaluation, so long as the original scene does not contain collisions between objects.
4.1 Synthetic shadows and reflections
For all synthetic videos, we use a randomly selected high dynamic range image (HDRI) as the background texture. Objects are randomly sampled from the Google Scanned Objects (GSO) dataset of 3D scanned household items. We split the HDRIs into 450 train and 59 test images. For the GSO objects, we use Kubric's default train/test split (90/10) of the 1033 objects.
We render all videos under a fixed camera position with three light sources at random positions in the scene, and an additional 4th fill light fixed location. We default settings to initialize the position and velocity of objects. Each video is 2s long at 12 fps, totalling 24 frames.
Shadow dataset is generated to understand our model's ability to associate objects with their shadows. For training, we render 10k videos with 2 objects, while for numerical evaluation, we only keep the generated test videos that do not contain object collisions, resulting in 323 2-object test videos and 86 4-object test videos, with the latter to test our model's is ability to generalize to more objects than seen during training.
Reflection dataset is generated with under the same physical conditions as the Shadows dataset, but with an additional mirror object that reflects about 50% of the scene!" We randomly jitter the placement and orientation of the mirror. As with the Shadows dataset, we generate 10k 2-object training videos, and 315 2-object videos and 38 4-object videos with objects not colliding.
4.2
Real videos
Apart from evaluating on the synthetic data, we also test on the challenging real sequences with various effects. We use real videos from Lu et. al [12] for comparison. including Reflections, which contains two people passing each other with their reflections visible; Trampoline, which involves six people jumping both cast long shadows; and Soccer, which contains player kicking a ball and also casting long shadows. These videos range in length from 80 200 frames, sampled at 30 fps.
trampolines; Dogwalk, which involves person walking dog while
At the time of submission, Kubric did not support reflections for HDRI backgrounds; thus the mirror contains the reflections of the scene objects but not that of the textured background.

---

Experiments
5.1
Training details
For our experiments, we first pretrain the model synthetic 2-object dataset at for 60k iterations, as inputs, we sample 5-frame sequences with a 3-frame gap, each frame of 128x 128 resolution. We randomly mask out 3 out of the 5 frames during training, which we have explored the effects of varying these choices in Sec. 5.4
During the second stage, we adapt our model to a test video by test-time training on the single video. We use the same frame sampling and masking strategy as the pretraining stage, with the exception that real videos are sampled with a frame gap of 1, which we empirically find yields better results. We implement all of our experiments in JAX and train on TPU v3 with 16 cores. We refer the reader to supplementary material for additional details.
5.2 Evaluation
At inference time, we provide the model with the same number of frames as during training. We always mask out the last 3 frames in the sequence. and keep the middle frame prediction, and run the model over the entire video in a sliding window fashion. For numerical evaluation, we report IoU on binarized masks of each object layer. We render each object individually and use the resulting frames as the ground-truth alpha layer. To obtain the ground-trutt binary mask Bi for object i at time 6, we do a background subtraction, i.e. compute the squared distance between the ground-trult single-object 1. if (Compp((CC) o,) - C)) 2 B 0, otherwise
scene and background layer, and threshold it: B = {
where C indicates the RGB channels. We obtain the predicted binary mask using the same procedure.
5.3 Synthetic results and generalization to more objects
We train separate models on the 2-object Shadows and Reflections datasets and show results on both the 2-object and 4-object test sets in Table For pretraining (see supplementary for details), and for test-time training, we report both results from using the estimate (Ours-B) and from using the ground truth background (Ours-C). Using a more accurate background results in improved performance, but our model obtains strong results even with the estimated background. Despite only seeing 2-object videos during training, the model is able to generalize to scenes containing 4 objects due to it being structured to predict each layer separately, without knowledge of the other objects the scene. Even without the test-time training stage, our data-driven method outperforms the single-video optimization method of Lu et al. [ by large margin. Finetuning on the test videos brings about further improvements in as few as 50 training steps, with additional performance gains as training time increases (see Figure 4
the estimated background
Table 1: IoU for 2- and 4-object test sets. For 2-object videos, we evaluate random subset of 50 videos. 'TTT" refers the number of test- time training iterations. 'Masks' refers to using the input object masks directly.
Figure 4: Performance (loU) vs. number of test-time training iterations (TTT) for 4-ojeect Shadows and Reflections test sets, using the ground truth background.
1.0
Ours (Shad.)
Shadows
Reflections
Ours (Refl.)
Model
TTT it.
ob t ob
2 ob
4 ob
0.8
Lu et. al (Shad.)
Lu et. al (Refl.)
Masks
0.372
0.368
0.348
0.317
3
Lu et
12
0.469
0.420
0.451
0.353
Ours-A
0.607
0.572
0.593
0.569
0.4
Ours-B (est. bg)
200
0.802
0.752
0.806
0.747
Ours-C (gt bg)
200
0.843
0.814
0.843
0.798
0.2
100
200
6000
Iterations
We show qualitative results on synthetic videos from the Reflections dataset in Figure 5 The Omnimatte method of Lu et al. [12] struggles to correctly associate objects with their reflections,
7

---

t=
obj
/= 12
obj 2
obj M
obj 2
=
obj
== 17
obj
(a) Original
(b) Input Masks
(c) Lu, et al.
(d) Ours (zero-shot)
(e) Ours (finetuned)
(f) Ground Truth
Frames
Figure 5: Results of training our model on 2-object scenes and testing on 2- and 4-object scenes. We show (a) the original video frames input to our model and the masked frame (red border) that must be reconstructed by our model, (b) the input object masks for the masked frame, (c) the result from [12] (d) our zero-shot result (pretrained network without test-time training), (e) our result after test-time training, and (f) the ground truth rendering. Whereas [12] places certain reflections in the incorrect object layer (yellow boxes), our pretrained network makes accurate coarse association while test-time training further improves reconstruction quality. Best viewed on display, zoomed-in.
often assigning elements based on proximity (Figure rows and 2, column (c)). Our method, however, coarsely places the reflections in the correct layer out of the box (column (d)), reconstruction quality is further refined by test-time training, capturing additional details such as the shadows the reflections (last row, column (e)).
The Omnimatte method 12 is randomly initialized network that is optimized on single video, and thus shorter sequences such as these provide greater challenge due to the ambiguity (e.g. the motions of objects can appear correlated under short observation window). Pretraining on many such short sequences, however, allows our method to overcome this limitation, and to do so in much fewer training iterations on the test video than required for 12
5.4 Ablations
We investigate the effects of pretraining with different numbers of masked input frames and different masking strategies. For all experiments, report IoU the 2-object datasets. In Table 2

---

sample 5 frame sequences during training and vary the number of frames K that we mask. For these experiments we mask either the first or the last K frames in the sequence. For K O, we apply the reconstruction loss to all of the frames, whereas when K >> 0, reconstruction loss is applied only to masked frames. In [3] we set K = 3 and vary the method of selecting the masked frames: (i) random, (ii) inpainting (masking the middle frames), (iii) 'prediction' (masking the last or the first frames). For models trained under the random or prediction schemes, during inference time we generate masks under the prediction scheme and use the model output for the first masked frame only (i.e. frame index 5 - K), running in a sliding-window fashion. As seen in Table 3 the choice of masking strategy does not have significant effect. The number of masked frames similarly does not have a strong impact, as long as at least one frame is masked. When K = O, performance drops significantly, showing the importance of the frame prediction task.
Table 2: Varying the number of input frames masked during training (out of 5). We report IoU and stan- dard deviation computed from 3 random seeds.
Table 3: Varying the strategy for masking the frame. We report IoU and standard deviation computed from 3 random seeds.
# Masked
Shadows f
Reflections
Strategy
Shadows f
Reflections t
0/5
0.453 + 2c-4
0.440 0.002
Random
0.599 0.005
0.611 = 0.004
1/5
0.679 + 0.012
0.605 + 0.007
Inpainting
0.615 + 0.017
0.605 = 0.004
3/5
0.638 + 0.002
0.625 + 7c-4
Prediction
0.585 + 0.003
0.596 = 0.002
4/5
0.626 1 0.003
0.617 1 0.004
5.5
Transfer to real videos
We adapt our pretrained model to real videos containing complex shadows, reflections, and trampoline deformations. We finetune our pretrained model for Ik iterations on each real video, taking about 12 minutes, compared with two hours for the optimization- comparable results to single-video optimization (Fig. 6). The separation of effects in the Trampol inne video improved (rows and 2), even though no trampoline effects are present in the training data, suggesting pretraining helps the network generalize to new effects. Other real videos are partially successful. In Dogwalk, the network separates the shadows of the dog and person correctly and even inpaints the shadow for most of the vidco (rows 3 and 4), but becomes confused as the dog obscures more of the person's 's shadow (rows 5 and 6). In this case, the single-video optimization method produces superior results, possibly due our synthetic training data lacking large shadow occlusions. We created an additional synthetic dataset with larger shadows to better mimic these videos. Please see supplementary for additional results using this new dataset.
approach [12]. Our method achieves
6 Discussion and Impact
We have demonstrated that a feed-forward network may be trained to successfully perform the Omnimatte task. On our synthetic dataset the feed-forward network even performs better than single- video optimization, showing the utility of the learned prior. Our model is trained with self-supervised losses, it can be applied to real videos where ground-truth labels are not available, dramatically reducing the time required to produce video layer decompositions. Viewing the decomposition problem as coordination game provides intuitive explanation for the success of this approach: the network must learn to find a focal point of the game, and since each instance is tasked with predicting single layer from a single object mask, the most prominent focal point lies where each instance predicts the effects of its object and nothing else.
Ethical considerations. Any technique that allows for new image editing effects could possibly be misused to produce fake or misleading images and videos. A layered decomposition video only allows rearrangement of content already present, but even simple rearrangement can significantly alter the effect of video. As with other research work in image editing, we hope publicly presenting the technique can help inform readers about editing capabilities that may exist in the near future.

---

Object
Trampoline
Object
Object
Object 2
Dogwalk
Object
Object
failure case
(a) Original Frames
(b) Input Masks
(c) Ours
(d) Lu, et al.
Figure 6: Results of our method on real videos. Our network pretrained on synthetic data requires much fewer training iterations (<10%) test videos than Lu, et al., and achieves comparable results in many cases (trampoline deformations and shadows grouped with the correct person/animal, top rows). However, method can fail in challenging scenarios involving heavy occlusion (part of the person's 's shadow is incorrectly placed in the dog's layer, rows 3 and 4).
References
[1] Jean-Baptiste Alayrac, João Carreira, and Andrew Zisserman. The visual centrifuge: Model-free layered video representations. In CVPR, 2019.
[2] Gabriel Brostow and Irfan A Essa. Motion based decompositing of video. In ICCV, 1999.
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Ali Eslami, and Oriol Vinyals. Synthesizing programs images using reinforced adversarial learning. International Conference on Machine Learning, pages 1666-1675. PMLR, 2018
Anand Gopalakrishnan, Sjoerd Steenkiste, and Jürgen Schmidhuber. Unsupervised object keypoint learning using local spatial predictability. International Conference on Learning Representations, 2021.
[5] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo. Charles Hermann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mchdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun. Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong. and Andrea Tagliasacchi. Kubrie:
scalable dataset generator. In CVPR, 2022.
[6] Xiaowei Hu, Tianyu Wang. Chi- Wing Fu, Yitong Jiang, Qiong Wang, and Pheng Ann Heng. Revisiting shadow detection: A new benchmark dataset for complex world. IEEE Transactions on Image Processing, 30:1925 1934, 2021.
10

---

[7] Naoto Inoue and Toshihiko Yamasaki. Learning from synthetic shadows for shadow detection and removal. IEEE Trunsactions on Circuits and Systems for Video Technology, pages 4187-4197, 2020.
[8] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 4020-4031, Red Hook, NY, USA, 2018. Curran Associates Inc.
[9] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. SIGGRAPH, 2021.
[10] Zhihao Liu, Hui Yin, Xinyi Wu, Zhenyao Wu, Yang Mi, and Song Wang. From shadow generation shadow removal. In CVPR, pages
2021.
[11] Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William T Freeman, and Michael Rubinstein. Layered neural rendering for retiming people in video. In SIGGRAPH Asia, 2020.
[12] Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William T. Freeman, and Michael Rubinstein. Omnimatte: Associating objects and their effects in video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4507-45 4515,, June 2021.
[13] Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. Advances Neural Information Processing Systems, 32, 2019.
[14] Martin Oshome and Ariel Rubinstein. Course Game Theory. The MIT Press, 1994.
[15] Thomas Porter and Tom Duff. Compositing digital images. In Proceedings of the I 11th annual conference Computer graphics and interactive techniques, pages 253-259, 1984.
[16] T.C. Schelling. The Strategy of Conflict. Harvard University Press, 1960.
[17] Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pages 231-242, 1998.
[18] John Wang Edward Adelson. Representing moving images with layers. IEEE Trans. on Image Processing, 1994.
[19] Tianyu Wang. Xiaowei Hu, Chi- Wing Fu, and Pheng-Ami Heng. Single-stage instance shadow detection with bidirectional relation learning. In Proceedings of the IEEL/CVE Conference on Computer Vision and Puttern Recognition, pages 1-11, 2021.
[20] Tianyu Wang, Xiaowei Hu, Qiong Wang, Pheng- Ann Heng, and Chi- thig Fu. Instance shadow detection. In CVPR, 2020.
[21] Jiajun Wu, Joshua Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 699-707, 2017.
[22] Edward Zhang, Ricardo Removing objects and their shadows using approximate lighting and geometry. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16397-16406, 2021.
Janne Kontkanen, and Brian Curless. No shadow left behind:
[23] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable free-viewpoint vidco using a layered neural representation. ACM Transactions Graphics (TOG), 2021.
[24] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.

---

Did you include the license to the code and datasets?
[No]
The code and the data are
proprictary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Figure 6
(C) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section [.]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' 'r using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "Mmp7WoNa8Ul", "reviewer": "Reviewer_c3fb", "summary": "The goal of this work is to decompose videos into different layers, for example, objects of interest and their shadow, reflection, and other visual effects. It is a challenging problem due to the complexity of the 3D geometry and lighting conditions in the real world, as well as the difficulties to get the ground truth. This paper proposes a self-supervised method to solve this problem. They borrow the idea from game theory and train networks to achieve this focal point. The experiments show the effectiveness of their design choice.", "questions": "Please refer to the weakness", "limitations": "Please refer to the weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The idea of using Focal Point is interesting and reasonable.\n+ This method achieves promising visualization results for video decomposition.\n+ The paper is easy to follow and the method is demonstrated detailedly.\nWeakness:\n- Object number: In this paper, the authors only show the results of 2 or 4 object scenarios. I am not sure this method can handle the scenarios with arbitrary objects. It might limit the generalization ability of this method.\n- It is better to show how this network achieves this focal point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "K9WOv_C2V2Z", "reviewer": "Reviewer_7Czt", "summary": "Given a short video with some moving objects and a rough mask for each object:\nThis paper tackles the problem of generating a per-object color and alpha mask for each object, containing all the effects on the image caused by that object (including e.g. shadows and reflections)\nThis paper achieves this via a network which plays a 'coordination game'; each 'copy' of the network is supplied with a different object's input mask, and attempts to reconstruct the mask and pixels corresponding to this object. The network is trained via a self-supervised reconstruction loss.", "questions": "Line 158: 2*sigmoid(5x); it seems there are two hyperparameters here, 2 and 5. Could the authors explain how these came about?\n\nHow sensitive overall is the network to the design choices? What would happen e.g. if any part of the overall loss was removed or hyperparameters changed?", "limitations": "Yes I think they have.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Originality\n\nThe problem being tackled is not original, but the proposed solution is, to the best of my knowledge, novel and interesting.\n\n## Quality\nReally nice to see experiments with real data rather than just rely fully on synthetic experiments.\n\nI would have liked to have seen more 'simple, heuristic-driven' baselines. For example:\nGiven the videos are from a fixed camera, I could imagine using simple median-differencing to find a background image for each video. \nThen this background image could be used to find per-frame, per-pixel differences from the background image; these must mostly be due to effects of foreground objects.\nFinally each of these pixels can be associated to the foreground masks via e.g. nearest neighbour assignment.\nI don't expect this simple heuristic to beat the proposed approach, but it might give better context to the numbers.\n\nI would also have liked to see more ablations on the components of the algorithm. For example, what would happen if $W_t$ was set to ones everywhere? What about varying the hyperparameters in line 158: 2*sigmoid(5x)?\n\n## Clarity\n\nThe overall writing of the paper is clear and easy to follow.\n\nI enjoyed section 3.5: an explanatory view of the model being used.\n\nI was very pleased to see the video results, I felt these made the overall system and quality of results clear.\n\nI would have liked a little more justification of the formulation of $W_t$ (line 155). The idea of this is to give the network more emphasis on reconstructing areas *outside* the mask of the current object?\n\n\n## Significance\n\nThe problem tackled is an interesting one, and the authors propose a thought-provoking solution.\nI hope that the paper shows other researchers that this solution (of multiple identical networks playing a coordination game) can be of use in these types of scenarios.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7OeqDxMv9G-", "reviewer": "Reviewer_Gkjr", "summary": "Authors propose a method that takes a video with multiple foreground objects as input, along with the corresponding background frames/image, and rough segmentation masks for each object. It then outputs an alpha decomposition of the video, where one layer corresponds to background, and the others to all visual effects from each object – so e.g. one layer includes a person plus their shadows and reflections. The method is trained for the task of missing-frame reconstruction, and relies on inductive biases (ease of learning) to ensure the correct shadow/etc. are paired with their respective objects, rather than any sophisticated constraints. Synthetic data (with plenty of shadows and reflections) is used for training. At test time, the method is further tuned to optimise the decomposition of a given video. It is demonstrated on both synthetic and real data; quantitative results on synthetic data are better than a recent baseline, and qualitative results on real data look reasonable.\n", "questions": "See issues raised above under 'weaknesses'.\n\nHow sensitive is the method to the accuracy of the provided object segmentations? If it can recover from imperfect masks and still capture the complete object, this might also be of interest in itself.\n\nIt would be interesting to see a more detailed analysis of failure cases. For example, bin the data according to how close the relevant 'effect' is to the object itself, and see how accuracy varies with distance – or see how problematic proximity/overlap with other objects (as in the Dogwalk failure) is in general\n\nWhy are the losses denoted mathbf capital E, when in fact they're just scalars?\n", "limitations": "There is minimal discussion of limitations; the paper would benefit from adding an explicit subsection for this. There is adequate discussion of broader impacts.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- the proposed approach is novel\n\n- the proposed method achieves significantly better quantitative results on synthetic data than a recent baseline\n\n- the method is also demonstrated on real-world data, where it often achieves visually acceptable results\n\n- the paper is clear and easy to read throughout; it is well structured, and the figures are appropriate\n\nWeaknesses:\n\n- the assumption of known background seems rather restrictive (and makes the problem considerably easier)\n\n- given the background is assumed known a-priori, what is the actual practical purpose/use of the proposed method? Authors should identify this clearly in the introduction, and add an experimental evaluation that shows performance on this task\n\n- on real data, the method is not significantly better than the baseline (though it is faster if one disregards training time)\n\n- there is no attempt to measure quantitative performance on real-world data. While I appreciate that this would require some manual annotation, it would only need to be a handful of frames from the few videos, indicating which regions are indeed moving with which object\n\n- the technical contribution is small, compared with the baseline [12], and also considering similar-in-spirit works such as Video Centrifuge – particularly given the method uses fairly standard architectures and doesn't have a strong justification for its own successes (see below)\n\n- there is no convincing reason presented for the method to work – the 'correct' result is one of several equivalent local optima (assigning shadows to arbitrary layers), and the arguments about Schelling points do not resolve why the 'correct' result is found (merely why the layers should find some arbitrary valid joint decomposition). It seems that the correct optimum is found simply because it is 'easier' for the network to learn, but it'd be nice to see a proper analysis of this. At minimum, does the model training ever converge to 'incorrect' solutions – and for what fraction of training runs if so?\n\n- the method assumes (I think) data is in sRGB (normalised to 0–1) and that different object contributions can be combined by alpha blending. However, this is not true in general – reflections should be treated as strictly additive in linear color space, and shadows darken surfaces rather than alpha-overlaying. This may limit applicability to scenes where the lighting and exposure are fairly well-behaved\n\n- resolution is limited (only 128x128 even training/testing on TPUs), thus limiting applicability in practice\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6RsUhbJrFoL", "reviewer": "Reviewer_4zWN", "summary": "This paper presents a novel frame for video layer decomposition, where they borrow the idea from game theory concept of focal points to frame this problem as a coordination game and let the networks reach consensus on their predictions. ", "questions": "I wonder how this method deals with occluded scenes or objects that might be combined together or split up.", "limitations": "The presented method might fail with more complex scene with heavy occlusion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The presented idea is both novel and interesting.\n\n- The paper is well written and easy to read.\n\n- Extensive experiments are conducted and improved results are shown.\n\nWeakness:\n\n- The results on real dataset is a little bit worse than the one on synthetic data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept