Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Authors propose a method that takes a video with multiple foreground objects as input, along with the corresponding background frames/image, and rough segmentation masks for each object. It then outputs an alpha decomposition of the video, where one layer corresponds to background, and the others to all visual effects from each object – so e.g. one layer includes a person plus their shadows and reflections. The method is trained for the task of missing-frame reconstruction, and relies on inductive biases (ease of learning) to ensure the correct shadow/etc. are paired with their respective objects, rather than any sophisticated constraints. Synthetic data (with plenty of shadows and reflections) is used for training. At test time, the method is further tuned to optimise the decomposition of a given video. It is demonstrated on both synthetic and real data; quantitative results on synthetic data are better than a recent baseline, and qualitative results on real data look reasonable.

strengths_and_weaknesses: Strengths:

- the proposed approach is novel

- the proposed method achieves significantly better quantitative results on synthetic data than a recent baseline

- the method is also demonstrated on real-world data, where it often achieves visually acceptable results

- the paper is clear and easy to read throughout; it is well structured, and the figures are appropriate

Weaknesses:

- the assumption of known background seems rather restrictive (and makes the problem considerably easier)

- given the background is assumed known a-priori, what is the actual practical purpose/use of the proposed method? Authors should identify this clearly in the introduction, and add an experimental evaluation that shows performance on this task

- on real data, the method is not significantly better than the baseline (though it is faster if one disregards training time)

- there is no attempt to measure quantitative performance on real-world data. While I appreciate that this would require some manual annotation, it would only need to be a handful of frames from the few videos, indicating which regions are indeed moving with which object

- the technical contribution is small, compared with the baseline [12], and also considering similar-in-spirit works such as Video Centrifuge – particularly given the method uses fairly standard architectures and doesn't have a strong justification for its own successes (see below)

- there is no convincing reason presented for the method to work – the 'correct' result is one of several equivalent local optima (assigning shadows to arbitrary layers), and the arguments about Schelling points do not resolve why the 'correct' result is found (merely why the layers should find some arbitrary valid joint decomposition). It seems that the correct optimum is found simply because it is 'easier' for the network to learn, but it'd be nice to see a proper analysis of this. At minimum, does the model training ever converge to 'incorrect' solutions – and for what fraction of training runs if so?

- the method assumes (I think) data is in sRGB (normalised to 0–1) and that different object contributions can be combined by alpha blending. However, this is not true in general – reflections should be treated as strictly additive in linear color space, and shadows darken surfaces rather than alpha-overlaying. This may limit applicability to scenes where the lighting and exposure are fairly well-behaved

- resolution is limited (only 128x128 even training/testing on TPUs), thus limiting applicability in practice

questions: See issues raised above under 'weaknesses'.

How sensitive is the method to the accuracy of the provided object segmentations? If it can recover from imperfect masks and still capture the complete object, this might also be of interest in itself.

It would be interesting to see a more detailed analysis of failure cases. For example, bin the data according to how close the relevant 'effect' is to the object itself, and see how accuracy varies with distance – or see how problematic proximity/overlap with other objects (as in the Dogwalk failure) is in general

Why are the losses denoted mathbf capital E, when in fact they're just scalars?

limitations: There is minimal discussion of limitations; the paper would benefit from adding an explicit subsection for this. There is adequate discussion of broader impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The goal of this work is to decompose videos into different layers, for example, objects of interest and their shadow, reflection, and other visual effects. It is a challenging problem due to the complexity of the 3D geometry and lighting conditions in the real world, as well as the difficulties to get the ground truth. This paper proposes a self-supervised method to solve this problem. They borrow the idea from game theory and train networks to achieve this focal point. The experiments show the effectiveness of their design choice.
strengths_and_weaknesses: Strengths:
+ The idea of using Focal Point is interesting and reasonable.
+ This method achieves promising visualization results for video decomposition.
+ The paper is easy to follow and the method is demonstrated detailedly.
Weakness:
- Object number: In this paper, the authors only show the results of 2 or 4 object scenarios. I am not sure this method can handle the scenarios with arbitrary objects. It might limit the generalization ability of this method.
- It is better to show how this network achieves this focal point.
questions: Please refer to the weakness
limitations: Please refer to the weakness
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a novel frame for video layer decomposition, where they borrow the idea from game theory concept of focal points to frame this problem as a coordination game and let the networks reach consensus on their predictions. 
strengths_and_weaknesses: Strengths:

- The presented idea is both novel and interesting.

- The paper is well written and easy to read.

- Extensive experiments are conducted and improved results are shown.

Weakness:

- The results on real dataset is a little bit worse than the one on synthetic data.
questions: I wonder how this method deals with occluded scenes or objects that might be combined together or split up.
limitations: The presented method might fail with more complex scene with heavy occlusion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Given a short video with some moving objects and a rough mask for each object:
This paper tackles the problem of generating a per-object color and alpha mask for each object, containing all the effects on the image caused by that object (including e.g. shadows and reflections)
This paper achieves this via a network which plays a 'coordination game'; each 'copy' of the network is supplied with a different object's input mask, and attempts to reconstruct the mask and pixels corresponding to this object. The network is trained via a self-supervised reconstruction loss.
strengths_and_weaknesses: ## Originality

The problem being tackled is not original, but the proposed solution is, to the best of my knowledge, novel and interesting.

## Quality
Really nice to see experiments with real data rather than just rely fully on synthetic experiments.

I would have liked to have seen more 'simple, heuristic-driven' baselines. For example:
Given the videos are from a fixed camera, I could imagine using simple median-differencing to find a background image for each video. 
Then this background image could be used to find per-frame, per-pixel differences from the background image; these must mostly be due to effects of foreground objects.
Finally each of these pixels can be associated to the foreground masks via e.g. nearest neighbour assignment.
I don't expect this simple heuristic to beat the proposed approach, but it might give better context to the numbers.

I would also have liked to see more ablations on the components of the algorithm. For example, what would happen if $W_t$ was set to ones everywhere? What about varying the hyperparameters in line 158: 2*sigmoid(5x)?

## Clarity

The overall writing of the paper is clear and easy to follow.

I enjoyed section 3.5: an explanatory view of the model being used.

I was very pleased to see the video results, I felt these made the overall system and quality of results clear.

I would have liked a little more justification of the formulation of $W_t$ (line 155). The idea of this is to give the network more emphasis on reconstructing areas *outside* the mask of the current object?


## Significance

The problem tackled is an interesting one, and the authors propose a thought-provoking solution.
I hope that the paper shows other researchers that this solution (of multiple identical networks playing a coordination game) can be of use in these types of scenarios.



questions: Line 158: 2*sigmoid(5x); it seems there are two hyperparameters here, 2 and 5. Could the authors explain how these came about?

How sensitive overall is the network to the design choices? What would happen e.g. if any part of the overall loss was removed or hyperparameters changed?
limitations: Yes I think they have.
review_title: null

Meta-review: metareview: All reviewers found that the paper provides a novel, interesting solution, and is well written. They appreciated that the proposed method outperforms prior work on synthetic experiments and shows reasonable results on real data. The video results were particularly helpful in judging the results.

The majority of the reviewers were concerned about the convergence of the proposed coordination game to the correct solution. While the authors provided some empirical evidence, a more formal analysis could alleviate concerns much more easily and would provide a strong justification for the proposed method. 

The requests by reviewers for more ablations, simple heuristic baselines, and quantitative results on real data were simply ignored by the authors. This does not induce confidence that any of these requests will be addressed in a final version.
confidence: Less certain
Decision: Accept