Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: 
The paper deal with the problem of objections (or object centric) learning by using differentiable iterative refinement to improve the stability and tractability during training. By that, the method seems to improve the stability of a vanilla slot attention architecture such as SLATE. The paper evaluates the proposed extension on the CLEVR-Mirror, Shapestacks, and COCO-2017 datasets.
strengths_and_weaknesses: Strengths:

- The paper addresses a not so well explored topic, but nevertheless important and interesting topic that would justify more research.

- The idea of extending the vanilla SLATE by a differentiable component during training seems interesting on a high level, but I'm not an expert, so it would be ideal to consider feedback from someone with hands-on experience in this domain.

- The paper provides an extensive evaluation of the different aspects of the proposed method.

Weaknesses:

- No coverage of related work in object discovery or unsupervised object detection:
There seems to be a decent body of work on this topic, that's not covered in the paper (e.g. for some older work: Pixel Objectness: Learning to Segment Generic Objects Automatically in Images and Videos, Bo Xiong et al. TPAMI 2018). There is also no comparison to any of those methods. 

- No comparison to related work:
It would be good to see and understand the general state-of-the-art in the field and how the proposed method compares to this on some benchmarks. 

- Limited scope?
Only framing this paper as a valid SLATE extension would be a bit narrow and might serve only a very small niche. If Slot attention architecture has a wider application that this idea might benefit from, it would be good to mention it and show the impact on respective experiments.


questions: 
I'm not an expert in the field, so I would be one of the readers who would highly profit from a broader introduction of the topic and a bit more context with respect to related works in the application area and comparison to respective benchmarks.

I get and like the idea of implicit differentiation, but I#m wondering if objectness in slot attention is really the only use case here.
limitations: The method seems to be very generic. Therefore, I don't see any direct limitations and potential negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper notes that the a number of state-of-the-art object-centric learning models rely on an iterative refinement step that can be understood as a fixed point algorithm. It then uses implicit differentiation to avoid a costly and unstable unrollment to increase the performance of SLATE, a state-of-the-art object-centric learning method, on a range of metrics and datasets with minimal changes to the code.
strengths_and_weaknesses: - Originality: At its core, the technical innovation itself is limited to transferring existing methods from implicit differentiation to object-centric learning.
- Quality: The range of experiments is excellent, with carefully chosen modifications from the original code of previous methods to ensure reliable and fair comparisons.
- Clarity: The manuscript was one of the most enjoyable reads in a while, and I complement the authors for their careful, structured and clear presentation of their methods, the background and the experiments.
- Significance: While technically simple, the insight yields how performance gains that make it likely that this method will become a standard in the further development of the field.
questions: - On Coco 2017 (Fig. 4), the solutions with iterative refinement seem to settle on a very different minimum: while standard SLATE finds pixel blobs in each slot (but doesn't reach good reconstruction), implicit SLATE seems to find stripes of small blobs. Do you have an explanation for this behaviour?
- In Figure 5, what dataset are you using here?
limitations: I don't see potential negative impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Summary:
* The paper combines existing techniques from two groups of works. First, unsupervised object-centric learning with sets representation. Second, implicit differentiation techniques. 
strengths_and_weaknesses: 
Strength:
* The paper is very nicely written overall. The logic flow take us through the author's thinking and reasoning clearly. Relevant sources are all cited at the correct places, making understanding the paper an enjoyable journey. The experiments and results are clearly presented with some helpful color-coding to aid the process. I really enjoy reading the paper. 
* Though similar idea has been empirically explored in prior work [44], this paper does a much more fine-grained investigation, analysis and evaluation.
* Impressive empirical results are achieved by the techniques. The authors do a great job evaluating the performance quantitatively and qualitatively.


Weakness:
* While I do think the paper shows very interesting results, I don't think it's reasonable to motivate them with "Despite their conceptual elegance, it has been difficult to scale iterative refinement methods **beyond modeling simple static scenes or short video sequences** because differentiating through the unrolled forward iteration makes training unstable." The paper do not showcase beyond modeling simple static scenes or short video sequences (which is fine), but promising it in the intro makes it an empty promise. 
* The techniques in implicit differentiation are by themselves not novel. The authors are applying existing techniques to a recurrent problem that has yet to see these techniques being applied. 
* I personally think that mentioning that "in one instance improves reconstruction mean-squared error by almost 7x" feels like an oversale.
* If I'm not mistaken "BB" variant is basically applying Bai et al.'s DEQ [6] to the current setting. And from what I can tell from the training curve Fig 3b, they perform similarly. I think it might be interesting to show how previous technique performs quantitatively. 
questions: * How do you expect the proposed approach to perform with more complex visual scenes / videos? 
* Do you have any intuition of how changing Neumann series expansion affect he model? Will higher order approxmiation benefits the model?
limitations: I don't believe there is a limitation section explicitly in the main paper. But there are some in the supplementary.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to treat object-centric models with iterative refinement procedures as fixed point operations and optimize them using implicit differentiation. In their experiments, the authors show that the training of SLATE becomes more stable when truncating the gradients after the last refinement step instead of propagating it through the unrolled refinement procedure and show performance increases on three datasets (CLEVR, ShapeStack, COCO).

strengths_and_weaknesses: The paper is clearly structured, and the experiments nicely build on each other to support the claims of the paper.  Object-centric models are known to be brittle in training and with regard to hyperparameters. The authors show this problem and propose a solution. They convincingly show that slot attention can be trained using implicit differentiation while retaining/ improving its performance on synthetic datasets. As the authors rightfully state, object-centric model currently do not scale to natural datasets and are therefore of limited use at the moment. Unfortunately, the changed optimization procedure proposed here does not help with scaling to more natural data but fails to learn to identify objects in COCO. 
Furthermore, I don’t see evidence that the optimization procedure helps with getting rid of optimization tricks. While implicit SLATE does show a more well-behaved Jacobian norm with and without learning rate decay, gradient clipping and learning rate warmup, it does not show a qualitatively different behavior when removing these tricks compared to vanilla SLATE. The results of the experiments rather seem to suggest that these tricks did not have a significant effect in the first place and could be in general removed (except maybe lr warmup).

Strength:
- Clear motivation of the work
- Well written and clearly structured
- Experiments on several datasets to support their claims including a natural image dataset


Weaknesses:
- The idea of using implicit differentiation to optimize neural networks is not new and the technical contribution limited.
- The paper would be stronger if the authors would show that their claims hold for more than one model. 
- The model is still of very limited use since it only works on synthetic datasets. 
- Experiments do not support claims regarding optimization tricks.
- Segmentation performance should be quantified in order to make a claim about it (Sec. 5.6)

questions: Can this methods we applied to other object-centric models and if so, do the claims hold there?
limitations: N/A
review_title: null

Meta-review: metareview: The paper proposes to treat object-centric models with iterative refinement procedures as fixed point operations and optimize them using implicit differentiation.

Overall, the reviewers find that the contribution of the paper is somewhat novel, although similar ideas have been presented in prior work in different contexts (supervised settings). Only one reviewer was more negative before the rebuttal, eventually increasing their score after discussion with the authors.

I, therefore, recommend acceptance and encourage the authors to address the comments raised by the reviewers in the final version.
confidence: Less certain
Decision: Accept