Venue: ICLR 2022 Workshop DGM4HSD

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper starts with the slot attention, which can be viewed as a fixed point operation. Then the authors propose implicit slot attention with improved performance in terms of accuracy and efficiency.

This key contribution is introducing a smart acceleration trick, which is quite effective in terms of the empirical results. Yet I'm wondering how universal this module is? Can it apply to other tasks/backbone generative models? And how this is useful (with some intuitional explanation in Sec 3)?

Minor points:
- The first sentence in Sec 3 is too long. Can be further split.

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary & Contributions: This paper proposed implicit slot attention for training the slot attention module in a simpler and more effective way by applying the first-order Neumann approximation to approximate the implicit gradient. The results of experiments on the task of image reconstruction show that the method stabilized the training  with lower space and time cost without optimization tricks.

Pros:
1. The method proposed is significant which can be easily implemented with only one additional line of code.
2. The experiments are conducted in multiple perspectives to show both the robustness and effectiveness.
3.  The paper is well-written and easy to follow.

Cons：
1.The experiments can be conducted on more tasks.
2. The complexity of time and space is suggested to be further clarified in details. 



review_title: A More Effective Way for the Training of Slot Attention 

final_score: rating: 6: Marginally above acceptance threshold
review_content: From what I understand, the main goal of this paper is to perform approximate inference in generative models that use slot attention, where pixels are "clustered" into groups using an iterative algorithm. Because differentiating through the iterative algorithm is difficult, the authors propose to use Cauchy's implicit function theorem to calculate the gradient at the fixed point. Instead of inverting the Jacobian, they instead propose to use just the first term in its Neumann approximation.

Originality: I do not know enough to say whether this insight is original. Other papers have used the implicit function theorem before, but I do not know this field well enough to say whether it has been used in this context before.

Quality: the paper is well written and the contribution is legitimate. I believe it is in the scope of this workshop. I am unsure whether the authors have made an error in their implementation or not, and if they have it would be a major detraction, but the work still has merit.

Significance: this is a small but non-trivial contribution (assuming that it is original), although admittedly I do not feel qualified to evaluate the significance of the experimental results. I believe it is enough for acceptance to the workshop, although if this were a full conference paper I would not feel that the significance is high enough to be accepted. Additionally, if the work contains an error as I ask about below, this would be a major detraction.

## Possible error?

I was slightly confused about the notation in equation 1. If my interpretation is correct I believe that there is an error in the code in Figure 2, but I could be wrong. I think that the author's should double check this.

**My understanding**: you change notation between the introduction and equation 1. My understanding is that $\lambda$ is $\theta^n$ which is a vector in $\mathbb{R}^n$. The Jacobian $J_{f_w}$ in equation 1 is a $n\times n$ matrix. Therefore, equation (1) multiplies a $1\times n$, $n\times n$, and $n\times |w|$ matrix. Written explicitly, the approximation in equation (2) seems to be $(1-J_{f_w})^{-1} \approx 1 + J_{f_w}$. 

**The discrepancy**: the matrix above seems to be a $n\times n$ matrix, and therefore it should at least scale as $O(n^2)$ not $O(n)$ as the authors claim. This is the first potential mistake. The second possible mistake is the code in Figure 2. That code appears to solve for $\theta_t^n=f_w(\theta_{t-1}^n, x)$ and then compute $\frac{\partial l}{\partial f_w(\theta_{t-1}^n, x)} \frac{\partial f_w(\theta_{t-1}^n, x)}{\partial w}$. I don't think that this is equivalent to computing the Jacobian.

I could be totally wrong here, and if so then please take this as an invitation to clarify your notation and the sizes of all the relevant matrices. But, if this is a mistake I think it would be important to correct it.

## Pros/cons

Pros:

- nice idea
- Minimal implementation (if correct)
- Seemingly good experiments

Cons:

- possible error (or at least unclear notation)
- Another thing that is unclear: In Figure 5a on the left hand side, are these the true gradients or approximate gradients (i.e. using your method)? If they are approximate gradients, then I think it is an unfair comparison, since the "real" gradients could also be quite high on the left hand side too, but it would not be shown.
review_title: interesting paper with good insight, good fit for workshop. Would benefit from more experiments and small clarifications. Possible implementation error.

Decision: Accept (Poster)