Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents a way to solve a localization task for in images without providing any localization supervision. This allows tackling image reconstruction and classification problems that involve multiple object instances. At the core of the approach is a suggestion to resolve the non-differentiable top-k selection process: It is done by introducing axuilliary variables which allows the derivation of an alternating optimization procedure for their framework.

I like the paper and I think the problem it's tackeling is at the core for all many important image understanding tasks. When thinking about reasons that speak against the paper, I'm mostly concerned about the fact that the idea is not yet fully worked out:
* The biggest unresolved aspect is the number 'k' of instances needs to be provided. Assuming real-world relevant instance detection tasks, images in the test set never have 'k' provided.
* All patches need to be squared -- again, for real-world tasks this is not the case.
* In the case of image reconstruction, what is the 'reconstruction error' exactly? Assuming a scene with background clutter, how would this work here? It seems that the demonstrated examples have a black background so working additively is not a problem.

The conclusion mentions 'a first step', so I understand that the authors may be aware of the shortcomings. As such, I'd like to have limitations (and potential resolutions) to be pointed out in the paper.

With respect to related work, I'm missing "Attention-based Deep Multiple Instance Learning", by Ilse et al. I'm not certain if it could be applied to the reconstruction task, but it seems that it should be a baseline for the classification task. When talking about weak supervision, the paper "Weakly supervised object recognition with convolu-tional neural networks" by Oquab should also be mentioned in the related work section.

On a more detailed level, Figure 1 has no symbols for the newtork H_n and T_tau.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method for multi-instance object classification and reconstruction that does not require any location-based supervision. The main contribution is the introduction of a differentiable top-K region proposal that allows to train the whole model with only a supervision of the total number of instances (and their class) in the image. They test the performance of their method in simple visual tasks like cluttered MNIST, street digit recognition and finding the basis of procedural texture generation.

The paper is well written and motivated. The proposed method is clear and well formalized. Their reported results seem substantially better than the baselines they compare against. The additional experiments in their Appendix Fig. 7, analyzing the evolution of the heatmap loss, is interesting, although I think the heatmap visualization could be improved to better understand what the model is learning across iterations (improve legend to include at what iteration the heatmap is taken, give on what image this is evaluated).

Unfortunately, their tasks seem quite easy, and it is hard to assess the impact of their method when working with more real-world data-sets, where the number of instances of every class is more loosely defined (we could always describe more objects in a real image from the COCO dataset for example). It seems of great importance to evaluate the limitations of their method in this direction, as the source of supervision might be too weak in the cases where the generated dataset might not have all the combinations of number of instances per image (as the cluttered MNIST has given that it’s procedurally generated). The results on SVHN are a little bit confusing, and it’s unclear what the “Supervised” method is, specially knowing that there are available methods that do obtain much better performance on this task using all the supervision available. It should also be better explained why the performance drops so drastically when the IoU threshold is increased. Finally, the texture generation experiments are very hard to interpret and are even further from real-world tasks.

Given my concerns on how this unsupervised approach can scale to real-life datasets, I suggest a weak reject, but I think the  proposed method has some interest for the community and I strongly encourage the authors to provide further evidence of performance of their method on more complex vision tasks.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The article introduces the novel MIST architecture which tries to solve the problem of multiple-instance classification and image generation from multiple objects. It employs two submodels, where the first generates a heatmap of intereting region and the second model is a task-specific model that works on image-patches, for example a classifier or an autoencoder. Both models are connected by a patch-extraction routine. The main contribution of this paper is to provide a way to propagate errors through this non-differentiable patch-extraction scheme. This is done by introducing slack-variables.
------------------

The paper is overall relatively easy to follow and the results are very good. However, it suffers from the fact that it does not differentiate between model-architecture and the overall approach. While the main contribution is described in Section 4, the paper spends a lot of space beforehand to introduce the task-dependent models as well as the heatmap architecture - things that i can imagine will vary a lot in different applications. The real important part is how to train the model and this is unfortunately only half described. A good deal of abstraction from the network architecture would have made the paper a lot better.  Further, I think that the loss-function for the classification task does not work in the general case.


On my first read-through, i completely misunderstood Section 4. Here is an unsorted list of issues i had with this:
- since E_K is not truly invertible, writing the approximate inverse as E_K^{-1} is misleading. 
- It might help to stress that you treat {x_k} as continuous and the sampling as differentiable.
- In (7) it would be better to explicitly write E_K^{-1}(x_k) instead of introducing \bar{h}. The line below is not clear.
- It is also misleading, because the choice of \bar{h}=E_K^{-1}(x_k) is not the minimizer of (5) given that all other variables are fixed. You can see this by observing that assuming that when {x_k}=E_K(H(I)) holds, we can choose all other pixels
to be exactly the value returned by H(I).
- I am not sure where the alternating part comes from because this usually involves taking your solution from (7) and feeding it into (6). 
- I am pretty sure that in (6)+(7), as well as lines 3+6 of the algorithm, you actually don't want to optimize for tau or eta from scratch but only perform a single SGD step. I think that is what you are doing, but right now it is written as "find a complete new model for each batch".
- Since this is performed batch-wise: is x_k a variable kept between iterations or do you use H(I) for an initial estimate of x_k for the batch?

Regarding the classification objective:
-since (3) uses the MSE of the mean class-label and the mean-prediction, a dataset where all objects always appear with the exact same amount will not work since than for each image the mean label is identical.
- Therefore, the MNIST-easy dataset should be unsolvable for the proposed architecture since every digit occurs exactly once.

review_title: Official Blind Review #2

Meta-review: Two reviewers are negative on this paper while the other one is slightly positive. Overall, this paper does not make the bar of ICLR. A reject is recommended.
Decision: Reject