Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper described an approach of performing multiple instance learning (MIL) by using a network branch to weight instances and then using a Gaussian normalization layer on top of it, where the weightings are predicted based on in-bag variances. The instance weighting scheme, a classic in MIL, has been proposed in deep networks by [3]. Hence, I don't see much novelty here except that there is a Gaussian normalization layer after the instance weighting in the MIL framework. 

I'm a bit worried that sigma seems to be unnormalized in the first case -- what would happen if the bag score distribution is non-Gaussian? GP1T1 seems more sensible by learning all these weights.

However, I see significant issues in terms of evaluation which makes hard to accept this paper.

I firmly believe that 22 years after the (Dietterich 1997) paper, it's no longer enough to only use the original MIL datasets and classification datasets such as the CIFAR-10 bags to evaluate MIL. I may be alone here which is open to discussions, but the original motivation for MIL is for the problem to be weakly-supervised where we only know a high-level label but no low-level labels. There exist many realistic image problems that are similar to this (e.g. weakly-supervised detection and semantic segmentation) and have received a plethora of work, hence it's unclear to me whether still using these arbitrarily generated CIFAR-10 bags and the 5 old datasets would still apply to MIL as an approach. After all, MIL is almost dying and "weakly-supervised learning" has risen in popularity with almost being the same problem as MIL. I think for MIL to work its way back, it should first start by using the right datasets to test (e.g. the datasets in [3] would be a great starting point) and comparing with other weakly-supervised detection approaches that do not use additional information.

Besides the philosophical point, a practical issue is that the numbers seem to be bold arbitrarily according to the authors' whim. In table 3, MI-Net DS is better than every other method in the last 2 columns, but not bold, and also it seems that no t-test was performed to determine the significance of differences. This also happens in Table 1, where I think WSDN should be equivalent with the proposed method in the last 2 rows.

Finally, the choice of excluding MI-Net in the CIFAR experiments is dubious as well, given its performance in the simple datasets. Why is MI-Net not tested on the CIFAR experiments?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper describes a deep multiple instance learning method. It uses
two stream neural networks, one for instance classification and one
for learning normalized weights. The proposed method can be used for
binary and multi-class bag classification.

The authors experimentally show the performance of their proposed
method on several datasets and compare against other state-of-the-art
methods. Also they created a new data set for multi-class bag
classification.

The results are very convincing but the description of the method is
not very clear. In particular, the description of the instance
classifier is confusing. It says that "f" is the instance classifier,
which is a composition of a feature extractor (\phi) and a classifier
(\psi), both modeled as deep neural networks. What are those neural
networks? 

Also the weighting function is a composition of three functions, the
Gaussian normalization, the instance weighting, and the feature
extractor. Again the instance weighting process, which is another key
feature of the paper, is not clearly explained. 

The paper has several English errors which makes sometimes difficult
to follow all the ideas. 

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper proposes a new approach to weighting in multiple-instance learning scenario. They multiply scores for instances by Gaussian normalized weights. The hyperparameters of the Gaussian RBF are either estimated locally (i.e., within a bag) or trained across bags. Additionally, the authors verify whether it is better to have two separate neural networks (one for instance scoring and one for instance weighting) or share weights between them. Eventually, a variant with globally trained hyperparameters of the Gaussian RBF and separate weights of neural networks performs the best.

In general, the paper is very well written and easy to follow. The proposed solution is reasonable and, importantly, performs better than other SOTA approaches. The organization of the paper is proper, all concepts are well explained. The experiments are meaningful and answer important questions (i.e., which of the considered variants is the best, how the proposed approach compares to SOTA methods). In my opinion, the paper could be accepted.

REMARKS:
- What is the temperature value used in the log-sum-exp?

- The authors claim that the log-sum-exp pooling operator is theoretically more relevant for MIL. Could you comment more on that?

- Why do the authors compare to ATT instead of Gated-ATT? In [8] it was shown that Gated-ATT performs better.

======== AFTER REBUTTAL ========
I would like to thank the authors for their rebuttal. I appreciate their answers and updates. I still believe that the paper is interesting and important for the MIL community. I sustain my decision.
review_title: Official Blind Review #1

Meta-review: The authors propose a novel MIL method that uses a novel approach to normalize the instance weights. The majority of reviewers found the paper lacking in novelty and sufficient experimental performance evidence.
Decision: Reject