Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors investigate a generative model for MIL data where instances are generated from bag level and instance level latent factors. They develop variational auto encoders to model this generative process and show an identifiability condition. Experimental results are presented on some synthetic MI datasets and one real world problem which show advances on the state of the art.

strengths_and_weaknesses: 
+ Paper attempts to explore the use of causal models in MIL, which is a nice, not well explored direction.
+ The experimental results are very impressive and it seems the approach can work well in the real world domain investigated.

- The paper is dense and hard to read. There are many typos. 
- The paper borrows heavily from two prior papers, one on identifiable VAEs and one on a multiple instance VAE. It was not clear to me that the advances made beyond these were significant in algorithmic or theoretical terms. While the identifiability result is nice, it was not clear if it required analytical tools that were novel beyond the known result for iVAE. The distinction between factorized/non-factorized seems minor; it is not clear how significant this is. There was also very little comparison to MIVAE, although the graphical models are extremely similar. It is stated that the reason this approach could do better than MIVAE was identifiability, but why that is and why the MIVAE was not identifiable was not clear to me. In general, I found the discussion and comparison to prior work lacking both in the theory and in the experiments. This is one significant area for improvement.
- It is not clear if the assumptions in theorem 1 make sense for MIL, especially (iv). Not enough intuition is provided to help understand the necessity/sufficiency of the assumptions.
- in Eq (8), where is the "weighting hyper-parameter alpha"?
- "The maximum operator of Equation 8 is fundamentally different from the pooling operators used"---this was not very convincing to me.
- Only one experiment is done with an instance labeled MI datasets. There are many MIL datasets available with instance labels. The authors should do additional experiments with actual MI datasets rather than synthetic data such as digit recognition. 
- From the appendix, it is not clear how hyperparameters were chosen. There did not seem to be a common tuning process for all algorithms.
- As mentioned above, the comparison with MIVAE was not convincing to me. The ablation study in the appendix is helpful but the claim "this is caused by not having identifiability" lacks justification, and does not really explain why. Further explanation is needed to explain the differences between these approaches. 
- No information is given about causal interpretations on the clinical dataset. Are causal relationships found clinically validated?
-Limitations are not really included except one line about the standard MIL assumption, which is not a sensible limitation in as much as every model must make some such assumption. 

To summarize, I found the approach promising, but the paper has significant room for improvement.

questions: see above
limitations: see above
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work solves multi-instance learning (MIL) via utilizing bag-level weak supervision. Specifically, the authors propose a general graphical model which disentangles each instance as generated from instance-specific factors and bag-inherited factors. The proposed TargetedMIL algorithm is solved by identifiable variational autoencoder (iVAE).
strengths_and_weaknesses: Pros:
1. The graphical model seems novel and rational. Under the standard assumption of MIL, instance labels are not observed and their labels determine the bag-level label. Hence, decoupling instance labels to both instance-specific factors and bag-level factors seems reasonable.
2. A novel TargetedMIL algorithm is proposed. TargetedMIL takes advantage of permutation invariant set transformation networks to identify the latent factors and learn instance feature representations.
3. The paper organization is well and clear to understand. Theoretical results are provided.

Cons:
1. The utilized datasets are slightly simple. MNIST, FaMNIST, and KuzushijiMNIST are simple datasets that are not so representative to verify the effectiveness of the proposed method. More complex datasets should be explored.

questions: 1. Does this result apply to collective MIL assumption proposed in [1]? The collective assumption assumes that several instances work together to determine the bag label. This assumption is explored in a related work that generalizes Attn-MIL [2]. Hence, could this work generalize to this MIL assumption?

[1] Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition.
[2] Deep multiple instance selection. Sci. China Inf. Sci.

limitations: Yes
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper synergizes identifiable variational autoencoder with multi-instance learning, an important weakly supervised learning problem. By utilizing the instances in the multi-instance bags as auxiliary information, the proposed method provably identifies the latent factors of the positive instances up to affine transformations. The proposed method then utilizes the inferred latents and achieves significantly better results in downward tasks such as instance label prediction and out-of-distribution generalization. Empirical evaluations of qualitative latent reconstructions support the identifiability claim. Quantitative results on several benchmark datasets also show that the proposed method is significantly better at predicting instance labels and out-of-distribution generalization than the baselines.
strengths_and_weaknesses: Strengths:
--	This paper presents a novel method for integrating multi-instance learning with identifiable latent representation learning.
--	The paper is well written and nicely presented.
--	The qualitative latent reconstruction results are novel and interesting, and the quantitative results show significant improvement over the baselines.
--	The out-of-distribution generalizability brought by the identified latent factors broadens the impact of MIL methods to other machine learning subfields.

Overall, the strength of this paper is in the formulation of multi-instance learning as an identifiable VAE with auxiliary information problems. This provides a novel perspective for MIL band motivates better methods not only for instance label prediction but also for out-of-distribution generalization. The focus on multi-instance learning is important as it has been much overlooked compared to how prevalent it is in important real-world applications such as whole-slide medical imaging and fine-grained prediction, and the focus on identifiability has been shown to be useful to further the MIL methodology. In terms of writing, it nicely addresses the challenges of MIL and identifiability and builds up its case coherently. Identifiability is first analyzed under the correct graphical model, then relaxed to accommodate the multi-instance assumption. The model assumptions are clearly stated as well as the assumptions about the data generating process.

Weakness:
--	Some minor issues in the text could be improved with further proofreading. Please see the detailed comments below.

Minor Points:
Line 147, “In Equation 2”, -> “In Equation 2,”
Line 160, “can then be written as” -> “can be written as”
Line 226, “and thus ensures identifiability” -> “and to ensure identifiability”
Line 263, “Appendix” -> “Appendices”
In Table 2, “TargetedMIL” should be in bold font.
Line 628 (in the Appendix), “Table ??” should be “Table 3”.

questions: 1.	What is the scope of domains for the proposed causal graph in Figure 1? Is it applicable to weakly-supervised image classification problems? Discussing some practical problems for which this causal graph is suitable would be preferable.
2.	At the high level, how does the proposed VAE-based MIL method compare to the methods that are based mainly on attention, such as [16] and its follow-up works? As VAE-based MIL algorithms is very different from the current trend of attention-based MIL algorithms, what are the considerations when choosing one over another?

limitations: Yes. The authors have discussed that the method only applies to the standard multi-instance assumption. Solving the identifiability problem in other MIL assumptions remains to be explored.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a causal representation learning algorithm for multi-instance learning, called TargetedMIL, which aims to identify invariant causal representations of instances from bag-level weak supervision. TargetedMIL separates each instance into a causal and a non-causal parts and estimates labels for each instance. Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithm for two tasks. 
strengths_and_weaknesses: Strengths:

1.	This work is well-motivated with theoretical analyses. 

2.	The assumptions used are described in detail.

3.	The experiments are conducted on real-world datasets the empirical results outperform some classical methods.

Weaknesses:

1.	The conditions (b) and (c) in Assumption 1 may not hold in real-world applications. 

2.	Lack of theoretical analyses about estimation errors using Equations (7) and (8), while Theorem 1 only holds in ideal environment.

3.	Lack of analysis of computational complexity.

4.	The relationship between the two tasks (instance label prediction and OOD Generalization) is unclear.

questions: How to ensure conditions (b) and (c) in Assumption 1 always hold?
limitations: My major concern is about the reasonability of the proposed algorithms. The conditions in assumptions used in this paper seems very strict that may be violate in real world. Furthermore, the latent factors identified by the proposed algorithm are hard to evaluate which may violate the assumptions.
review_title: null

Meta-review: metareview: The paper studies multiple instance learning (MIL) by treating bags as auxiliary information, aiming to identify invariant causal representations using only bag labels available in the MIL setting. To achieve identifiability, it is assumed that the prior distribution over the instance latent variables belongs to the non-factorized exponential family conditioning on the bags. This allows the disentanglement between the causal and non-causal factors and only the causal ones are supposed to contribute to the instance labels (while the bag-level labels are used in the proposed objective function in Eq. 8 to accommodate the MIL setting).  Experiments are conducted on multiple datasets to demonstrate the instance prediction and out-of-distribution generalization performance of the proposed TargetedMIL algorithm. 

The perspective of learning invariant causal representations is new in the context of multiple instance learning. Reviewers have acknowledged this interesting aspect of the proposed work. Authors and reviewers engaged in a detailed discussion and the authors' rebuttal helped to address some major confusions, which further improved the qualify of the paper. The authors are encouraged to more clearly highlight the key difference from two important references in the final version of the paper, including identifiable VAEs and multiple instance VAE, which are relevant to the proposed work. The causal inference related assumption could also be further clarified as suggested by one reviewer. 

confidence: Certain
Decision: Accept