Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper investigates influence functions for several classical unsupervised learning methods and VAE. Influence functions in unsupervised learning can reveal the most responsible training samples that increase the likelihood (or reduce the loss) of a particular test sample. A particular usage of the proposed approach is for data cleaning. The authors demonstrated their approach on the MNIST and CIFAR-10 datasets. 
main_review: I am not an expert on instance-based interpretation methods so I cannot evaluate the significance of this work. But the methods proposed here look useful to me. The extension of instance-based interpretation from supervised learning models to unsupervised learning models is non-trivial. A potential drawback of this work is that the authors do not validate their experimental results with other existing data cleaning methods. Therefore, I am not sure the approach proposed here provides meaningful practical usage in data cleaning. I, therefore, refer to other reviewers on the evaluation of the significance of this work. 
limitations_and_societal_impact: See my comments in "Main Review".
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper applies the method TracIn to VAEs (beta-VAE) in order to estimate influence functions of training samples with new ones.
main_review: The novelty of the paper is not clear. Looking at the technical part, section 3 presents at length classical influence functions, but there is no novel component. Section 4 begins by explaining beta-VAEs for the most part, which is also background. Section 4.1 is where the new approach is explained, but is less than half a page long and, as far as I can tell, it is just straight application of TracIn. 

The authors do outline as a unique challenge the impossibility to directly compute the expectation over the encoder Q, and proposed as a novel insight/method using samples to estimate the IF over this expectation (theorem 1). However, the estimation of the expectation over Q through sampling is just a standard feature of VAEs and the fact that this remains a valid approximation in estimating the IF instead of the ELBO directly derives from plugging in the ELBO into TracIn, and therefore it is not a technical novelty.
limitations_and_societal_impact: No unstated limitations or societal concerns.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper characterizes influence estimation techniques to understand instance based interpretations for deep generative models.  Specifically, the authors carry out a comprehensive empirical and theoretical study about influence functions for generative models. 

main_review: Influence estimation techniques have not been studied well for unsupervised learning and this paper does a good job of filling that gap.  The authors primarily study the effectiveness of influence function based methods to understand example based explanations and data cleaning. In the paper, influence functions for classical unsupervised learning (e.g. KDE/GMM) is first studied where the authors provide closed form solutions for influence estimations. Then, to compute influence estimates for deep generative models,  instead of using the classical influence function (Koh et al. , Cook et al.), the authors use TracIn (Pruthi et al. ) and design it specifically for VAEs. With VAE-TracIn, the authors conduct multiple empirical studies to understand in details about self-influence and top/bottom influential samples. 

Although the general influence estimation technique used in the paper (TracIn) is not entirely novel and borrowed from Pruthi et al. , the authors adapt it nicely for generative models which is not trivial to do. The empirical studies on VAE models give relevant information about what influential samples constitutes for in generative models (e.g. samples having high self-influence scores are generally atypical examples are important takeaways specifically for generative models and could be used in data cleaning). The related works section is comprehensive and the authors have covered most of the recent works. Overall, the paper is solid and can open up more research directions on using influence function based methods for generative models. However I have the following concerns/ questions to the authors:

(1) While the general influence function based method (Koh et al. 2017) is computationally expensive, it has a principled theory behind it. What is the relationship between VAE-TracIn and influence function? Would using both the methods give similar inferences? 

(2) The experiments on VAE are on small networks and from a practical standpoint how can VAE-TracIn be scaled up for large networks ?

(3) How can VAE-TracIn be used for applications other than data cleaning?

(4) How precise is VAE-TracIn to quantify the ground-truth influence (e.g change in loss for a test sample when a training sample is removed)? 


Although the experiments are relatively small scale, I feel this paper is a very good starting point for instance based interpretability for generative models and I lean towards an acceptance. I also hope the authors provide answers to (1)-(4) in the rebuttal. 

limitations_and_societal_impact: Although the paper is a very good starting for instance based explanations for generative models, the paper has the following limitations:

(a) Experiments are relatively small scale on small networks/datasets. More experiments on modern VAEs will improve the paper from a practical standpoint.

(b) No solid discussion about how VAE-TracIn compares with other influence estimation technique (Koh et al. 2017). A small discussion would be useful to understand if the generally used influence technique used in supervised learning can be readily adapted for VAEs.


No negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper provides the mathematical frame for influence functions (IFs) for several unsupervised machine learning approaches such as kNN, KDE, and GMM, and proposes a way to empirically compute VAE's IF, and performs several experiments to describe the rationale for using IF for unsupervised tasks. Their approach, VAE-TracIn, can answer which training samples are most responsible for increasing the likelihood of itself (i.e. self-influences) and/or a test sample. The authors claim that the self-influences can be used to discard undesired training samples before training and provide several experimental results showing that IFs for a test sample can identify most and least responsible training samples.

main_review: - Originality: This paper proposes a way to calculate IFs for VAE and performs several experiments showing the ability of IFs and what they reveal about the impact of training samples. There is no big concern about originality.

- Quality and Significance: This paper provides experimental results describing how VAE-TracIn answers the three questions (L186-L189). However, an experiment should be performed in section 5.2 to evaluate how much can a VAE be improved by removing x% of samples with the highest self-influence from the training samples. This can explain why it is important to identify training samples with the highest and lowest self-influence training samples. Also, it is not clear what are the benefits of knowing which training samples have the highest/lowest influence over a test sample. What is the possible application of IFs over test data in unsupervised learning tasks?

- Clarity: The technical challenge and rationale are well described in the introduction. However, the benefit of using VAE-TracIn interpretation from the user's perspective is not well described in the introduction and abstract. The authors describe these in the later sections but I recommend the authors clearly describe the benefits and potential applications of VAE-TracIn in the introduction and/or abstract. Also, here are minor comments for improving the clarity. 
  1. L49: Make italic or use different font styles for the terms "proponents" and "opponents" when they're first introduced.
  2. L84: Complimentary - Is this a typo?
  3. L363: Please update the journal/conference name. 

- In summary, the main concern is that there is no or weak experimental evidence for illustrating the benefits and potential applications of using VAE-TracIn in training and test sample. The reviewer's score can be modified upon the authors' response and additional experimental results.
limitations_and_societal_impact: The author briefly mentions in the conclusion that it is challenging to apply their approach to millions of samples.
review_title: null

Meta-review: The paper investigates the instance-based interpretation method based on influence functions in the Variational Auto-Encoder framework. The paper is well-organized and rather easy to follow. The reviewers tend to agree on the positive aspects of the paper, such as:
- Even though some ideas were introduced elsewhere, the application to VAEs is novel and interesting.
- The paper is well-positioned in the literature review.

The main disadvantage of the paper is the experimental part:
- The paper does not compare their experimental results with other existing data cleaning methods.
- There is little experimental evidence for illustrating the benefits and potential applications of using VAE-TracIn in training and test samples.

Overall, all reviews have a tendency towards the acceptance with one reviewer being positive about the acceptance. Therefore, I believe that the paper could be accepted.

Decision: Accept (Poster)