Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new class of interpretable models, instead of limiting model complexity (like using simple interpretable models) or limiting the feature interaction (like in GAM), the paper constrains the amount of information in the model by using Distributed Information Bottleneck IB. IB introduces a constraint after every feature is processed and before any interaction effects between the features can be incorporated. The features are compressed to optimal representations which communicate the most relevant information to the rest of the model This can be viewed as a penalty for the amount of information used by a model. 
strength_and_weaknesses: Strength:
---

- The idea is quite novel, constraining the amount of information going into model for interpretability is very interesting.
- The idea is well motivated.
- Unlike GAMs the approach allows feature interaction since there are no restrictions on how the features can interact after IB.
- Using such an approach on tabular data does not cause model accuracy degradation.


Weakness:
---
- I am concerned about the interpretability from the following aspects:
     - Is this model actually interpretable since we are not sure how features interact?
     - What does it mean (in terms of interpretability) to need part of a feature? Like using some information about the age of humans important?
    - Explanations produced by the model are very hard to interpret, I read the paper multiple times and I don't fully understand the interpretations produced in figure 5.

- The paper is well written but the figures are very difficult to understand, better interpretations should be included,
- Given that this paper is mainly about interpretability more interpretability results are needed, for example, the use of a synthetic dataset showing that the optimal amount of feature information is actually extracted by IB. A user study can also be useful.
- The proposed model only applies to tabular datasets and can not be used in other domains like images, language and time series (although this is the case for NAM).


clarity,_quality,_novelty_and_reproducibility: Clarity/Quality-- Medium:
The text is clear but the figures are very difficult to understand.

Novelty--High:
The paper introduces a new class of interpretable models.

Reproducibility-- Medium:
The code for the experiments was included but it is not well documented i.e there were no instructions on how to run it or which functions to call..etc
summary_of_the_review: The overall idea is very interesting. However, I am not completely convinced that the proposed models are in fact interpretable since we do not know how features actually interact. Also, the interpretations provided by the paper were very hard to understand which again questions the overall interpretability of the method.  Interpretations are generally created for humans to have a better understanding of the models and how it uses different features. I am not sure this model achieves this.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a technique to improve interpretability of models by compressing the information in features and feature values using Distributed Information Bottleneck. The redundant and useless features/feature values are intended to be removed by the bottleneck layer so that only the remaining relevant features/values are used for model prediction. This promotes model sparsity which makes the predictions more comprehensible.
strength_and_weaknesses: 1. Overall, the paper needs to make the language clearer and improve the presentation of the algorithm.

Examples of text that are unclear:

    a. Abstract: "The central object of analysis is not a single trained model, but rather a spectrum of models serving as approximations that leverage variable amounts of information and range from the uninformative to the most performant model obtainable."
    
    b. Abstract: "tackling the problem of feature selection with inclusion/exclusion existing on a learned continuum."

    c. The paper keeps referring to 'a spectrum of models'. But it is not clear what that means in this context. Please provide a clear definition of it in the beginning.
    
    d. Section 4.1: "The signal obtained by the Distributed IB can help decompose through datasets with a large number of features and complex interaction effects."
    
    e. Section 3: "The Distributed IB (Estella Aguerri & Zaidi, 2018) concerns..." -- there is some confusion. Distributed IB is a reference to a previous work as well as being used as the name of the proposed algorithm (Table 1). Should disambiguate in text.


2. Two of current most popular techniques for interpretability LIME [r1] and SHAP [r2] are missing in discussions and experiments.


3. In the experiments section, there is no comparison with existing techniques for interpretability.


4. Section 3 should present the techniques for training and information plane visualization in Algorithm formats. A schema of the overall architecture of the algorithm should also be illustrated as a picture for clarity.


5. It is hard to understand whether the proposed technique is a 'global' interpretability technique or 'local'. As per my understanding from statements like in Section 3 "...The penalty has the effect ... information that is least helpful ... is discarded.", the technique is 'local'.

A 'global' (e.g., LASSO) technique allows us to identify the most important features/feature values across all training data.

A 'local' (e.g., LIME [r1]) technique allows us to identify the features/feature values that lead to a specific prediction for a specific input data.


References

[r1] Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "Why should I trust you?: Explaining the predictions of any classifier." Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM (2016).

[r2] Lundberg, Scott M., and Su-In Lee. "A unified approach to interpreting model predictions." Advances in Neural Information Processing Systems (2017).

>>>>>>>>>>> Update after author response

I thank the authors for responding to my comments. At this time I do not have sufficient reasons to change my scores. Some comments:

1. "...from the uninformative to the most performant model obtainable..." -- The range 'from uninformative .... to ... performant' makes little sense (like saying 'from brightest ... to hottest'). It is not clear what the two ends of the scale 'uninformative' and 'performant' mean i.e., are these in terms of interpretability or accuracy?

2. Thanks for pointing out that the method is 'global'. Global methods are not as interesting (w.r.t research and application) as 'local' methods that explain model response on specific examples, especially when the number of training examples is large and the number of features is small. When the number of features is large (few hundreds), the global feature selection methods are useful, but usually the objective is then to clean the data rather than to enhance interpretability.

3. Figure 4 needs additional labels/captions on the figures to improve readability. E.g., Distributed IB/LassoNet needs to be displayed on the plots, else it is difficult to follow the dense caption text.

4. I am still of the opinion that the language needs to be made clearer and straight-forward, figures should be better presented, and the 'interpretability' aspect is not convincing -- for a 'global' interpretable algorithm, it outputs too much information and requires too much effort from an end user.


clarity,_quality,_novelty_and_reproducibility: The language and presentation needs to be improved as already mentioned above. The algorithm is novel, but the merits are not clear.
summary_of_the_review: While the application of the information bottleneck is novel, relevant literature has been missed out and comparison with existing techniques for interpretability needs to be covered in the experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors develop an interpretability framework based on information theory. The advantage/novelty of the approach being that it works directly on the 'informativeness' of the individual features and their joint interactions w.r.t. a given target under potentially a variety of losses and a variety of neural network architectures as predictive models.  
strength_and_weaknesses: Good:
The approach goes beyond selecting a (discrete) number of active features but rather continuously varies the degree of compression. 
An in depth analysis can be performed by looking at a confusion matrix type of representation that is in principle richer.

Bad:
The claim that the approach improves the interpretability of complex models needs to be better defended: the type of analysis offered in the end is often unclear or complicated: 
- capacity to extract relevant features: the method allows to quantify <<how much information is required - total, and from each feature - for a given level of predictability>>, ok how is this different from the information that we can get from the Lasso Path along the regularization parameter using the LARS algorithm? How does one use this information in practice?
- as an example of analysis, the author report a confusion matrix type of representation that allows to conclude that the model is first paying attention to discriminating specific ranges of values for some variables. How different would this be from partitioning in a few bins each variable and then run a recursive feature elimination selection technique to identify which (interacting) ranges of which variables are important? 
- <<The capacity for successive approximation, allowing detail to be incorporated gradually >> how can this in practice be used to offer insights? is it possible to identify some prototypical scenarios, i.e. if this happens than this can be concluded?
- the experimental section would benefit from including some artificial cases where the known important interactions are known a-priori and it can be shown that they are recoverable by the proposed approach but not by other baselines (such as Lasso or recursive feature elimination selection techniques).
clarity,_quality,_novelty_and_reproducibility: The paper is clear and proposes a novel variation of the Distributed Information Bottleneck framework. 
summary_of_the_review: The ideas presented are of interest but more effort should be put in demonstrating the capacity of the framework to offer useful (and understandable) insights.  

After the authors' modifications I feel the paper has improved and is ready for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for interpretability based on a feature-wise Information Bottleneck. It allows training classifiers that use optimal feature transformations maintaining only the required information. Two forms of visualizations of this information content are presented as global model explanations: Quasi-information plane trajectories with feature-wise information curves and confusion matrices for a single feature to show which information is used by the model. The interpretations are discussed for three tabular datasets with categorical features.
strength_and_weaknesses: Strengths:

1. Overall, I found the presentation well-structured and clear. 
2. Interpretability of AI algorithms is an important and timely topic. Specifically, the tradeoff between model complexity and interpretability is an important subject of study.
3. The presented techniques can provide reasonable insights into feature usage and feature selection. I think the visualizations, as shown in Figures 4 and 5, provide some useful insights into feature importance and can be useful for feature selection.
4. Many information-theoretic notions, such as entropy or mutual information, are sometimes hard to grasp. The distinguishability matrices used in this method help to better understand which aspects are important.

Weaknsses:

1. For the proposed method, an additional compression model needs to be trained, and this combination of models also needs to be trained for a sufficiently high number of betas. This is a lot more than required by other post-hoc techniques and may limit the usability of the method in practice.
2. While the method can quantify the information represented by the features, it does not give insights into how the information is used. For some applications, it might also be beneficial to see the effect of the information on the prediction. It would also be useful to analyze, whether there could be a faulty model that uses the right information but leads to wrong conclusions. Such behavior could not be discerned by the proposed method. Also, interactions are not visualized at all.
3. The motivation and the advantages of the proposed framework over other black-box explanation methods need to be improved. I am not exactly sure what the advantage of the proposed framework over other feature selection / attribution frameworks is. There are other methods that compute attributions, which also allow for more complex dependencies than GAMs or linear models. An advantage of GAMs / linear models is that these models are white box models. However, the proposed method is a black-box method. In this class a variety of methods including general constructs such as SHAP or the information bottleneck by Schulz et al. can also be applied with interactions between features.
4. So far, the method is restricted to categorical tabular data. In the paper, only categorical features are discussed. It would be nice to generalize and deploy the framework for continuous features and other data types, such as image or text data.


Minor comments:

1. Section 2 (Related work): The paragraph that discusses CCA, ANOVA and how linear correlation can be replaced by MI for feature selection seems very far away from the present work. Instead, the space could be used better to discuss works (e.g., those mentioned in the next box) that deploy the IB for interpretability.
2. Section 3 is very long. I would have appreciated subsections to better organize the content.
3. Sum H(X) and Sum I(X,U) (in Figure 2a) only denote the total (mutual) information in all features when the features are independent. Otherwise, it is only an upper bound. As far as I see independence was never assumed, so I was a bit confused here.

Questions:

1. Is the sparsity enforced (that the KLD for some features is 0 bits for some features, BikeShare in particular) or is it a side-product of the IB approach?
2. How long is the computational time of the method compared to other approaches? How many values for the KLD were used to compute the curves shown in Fig. 4?


clarity,_quality,_novelty_and_reproducibility: Clarity: In general, this work is well-presented. The method and experimental setup are well-described. 
Novelty: The information bottleneck has become an established tool in Deep Learning (see https://github.com/ZIYU-DEEP/Awesome-Information-Bottleneck for a long list of works). In particular, its properties have been used for to devise attribution and feature selection methods for interpretable ML before, for instance by Chen et al., Schulz et al., and Koh et al.
While the devised method can be seen as a global explanation technique and the others are local, they are still very similar in spirit. I have not seen the confusion matrix plots before and the quasi-information plots before, but to me it seems to be a combination of existing techniques.
The work is also highly similar to Murphy et al., where a) the distributed bottleneck is already proposed and b) the feature-wise information curves are presented. 

Jianbo Chen, Le Song, Martin Wainwright, Michael Jordan: Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. ICML 2018
Karl Schulz, Leon Sixt, Federico Tombari, Tim Landgraf: Restricting the Flow: Information Bottlenecks for Attribution. ICLR 2020
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, Percy Liang: Concept Bottleneck Models. ICML 2020
Kieran A. Murphy, Dani S. Bassett: The Distributed Information Bottleneck reveals the explanatory structure of complex systems, arXiv 2204.07576

Reproducibility. I have no reproducibility concerns.

summary_of_the_review: The research questions addressed in the paper are timely and interesting to the XAI community. The work presents some promising results and interesting visualizations for the information accessed by a model. However, its computational overhead, the close relation to other works that use IB, and unclear benefits over these works limit the possible impact, which is why I am currently leaning towards rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to track the information between features and model outputs and, through compression/distortion, identify a spectrum of models utilizing a variable amount of information. This spectrum of models then provides a global type of interpretability. 

The reviews initially penalized the work for clarity. Through considerable efforts on the part of the authors and good engagement by the reviewers, these clarity issues were resolved to enough of an extent that three reviewers raised their scores to recommend acceptance. It seems that the work is recognized as novel and impactful---certainly interpretability is a key problem facing blackbox methods in ML.

Regard weaknesses, the reviewers are perhaps not entirely convinced that the paper is in its clearest form, but I think the authors have to be allowed some control. 
justification_for_why_not_higher_score: The reviewers are not unanimous in their assessments of novelty, clarity, impact. I do wonder whether the scores may reflect some amount of anchoring (reticence on reviewers to change their scores/impression too much) and so, had the paper been in its final form, it is conceivable it would have been received better. 
justification_for_why_not_lower_score: Three reviewers moved from reject to accept recommendations and so I think that reflects a strong signal that the work is viewed as worthy of the broader community's attention. The fourth review recommend a strong accept. I personally see an interesting new take on an important problem. It may not be the cleanest version of this idea, but I think the community needs new ideas like this one.
Decision: Accept: poster