Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
This paper adapts the concept of information flows, develops by Venkatesh et al for general computational systems, to (artificial) neural networks with the goal of studying how to "intervene" in an already trained NN to change the way its output is affected by different inputs. The authors then apply this approach, namely a pruning-based approach for altering information flows, in order to manage the bias-accuracy tradeoff in fair classifiers that take sensitive features as inputs.
main_review: 
Overall, the paper presents a very interesting approach to manipulating the label-feature interactions for the sake of tuning the fairness-accuracy tradeoff in neural nets. There are some interesting contributions in the paper, namely the formulation of an information-theoretic counterpart of (demographic) parity, and the development of a simple pruning-based algorithm for evaluating the bias-accuracy tradeoff. On the flip side, the following drawbacks makes me reluctant to fully recommend the paper for acceptance. These issues are listed below:

**+ Confusing introduction, focus and an unclear motivation/objective**

The paper's title, abstract and introduction do not fully reflect what the paper is about. In my view, the key contribution of the paper is proposing an information-theoretic approach to evaluating/ensuring fairness of a neural net **classifier** for a given level of accuracy. The abstract and introduction discuss "interventions" from observational data---in the clinical and neuroscientific context---in a way that suggests learning the causal pathways of clinical interventions (Lines 4-7, Lines 21-24). But the concept of an intervention here is very different as it refers to alterations to the network architecture rather than manipulation of actual interventions in the real world. 

Also, the paper presents fairness as a potential application of the broader methodology of information flows. However, it seems to me that the proposed method has very limited applications beyond that, and it only works for simple binary classifier as estimating the conditional mutual information $I(Y|Z)$ is practically impossible for more complex tasks. I think it would have been better if the paper was written with a focus on the "fairness of classification" application.

**+ Experiments: absence of baselines** 

The authors used the experiments section to show that information flows can be used to find targets for intervention in the context of evaluating the bias-accuracy tradeoff. However, I think a more appropriate research question is whether this is the right way, or a better way, to evaluate this tradeoff. There are many methods for evaluating and ensuring fairness in classifiers that have been proposed in the past few years, that also have the advantage of being model-agnostic and not restricted to neural nets. I think the authors should have compared with these. 

Minor issue: I am also unsure about the validity of the linear fits in Figure 4 given the massive variance and the lack of a clear trend.
















limitations_and_societal_impact: Limitations and social impact are adequately addressed in the discussion section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies whether observational data can be used to estimate the outcomes of certain interventions. This is especially important since cost of interventions can be high and in some cases irreversible. This paper empirically demonstrates the effectiveness of using the M-information flows framework for neural networks to assess different kinds of information flows in the network, and then measuring the change in outcome by intervening on different types of flows. They consider fairness in ML as a case study where there are two types of distinct information flows: the label flow and the bias flow. By intervening on paths (by pruning nodes or edges or reducing weights of edges) which contribute to bias in the outcome, the authors show that they can achieve the desired effect of reducing bias. Thus, the paper shows that information flows can offer useful information about where to intervene to achieve certain outcomes.
main_review: This paper addresses a very important problem that comes up in all kinds of application areas (fairness, healthcare, drug discovery etc). This  paper empirically demonstrates how to use information flows in finding neurons/edges which can be intervened upon to produce a desired outcome. I like using fairness as a use case for such a framework since there are two distinct kinds of flows that can be measured using well defined metrics. This also offers a new perspective on the role of model's structure in contributing to biased outcomes. The methods used in this paper are broadly applicable to other areas of interest in ML and Neuroscience. The paper is also very well written and provides all necessary details, and is thus accessible to a broad audience. Overall, I think this is a very interesting paper.

### Scalability

 * All the neural nets used in experiments are quite toy-ish, with very few layers and very few nodes. I understand that this allows the authors to extensively evaluate all kinds of flows, however, how would this scale to larger, deeper nets with number of paramaters that are beyond the scope of manual inspection?

 * How would size of NNs affect the tradeoffs presented in Figures 3 and 5?

 * In reality, even for the case of fairness, one would want to consider intersectional attributes. This would be true even in domains such as drug discovery. How would the proposed method scale for non-binary outputs and non-binary protected features?

### Other

 * The paper only considers demographic parity (which is fine), however can this be adopted for other definitions like equal odds and equal opportunity?

 * I know this isn't the goal of your paper, but what you're doing by intervening to on Z-information flows sounds similar to some of the adversarial learning methods to remove the effects of the sensitive attribute on the output (eg: https://arxiv.org/abs/1807.00199; https://arxiv.org/abs/1902.00334). It would be interesting to see the effectiveness of pruning (your method) vs adversarial methods.
limitations_and_societal_impact: The authors have acknowledged the limitations and potential societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the possibility of using the measures of information flows to intervene with ANN for reducing bias while improving/maintaining accuracy. It adopts the M-information flow framework proposed by Venkaatesh et al., and proposes a quantitative notion of information flow and a method for estimating such flows. The magnitude of information flows can then be used to understand and compare the impact of different pruning strategies on the outputs, as well as their fairness-accuracy tradeoffs. 
main_review: Overall the paper is well-written and easy to understand. The methods are simple and easy to use. 

My main concerns are as follows. 

1. Most techniques are adopted from the existing works such as the concept of M-information flow, method of estimating information flow based on Fano’s inequality, and typical pruning methods, etc. One major concern I have is the novelty of the paper. 
2. Although the methods seem to be effective on synthetic data and modified adult data, there is no theoretical support/guarantee of the method. Moreover, the experiments only consider relatively small networks (3 features, one hidden layer & 5 features, 2 hidden layers). It is not clear how this method performs on other datasets and larger networks, whether the improvement and effectiveness still preserve in more general settings. 
3. It seems that the method can only be used for the notion of demographic parity fairness, which may limit the generality of the proposed model. 

Additional questions:

1. Information flows are quantified by taking maximum over all subsets of nodes in a layer (Eqn (3)). Are there any particular reasons for taking maximum rather than other operations such as average? 
2. The experiment on synthetic data only considers a small neural network (3 features, one hidden layer). Figure 3 shows that there is no significant difference between different pruning methods regarding the fairness-accuracy tradeoff. Is it possible it’s because the network is too small? I suggest authors performing experiments on a larger-scale network and compare accuracy/fairness between different methods. Intuitively, why does fairness-accuracy tradeoff show the contrarian patterns using pruning on the basis of accuracy-to-bias flow ratio and pruning on the basis of bias-to-accuracy flow ratio?  


limitations_and_societal_impact: Yes
review_title: null

Meta-review: The paper considers how to use information flows to "debug" a neural net model toward desired fairness objectives. The idea seems novel and connects disparate fields in an interesting way. The reviewers appreciated the paper and agree it should be accepted, albeit their ratings were mostly borderline. The reviewers brought up many important concerns that the authors for the most part appropriately responded to. The authors are expected to address these points raised by reviewers in a final version, including as they outlined in their response and incorporate additional discussion or results from their responses into their paper in an appropriate manner.
Decision: Accept (Poster)