Can Information Flows Suggest Targets for Interventions in Neural Circuits?
Authors: Praveen Venkatesh
Sanghamitra Dutta
Neil Mehta
Pulkit Grover
Venue: NeurIPS 2021 Poster (2021)
Keywords: information flow
interventions
neuroscience
information theory
explainability
fairness

Abstract: Motivated by neuroscientific and clinical applications, we empirically examine whether observational measures of information flow can suggest interventions. We do so by performing experiments on artificial neural networks in the context of fairness in machine learning, where the goal is to induce fairness in the system through interventions. Using our recently developed M-information flow framework, we measure the flow of information about the true label (responsible for accuracy, and hence desirable), and separately, the flow of information about a protected attribute (responsible for bias, and hence undesirable) on the edges of a trained neural network. We then compare the flow magnitudes against the effect of intervening on those edges by pruning. We show that pruning edges that carry larger information flows about the protected attribute reduces bias at the output to a greater extent. This demonstrates that M-information flow can meaningfully suggest targets for interventions, answering the title's question in the affirmative. We also evaluate bias-accuracy tradeoffs for different intervention strategies, to analyze how one might use estimates of desirable and undesirable information flows (here, accuracy and bias flows) to inform interventions that preserve the former while reducing the latter.

Can Information Flows Suggest Targets for Interventions in Neural Circuits?
Praveen Venkatesh"* X Sanghamitra Dutta²*, Neil Mehta and Pulkit Grover+ 'Allen Institute, 'University of Washington. Seattle; JJ Morgan Chase AI Research; Deepartment of Electrical and Computer Engineering, "Neuroscience Institute, Carnegie Mellon University
'praveen. "pwaveen. wew.kstese@lllleensstttuuu org, saangham tratt wwwraa2222gmmal...mmmmm com, "neilashn@andrev, cmu, edu, "pulkit@cmu. edu
Abstract
Motivated by neuroscientific and clinical applications, we empirically examine whether observational measures of information flow can suggest interventions. We do so by performing experiments on artificial neural networks in the context of fairness machine learning, where the goal is to induce fairness the system through interventions. Using our recently developed M M-information flow frame- work, we measure the flow of information about the true label (responsible for accuracy, and hence desirable), and separately, the flow of information about a protected attribute (responsible for bias, and hence undesirable) on the edges of a trained neural network. We then compare the flow magnitudes against the effect intervening on those edges by pruning. We show that pruning edges that carry larger information flows about the protected attribute reduces bias the output to a greater extent. This demonstrates that M -information flow can meaningfully suggest targets for interventions, answering the title's 's question in the affirmative. We also evaluate bias-accuracy tradeoffs for different intervention strategies, to analyze how one might estimates of desirable and undesirable information flows (here, accuracy and bias flows) to inform interventions that preserve the former while reducing the latter.
Introduction
The "reward circuit" of the brain controls much of our behavior [1, 2]. is believed that addiction is characterized by a strong bias towards immediate rewards the reward circuit [3], while a different bias, called affective bias, in the same circuit, can lead to depression [4]. Broadly, such biases in the output of the reward circuit control our behavior and responses to stimuli. Recent technological advancements in neural probes and optogenetics [e.g.. see 5, 6] are increasingly enabling us to record neural circuits at a high resolution and alter the network by activating or suppressing nodes and links, giving powerful tools to understand and affect how this circuit processes information. Even weak understanding of the reward circuit is motivating clinical interventions for individuals suffering from depression, addiction, obsessive compulsive disorder, obesity, etc.; patients are starting to receive surgeries for long-term deep-brain electrode implantation in brain regions that are involved in the reward circuit [7-10]. We identify new and largely unexplored problem within this context: how do we perform minimal interventions that "correct" undesirable biases without affecting other functions of this important network much possible?
In this paper, we propose to use the M-information flow framework, a recent advancement of ours, that enables tracking the information flow about specific messages within computational
This work was done while P. Venkatesh and S. Dutta were Ph.D. students at Cameegie Mellon University. ts. Dutta and N. Mehta contributed equally to this work.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

circuit [[1]. We aim to use this framework to identify interventions that can correct undesirable biases within the circuit, while preserving desirable ones. A difficulty that we face, however, is finding a sufficiently comprehensive dataset with simultaneous recordings of the various brain regions involved in the reward circuit. Although advances in experimental techniques suggest that we are on track to have such datasets in the coming years [12, 13], simultaneous multi-area recordings of specific circuits are still rare. To overcome this issue, we use the context of fairness in artificial intelligence [14, 15] to study the above question, which, as we next discuss, provides an excellent analogy to the neuroscientific context.
The problem of reducing biases to improve fairness in decision-makinn system is much like that of reducing undesirable biases in the reward circuit: (i) both systems rely on learned associations from stimuli and responses/labels (i.e., training data); (ii) both systems have intended objective (to learn a desired association between the features and the true labels, or healthy reward- -based learning); and (iii) both systems often learn undesirable associations (e.g., racial gender bias in artificial neural networks, and biases that cause addiction, depression, or obsessive compulsive disorder in the reward circuit). This suggests that if we want to understand how information flows inform interventions to correct biases in the reward network, we can simulate such experiments by examining the relationship between information flows and interventions in artificial neural networks (ANNs) trained on biased datasets.
A distinct advantage offered by our M-information flow framework [11] is that it can track the flows of multiple messages in a neural circuit. In this paper, we adapt our information flow measure to ANNs, and evaluate this measure in an empirical setting involving multiple messages (one each pertaining to bias and accuracy) for the first time. We then perform interventions using different strategies to prune nodes or edges of the ANN, and measure the effect of each intervention by quantifying the change in accuracy or bias at the output of the network.
Goals and Contributions. Concretely, the main goals and contributions of this paper are:
1. Our primary goal is to study whether measuring information flows about a message can suggest targets for interventions in an ANN, to change how its output is affected by that message. Towards this, our first contribution is in adapting M-information flow to the ANN context, developing meaningful way to quantify it.
2. Secondly, we wish to understand the correlation between the magnitude of information flow on an edge and the degree to which an intervention on that edge affects the output. For this, our contribution is an empirical examination of information flows in an ANN, showing that they can predict the effect of interventions in fairness context.
3. Lastly, we wish to understand how we can use information flows corresponding to two different messages, to remove undesirable behaviors while preserving desirable ones. Towards this, we compare different intervention strategies that are informed by information flows, and compare the bias-accuracy tradeoffs achieved by each of these strategies.
The rest of the paper is organized as follows: we discuss related work below, following which we summarize the fundamentals of the M-information flow framework in Section 2. We also show how M-information flow can be adapted to ANNs, discuss the setup of the faimeess context, and explain the rationale for adopting an empirical approach. Section 3, we describe how we empirically discuss the process of estimating M -iiffrmmation flow and the
evaluate our goals: specifically, design of different intervention strategies. Finally, we present our results on synthetic and real datasets in Section 4, and conclude with discussion of the implications of our results in Section 5.
Related work. The idea of using information flow to understand an ANN (and neural circuits more broadly), could be interpreted as new type of "explanation" of ANNs. A large number of works have dealt with problems in the fields of explainability, transparency and fairness in machine learning broadly. as well as for ANNs specifically. Molnar [16] provides good summary of the different approaches taken by many of these methods for explainability. Most of these approaches seek to understand the contribution of individual features [17-20] individual data points [21, 22] to the output. For ANNs specifically, these can also take the form of visualizations to describe what features or what abstract concepts an ANN has learned [23, 24]. There have also been number of information-theoree approaches for measuring bias and promoting faimess in AI systems [25-29].
Our approach in this paper is quite different from these prior works: want to understand what it is about the network structure itself that leads to a certain output. In this sense, our work is similar

---

in spirit to the idea of "lottery tickets" [30], although we take a very different approach involving information flows. In particular, we want to understand which edges carry information relevant to classification, as well as information resulting in bias, to the output. We also want to know which edges need to be changed in order to produce a desired output, e.g., fairness towards protected group with minimal loss of accuracy.
Although we examine how information flow can be used to obtain a bias-aceuracy tradeoff in a fairness context, our goal is not to find optimal tradeoffss between fairness and accuracy, which is the focus of many recent works [31 -31-. Approaches that directly seek to optimize this tradeoff (e.g., by including bias in the training objective, or by using adversarial methods [39, 40)) will likely perform much better than our method, which uses a much more indirect approach to inducing fairness in an ANN. However, those methods would not tell how to edit the network towards particular goal, which is what is needed in clinical and neuroscientific application domains. We use the fairness context only as concrete setting in which to understand whether M -iinfformation flow can inform interventions in a network with complex flows involving multiple messages.
Background and Problem Statement
In this section, we provide brief introduction to M -iinformation flow framework [11], and show how it can be adapted and reinterpreted for ANNs. Then, we set up the fairness problem, and describe how the adapted information flow measure is related to commonly used measures of bias against a protected class, We also discuss why providing theoretical guarantees might be challenging and motivate the need for an empirical approach.
2.1 Adapting and Reinterpreting M-Information Flow for ANNs
Our M-information flow framework provides a concrete way to define information flow about a random variable M-called the message-in general computational system. By changing what M represents, we can understand the flows of any number of different messages (c.g., if Y and Z are two different variables, we can examine Y Y-information flows and Z-information flows in the same system). The computational system, which is modeled after the brain, is a graph consisting of nodes that compute functions, and edges that transmit the results of these computations between nodes. The definition of M -iinformation flow satisfies an important property: it guarantees the ability to track how information about the message M flows from the input to the output of a feedforward neural c Such a model is completely compatible with a feedforward neural network, where the neurons of the ANN act the nodes, and the weights connecting neurons from different layers act as the edges of the computational system.
A key feature of our computational system model is how it accounts for the inherent stochasticity of the brain as well as its inputs: nodes can generate noise intrinsically, and the transmissions on the edges are considered to be random variables. But in trained ANN, the computation every node is deterministic, specified completely by the weights on the incoming edges and the neuron's 's activation function. However, we can still continue to think of the edges' transmissions as being random variables because the input data for the neural network comes from a distribution (which could also be an empirical distribution, i.c., a dataset). We are now in a position to state what our original M-information flow definition [11] looks like in the context of ANNs, before proceeding to adapt it for our purposes in this paper.
Definition 1 (Original M -iinformation flow [1i]). Let an arbitrary edge of the neural network at layert t be denoted E, and let the "transmission" " this edge be denoted (EE). Similarly, let subset of edges at layert be denoted E6 and the set of transmissions on this subset be denoted X ((()) Then we say that information about M flows on the edge E,, i.e., edge E, has M-information flow, if
3 C 8, S.I. I(M;
X())) 0,
(1)
I 'The word "interventions" in this paper refers to editing the weights of an ANN, which may correspond to using targeted external stimulation the brain, e.g. through long-term potentiation [41]. We do not mean interventions in the sense of diffuse medications which have a more widespread impact on brain function.
"In the original framework, which we designed for the neuroscientific context, we considered the neural circuit as being "Feedforward time": i.e.. the computational graph is lime-unrolled in such a way that edges send transmissions from nodes at time t to nodes at time t + 1.
We avoid constructions that introduce stochasticity within an ANN for the sake of simplicity.
3

---

where Et is the set of all edges in layer t, ((;)) represents the set of transmissions on 8 and [M;; A| B) the conditional Shannon-mutun mutuual information between M and A, given B [42, Ch. 2).
The motivation behind this definition is two-fold: (i) One intuitively expects an edge E, to carry information flow about M if its transmission X(E)) depends on M, ie., if I [M; ((,)) > 0. Definition 1 satisfies this requirement. (ii) However, it is possible for two edges E, and E to carry information about M jointly in such a way that (M;; (EE))
0, but T(M; ( (E x(E > 0.* Therefore, Definition is designed to assign M-information flow to both F. and F.t in this case: we previously showed [11, 43] that such assignment is imperative for consistently tracking the information flow about M in a computational system.
[MM (A;:)(EE)
To adapt this definition of information flow to ANNs, we start by recognizing that our computational system model allowed each outgoing edge of a given node to carry a different transmission. In an ANN, however, the outgoing edges of a given neuron all carry the same activation, but with different weights. Consequently, the random variables representing the transmissions are all scaled versions of each other, and have precisely the same information content. Therefore, can define information flows for the activations of every node, rather than for the transmissions of every edge. We then use the properties of edges (specifically, their weights) to construct definition of information flow for edges, so as to identify the most important outgoing edges of each node and intervene in a more selective manner. Furthermore, Definition only specifies whether or not a given edge has M-information flow. However, we require a quantification of Miinformation flow that will let us compare different nodes or edges, and decide which ones to intervene upon. Keeping these aspects in mind, we propose M-information flow for the nodes of an ANN, followed by quantification of M-information flow, and finally a weighted version that assigns different flows to each outgoing edge of given node:
Definition (M-information flow for ANNs). Let an arbitrary node of the neural network at layer t be denoted Ve, and let the activations of this node by represented by the random variable X (V(). Similarly, let an arbitrary subset of nodes at layer be denoted V. and the set of activations of this subset be denoted X (VV)). Then, we say that the node V₁ has M -iinformation flow if
3 s.t. XVM) XTE)) > O.
We quantify M-information flow by taking a maximum" over all subsets of nodes V.' in layer t:
FM (Vi)= max T X(V)) ((())).
'V²
Finally, if E, is an outgoing edge of the node Vs that has weight w E,), then we define the weighted M M-information flow on that edge as
FM (F4) =
44)
2.2 Fairness Problem Setup
The central problem in the field of fair machine learning is to understand how train models for classification or regression without learning biases present in training data. Recent examples in the literature have shown why algorithms biased against a protected group can be of great concern [15, 46), with the rise of automated algorithms in application domains such as hiring [15], criminal recidivism prediction [47] and predictive policing [48].
We consider the problem of training artificial neural networks for classification using datasets that have bias in their features and/or labels. The dependencies between the protected attribute (e.g.. race, gender, nationality, etc.), the true labels, and the features may be described using a graphical model as shown in Figure 1. We assume that the protected attribute Z influences the features (X+), and possibly the label Y. along with some other latent factors encoded in U. We then train an ANN using the labels and features to acquire the predicted label Y.
*Simply take M, X(EF) ~ i.i.d. Ber(1/2) /2) and X(E)) = M (E where @ represents the exclusive-OR operation between two binary variables.
'The intuition behind using a maximum comes from notion called "synergy" [44, 45]. Essentially, if I(M;A A BB) > [M;; A), then B is said to contribute synergistic information about M to A. By taking a maximum, attempt include synergistic contributions from all other activations in the same layer.
'Our definition of weighted M -information flow for the edges of the ANN is admittedly heuristic, however it has a simple rationale: an edge with larger weight has a greater influence on the activation of a target node as compared to an edge with smaller weight, given the same input both.

---

Recall that our goal is to measure the information flows of two different messages: (i) information flows about the protected attribute, i.e., Z-information flows, which we also refer to as bias flows; and (ii) information flows about the true label, i.e., Y-information flows, which we also refer to as accu- racy flows, as these are responsible for accuracy at the output. The measure of bias we consider at the output is an information theoretic version of statisti- cal (or demographic) parity [49], which has also been used in many previous works [36]. This is because Z-information flow at the output is simply T(Z:Y), since there are other edges to condition upon."
Z
X1
X
Figure 1:
graphical model representing the causal relationships assumed between the variables used in the fairness problem. The ANN's output, Y, depends on the features {X;} and the true labels Y, which turn may be influenced by the protected attribute Z and some latent variables (where Z 1 U). Here, U encodes latent factors responsible for "accuracy", i.c., the relationships between [X,] and Y that want to capture. The dashed line from Z to Y indicates that the true labels may or may not be biased (e.g., assume Y is unbiased in our synthetic dataset, but real datasets will likely have bias).
2.3 Theoretical Counterexamples
on M- M-Information Flow and Interventions
This section explains why we adopt an empirical approach in this work, and why our results might be considered non-obvious. Intuitively, we might expect that intervening on an edge with large M- information flow (e.g.. by deleting it) will have a large effect on the dependence between Y and M.
However, there exist counterexamples showing that both the above statement and its converse do not always hold true.
Counterexample 1 (M-information flow does not imply interventional effect). Consider an ANN
that generates an intermediate feature some node V. which is strongly correlated with Z, and thus has large Z- Z-information flow, However, this feature need not contribute to Y: for example, all weights leading out of V2 could be zero. In such case, pruning V/ will have no effect on the bias at the output, even though Fz (V)) can be arbitrarily large.
Counterexample 2 (Absence of M Minnoormation flow does not imply absence of interventional effect). Consider an ANN (and a suitable data distribution) in which the flow of a binary variable M E {0, 1] to Y is "gated" by an independent random variable W. More precisely, suppose the activation of a node V is a variable W that satisfies Fas (V)) = O. Further, let a different node V4+1 (say with ReLU activation) receive both M and W inputs with equal weights. If O, then ((++1)) will depend on the value of M; on the other hand if W < -1, then X(V++1) will always be zero, and hence independent of M. If X(V ), then pruning the node V, (i.e., intervening on W) can change how Y depends on M, even though V₁ itself has no M-information flow.
The title of the paper asks non-obvious question for yet another reason: it asks whether an obser- vational measure of information flow can predict the effect of interventions. The field of Causal Inference [51] shows that it is impossible, in general, to predict causal effects using purely observa- tional methods (i.e., by observing empirical data from a joint distribution of random variables, without manually perturbing them in some way). This fact, along with the aforementioned counterexamples, illustrates why it is difficult to provide theoretical guarantees about whether information flows can predict interventions (we also explored this issue in [43]). This is why we adopt an empirical approach in this paper, to examine whether such predictions can made in practice.
3 Methods: Estimating Information Flow and Implementing Interventions
In this section, we discuss how we estimate the information flow measure described earlier, and propose few different intervention strategies for "editing" trained neural network.
'I is also straightforward to extend our work to alternatiee detinitions of bias such as Equalized Odds 50] by suitably modifying Definition 2. However, it is not clear if such measure can still be meaningfully interpreted an "information flow".
5

---

3.1 Estimating Information Flow
Conditional mutual information is a notoriously difficult quantity to estimate [52]. However, in our empirical study, we only consider datasets where the true label Y and the protected attribute Z are binary. which makes estimation considerably easicr.* 8 We use a classification based method to estimate the conditional mutual information that appears in Definition 2.
First, we construct separate classifiers for predicting and Z from the intermediate activations of a node (or subset of nodes) in chosen layer, which we call X The generalization accuracy of this classifier indirectly tells us the extent to which information about Z (say) is present in Xi"i, More precisely, if the generalization accuracy of classifying Z from Xtint is that means the expected probability of error in correctly guessing Z from Xi is Pe == 1- a. Then, from Fano's 's inequality [42, Ch. 2]. we have:
H(Z Hp(P.) + Pe log(1 - 1)
(5)
where Hy is the binary entropy function, and Z is the sample space of Z. Since Z is binary, 2 E 1). hence [2] = 2. If we further assume Z ~ Ber(1/2), then H() = 1 bit. This simplifies
the above equation to:
H Xi < (P.))) = Ho(1 - a) [Z; =
(6)
H X _t) = = H x 2 1 H1(1-a) a).
(7)
Therefore, given any classifier that can predict Z from imt with generalization accuracy a, we can compute lower bound on the mutual information between Z and Xini, with a better classifier providing a tighter lower bound. This allows us to compute all conditional mutual information quantities required by Definition 2 using the chain rule [42, Ch. 2]:
[( x
| = (2; x x - ((;
(8)
where ximtt and X are the activations of two intermediate nodes (or subsets of intermediate nodes) in the ANN. We use both linear and kernel Support Vector Machines for classifiers. Further details on estimating the mutual information can be found in Appendix A in the supplementary material.
3.2 Intervention strategies
We perform interventions on an ANN by pruning its edges. To evaluate bias-accuracy tradeoffs, we use "soft-pruning" of edges, i.c, gradually reducing their weights rather than removing them outright. We compare a number of pruning strategies, which differ based on the following factors:
1. Pruning method. We consider three methods: (i) pruning entire nodes; (ii) pruning individual edgex; and (iii) pruning paths from the input layer to the output layer. Pruning nodes, i.e., simultaneously decreasing the weights of all outputs of a particular node, can be interpreted as decreasing the dependence of future computations on the derived feature that this node represents. Pruning individual edges has the advantage of being extremely specific, potentially allowing for similar results with minimal changes to the network. Pruning paths can be interpreted as stopping the largest net flow of information from the input features all the way to the output.
2. Pruning metric. For each of the above methods, we use two different metrics: the primary metric based on the ratio of bias flow to accuracy flow, and a secondary metric the ratio of accuracy "control" to show that our tradeoff results are not merely
flow to bias flow, which product of chance. When soft- pruning nodes, the primary metric we use is the ratio of flows at the node, i.c. Fzz (V (V²). When soft-pruning edges, our primary metric is the weighted ratio of flows on the edge, i.e., (E,)) F / (E,) (note that we need to re-weight the ratio since weights in Fz (E₂) and Fy (E,) would otherwise cancel). Finally, when
paths, score each path using the smallest adge-metric, and choose the path P having the largest score:
P === argmax min Iw Ex)] Fz / Fy (Ex).
P
E,EP
3. Pruning level. In each of the above cases, we soft- the top "k" nodes, edges paths, which exhibit the largest value of the respective pruning metric. Pruning too much will likely hinder all flows, so we test a few different values of k to find optimal "level" of pruning.
'Extensions to non- binary M (e.g.. intersectional attributes) are certainly possible, and would require the use of different mutual information estimators, suitably tailored to the dataset [53-57].

---

Aue: weighted acc flow (Dataset: Synthetic, MI est: Kernel SVM)
Asi weighted blas flow
Change in output acc upon pruning (new acc old Sk-points)
Change in outout bias upon pruning (new bias old bias; points)
(Dataset: Synthetic, MI est: Kernel SVM)
Mf = 0.22
RG 0.00
0-
1
1
-10
0.04
10
Weighted accuracy flow, (F())))
Weighted bias flow, Fa(
Figure 2: (Left) A plot showing the dependence between the weighted Y-information flow of every edge and the change in accuracy (Aace) at the output of the ANN upon pruning that edge, for the synthetic dataset. (Right) The same for weighted Z-information flow and change in bias (Assos). ). In both figures, the information flow of an edge increases, there is greater decrease in the accuracy or bias at the output upon pruning that edge. The lighter data points indicate edges in the first layer, while the darker points represent edges in the second (final) layer, with grey and black lines representing linear fits. R2 is the fraction of variance explained by the linear fit, for points in layer
Bias-accuracy tradeoff
Bias-accuracy tradeoff
(Dataset: Synthetic, MI est: Kernel SVM)
(Dataset: Synthetic, MI est: Kernel SVM)
(%6)
64
Accuracy (%))
63
63
62
62
Configuration
Configuration
61
Blas/Acc: nodels)
61-
Acc/Bias: node(s)
Blas/Acc: nodels)
60
Acc/Blas: node(s)
Blas/Acc: edge(s)
Acc/Blas: edge(s)
Blas/Acc: edge(s)
Acc/Bias: edge(s)
52
54
58
60
62
64
66
68
70
72
74
76
Bias (%)
Bias (%)
Figure 3: Figures showing the tradeoff' between fairness and accuracy when gradually pruning nodes or edges of an ANN trained on the synthetic dataset. * denotes the accuracy and bias after training but before pruning. The legend indicates the pruning strategy used: pruning based on to-acurcy flow ratio (left) or accuracy- bi-b flow ratio (right); pruning nodes edges; and the number of nodes or edges pruned. Error bars represent one standard error of the mean. Pruning based on the bias- to-accuracy ratios causes bias to fall faster than accuracy, while pruning based accuracy-to to-bass ratios causes bias to increase while accuracy falls.
Results
4.1 Synthetic Dataset
First, we examine information flows for a small neural network trained on synthetic dataset. The synthetic dataset is generated in a manner similar to Figure (see Appendix B.1 in the supplementary material for details). The dataset has three continuous-valuee features X1, X2 and X3. a binary label Y. and a binary protected attribute Z. X1 and X, are designed to receive large causal influence from Z, while X3 and the labels Y have no influence from Z. Lastly, all three features independently provide information about Y, i.e., they are noisily correlated with Y. with independent noise terms.
For simplicity, the neural network was taken to have just one hidden layer with three neurons, with leaky ReLU activations. The output layer is a one-hot encoding of the binary Y and cross-entropp loss was used for training. The data was split, with 50% used for training the neural network, and 50% for estimating information flows on the trained network. We used kernel SVM classifier to estimate information flow (as described Section 3. and nested cross validation to fit SVM hyperparameters and estimate generalization accuracy. Finally, all analyses are repeated across 100

---

Aur vs. welghted flow
Atios weighted blas flow (Dataset Adult, MI est: Linear SVM)
Change in output acc upon pruning
(Dataset: Adult, MI est: Linear SVM)
Change in outout tlas upon pruning (new bias old bias; points))
(new acc old Sk-points)
-2-
2-
-2
Ri 00.00 R 0.01
R] =027 Rg 0.03
10
Weighted accuracy flow, Fy(E)
Weighted bias flow, (P(
Figure 4: Plots analogous to those in Figure 2, for the Adult dataset trained on the smaller ANN.
Bias- accuracy tradeoff (Dalasel: Adult, MI est: Linear SVM)
Bias-accuracy yy tradeoff (Dataset: Adult, Mi est: Linear SVM)
73
73-
Configuration
AcciBlas: 1 node(s)
72
72-
AccBilas: 2 nodels)
AcciBias: 1 edgels)
Accuracy (66)
72
71
AcciBlas: 2 edgets)
Configuration
BiasMAce:1 nodets!
B
Blan/ee: edge(s)
Bias/Acc:2 2 edge(st
Figure 5: Tradeoff plots analogous to Figure 3, for the Adult dataset trained on the smaller ANN.
neural networks trained on the same data but with different random weight initializations. Further details are in Appendix C.1 in the supplementary material.
We first analyzed whether the extent of change in output bias (Abias) accuracy (Auce) upon pruning an edge is related to the magnitude of Z- or Y -iinformation flow on that edge. We completely pruned each edge of the trained network, keeping all other edges intact, and examined the change in accuracy and bias at the output (see Figure 2). The results show that pruning edges with larger magnitudes of weighted accuracy bias flow tends produce larger reductions in accuracy bias at the output, respectively. We also see a marked difference in the slopes of the black and grey lines, indicating that edges from the second layer are less likely change accuracy or bias upon being pruned: this is discussed in the following subsection. While the linear fits do not capture a large fraction of the overall variance, the dependence between information flow magnitude and Absas or Aace is evident: for completeness, correlation values and statistical significance tests are provided in Appendix D in the supplementary material.
Next, we analyzed how bias and accuracy evolve upon "soft-pruning' the edges gradually, for different pruning strategies outlined in Section 3.2. Figure shows that, when pruning on the basis of bias-to-accuracy flow ratio, bias initially falls much faster than accuracy: on average, bias can be reduced almost 10 percentage points, at a cost of just 2 percentage points in accuracy. The tradeoff curves produced by different pruning methods and levels were similar for this dataset, but interestingly, pruning on the basis of the accuracy-to-bl
flow ratio causes accuracy to remain steady before falling, while bias increases. This may suggest the presence of some redundancy: when edges most responsible for accuracy are pruned, other edges (presumably from nodes that have larger bias) are able to compensate, but at the cost of increasing bias.
4.2 Real Datasets: Modified Adult and MNIST
We also performed the same analyses on the Adult dataset from the UCI machine learning reposi- tory [58] and the MNIST dataset [59]. The Adult dataset, which comes from the 1994 US census, consists of a mix of numerical and categorical features for classifying people with annual incomes
8

---

Bias-accuracy tradeoff in a larger (5x3x3x2) (Dataset: Adult, MI est: Linear SVM)
Blas-accuracy tradeoft
(Dataset: MNIST, MI est: Correlation)
75-
Confguration
Configuration
Bias/Acc: nodels)
Blas/Acc: node(s)
74-
Blas/Aoc: nodels)
Bias/Ace: 10 node(s)
Blas/Arc: edgets)
95
Blas/Acc: 15 edge(s)
73
Bias/Acc: edgels)
Accuracy (%)
Bias/Acc: edge(s)
Accuracy
72-
Blas/Arc: pathls)
Blas/Acc: pathis)
00-
Blas/Acc: pathis)
Blas/Acc: 10 path(s)
11-
Acc/Bias: nodels)
Acc/Bias: node(s)
AcciBlas: nodels)
Acc/Blas: 10 node(s)
Acc/Blas: 3 edge(s)
Acc/Blas: 15 edgets)
AcciBlas: edgels)
Acc/Bias: edge(s)
69
AcciBlas: pathis)
80-
Acc/Blas: pathis)
68-
Acc/Bias: pathls)
Acc/Bias: 10 path(s)
58
60 62
64
60
65
70
Blas (%)
Blas (%)
(a) Tradeoff plot for the modified Adult dataset, trained (b) Tradeoff plot for the MNIST dataset, as on slightly larger ANN, with five input features and described in Appendix B.3 and C.I.
two hidden layers with three neurons each.
Figure 6: Tradeoff plots for the Adult and MNIST datasets. In both plots, the green lines are for the path-basee based pruning method (described in Section 3.2). Note that both the bias-to-accuract and accuracy-to tobs flows have been shown in the same plot.
less than and greater than S50k. We perform our experiments on two ANNs: small neural network, identical to the one used for the synthetic dataset, and slightly larger network having five input fea- tures, two hidden layers with three neurons each, and a one-hot-encoded output layer. For the smaller network, we use three numerical features sex to be the protected attribute. For the larger network, we additionally use continuous-valuee embeddings derived from two categorical features, occupation and workclass, to test that our observations are not limited to specific topology. We also used only a subset of the records order to equalize the number of individuals with high and low incomes and the number of male and female individuals. However, we introduced bias in the true labels by skewing the dataset towards higher incomes among males and lower incomes among females (at ratio of 2:1). In the figures, simply use "Adult dataset" to refer to this modified version of the Adult dataset. Information estimates were performed using linear SVM (refer to Section 3.1).
dduuatiionnan hours per- and age and take
Results analogous to the synthetic dataset are shown in Figures 4 and 5 for the smaller ANN, and in Figure 6a for the larger ANN trained the modified Adult dataset. These results reflect the trends seen in the synthetic dataset. The smaller ANN shows a stronger dependence between information flow and interventional effect for the Adult dataset than when trained on the synthetic dataset (compare Figures 2 and 4). This might be because the synthetic data had two highly biased features, whereas the modified Adult dataset likely has much less bias in its features. Figures 2 and 4 also show that interventions in the first layer (lighter points) tend to have a greater effect on output accuracy/bias than those in the second layer (darker points)-thh this is particularly pronounced in Figure 4 (left). We believe this is because of the intrinsic redundancy in one-hot encoding: pruning a single edge in the second layer will only affect one of the two output neurons, which encode the binary random variable Y, and therefore provide the same information.
In Figures 5 and 6a, pruning edges does worse than pruning nodes (e.g., compare 2-nodes with 6-edges in Figure 6a, both of which prune 6 edges). This suggests that reducing the impact of an entire (derived) feature at a node is a more robust intervention than attempting to edit individual edges. We also see that when pruning using accuracy-to-bias flow ratios (dashed lines), accuracy falls quickly and bias increases only modestly (in contrast to Figure 3 for the synthetic dataset). We believe this indicates that the modified Adult dataset is dominated more by accuracy flows while the synthetic dataset is dominated by bias flows (this also substantiated by the magnitudes of information flow seen in Figures 2 and 4). Finally, the more sense for deeper networks) is shown in Figure 6a, and is similar to the edge-based methods in its tradeoff performance when using weighted bias-to- to-accurycy flow ratios. Interestingly however, when pruning based on weighted accuracy-tor to-yy--toiiiiee bias flow ratios, there is no increase in bias, suggesting that path-based methods might have greater specificity to particular message.
pruning strategy (which makes

---

To test the scalability of our methods to wider and deeper ANNs, we also repeated our analyses on an ANN with 5 hidden layers consisting of 6 units each, trained on the MNIST dataset (details are provided in Appendix B.3 and C.1 in the supplementary material). Information flow was estimated using a fast correlation- estimator (see Appendix A.3), and the results mirror the trends from the other datasets (see Figure 6b). An interesting observation is that, as the size of the network increases, one has to prune a larger number of nodes, edges or paths to achieve similar trade-offs. We also performed a dependency analysis (not depicted), similar to that of Figures 2 and 4. However, these showed a very poor linear fit; correlation values were also low with inconsistent statistical significance (see Appendix D for details). These observations might be attributed the fact that pruning individual edges is unlikely to have a significant impact on the output in such a large network.
Further results, including more extensive sampling of pruning strategies and fast correlation bbssed information flow estimates, are provided in the supplementary material. Code to generate these results is available online at https // //ggtt/t../////////////tt.thht com/praveenv253/ anwaaann-----ia-aaaaae-ia--iieoe
5
Diseussion
Our results show that M Miinformation flow can indeed suggest targets for intervention in trained artificial neural network, to change its behavior at the output. We can simultaneously measure the information flows of multiple variables within the system, and make edits to prevent undesirable behaviors while preserving desirable ones.
This has important implications for neuroscience and clinical practice, where an observational understanding appears to be necessary prior to deciding the form of targeted intervention. Today, studies typically report results from interventions at a small number of sites (usually just one), while the number of sites for potential interventions is large. The number of possible interventions grows exponentially with the number of sites, which, along with the amount of data required for each, and the irreversibility of interventions, necessitates the use of new approaches that do not iterate through candidate interventions. On the other hand, there are several well-established neural recording modalities for making observations, and there have been rapid advances in non-invasive recording and stimulation techniques [60-65]. Thus, neuroscientists who wish explore causal mechanisms in neural circuits can measure information flows to narrow down target brain areas for confirmatory experiments that use optogenetic stimulation- in this way, information flows can inform experiment design. Developing such an understanding is crucial for the eventual goal of treating brain disorders (e.g., in the reward circuit); furthermore, mapping information flows might provide a roadmap to personalized deep-brain stimulation strategies [66].
Our work is intended to be an initial exploration of how we might make interventions by studying information flows. Our results show that there is substantial room for improvement: there is large variance in how well the interventions suggested by our approach perform (note that this not entirely unexpected, as discussed in Section 2.3). Future studies may examine other intervention strategies, datasets, and more sophisticated networks. Some challenges remain in extending these results to neuroscientific setting. Firstly, the computational system model (nodes and edges) [11] may be too simplistic for biological neurons where even single cell has sophisticated dynamics. Secondly, measurements in real neural systems are firquently noisy, corrupted by background brain activity as well as sensor noise, while we assume here that noiseless measurements can be made. Similarly, although technologies for making interventions on specific cells are rapidly advancing [6], it is still difficult to perform precise neuromodulation, especially without genetic manipulation of cells.
Potential negative societal impacts. This study discusses approaches for making informed inter- ventions in neural circuits, with potential applications to the brain. As with any new technology, this carries potential risks of misuse or abuse, c.g., imprudent overzealous interventions in clinical or non-clinical settings, with or without the subject's consent. Proper regulation and oversight will be key in preventing such misuse. On more imminent time-scale, the techniques developed here might be misused in fairness settings, despite having no formal guarantees. Although our paper uses the context of fairness understand whether information flows can suggest interventions, we have not tested this approach rigorously enough for it to be employed to induce faimess in real systems-indeed this was not our goal. While we believe that information flow analyses may, in future, be used to guide the systematic editing of neural networks for purposes such as bias reduction, that would require more thorough analysis of optimal information measures and pruning strategies.
10

---

Acknowledgments
We thank Cosma R. Shalizi for suggesting comparison between the magnitudes of information flow and interventional effect. We also thank Robert E. Kass and José M. F. Moura for useful discussions.
Funding disclosures. This material is based upon work supported by the National Science Founda- tion under Grant No. CCF-1763561. P. Venkatesh was supported in part by Fellowship in Digital Health from the Center for Machine Learning and Health at Carnegie Mellon University. S. Dutta was supported in part by Cylab Presidential Fellowship and a K&I. Gates Presidential Fellowship. N. Mehta was funded in part by fellowship supported by the CMU-Portugal program.
This paper was prepared for informational purposes and is not a product of the Research Department of J.P. Morgan. J.P. Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, recommendation. offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
Disclaimer.
References
[1] S. Cooper, A. Robison, and M. S. Mazei-Rohison, "Reward circuitry addiction," Neurotherapeutics. vol. 14, no. 3, pp. 687-697, 2017.
[2] N. D. Volkow, M. Michaelides, and R. Baler, "The neuroscience of drug reward and addiction." Physiologi- cal Reviews, vol. 99, no. 4, pp. 21 15 2140, 2019.
[3] C. A. Boettiger, J. M. Mitchell, V. C. Tavares, M. Robertson, G. Joslyn, M. D D'Esposito, and H. L.. Fields, "Immediate reward bias in humans: fronto-parictal networks and role V58VallVal genotype," Journal of Neuroscience, vol. 27, no. 52, pp. 14 383-14 391, 2007.
[4] E. J. Kilford, L.. Foulkes, R. Potter, S. Collishaw, A. Thapar, and E. Rice, "Affective bias and current, past and future adolescent depression: familial high risk study," Journal of Affective Disorders, vol. 1 174, pp.
265-2711 2.51271,222 2015,
[5] N. A. Steinmetz, C. Aydin, A. Lebedeva, M. Okun, M. Pachitariu, M. Bauza, M. Beau, J. Bhagat, C. Böhm, M. Broux et al., "Ncuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings," Science, vol. 372, no. 6539, 2021.
[6] E. S. Boyden. "A history of optogenetics: the development of tools for controlling brain circuits with light," F1000 biology reports, vol. 3, 2011.
[7] J. v. Luigjes, W. Van Den Brink, M. V. Feenstra, P. Van Den Munckhof, Schuurman, R. Schippers,
A. Mazaheri, T. De Vries, and D. Denys, "Deep brain stimulation in addiction: a review of potential brain targets," Molecular psychiatry, vol. 17, no. 6, pp. 572-583, 2012.
[8] S. Delaloye and P. E. Holtzheimer, "Deep brain stimulation in the treatment of depression." Dialogues in clinical neuroscience, vol. 16, no. 1. p. 83, 2014.
[9] C. Borders, F. Hsu, A. J. Sweidan, E. S. Matei, and R. G. Bota, "Deep brain stimulation for obsessive compulsive disorder: A review of results by anatomical target," Mental illness, vol. 10, no. 2, 2018. [10] D. A. Formolo, J. M. Gaspar, H. M. Melo, T. Bichwald, R. J. Zepeda, A. Latini, M. S. Okun, and R. Walz, "Deep brain stimulation for obesity: review and future directions." Frontiers Neuroscience, vol. 13, p. 323, 2019.
[11] P. Venkatesh, S. Dutta, and P. Grover, "Information flow in computational systems." IEEE Transactions on Information Theory, vol. 66. no. 9, pp. 5456-5491, September 2020. [Online]. Available: https://doi.org/10.1 (09/TTT:2020.2987788
[12] I. H. Stevenson and K. P. Kording, "How advances in neural recording affect data analysis." Nature neuroscience, vol. 14, no. 2, pp. 139-142, 2011.
[13] A. E. Urai, B. Doiron, A. M. Leifer. and A. K. Churchland, "Large-scale neural recordings call for new insights to link brain and behavior." arXiv preprint arXiv:2103. 2021.
[14] S. Barocas, M. Hardt, and A. Narayanan, Fairness and Machine Learning. fairmlbook.org. 2019. [Online]. Available: http://www.ffiit http://wwi.mmlbboooorggggg
[15] T. W. House, "Big data: report on algorithmic systems, opportunity, and civil rights." 2016. [Online]. Available: http://wwww.....e..e//////eettan data discrimination.pl
0504
[16] C. Molnar, Interpretable Machine Learning. 2019, http::///chiiishhnnngggbbbbooooe bloookkk/
11

---

[17] M. Sundararajan. A. Taly, and Q. Yan. "Axiomatic attribution for deep networks." in International Conference on Machine Learning. PMLR, 2017, pp. 3319-3328.
[18] P. Dabkowski and Gal, "Real time image saliency for black box classifiers." in Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017, pp. 6970-6979. [Online]. 210000000000020000109900000000 19800999999922hhh pf
Available: httpss//proceedings. https//poooodingggnnnsssccce/ee
[19] U. Bhatt, A. Weller, and J. M. F. Moura, "Evaluating and aggregating feature-based model explanations." in Proceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI-20. International Joint Conferences on Artificial Intelligence Organization. 7 2020. pp. 3016 30222.
[20] A. Datta, S. Sen, and Y. Zick, "Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems," in 2016 IEEE Symposiun on Security and Privacy, 2016, pp. 598-617.
[21] P. W. Koh and P. Liang. "Understanding black-box predictions via influence functions," in International Conference on Machine Learning. PMLR, 2017, pp- 1885-1894.
[22] B. Kim. O. Koyejo, R. Khanna et al.. "Examples are not enough, learn to criticize! Criticism for interpretability." in NIPS, 2016, pp. 2280-2288.
[23] C. Olah, A. Satyanarayan, I. Johnson, S. Carter, I.. Schuhert, K. Ye, and A. Mordvintsev, "The building blocks of interpretabilityy" Distill, 2018. [Online]. Available:
httplluuuhh///uullllai [24] C. Olah, A. Mordvintsev, and L.. Schubert, "Feature visualization." Distill, 2017. [Online]. Available:
http:///istill.pub/0////// 7ae///00////auuee-ee /fauo/////eauueessvuuiiiiiooooo visualization
[25] S. Dutta, P. Venkatesh, P. Mardziel, A. Datta, and P. Grover, "An cntomatiio-theoreete quantification of discrimination with exempt features," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 3825-3833.
[26] H. Wang. B. Ustun, and F. Calmon, "Repairing without retraining: Avoiding disparate impact with counterfactual distributions,"
International Conference on Machine Learning. PMLR, 2019, pp.
0618-7627.
[27] H. Wang, H. HsU, M. Diaz, and F. P. Calmon, "To split or not to split: The impact of disparate treatment in classification," arXiv preprint trXiv:2002.04788. 2020.
[28] A. Ghassami, S. Khodadadian, and N. Kiyavash, "Faimess in supervised learning: An information theoretic approach," in 2018 TEFE International Symposium on Information Theory (ISIT), 2018, pp. 176-180.
[29] S. Dutta, P. Venkatesh, P. Mardziel, A. Datta, and P. Grover, "Fairness under feature exemptions: Coun- terfactual and observational measures," IEEE Transactions on Information Theory, vol. 67, no. 10, pp. 6675-67 2021.
[30] J. Frankle and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks," International Conference on Learning Representations, 2019. [Online]. Available: wtw.reerreeie.e.e/oouummmooe
[31] A. K. Menon and R. C. Williamson, "The cost of fairness in binary classification," in Proceedings of the Conference on Fairness, Accountability and Transparency. 2018, pp. 107-118.
[32] H. Zhao and G. Gordon, "Inherent tradcoffs in learning fair representation." arXiv preprint Xiv: 10w..9960.0 08366, 2019.
[33] I. Y. Chen, F. D. Johansson, and D. Sontag, "Why is my classifier discriminatory?" in Advances in Neural Information Processing Systems. 2018, pp. 3539-352 -3500.
[34] S. Sabato and E. Yom-Tov, "Bounding the faimess and accuracy of classifiers from population statistics." in International Conference on Machine Learning. PMLR, 2020, pp. 8316-8325.
[35] J. S. Kim, J. Chen, and A. Talwalkar, -Model-agnostic characterization of faimness trade-offs." arXiv preprint arXiv:2004.03:: 203444 2020.
[36] A. Agarwal, A. Beygelzimer, M. Dudík, J. Langford, and H. Wallach, "A reductions approach to fair classification," in International Conference on Machine Learning. PMLR, 2018, pp. 60 69.
[37] A. Blum and K. Stangl, "Recovering from biased data: Can fairness constraints improve accuracy?" 2019. [38] S. Dutta, D. Wei, H. Yueksel, P.-Y. Chen, S. Liu, and K. Varshney, "Is there a trade-off between fairness and accuracy? A perspective using mismatched hypothesis testing," in International Conference on Machine Learning. PMLR, 2020, pp. 2803-2813.
[39] C. Wadsworth. F. Vera, and C. Piech. "Achieving faimess through adversarial learning: an application to recidivism prediction," in 5th Workshop on Fairness, Accountability; and Transparency in Machine Learning (FAT/ML 2018), 2018. [Online]. Available: https://www.fattt hww.://wwwwgmeeeadoommmeneeeeeee cateesstttouuhhahdds aaddverearialllammnng..o
[40] A. Morales, J. Fierrez, R. Vera-Rodriguez, and R. Tolosana, "Sensitivenets: Learning agnostic representa- tions with application to face images," IEEE Transactions Pattern Analysis and Machine Intelligence.
vol. 43, no. 6. pp. 2158-2164, 2020.
12

---

[41] T. J. Teyler and P. DiScenna, "Long-term potentiation." Annual review of neuroscience, vol. 10, no. 1. pp.
131-161, 1987.
[42] T. M. Cover and Thomas, Elements of Information Theory. John Wiley & Sons, 2012.
[43] P. Venkatesh, S. Dutta, and P. Grover, "How else can we define information flow in neural circuits?" in
2020 IEEE International Symposium on Information Theory (ISIT). IEFE, 2020, pp. 2879-2884.
[44] N. Bertschinger, J. Rauh, E. Olbrich, Jost, and N. Ay. "Quantitying unique information." Entropy, vol. 16,
no. 4. pp. 2161-2183, 2014.
[45] T. Lizier, N. Bertschinger, J. Jost, and M. Wibral, "Information decomposition of target effects from multi-source source interactions: Perspectives on previous, current and future work," Entropy, vol. 20, no. 4, 2018. [46] S. Barocas and A. D. Selbst, "Big data's 's disparate impact," Calif. I.. Rev. vol. 104, p. 671, 2016.
[47] J. Angwin, J. Larson, S. Mattu, and L. Kirchner, "Machine bias: There's software used across the country predict future criminals. And it's biased against blacks." ProPublica, 2016. [Online]. Available: http://wwwwwooubicc..ooo wrg.aartiicceemmmiiieee----ee bias risk assessments- assesments-nn-ciiinna sentencing
[48] A. G. Ferguson, "Policing predictive policing." " Washington University Law Review, vol. 94, p. 1109, 2016.
[49] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, "Fairness through awareness," in Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, 2012, pp. 214-226.
[50] M. Hardt, E. Price, and N. Srebro, "Equality of opportunity in supervised learning," Advances in neural information processing systems, vol. 29, pp. 3315-3323, 2016.
[51] Peters, D. Janzing, and B. Schölkopf, Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[52] W. P. Bergsma, "Testing conditional independence for continuous random variables," EURANDOM report, no. 049, 2004.
[53] A. Kraskov, Stögbauer, and P. Grassberger, "Estimating mutual information." " Physical review E, vol. 69, no. 6, p. 066138, 2004.
[54] W. Gao, S. Kannan, S. Oh, and P. Viswamath, "Estimating mutual information for discrete-comiinuous mixtures." Advances in Neural Information Processing Systems, vol. 2017, pp. 5987-5998, 2017.
[55] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm, "Mutual information neural estimation," in International Conference on Machine Learning. PMLR, 2018, pp. 531-540.
[56] S. Mukherjee, H. Asnani, and S. Kannan, "CCMI: Classifier based conditional mutual information estimation," in Uncertainty in artificial intelligence. PMLR, 2020, pp. 1083-1093.
[S7] A. Mondal, A. Bhattacharjee, S. Mukherjee, H. Asnani, S. Kannan, and A. Prathosh, "C-MI-GAN: Estimation conditional mutual information using minmax formulation," in Conference on Uncertainty in Artificial Intelligence. PMLR, 2020, pp. 849-858.
[58] D. Dua and C. Graff, "UCT machine learning repository," 2017. [Online]. Available: http:
[59] Y. LeCun, C. Cortes, and C. Burges, "MNIST handwritten digit database." " 2010. [Online]. Available: htt::/yyann.lecum httllyaannecu..co//elttiit com.exdd//n/iiitttt
[60] A. K. Robinson, P. Venkatesh, M. J. Boring, M. J. Tarr, P. Grover, and M. Behrmann, "Very high density EEG clucidates spatiotemporal aspects of carly visual processing," Scientific Reports, vol. 7, no. 1, pp. 1-11, 2017.
[61] J. S. Phillips, A. Greenberg, A. Pyles, S. K. Pathak, M. Behrmann, W. Schneider, and M. J. Tarr, "Co-analysis of brain structure and function using fMRI and diffusion-weiighted imaging." Journal of Visualized Experiments: JoVE, no. 69, 2012.
[62] A. K. Fishell, T. M. Yocuu, K. M. Bergonzi, A. T. Eggebrecht, and J. P. Culver, "Mapping brain function during naturalistic viewing using high-density diffuse optical tomography," Scientific Reports,
vol. 9, no. 1, pp. 1-11, 2019.
[63] J. Cao and P. Grover, "Stimulus: Noninvasive dynamic patterss neurostimulation using spatio-temporal interference," IEEE Transactions on Biomedical Engineering, vol. 67, no. 3. pp. 2019997,001.
[64] M. Forssell, C. Goswami, A. Krishnan, M. Chamanzar, and P. Grover, "Effect of skull thickness and conductivity current propagation for noninvasively injected currents," Journal of Neural Engineering vol. 18, no. 4, p. 046042, 2021.
[65] M. Forssell, Jain, C. Goswami, S. Caldas-Martinez, Grover. and M. Chamanzar, "Effect of focality of transcranial currents on neural responses," in 2021 10th International JEEE/EMBS Conference on Neural Engineering (NER). IEEE, 2021, pp. 289-292.
[66] P. Venkatesh, D. Sneider, M. Danish. N. D. Sisterson, N. Zaher, A. Urban, P. Grover, R. M. Richardson, and V. Kokkinos "Quantifying frequency modulation response biomarker in responsive neurostimulation," "
13

---

Journal of Neural Engineering, vol. 18, no. 4, 046017, 2021.
[67] F. Pedregosa al., "Seikit-leam: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825 2830, 2011.
[68] L. Van der Maaten and G. Hinton, "Visualizing data using t-SNE," Journal of machine learning research,
vol. 9, no. 11, 2008.
[69] A. Paszke et al., "PyTorch: An imperative style. High-performance deep learning library." in Advances
in Neural Information Processing Systems 32, 2019. pp. 8024-8035. [Online]. Available: http://papers. neurips.cc/pag srrrips.ceppape/900555 1-5--yyorch-hee haaprr9000--tttoorca-a--eeeeeee iaaiimeratiiee-eee style-high hepprraawesseeeggggipprnneeeeeee ceanaance-dee--eaannnn learning-library.dee
[70] P. Virtanen et al., "SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python," Nature Methods, vol. 17, pp. 261-272, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The contributions are concretely listed in the Introduction.
(b) Did you describe the limitations of your work? [Yes]] See the Discussion section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the Discus- sion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main experimental re- sults (either in the supplemental material or as a URL)? [Yes] These details are provided in the supplementary material. The code can be found https: h/pp::///h //gi h/gggtttubb... com praveenv2v////////// ann info-flow
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] These details are provided in the supplementary material.
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See Figures 3, 5 and 6.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] These details provided in the supplementary material.
If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We used the Adult dataset from the UCT MI. repository and the MNIST dataset, and cited both appropriately.
(b) Did you mention the license of the assets? [N/A] Neither dataset appears to explicitly specify license, but they have been widely used by the research community; we assume they are public domain.
Did you include any new assets either in the supplemental material or URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica- ble? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "c1gkCisbpXL", "reviewer": "Reviewer_YNbt", "summary": "\nThis paper adapts the concept of information flows, develops by Venkatesh et al for general computational systems, to (artificial) neural networks with the goal of studying how to \"intervene\" in an already trained NN to change the way its output is affected by different inputs. The authors then apply this approach, namely a pruning-based approach for altering information flows, in order to manage the bias-accuracy tradeoff in fair classifiers that take sensitive features as inputs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOverall, the paper presents a very interesting approach to manipulating the label-feature interactions for the sake of tuning the fairness-accuracy tradeoff in neural nets. There are some interesting contributions in the paper, namely the formulation of an information-theoretic counterpart of (demographic) parity, and the development of a simple pruning-based algorithm for evaluating the bias-accuracy tradeoff. On the flip side, the following drawbacks makes me reluctant to fully recommend the paper for acceptance. These issues are listed below:\n\n**+ Confusing introduction, focus and an unclear motivation/objective**\n\nThe paper's title, abstract and introduction do not fully reflect what the paper is about. In my view, the key contribution of the paper is proposing an information-theoretic approach to evaluating/ensuring fairness of a neural net **classifier** for a given level of accuracy. The abstract and introduction discuss \"interventions\" from observational data---in the clinical and neuroscientific context---in a way that suggests learning the causal pathways of clinical interventions (Lines 4-7, Lines 21-24). But the concept of an intervention here is very different as it refers to alterations to the network architecture rather than manipulation of actual interventions in the real world. \n\nAlso, the paper presents fairness as a potential application of the broader methodology of information flows. However, it seems to me that the proposed method has very limited applications beyond that, and it only works for simple binary classifier as estimating the conditional mutual information $I(Y|Z)$ is practically impossible for more complex tasks. I think it would have been better if the paper was written with a focus on the \"fairness of classification\" application.\n\n**+ Experiments: absence of baselines** \n\nThe authors used the experiments section to show that information flows can be used to find targets for intervention in the context of evaluating the bias-accuracy tradeoff. However, I think a more appropriate research question is whether this is the right way, or a better way, to evaluate this tradeoff. There are many methods for evaluating and ensuring fairness in classifiers that have been proposed in the past few years, that also have the advantage of being model-agnostic and not restricted to neural nets. I think the authors should have compared with these. \n\nMinor issue: I am also unsure about the validity of the linear fits in Figure 4 given the massive variance and the lack of a clear trend.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and social impact are adequately addressed in the discussion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5+ hours"}, "extra_text": {}}, {"review_id": "2yogCHHZ-gg", "reviewer": "Reviewer_BPew", "summary": "This paper studies whether observational data can be used to estimate the outcomes of certain interventions. This is especially important since cost of interventions can be high and in some cases irreversible. This paper empirically demonstrates the effectiveness of using the M-information flows framework for neural networks to assess different kinds of information flows in the network, and then measuring the change in outcome by intervening on different types of flows. They consider fairness in ML as a case study where there are two types of distinct information flows: the label flow and the bias flow. By intervening on paths (by pruning nodes or edges or reducing weights of edges) which contribute to bias in the outcome, the authors show that they can achieve the desired effect of reducing bias. Thus, the paper shows that information flows can offer useful information about where to intervene to achieve certain outcomes.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses a very important problem that comes up in all kinds of application areas (fairness, healthcare, drug discovery etc). This  paper empirically demonstrates how to use information flows in finding neurons/edges which can be intervened upon to produce a desired outcome. I like using fairness as a use case for such a framework since there are two distinct kinds of flows that can be measured using well defined metrics. This also offers a new perspective on the role of model's structure in contributing to biased outcomes. The methods used in this paper are broadly applicable to other areas of interest in ML and Neuroscience. The paper is also very well written and provides all necessary details, and is thus accessible to a broad audience. Overall, I think this is a very interesting paper.\n\n### Scalability\n\n * All the neural nets used in experiments are quite toy-ish, with very few layers and very few nodes. I understand that this allows the authors to extensively evaluate all kinds of flows, however, how would this scale to larger, deeper nets with number of paramaters that are beyond the scope of manual inspection?\n\n * How would size of NNs affect the tradeoffs presented in Figures 3 and 5?\n\n * In reality, even for the case of fairness, one would want to consider intersectional attributes. This would be true even in domains such as drug discovery. How would the proposed method scale for non-binary outputs and non-binary protected features?\n\n### Other\n\n * The paper only considers demographic parity (which is fine), however can this be adopted for other definitions like equal odds and equal opportunity?\n\n * I know this isn't the goal of your paper, but what you're doing by intervening to on Z-information flows sounds similar to some of the adversarial learning methods to remove the effects of the sensitive attribute on the output (eg: https://arxiv.org/abs/1807.00199; https://arxiv.org/abs/1902.00334). It would be interesting to see the effectiveness of pruning (your method) vs adversarial methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have acknowledged the limitations and potential societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1lNGLx4YZxx", "reviewer": "Reviewer_GutV", "summary": "The paper studies the possibility of using the measures of information flows to intervene with ANN for reducing bias while improving/maintaining accuracy. It adopts the M-information flow framework proposed by Venkaatesh et al., and proposes a quantitative notion of information flow and a method for estimating such flows. The magnitude of information flows can then be used to understand and compare the impact of different pruning strategies on the outputs, as well as their fairness-accuracy tradeoffs. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well-written and easy to understand. The methods are simple and easy to use. \n\nMy main concerns are as follows. \n\n1. Most techniques are adopted from the existing works such as the concept of M-information flow, method of estimating information flow based on Fano’s inequality, and typical pruning methods, etc. One major concern I have is the novelty of the paper. \n2. Although the methods seem to be effective on synthetic data and modified adult data, there is no theoretical support/guarantee of the method. Moreover, the experiments only consider relatively small networks (3 features, one hidden layer & 5 features, 2 hidden layers). It is not clear how this method performs on other datasets and larger networks, whether the improvement and effectiveness still preserve in more general settings. \n3. It seems that the method can only be used for the notion of demographic parity fairness, which may limit the generality of the proposed model. \n\nAdditional questions:\n\n1. Information flows are quantified by taking maximum over all subsets of nodes in a layer (Eqn (3)). Are there any particular reasons for taking maximum rather than other operations such as average? \n2. The experiment on synthetic data only considers a small neural network (3 features, one hidden layer). Figure 3 shows that there is no significant difference between different pruning methods regarding the fairness-accuracy tradeoff. Is it possible it’s because the network is too small? I suggest authors performing experiments on a larger-scale network and compare accuracy/fairness between different methods. Intuitively, why does fairness-accuracy tradeoff show the contrarian patterns using pruning on the basis of accuracy-to-bias flow ratio and pruning on the basis of bias-to-accuracy flow ratio?  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers how to use information flows to "debug" a neural net model toward desired fairness objectives. The idea seems novel and connects disparate fields in an interesting way. The reviewers appreciated the paper and agree it should be accepted, albeit their ratings were mostly borderline. The reviewers brought up many important concerns that the authors for the most part appropriately responded to. The authors are expected to address these points raised by reviewers in a final version, including as they outlined in their response and incorporate additional discussion or results from their responses into their paper in an appropriate manner.
Decision: Accept (Poster)