Can Information Flows Suggest Targets for Interventions in Neural Circuits?

1 Introduction . The “ reward circuit ” of the brain controls much of our behavior [ 1 , 2 ] . It is believed that addiction is characterized by a strong bias towards immediate rewards in the reward circuit [ 3 ] , while a different bias , called affective bias , in the same circuit , can lead to depression [ 4 ] . Broadly , such biases in the output of the reward circuit control our behavior and responses to stimuli . Recent technological advancements in neural probes and optogenetics [ e.g. , see 5 , 6 ] are increasingly enabling us to record neural circuits at a high resolution and alter the network by activating or suppressing nodes and links , giving us powerful tools to understand and affect how this circuit processes information . Even a weak understanding of the reward circuit is motivating clinical interventions for individuals suffering from depression , addiction , obsessive compulsive disorder , obesity , etc . ; patients are starting to receive surgeries for long-term deep-brain electrode implantation in brain regions that are involved in the reward circuit [ 7–10 ] . We identify a new and largely unexplored problem within this context : how do we perform minimal interventions that “ correct ” undesirable biases without affecting other functions of this important network as much as possible ? In this paper , we propose to use the M -information flow framework , a recent advancement of ours , that enables tracking the information flow about specific messages within a computational * This work was done while P. Venkatesh and S. Dutta were Ph.D. students at Carnegie Mellon University . †S . Dutta and N. Mehta contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . circuit [ 11 ] . We aim to use this framework to identify interventions that can correct undesirable biases within the circuit , while preserving desirable ones . A difficulty that we face , however , is finding a sufficiently comprehensive dataset with simultaneous recordings of the various brain regions involved in the reward circuit . Although advances in experimental techniques suggest that we are on track to have such datasets in the coming years [ 12 , 13 ] , simultaneous multi-area recordings of specific circuits are still rare . To overcome this issue , we use the context of fairness in artificial intelligence [ 14 , 15 ] to study the above question , which , as we next discuss , provides an excellent analogy to the neuroscientific context . The problem of reducing biases to improve fairness in a decision-making system is much like that of reducing undesirable biases in the reward circuit : ( i ) both systems rely on learned associations from stimuli and responses/labels ( i.e. , training data ) ; ( ii ) both systems have an intended objective ( to learn a desired association between the features and the true labels , or healthy reward-based learning ) ; and ( iii ) both systems often learn undesirable associations ( e.g. , racial or gender bias in artificial neural networks , and biases that cause addiction , depression , or obsessive compulsive disorder in the reward circuit ) . This suggests that if we want to understand how information flows can inform interventions to correct biases in the reward network , we can simulate such experiments by examining the relationship between information flows and interventions in artificial neural networks ( ANNs ) trained on biased datasets . A distinct advantage offered by our M -information flow framework [ 11 ] is that it can track the flows of multiple messages in a neural circuit . In this paper , we adapt our information flow measure to ANNs , and evaluate this measure in an empirical setting involving multiple messages ( one each pertaining to bias and accuracy ) for the first time . We then perform interventions using different strategies to prune nodes or edges of the ANN , and measure the effect of each intervention by quantifying the change in accuracy or bias at the output of the network . Goals and Contributions . Concretely , the main goals and contributions of this paper are : 1 . Our primary goal is to study whether measuring information flows about a message can suggest targets for interventions in an ANN , to change how its output is affected by that message . Towards this , our first contribution is in adapting M -information flow to the ANN context , developing a meaningful way to quantify it . 2 . Secondly , we wish to understand the correlation between the magnitude of information flow on an edge and the degree to which an intervention on that edge affects the output . For this , our contribution is an empirical examination of information flows in an ANN , showing that they can predict the effect of interventions in a fairness context . 3 . Lastly , we wish to understand how we can use information flows corresponding to two different messages , to remove undesirable behaviors while preserving desirable ones . Towards this , we compare different intervention strategies that are informed by information flows , and compare the bias-accuracy tradeoffs achieved by each of these strategies . The rest of the paper is organized as follows : we discuss related work below , following which we summarize the fundamentals of the M -information flow framework in Section 2 . We also show how M -information flow can be adapted to ANNs , discuss the setup of the fairness context , and explain the rationale for adopting an empirical approach . In Section 3 , we describe how we empirically evaluate our goals : specifically , we discuss the process of estimating M -information flow and the design of different intervention strategies . Finally , we present our results on synthetic and real datasets in Section 4 , and conclude with a discussion of the implications of our results in Section 5 . Related work . The idea of using information flow to understand an ANN ( and neural circuits more broadly ) , could be interpreted as a new type of “ explanation ” of ANNs . A large number of works have dealt with problems in the fields of explainability , transparency and fairness in machine learning broadly , as well as for ANNs specifically . Molnar [ 16 ] provides a good summary of the different approaches taken by many of these methods for explainability . Most of these approaches seek to understand the contribution of individual features [ 17–20 ] or individual data points [ 21 , 22 ] to the output . For ANNs specifically , these can also take the form of visualizations to describe what features or what abstract concepts an ANN has learned [ 23 , 24 ] . There have also been a number of information-theoretic approaches for measuring bias and promoting fairness in AI systems [ 25–29 ] . Our approach in this paper is quite different from these prior works : we want to understand what it is about the network structure itself that leads to a certain output . In this sense , our work is similar in spirit to the idea of “ lottery tickets ” [ 30 ] , although we take a very different approach involving information flows . In particular , we want to understand which edges carry information relevant to classification , as well as information resulting in bias , to the output . We also want to know which edges need to be changed in order to produce a desired output , e.g. , fairness towards a protected group with minimal loss of accuracy . Although we examine how information flow can be used to obtain a bias-accuracy tradeoff in a fairness context , our goal is not to find optimal tradeoffs between fairness and accuracy , which is the focus of many recent works [ 31–38 ] . Approaches that directly seek to optimize this tradeoff ( e.g. , by including bias in the training objective , or by using adversarial methods [ 39 , 40 ] ) will likely perform much better than our method , which uses a much more indirect approach to inducing fairness in an ANN . However , those methods would not tell us how to edit the network towards a particular goal , which is what is needed in clinical and neuroscientific application domains . We use the fairness context only as a concrete setting in which to understand whether M -information flow can inform interventions1 in a network with complex flows involving multiple messages . 2 Background and Problem Statement . In this section , we provide a brief introduction to our M -information flow framework [ 11 ] , and show how it can be adapted and reinterpreted for ANNs . Then , we set up the fairness problem , and describe how the adapted information flow measure is related to commonly used measures of bias against a protected class . We also discuss why providing theoretical guarantees might be challenging and motivate the need for an empirical approach . 2.1 Adapting and Reinterpreting M -Information Flow for ANNs . Our M -information flow framework provides a concrete way to define information flow about a random variable M—called the message—in a general computational system . By changing what M represents , we can understand the flows of any number of different messages ( e.g. , if Y and Z are two different variables , we can examine Y -information flows and Z-information flows in the same system ) . The computational system , which is modeled after the brain , is a graph consisting of nodes that compute functions , and edges that transmit the results of these computations between nodes . The definition of M -information flow satisfies an important property : it guarantees the ability to track how information about the message M flows from the input to the output of a feedforward neural circuit.2 Such a model is completely compatible with a feedforward neural network , where the neurons of the ANN act as the nodes , and the weights connecting neurons from different layers act as the edges of the computational system . A key feature of our computational system model is how it accounts for the inherent stochasticity of the brain as well as its inputs : nodes can generate noise intrinsically , and the transmissions on the edges are considered to be random variables . But in a trained ANN , the computation at every node is deterministic , specified completely by the weights on the incoming edges and the neuron ’ s activation function.3 However , we can still continue to think of the edges ’ transmissions as being random variables because the input data for the neural network comes from a distribution ( which could also be an empirical distribution , i.e. , a dataset ) . We are now in a position to state what our original M -information flow definition [ 11 ] looks like in the context of ANNs , before proceeding to adapt it for our purposes in this paper . Definition 1 ( Original M -information flow [ 11 ] ) . Let an arbitrary edge of the neural network at layer t be denoted Et and let the “ transmission ” on this edge be denoted X ( Et ) . Similarly , let a subset of edges at layer t be denoted E′t and the set of transmissions on this subset be denoted X ( E ′ t ) . Then we say that information about M flows on the edge Et , i.e. , edge Et has M -information flow , if ∃ E′t ⊆ Et s.t . I ( M ; X ( Et ) ∣∣X ( E′t ) ) > 0 , ( 1 ) 1The word “ interventions ” in this paper refers to editing the weights of an ANN , which may correspond to using targeted external stimulation in the brain , e.g . through long-term potentiation [ 41 ] . We do not mean interventions in the sense of diffuse medications which have a more widespread impact on brain function . 2In the original framework , which we designed for the neuroscientific context , we considered the neural circuit as being “ feedforward in time ” : i.e. , the computational graph is time-unrolled in such a way that edges send transmissions from nodes at time t to nodes at time t+ 1 . 3We avoid constructions that introduce stochasticity within an ANN for the sake of simplicity . where Et is the set of all edges in layer t , X ( E′t ) represents the set of transmissions on E ′ t , and I ( M ; A |B ) is the conditional Shannon-mutual information between M and A , given B [ 42 , Ch . 2 ] . The motivation behind this definition is two-fold : ( i ) One intuitively expects an edge Et to carry information flow about M if its transmission X ( Et ) depends on M , i.e. , if I ( M ; X ( Et ) ) > 0 . Definition 1 satisfies this requirement . ( ii ) However , it is possible for two edges E1t and E 2 t to carry information about M jointly in such a way that I ( M ; X ( E1t ) ) = I ( M ; X ( E 2 t ) ) = 0 , but I ( M ; { X ( E1t ) , X ( E2t ) } ) > 0.4 Therefore , Definition 1 is designed to assign M -information flow to both E1t and E 2 t in this case : we previously showed [ 11 , 43 ] that such an assignment is imperative for consistently tracking the information flow about M in a computational system . To adapt this definition of information flow to ANNs , we start by recognizing that our computational system model allowed each outgoing edge of a given node to carry a different transmission . In an ANN , however , the outgoing edges of a given neuron all carry the same activation , but with different weights . Consequently , the random variables representing the transmissions are all scaled versions of each other , and have precisely the same information content . Therefore , we can define information flows for the activations of every node , rather than for the transmissions of every edge . We then use the properties of edges ( specifically , their weights ) to construct a definition of information flow for edges , so as to identify the most important outgoing edges of each node and intervene in a more selective manner . Furthermore , Definition 1 only specifies whether or not a given edge has M -information flow . However , we require a quantification of M -information flow that will let us compare different nodes or edges , and decide which ones to intervene upon . Keeping these aspects in mind , we propose M -information flow for the nodes of an ANN , followed by a quantification of M -information flow , and finally a weighted version that assigns different flows to each outgoing edge of a given node : Definition 2 ( M -information flow for ANNs ) . Let an arbitrary node of the neural network at layer t be denoted Vt , and let the activations of this node by represented by the random variable X ( Vt ) . Similarly , let an arbitrary subset of nodes at layer t be denoted V′t and the set of activations of this subset be denoted X ( V′t ) . Then , we say that the node Vt has M -information flow if ∃ V′t s.t . I ( M ; X ( Vt ) ∣∣X ( V′t ) ) > 0 . ( 2 ) We quantify M -information flow by taking a maximum5 over all subsets of nodes V′t in layer t : FM ( Vt ) : = max V′t I ( M ; X ( Vt ) ∣∣X ( V′t ) ) . ( 3 ) Finally , if Et is an outgoing edge of the node Vt that has weight w ( Et ) , then we define the weighted M -information flow on that edge as6 FM ( Et ) : = |w ( Et ) |FM ( Vt ) . ( 4 )

Summary: This paper studies whether observational data can be used to estimate the outcomes of certain interventions. This is especially important since cost of interventions can be high and in some cases irreversible. This paper empirically demonstrates the effectiveness of using the M-information flows framework for neural networks to assess different kinds of information flows in the network, and then measuring the change in outcome by intervening on different types of flows. They consider fairness in ML as a case study where there are two types of distinct information flows: the label flow and the bias flow. By intervening on paths (by pruning nodes or edges or reducing weights of edges) which contribute to bias in the outcome, the authors show that they can achieve the desired effect of reducing bias. Thus, the paper shows that information flows can offer useful information about where to intervene to achieve certain outcomes.