Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Authors present a partial information decomposition (PID) method that can be applied to continuous variables thereby extending a popular approach for analysis of discrete variables. Authors validate their method  by leveraging known analytic results for Gaussian random variables. Additionally, the authors demonstrate that their approach is able to recover the effective connectivity of a neuronal network with chaotic dynamics. Finally, authors use their method to investigate a complex trade-off between redundant, synergetic, and unique information components in recurrent networks that have been trained to solve the XOR problem.
main_review: The algorithm presented in this manuscript is derived from an existing one for discrete variables, but the authors manage to extend the method to include continuous variables, thereby producing a novel and widely applicable technique. There are no obvious flaws that I can see in the theory and experiments presented in this paper. However, I did not review the theory in detail and it is not impossible that I have missed many important details. The paper is written clearly enough and provides enough experimental evidence to back the main claims introduced by the authors. PID for continuous variables can have a number of applications (specially in the field of neuroscience) as indicated by the authors and therefore the contributions presented in this manuscript are quite significant. An interesting addition to this paper would be to add an example application of the method using real data (from the brain), but I do not see this as requirement for this manuscript to be accepted in the conference.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper extends the popular PID measure of Bertschinger et al. (2014) to continuous random variables, and provides a method to compute and estimate the PID in this case. The authors use a novel method based on copulas to reparameterize the optimization problem used to compute the PID, and estimate it by using an approximation based on a variational upper bound. They also provide several examples to show that their estimates match known PID quantities and that the PID can be useful in various neuroscientifically relevant settings.
main_review: Overall, the technical section of the paper remains hard to read for an audience unfamiliar with copulas, despite the efforts of the authors to introduce these concepts. The notation is often times confusing and challenging to parse. However, I believe the paper is of substantial significance, since no methods to compute or estimate PIDs on general continuous distributions currently exist. As the experimental results outlined in the paper illustrate, such a method could be of great interest in several application domains, particularly in neuroscience.

Originality:
The paper is very original, since there are no methods to estimate PIDs for general continuous distributions. Most works discuss computation of the PID, assuming full knowledge of the distributions, rather than estimating the PID, which requires the additional step of inferring distributions from data.

(Technical) Quality:
Barring the issues with clarity of presentation and notation (which are discussed in detail below), I believe the paper is of high technical quality: the authors have used a number of concepts and have been thorough in their references: they provide a good sampling of the relevant background and literature for each concept. The idea of using copulas for the Bertschinger et al. PID is intuitively appealing.
I have one technical question: in Figure 1, why is there a large difference in the left bottom corner? The left-bottom should be where the question is easiest: $\rho_{Y, X_2}$ is close to 1 and $\rho_{Y, X_1}$ is close to zero, so it should be easy to determine that the unique information (in $X_1$) is zero.

Clarity:
I believe the technical section can be significantly improved in its presentation:
1. It would be useful to have an short outline at the start of Section 3 briefly describing the steps involved in estimating the unique information. Otherwise, it is hard to see what the various sub-parts are leading towards.
2. In equation 3.3, is $c(u_y, u_2)$ the distribution that results from summing $c(u_y, u_1, u_2)$ over $u_1$? Please clarify.
3. Line 107: What does the term "copula degrees of freedom" mean? This has not been introduced yet.
4. Equations 3.5 and 3.8: it is not immediately obvious how the conditional distribution can be written in terms of the copula. An extra line explaining this near Equation 3.1 may be useful.
5. Line 111: "Note that the function ..." - it might be useful to have this sentence appear a little earlier, just after Equation 3.8.
6. Line 119: "parameterizing by $\theta$"  - it remains unclear after this sentence exactly what $\theta$ parameterizes. If $\theta$ is the same as $Q$, why not use that instead?
7. Line 121: $\theta(y)$ is now being used as a function. Please clearly explain what $\theta$ is.
8. Equation 3.14: $I_q[\theta]$ is confusing notation, since $q$ does not appear in the RHS. Also, the second term in the RHS implicitly converts an expectation over three variables to one over two. This is only valid if summing $c_{\theta}(u_y, u_1, u_2)$ over $u_y$ is equal to $c_{\theta}(u_1, u_2)$.
9. Line 128: Does the $D \log D$ cost come from sorting the variables for computing the CDF? Please clarify.
10. Please mention how long the estimation algorithm took to run, and on what hardware. Running time is an important aspect of the reusability of this work.
11. Line 214: "We treated ... as iid samples and ran PID on these triplets" - why is this a sensible thing to do for the question being explored? Please explain briefly.

Minor comments:
1. The negative vspace after title should be removed.
2. Line 81: "exits" should be "exist"
3. Figure 3b: Bottom-right quadrant has $X_t$ in place of $Y_t$
4. Line 251: "... PID framework abstracts _and_ generalizes ..."

Significance:
I believe this paper is very significant: it provides the first usable PID estimator for general continuous random variables, and is quite likely to be used by a number of works seeking to apply PIDs in different contexts.

**Update after author responses**

Based on the authors' responses, I am happy to fully recommend accepting the paper, and I have raised my score accordingly.
limitations_and_societal_impact: Limitations have been adequately discussed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary:    This paper presents a novel method for computing the Partial
   Information Decomposition (BROJA flavour) in the case of three
   continuous random variables. The method uses a variational approach
   on a lower bound derived from a copula representation of the joint
   PDF of the three variables. The paper presents some application
   examples, including two where the proposed method is used to
   elucidate the properties of recurrent neural networks.
main_review: This paper addresses an important issue standing in the way of
practical application of PID to the analysis of real data. To the best
of my knowledge, the approach is novel, and I could not identify any
formal or technical problem. The paper is well written and for the
most part does a good job of describing the relationship of this work
to existing literature. I particularly appreciated the very accessible
summary of the decision-theoretic foundation of the BROJA PID. Overall
this is in my opinion a good and useful paper that would benefit the
NeurIPS community.

Having said the above, I would like to bring two points to the
attention of the authors.

1. While reading the paper, I got the distinct impression that the
   title of the paper is somewhat misleading, in the sense that to me
   the "meat" of the paper is clearly the novel PID estimation method,
   while the recurrent network side is more of a demonstration of
   potential usefulness. I was confused by this until I happened to
   see the submission history of the paper (that is, that the RNN side
   was added partly to address previous criticisms). While I think
   that the RNN results are interesting, I am of the opinion that the
   paper would be more likely to reach an interested audience / more
   likely to have a bigger impact if the title was changed to
   something more related to the PID itself (as I suspect that the
   original title must have been in the previous submission). In the
   same vein, the neural-networks-first presentation in the
   introduction may be softened.
2. I couldn't find a link to a software implementation of the proposed
   technique. I don't think this is a hard requirement, but it would
   certainly be of great help for the interested reader, even if it is
   just unpolished set of functions/scripts that allow to at least to
   reproduce some of the results in the paper.

Minor points:
- Lines 86--87: "PID has been previously applied to in vitro neural
  data to elucidate how single neuron computational properties depend
  on network topology [ref to Timme et al, PLoS comp bio 2016]". If I
  am not mistaken, the PID in Timme et al is based on the old Williams
  and Beer approach. By contrast, the method proposed by the authors
  in the current paper is based on the three-variable BROJA PID. An
  early application of the BROJA PID in neuroscience is Pica et al,
  NeurIPS 2017, "Quantifying how much sensory information in a neural
  code is relevant for behavior.", so the authors may consider adding
  this reference too.
- Line 187: missing period at the end of the paragraph.

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: (I am not an expert in information theory)
The authors extend partial information decomposition to the case of continuous variables with arbitrary distributions. 
My review will focus on the last section, where a recurrent neural network was trained on a generalized XOR task. By decomposing the unique, redundant and synergistic components of each neuron’s contribution to the output, the authors identify different strategies used by these networks. 

main_review: I am not an expert in information theory, and will therefore focus on the last section – studying a recurrent neural network trained on a generalized XOR task.

The question of algorithm identification in trained networks is both important and challenging. One aspect of this challenge is the difficulty to define what it means to identify an algorithm. Is it some connectivity motif? An arrangement of fixed points? The geometry of phase space trajectories? The response to novel inputs?

Given this state of affairs, novel suggestions on how to characterize algorithms can be useful. In the specific example given here, however, the authors did not clearly define what they mean by algorithm. The analysis shows that the properties of individual neurons differ both within networks and across networks. Specifically, some single neurons in the recurrent layer seem to already encode most of the desired solution to the XOR problem.

In terms of an algorithm, I find this description lacking. How do these neurons obtain such a representation? By definition, the output of the network will have this property. Finding such internal neurons merely moves the question, but does not answer it.

Furthermore, this notion is very similar to that of mixed selectivity (Fusi et al. Curr opinion neuro, 2016). This might imply that linear dimensionality, which is much simpler to compute, could provide similar information on the networks.

I also found figure 7C puzzling. For K=6, both neurons 1 and 5 seem to have very similar receptive fields and can be described as grandmother cells. Neuron 5 has a higher mutual information, and neuron 1 has a higher unique information. It could be useful to use this case to explain the differences and how they originate.

Furthermore, the networks analyzed here are very small (5 neurons) compared to those cited in line 227 (100 neurons). How does this method scale?

Overall, this specific example does not demonstrate the utility of this method in inferring algorithms from trained networks.

Once again, I would like to stress that this review only relates to one small aspect of the paper.


**EDIT: Score raised from 4 to 7 after reading all reviews and author responses.**


limitations_and_societal_impact: not relevant
review_title: null

Meta-review: This work presents a new method for partial information decomposition that was found to be novel and interesting. Given the narrowing of the scope (away from the RNN application and more focused on the PID estimation) proposed by the authors in response to the reviewers initial comments, the reviewers agreed that the work was relevant to the NeurIPS community and that the method and applications would be greatly appreciated. I thus am happy to recommend this paper be accepted. 
Decision: Accept (Poster)