Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors exploit the expressive power of Copula mixtures to model time-varying multi-modal data, and employ Gaussian Processes to model the time-varying copula parameters. They demonstrate the efficacy of their method using information theoretic metrics on a synthetic dataset and a real-world joint neural-behavioral dataset from a neuroscience experiment.  Results demonstrate that the proposed techniques are comparable to the state of the art nonparametric methods, while being more scalable due to the use of stochastic optimization based methods that are commonly used with parametric methods. 

The paper was quite thorough in the motivation, development and empirical analysis of the proposed technique. The use of Copulas, that are commonly employed in the Finance community, but are rare in statistical neuroscience, should interest the more theoretically inclined reader. Given the accelerating trend of collecting long-term joint neural and behavior in experimental neuroscience, the authors make an interesting and timely contribution to the statistical neuroscience literature.

I found that the motivations of the paper were difficult to extract from the Introduction, as there was substantial use of jargon and the intuition behind the technical results were not accessible. To encourage the adoption of their methods in the neuroscience community, the authors should consider improving the readability of their manuscript by making it more friendly to the non-statistician reader. 


review_title: application of copula mixtures to model time-varying multimodal data

final_score: rating: 5: Marginally below acceptance threshold
review_content: - This is an interesting neuroscience application where Copula estimation has shown to be effective. 

- Having said that, I believe that the technical novelty is minimal. To the authors' credit, they did not claim a huge theoretical edge. They honestly reported the findings of the paper. 

- The paper is well written. Ideas flow very neatly. 

- p2: "It was previously shown that such a combination of parametric copula models with GP priors ...": Precisely.. Therefore, given the work by Hernandez-Lobato et al. (2013), as well as the works which have eventually built on top of it, novelty of the proposed method is minimal. I am not saying the application is not useful though.  

- Along the same lines from above, equations (2), (3) and (4) which depict the core of the method, are all taken from seminal previous works of copula mixture estimation. 

- p3: "Since none of the aforementioned families alone could describe such conditional dependency, we combined multiple copulas into a linear mixture model": Is it possible to elaborate a little bit more on whether this is actually the best technical choice here? For instance are there any side effects (e.g. computational) resulting from using this linear mixture? 


review_title: Very interesting application but novelty is below the ICLR level

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors develop a Gaussian process vine copula model, very much in the flavor of the modeling approach of Lopez-Paz et al. (2013). The improvement to the earlier work seems to be a framework for flexible copula modeling including a copula mixture model, approximate inference, model selection, and calculation of mutual information. The paper on its own is a modest improvement on existing work and is both an engineering accomplishment and has the potential for a useful model. The paper is exceptionally well-written and clear. I found it a breeze to read and I credit the authors for that. However, both the validation and the motivation for the model (namely characterizing the probabilistic relationships for neural and behavioral variables) seems particularly thin and could be substantially improved. The paper falls short on these points to such a degree that I am hesitant to recommend acceptance since it notably impacts my evaluation of the significance of the works

## Major points:

First, the authors claim (page 7, paragraph 2) that their model out-performs all non-parametric methods. This is by no means obvious form Fig2b,c. Moreover, If the model performs "similarly" to MINE, then it is maybe not worth using when a convenient technique already exists. What is the advantage of the present model?

Second, the authors show that the estimation of mutual information is only unbiased for a narrow range  of distributions (Gaussian or transformed Gaussian for small dimensions according to Fig 2) and fails for the heavy-tailed Student's-T. However, many neural and behavioral variables are themselves heavy-tailed and the authors did not demonstrate that the real data are sub-gaussian.

Third, it seems like there was a wasted opportunity with this paper. The authors spent $\approx$ 2/3 of a page discussing the estimation of mutual information without motivating why that was a good example metric that could be derived from their model. This modeling framework is an opportunity to determine virtually any expectation over the entire distribution and it is entirely possible that MI is neither all that interesting, nor does it play to the strengths of the model. They then describe changes  to the pairwise distributions of variables from the copula model but we didn't need the copula model to estimate.

Finally, what are we to make of the real data results *vis-a-vis* the validation experiments with simulated data in Fig 2? Besides what I mentioned above regarding the tails of the real data, its not clear that the variance explained wouldn't behave differently. Could the authors report the variance explained for their simulation experiments?

## Minor points: 
The authors state (page 7, last paragraph) That the "stimulus-related changes in the joint variability of the two neuronal signals are commonly described as _noise correlations_." But, isn't that the definition of _signal correlations_?

the authors state (page 8, paragraph 5) 
>The Copula-GP “estimated” (dashed line) almost perfectly matches the “integrated” result, which suggests that the model was able to tightly approximate both $p(u^x|x)$ and $p(u^x)$, and, as a result, $I(x, \{u|^x_{i<N} \})$.

However, it is not clear at all that the later statement regarding $I(x, \{u|^x_{i<N} \})$ follows from the former. In fact Figure 2 demonstrates that the integrated result may not estimate $I(x, \{u|^x_{i<N} \})$ well at all.
review_title: Cool method, poor motivation.

final_score: rating: 7: Good paper, accept
review_content: This manuscript models the conditional joint distribution over variables by using Copula models and copula vines.  The experimental data shows that when the observed variables are highly correlated that the proposed approach improves estimation of entropy over competing benchmark approaches (MINE and KSG) when the variables are highly correlated. Synthetic results demonstrate good improvements, and application to real scientific data seems promising.

Strengths:
Application to real, novel scientific data shows potential utility of the model.
Synthetic results show a good improvement over competing methods, albeit in a limited setup (highly correlated variables)
Mixtures of copulas seems an effective way to produce model complexity, and by linking it to a GP can make sure that it's smooth over the conditioned variable x.

Weaknesses:
Not all modeling steps are clear.  In particular, the interaction between the GP prior and the model selection step is undescribed.  Since there are multiple ways to do this, needs a full description.
No comparison to more neuroscience focuses techniques.
Experimental setup and utility is not fully described.
Lacks ablation studies to elicit key model components.

Questions:
The choice to make the link function on the different copula families all dependent on the same $f$ seems like a strange and limiting choice.  Why was this choice made?

The choice to model calcium trace level and not neural spiking seems mathematically convenient for this method, but it's not clear to me that this is the correct scientific choice.  Typically, calcium imaging traces are preprocesses to extract spikes.  Is there a scientific rationale for using the raw data?  Or was this primarily motivated by avoiding discrete measurements that would be harder to model in the copula?

Please describe more of the scientific setup.  For example, why is there any relationship to licks outside of the reward?

What is the scientific question on this neuroscience application?  Why is entropy relevant, rather than using one of the many predictive problems?

Update after author response:
Most of my methodological concerns have been address (except for the ablation studies).  The scientific application here is not super well-motivated.  It would improve the article greatly to show a greater utility (or at least, clearly describing future utility for answering scientific questions).  It is mentioned that "A full application of the method to study the dependence of contextual signals in mouse visual cortex will be the focus of a follow-up publication."  That's vague; it would be nice to at least clearly discuss how this could be used to facilitate or enhance these scientific experiments.

review_title: Promising entropy estimation on real data but some fuzzy details

Meta-review: Summary: The authors built on existing work of GP vine copula
models. Some modifications are made, to conditional marginals and
mixing. Applications to mutual information estimation are discussed
and evaluated, and the approach is applied to joint
neural/behavioral data.


Discussion:
Strengths mentioned in the reviews are that the
application is (from a neuroscience perspective) interesting, that
estimating mutual information is an important problem, and that the
paper is very well-written. Weaknesses are the limited novelty (from a
machine learning perspective), and weak empirical validation. 

The authors have responded in detail, and were able to clarify a
number of unclear points. Clearly, however, the main criticisms noted
above are hard to address in discussion.

Despite the paper being overall clearly written, I agree with
reviewers that it is hard to tell from
abstract and introduction where the paper is going (even after
modifications made by the authors in the course of the discussion); of
the fairly long abstract, just about half a sentence relates to where
the proposed model differs from previous work. 


Recommendation:
I recommend rejection. Despite some clearly positive aspects, the two main criticisms voiced
by reviewers are serious: Weak validation and minimal 
novelty from a machine learning perspective. I agree that the
neuroscience application may be interesting, but requires more validation.

If the authors want to pursue this work further, I would suggest to
perhaps consider first where to position the paper's focus.
Estimation of mutual information is
a problem that is both hard and important. Any progress here would be
welcome, and simple usefulness could offset any lack of model novelty,
but it would have to be carefully and comprehensively
evaluated. On the other hand, a focus on neuroscience applications would
require more emphasis on, and presumably more space in the paper for,
relevant experiments. 

Decision: Reject