A Data-Efficient Mutual Information Neural Estimator for Statistical Dependency Testing
Authors: Xiao Lin
Indranil Sur
Samuel A. Nastase
Uri Hasson
Ajay Divakaran
Mohamed R. Amer
Venue:  (2020)
Keywords: mutual information
fMRI
inter-subject correation
mutual information neural estimation
meta-learning
statistical test of dependency

Abstract: Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications. Recent works have developed accurate MI estimators through provably low-bias approximations and tight variational lower bounds assuming abundant supply of samples, but require an unrealistic number of samples to guarantee statistical significance of the estimation. In this work, we focus on improving data efficiency and propose a Data-Efficient MINE Estimator (DEMINE) that can provide a tight lower confident interval of MI under limited data, through adding cross-validation to the MINE lower bound (Belghazi et al., 2018). Hyperparameter search is employed and a novel meta-learning approach with task augmentation is developed to increase robustness to hyperparamters, reduce overfitting and improve accuracy. With improved data-efficiency, our DEMINE estimator enables statistical testing of dependency at practical dataset sizes. We demonstrate the effectiveness of DEMINE on synthetic benchmarks and a real world fMRI dataset, with application of inter-subject correlation analysis.

Under review as a conference paper at ICLR 2020
MUTUAl INFORMATION NEURAL ESTIMATOR FOR STATISTICAL DEPENDENCY TESTING
Anonymous authors Paper under double-blind review
ABSTRACT
Measuring Mutual Information (MI) between high- gigh-dimensional, continuous, ran- dom variables from observed samples has wide theoretical and practical applica- tions. Recent works have developed accurate MI estimators through provably low- bias approximations and tight variational lower bounds assuming abundant supply of samples, but require an unrealistic number of samples to guarantee statistical significance of the estimation. In this work, we focus improving data efficiency and propose a Data-Efficiem MINE Estimator (DEMINE) that can provide a tight lower confident interval of MI under limited data, through adding cross-val validation to the MINE lower bound (Belghazi et al.) 2018). Hyperparameter search is em- ployed and a novel meta-learning approach with task augmentation developed to increase robustness to hyperparameters, reduce overfitting and improve accu- racy. With improved data-efficiency, our DEMINE estimator enables statistical testing of dependency at practical dataset sizes. We demonstrate the effectiveness of DEMINE on synthetic benchmarks and real world fMRI data, with application of inter-subject correlation analysis.
INTRODUCTION
Mutual Information (MT) an important, theoretically grounded measure of similarity between ran- dom variables. MI captures general, non-linear, statistical dependencies between random variables. MI estimators that estimate MI from samples are important tools widely used in not only subjects such as physics and neuroscience, but also machine learning ranging from feature selection and representation learning to explaining decisions and analyzing generalization neural networks.
Existing studies on MI estimation between general random variables focus on deriving asymptotic lower bounds and approximations to MI under infinite data, and techniques for reducing estima- tor bias such as bias correction, improved signal modeling with neural networks and tighter lower bounds. Widely used approaches include the k-NN-based KSG estimator (Kraskov et al. 2004) and the variational nower-bound-ae based Mutual Information Neural Estimator (MINE) family (Belghazi ett al. 2018 Poole et al. 2018).
Despite the empirical and asymptotic bias improvements, MI estimation has not seen wide adop- tion. The challenges are two-fold. First, the analysis of dependencies among variables- let alone any MI analyses for scientific studies requires not only an MI estimate, but also confidence inter- vals (Holmes & Nemenman] 2019) around the estimate to quantify uncertainty and statistical sig- nificance. Existing MI estimators, however, do provide confidence intervals. As low probability events may still carry a significant amount of information, the MI estimates could vary greatly given additional observations (Poole al. 2018). Towards providing upper and lower bounds of true MI under limited number of observations, existing MI lower bound techniques assume infinite data and would need further relaxations when limited number of observations are provided. Closest to our work, |Belghazi et al. (2018) studied the lower bound of the MINE estimator under limited data, but it involves bounds on generalization error of the signal model and would not yield useful confidence intervals for realistic datasets. Second, practical MI estimators should be insensitive to the choice of hyperparameters. An estimator should return a single MI estimate with its confidence interval irrespective of the type of the data and the number of observations. For learning- based approaches, this means that the model design and optimization hyperparameters need to not only be determined automatically but also taken into account when computing the confidence interval.

---

Under review as a conference paper at ICLR 2020
Towards addressing these challenges, our estimator, DEMINE, introduces predictive MI lower bound for limited samples that enables statistical dependency testing under practical dataset sizes. Our estimator builds top of the MINE estimator family, but performs cross-vaile voos-valldation to remove the need to bound generalization error. This yields a much tighter lower bound agnostic to hyper- parameter search. We automatically selected hyperparameters through hyperparameter search, and a new cross-validation meta-learning approach is developed, based upon few-shot meta-learning, to automatically decide initialization of model parameters. Meta-overfitingg strongly controlled through task augmentation, a new task generation approach for meta-learning With these improve- ments, we show that DEMINE enables practical statistical testing of dependency for not only syn- thetic datasets but also for real world functional Magnetic Resonance Imaging (fMRI) data analysis capturing nonlinear and higher-order brain-to-brain coupling-
Our contributions are summarized as follows: 1) A data-efficient Mutual Information Neural Esti- mator (DEMINH) for statistical dependency testing; 2) A new formulation of meta-learning using Task Augmentation (Meta-DEMINE) 3) Application to real life, data-scarce applications (fMRI).
2
RELATED WORK
2.1 MI ESTIMATION
widely used approach for estimating MI from samples is using k-NN estimates, notably the KSG estimator (Kraskov et al. 2004). [Ga et al.] (2017) provided comprehensive review and studied the consistency and of asymptotic confidence bound of the KSG estimator (Gao et al. 2018). MI estimation can also be achieved by estimating individual entropy terms through kernel density es- timation (Ahmad & Lim 1976) or cross-entropy (McAllester & Statos, 2018). Despite their good performance on random variables with few dimensions, MI estimation on high-dimensional random variables remains challenging for commonly used Gaussian kernels. Fundamentally, estimating MI requires accurately modeling the random variables, where high-capacity neural networks have shown excellent performance on complex nigh-dimensional signals such as text, image and audio.
Recent works on MI estimation have focused on developing tight asymptotic variational MI lower bounds where neural networks are used for signal modeling. The IM algorithm (Agakov) 2004) introduces variational MI lower bound, where a neural network (=([x)) is learned as a variational approximation to the conditional distribution X)) The IM algorithm requires the entropy, HZ)), and Exz log (<2|2=) zz) to be tractable, which applies to latent codes of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) as well as categorical variables. [Belghazi] et al. (2018) introduces MI lower bounds MINE and MINE-f which allow the modeling of general random variables and shows improved accuracy for high-dimensional random variables, with appli- cation to improving generative models. Pooole et al. (20T8) introduces spectrum of energy-based MI estimators based on MINE and MINE-f lower bounds and new TCPC estimator inspired by Contrastive Predictive Coding (Oord et al. 2018) for the case when multiple samples from X(Z[X) can be drawn.
Our work introduces cross- volss-valiationn to the MINE-f estimator. We derive the lower bound of MINE-f under limited number of samples, and introduce meta-learning and hyperparameter search to enable practical statistical dependency testing.
2.2 GENERAL STATISTICAL DEPENDENCY TESTING
Existing works in general statistical dependency testing (Bach & Jordan 2002 Gretton et al.) 2005a Berrett & Samworth 2019) have developed non-parametric independent criterions based on correla- tion and mutual information estimators equivalent testing [Xx; Z) = 0, followed by detailed bias and variance analyses. Our approach for independent testing suggest a different direction by har- nessing the generalization power of neural networks and may improve test performance on complex signals. The p-values provided by our test do not involve approximated distributions and hold for small number of examples and arbitrary number of signal dimensions. As different statistical depen- dency testing approaches have explicit or implicit assumptions and biases that make them suitable in different situations, a fair comparison across different approaches is a challenging task. Instead, we focus on self-contained presentation of our dependency test, and provide preliminary comparisons with a widely studied Hilbert-Schmile independence criterion (HSIC) (Gretton et al. (005a) in the appendix.
2

---

Under review as a conference paper at ICLR 2020
2.3
META LEARNING
Meta-learning "learning leam", seeks to improve the generalization capability of neural net- works by searching for better hyperparameters (Maclaurin et al. 2015)) network architectures (Pham al.
2018), initialization (Finn al. 2017a 2018 Kim al., 2018) and distance metries (Vinyals et al. 2016 Snell et al. 2017). Meta-fearning approaches have shown significant performance im-
provements in applications such as automatic neural architecture search (Pham et al. 2018). few-shot image recognition (Finn et al., 2017a) and imitation learning (Finn et al. 2017b).
In particular, our estimator benefits from the Model-Agnosee Meta-Learning (MAML) Finn et al., 2017a) framework which is designed to improve few-shot learning performance. network initial- ization is learned to maximize its performance when fine-tuned on few-shot learning tasks. Appli- cations include few-shot image classification and navigation.
We leverage the model-agnostic nature of MAML for MI estimation between generic random vari- able and adopt MAML for maximizing MI lower bounds. To construct collection of diverse tasks for MAML learning from limited samples, inspired by MI's invariance invertible transformations, propose aak-aaugmnttttion protocol to automatically construct tasks by sampling random trans- formations to transform the samples. Results show reduced overfitting and improved generalization.
3 BACKGROUND
In this section, we will provide the background necessary to understand our approact We define X and Z to be two random variables, P(X, 2) is the joint distribution, and P(X) and P(Z) are the marginal distributions over X and respectively. Our goal is to estimate MI, [(;; 2) given independent and identically distributed (i.i..)) sample pairs (Ia, 24), í 1, 2.2...m from P(X, Z). Let F = (To 2)) Tuee be class of scalar functions, where is the set of model parameters. Let 29919,8)
<<x]2) p(x) P (tt.. Results from previous works (Belghazi et al. 2018 Poole al, 2018) show that the following energy- based family of lower bounds of MI hold for any
Z)
w w (( log g(rlz)
p(x) = 2) Ex--Px log Exn. (ol()))) LEBI
Px To 2) - log Pe-Px.e e Tola,s) & IMINE ((x.)))Pxz Ta( Px Te(m,))
IMINE.LJER
where, E is the expectation over the given distribution. Based on IMINE, the MINE estimator
T(X, 2), i is defined as in Eq Estimators for TEBI, I and JEB can be defined similarly.
I(X; 2).
sup #CG nt
1 Σ To(o), log n²
ΣΣ To(m)1)5)
(2)
With infinite samples to approximate expectation, Eq 2 converges to the lower bound T(X, (XX,))os 2)
supage MINE Note that the number of samples n needs to be substantially more than the number of model parameters d = |0| to guarantee that TX(,, Y') does not overfit to the samples (xz, 24),
i = 1,2 n and overestimate MI. Formally, the sample complexity of MINE is defined as the minimum number of samples n in order to achieve Eq
Pr(
(3)
Specifically, MINE proves that under the following assumptions: 1) Tx Z) is L-Lipschitz; 2) TX, Z) [-M, MJ. 3) (0; E K], Vi E 1,....d). the sample complexity of MINE is
given by Eq 4
2M [d 2dM + log(2/6))
For example, neural network with dimension d = 10, 000, M K, = 0.1 and L 1, achieving a confidence interval of 6 0.1 with 95% confidence (ō = 0.05) would require n 2 18. 756, 256 samples. This is achievable for synthetic example generated by GANs like that studied in [Belghazi
et al. 2018). For real data, however, the cost of data acquisition for reaching statistically significant
We follow the same notations in Belghazi al. 2018), We encourage the review of Belghazi et al. 2018);
Poole al. (2088) for detailed understanding of MINE; TEBI, and Tga-
3

---

Under review as a conference paper at ICLR 2020
estimation can be prohibitively expensive. Our approach instead uses the MI lower bounds specified in [ from a prediction perspective, inspired by cross-validation Our estimator, DEMINE, im- proves sample complexity by disentangling data for lower bound estimation from data for learning a generalizable To(X, 2). DEMINE enables high-confidence MI estimation on small datasets.
APPROACH
Section 4.1
specifies DEMINE for predictive estimation and derives the confidence interval; formulates Meta-DEMINE, explains task augmentation, and defines the optimization
Section 4.2
algorithms.
4.1
PREDICTIVE MUTUAL INFORMATION ESTIMATION
In DEMINE, we interpret the estimation of MINE-f f lower bound Eg 1 as learning problem. The goal is given a limited numher of samples, infer the optimal network T (X, 2) with parameters 0* defined as follows:
0 arg max Expz (x -Ex e Tx(x.2) +
9EE
Specifically, samples from P(X, Z) are subdivided into a training set {(T 2iainan 1, ..... m} and a validation set { 24 v:)aal I,...,n]. The training set used for learning a network 0 as an approximation to O* whereas the validation set is used for computing the DEMINE estimation [X, n), defined as in 5
1(X, m).
E Toloe, Zalval n
e Talaeezzea
We propose an approach learn 0. DEMINE. DEMINE learns d by maximizing the MI lower bound on the training set as follows:
i arg min L(((a, 2) Juain 0), where. DE0
[8|
18 88
C(1(2, 2).
To (2 2 +
Tolo,
(6)
B
[B|
The DEMINE algorithm is shown in Algorithm in appendix.
Sample complexity analysis. Because i is learned independently of validation samples {1T,, Ri) 1, 1,,..,1,, the sample complexity of the DEMINE estimator does not involve the model class FF and the sample complexity is greatly reduced compared to MINE-f. DEMINE esti- mates [(X, 20, when infinite number of samples are provided, defined as:
(X,2) Erxz Tx(X, Er, Br. Tx(X.2) +1
< supace Epx To (X, Z) Ep, Ep To(
I(X;
We now derive the sample complexity of DEMINE defined as the number of samples n required for T(X, Z),, to be good approximation T(X, o). in Theorem 1.
Theorem 1. For (x(X, X, 2) bounded by [4, U], given any accuracy and confidence d, we have:
Pr([I(X, 2),, 1(X, 2) x 5e) 2 8-8
when the number of validation samples satisfies:
25 W-1 + de -otl-at
n 2 n f(n")(())))))))(())
min 2e os4
(8)
2 MINE lower bound can also be interpreted in the predictive way, but will result in higher sample com- plexity than MINE-f lower bound. We choose MINE-f in favor of lower sample complexity over bound tightness.

---

Under review as a conference paper at ICLR 2020
Proof. Since (X, Z) is bounded by [L, U1, applying the Hoeffding inequality to the first half of Eq yields:
Pr( -
To 2.) - Epx2 Tx,, 5) S 2e to E)
As eo((222 is bounded by [et, ). applying the Hoeffding inequality twice to the second half of 5
Pr(EExy Erg Ta(x Σ Epy (6( 20)
ce
26
Pr(EErz 1 E To(z..)) - 1 E (fole)))) 2()
S 2e
(o -chy
Combining the above bounds results in:
26
2c
Pr[] 2), I(X, Z) +(+2() 2 1- 2e
10 D7
4e
(60 16622
By solving { to minimize n according Eq 8 we have:
Pr([T(X, n,, T(X, 7) x), 5 5 2
Theorem also implies the following MI lower confidence interval under limited number of samples
(x( 2 T(X, 2), - 21
Compared to MINE, per the example shown in Section for M = (i.e. L= and U 1). 5 = 0.05, c = 0.1, our estimator requires n = 10, 742 compared to MINE requiring n = 18, 756, 256 i.idd validation samples to estimate lower bound, which makes MI-based dependency analysis feasible for domains where data collection is prohibitively expensive, e.g. fMRI scans. In practice, sample complexity be further optimized by optimizing hyperparameters U and L.
Note that unlike Eq
Theorem 1 bounds the closeness of the DEMINE estimate, T(X, Z) not
wards the MI lower bound supace MINES but towards the MI lower bound I(X, o0.. Therefore, the sample complexity DEMINE as in Eq makes fair comparison with the sample complexity of MINE as in Eq 4 MINE's s higher sample complexity stems from the need to bound the generaliza- tion error of (X, Z) on unseen (x)=)) Existing generalization bounds are known to be overly loose, over-vovvaa parameterized neural networks have been shown to generalize well in classification and regression tasks (Zhang et al. 2016). By using a learning- formulation, DEMINE not only avoids the need to bound generalization error, but also allows further generalization improvements by learning 0 through meta-learning
In the following section, we present meta-learning formulation, Meta-DEMINE, that learns 0 for generalization given the same model class and training samples.
4.2
Given training data {T,, 2; 2i)inin 1.... 1....m). Meta-DEMINE first generates MI estimation tasks each consisting of a meta-training split A and meta-val split through novel task augmenta- tion process. And then a parameter initialization Oun is then learned to maximize MI estimation performance on the generated tasks using initialization 6mit shown in Eq
Pnit = arg min B(A L((x, 2)B, )()), with g(t) MetaTrain (z,, 2)A, (
of
Here g(t) MeetaTrain((:; 2)A, 0(0) is the meta veta-training process of starting from an initialization g(0) and applying Stochastic Gradient Descent (SGD) over steps to learn where in every meta training iteration we have:
000 + g(t-1) - - VVL((x (),, g(e-1)).
In practice, the Adam optimizer (Kingma & Ba 2014) is used for faster optimization. The Adam optimizer uses first and second order momentums of the gradient speed up optimization. Illustrating SGD for simplicity.
5

---

Under review as a conference paper at ICLR 2020
Finally, 0 is learned using the entire training set (2 24 2t)unins i = 1, ...., m) with Omit as initializa- tion:
MetaTrain (zz, Jaaiii ama)
Task Augmentation: Meta-DEMIIE adapts MAML (Finn et al. 2017a) for MI lower bound max- imization. MAML has been shown to improve generalization performance in N-class K-shot im- age classification. MI estimation, however, does not come with predefined classes and tasks. A naive approach to produce tasks would be through cross-validation partitioning training data into meta-training and meta-validation splits. However, merely using cross-validation tasks is prone to overfitting - mitt which memorizes all training samples would result have memorized all meta- validation splits. Instead, Meta-DEMINE generates tasks by augmenting the cross-validation tasks through task augmentation. Training samples are first split into meta-train meta-training and meta- vela-validation splits, and then transformed using the same random invertible transformation to increase task diver- sity. Meta-DEMINE generates invertible transformation by sequentially composing the following functions:
Mirror
m(x) = (2n - 1)æ,
7 ~ Berouulií ());
Permute:
P(x) PPd
Permute dimensions.
Offset:
O(2) xr+6
c ((-0.1,0.1)
Gamma: : G(x)) = sign(x) ) |0
7~ (0..5.2)
Since the MI between two random variables is invariant to invertible transformations on each vari- able, MetaTTrain(;:-) is expected arrive at the same MI lower bound estimation regardless of the transformation applied. At the same time, memorization is greatly suppressed, as the same pair pla..) (x, 2) can have different log p(ar)p(s] under different transformations. More sophisticated invertible transformations (affine, piece-wise linear) can also be added. Task augmentation is an orthogonal approach to data augmentation. Using image classification as an example, data augmentation gener- ates variations of the image, translated, or rotated images assuming that they are valid examples of the class. Task augmentation on the other hand, does not make such an assumption. Task augmenta- tion requires the initial parameters mmt to capable of recognizing the same class in a world where all images are translated and/or rotated, with the assumption that the optimal initialization should easily adapt to both the upright world and the translated and/or rotated world.
Solving mmt using the meta-leariing formulation Eqpppsses a challenging optimiza- tion problem. The commonly used approach is back propagation through time (BPTT) which com- putes second order gradients and directly back propagates gradients from MetaTrain( (zx, (x, 2)A, o(9) to 0ma- BPTT is very effective for a small number of optimization steps, but is vulnerable explod- ing gradients and is memory intensive. In addition to BPTT, we find that stochastic finite difference algorithms such as Evolution Strategies (ES) (Salimans et al., (017) and Expelere-Exploringg Pol- icy Gradients (PEPG) (Sehnke et al., 2010) can sometimes improve optimization robustness. In practice, we switch betwen BPTT and PEPG depending on the number of meta-training iterations.
Optimization:
Meta-DEMINE algorithm is specified in Algorithm
EVALUATION ON SYNTHETIC DATASETS
Dataset. We evaluate our approaches DEMINE and Meta-DEMINE against baselines and state-of- the-art approaches on 3 synthetic datasets: ID Gaussian, 20D Gaussian and sine wave. For ID and 20D Gaussian datasets, following Belghazi et al. (2018), we define two k-dimensional multivariate Gaussian random variables and which have component- wise correlation corr(X,, Z,) = 8zpP. where P (-1, and 6ey is Kronecker's delta. Mutual information [(X; Z) has a closed form solution T(X; 7) = - 1: In(1 p²). For the sine wave dataset, we define two random variables X and Z, where X ~ 1)-
sin(a + 3) + 0.05c, and e A/ 1). Estimating mutual information accurately given few pairs of (X, Z) requires the ability to extrapolate the sine wave given few examples. Ground truth MI for sine wave dataset is approximated by running the the KSG Estimator (Kraskov et al. 2004) 1, 000, 000 samples.
Implementation. We compare our estimators, DEMINE and Meta-DEMINE, against the KSG estimator (Kraskov et al. 2004) MI-KSG and MINE-f (Belghazi et al. 2018 For both DEMINE and Meta-DEMING we study variance reduction mode, referred to -rr, where hyperparameters are selected by optimizing 95% confident estimation mean (μI 2a,,) and statistical significance mode, referred -sig, where hyperparameters are selected by optimizing 95% confident MI

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Meta-DEMINE
Input Data: {1x, 2 Juain (z, 2)val
Parameters: batch B. Meta Learning Iterations NM. Task Augmentation Iterations Ny. Opti-
mization Iterations No, Ratio r, Learning rate 7/, Meta Learning Rate 7/meta
Output: MI, To (x, 2) To(X,
for 1 N& do
2:
for j = 1 .Nr do
3:
train- A Split (Ir, 2)trai into (z, 2)x and (I, 2)B Transformation R₂ for æ, R.(-) m(P(O((())
A = r x train,
4:
5:
6:
Transformation R2 for z, R2(-) m m(P(O(G())))
( + mitt
7:
8:
for k 1: No do
9:
Sample batch of (x, =)18 (I,, Z)
Compute (((R Rg (2)) B, 0 (
10:
11:
Compute ( L gradient for Ometa
12:
Update mmcia using Adam Kingma& & Ba 2014 with end for
13:
Compute Limus (#) R2 (z)) ( end for
14:
15:
Compute Cmeta gradient to mii using BPTT
16:
17:
Update Oinit using Adam Kingma & Ba 2014 with Tmmeto
18:
end for
19:
((0)
0nit 1 No do
21:
Sample a batch of (x, 2)B ~ (xx, z)raai
22: Compute [(xx, z),, 0())
Compute gradient VoL
24: Update 0 using Adam with v
25:
end for
26:
Compute MI (((xx2))al) o(No))
return MI, fint g(No)
lower bound (μr - e). Samples (xr, z) are split 50%-50% into (I, train and (x, 2) We use separable network architecture (1 2) M (tanh(w (w cos (f(x).(()) + b) t). and g are MLP encoders that embed signals and into vector embeddings. Hyperparameters & E [-1,]] 1] and M control upper and lower bounds (z, 2) E [1 + t), M[ - t)}. Parameters w and b are learnable parameters. MLP design and optimization hyperparameters are selected using Bayesian hyperparameter optimization (Bergstra et al.) 2013) described below.
Hyperparameter search on DEMINE-VT and DEMINE-Sig was conducted using the hyperopt pack- age Seven hyperparameters were involved hyperparameter search: 1) number of encoder layers [1. 5T, 2) encoder hidden size [8, 256], 3) learning rate 77 [10-4, 3 x 10 in log scale, 4) number of optimization iterations No [5. 200] (sine wave [5, 5000) log scale, 5) batch size B [256, 1024],
6) M, [10-3 5] in log scale, 7) [-1, Mean and sample standard deviation of MI estiamte (x., 2)rrain DEMINE-Vr maximizes two sigma low μ 20p where 0 V3" due to 3-fold cross-validation. DEMINE-Sig maximizes statistical significance e where is two-sided 95% confidence interval of MI. Meta-DEMINE-vr and Meta-DEMINE.. sig subsequently reuse these hyperparameters as DEMINE-vr and DEMINE-sig.
computed over 3-fold cross-validation
Meta-leamiig hyperparameters
chosen outer loop NM 3, 000 iterations, task augmentation Ny = iterations, r = 0.8, Tmmeta = 3. with task augmentation mode m (P(O()))) No was capped 30 iterations for ID and 20D Gaussian datasets due memory limit. For the sine wave datasets with large No, we used PEPG (Sehnke et al. 2010)) rather than BPTT.
Hyperoppt http:://gggttbb.cc//////rrrrr/pp

---

Under review as a conference paper at ICLR 2020
For MI-KSG, use off-the-shelf implementation by Gao et al. (2017) with default number of nearest neighbors = 3. MI- MS- does not provide any confidence interval. For MINE-f, we use the same network architecture same DEMINE-VE. we implement both the original formulation which optimizes To on (It, 2) till convergence (10k iters), as well as our own implementation MINE-f-ES with early stopping, where optimization is stopped after the same number of iterations DEMINE- vr to control overfitting.
Results. Figure a shows MI estimation performance on 20D Gaussian datasets with varying p E {0, 00,, 0.3, 0.4, 0.5) using N 300 samples. Results are averaged over 5 runs to compare estimator bias, variance and confidence. Note that Meta- Meta-DEMMINE-iig detects the highest < 0.05 confidence MI, outperforming DEMINE-Sig which is a close second. Both detect p < 0.05 statisti- cally significant dependency starting p = 0.3, whereas estimations of all other approaches are low confidence. It shows that in contrary to common belief, estimating the variational lower bounds with high confidence can be challenging under limited data. MINE-f estimates MI > 3.0 and MINE-f-ES estimates positive MI when = 0, both due to overfitting, despite MINE-f-ES having the lowest empirical bias. DEMINE variants have relatively high empirical bias but low variance due to tight upper and lower bound control, which provides a different angle to understand birs-variance trade off in MI estimation (Poole al. 2018)
Figure D b.c.d) shows MI estimation performance ID, 20D Gaussian and sine wave datasets with fixed 0.8, 0.3 and a = 8I respectively, with varying N E {30, 100, 300, 1000, 3000} number of samples. More samples asymptotically improves empirical bias across all estimators. As opposed to 1D Gaussian datasets which are well solved by N = 300 samples, higher-dimensional 20D Gaussian and higher-complextty sine wave datasets are much more challenging and are not solved using N 3000 samples with a aggnal-agosttc MLP architecture. DEMINE-Sig and Mcta-DEMINE-siig detect p < 0.05 statistically significant dependency on not only 1D and 20D Gaussian datasets where and 2 have non-zero correlation, but also on the sine wave datasets where correlation between x and z is O. This means that DEMINE-Sig and MeMa-DEMINN--ig can be used for nonlinear dependency testing to complement linear correlation testing.
We study the effect of cross-validation meta-learning and task augmentation on 20D Gaussian with p = 0.3 and N = 300. Figure 2 plots performance of Meta-DEMINE-vr over NM = 3000 meta iterations under combinations of task augmentations modes and number of adaptation iter- ations No E {0, 203. Overall, task augmentation modes which involve axis flipping m(-) and permutation P(-) are the most successful. With No 20 steps of adaptation, task augmentation modes P(.). m( (P()) and m mPP(O())) prevent overfitting and improves performance. The perfor- mance improvements of task augmentation is not simply from change in batch size, learning rate or number of optimization iterations, because meta-learning without task augmentation for both No and 20 could not outperform baseline. Meta-learning without task augmentation and with task augmentation but using only O(-) or G(-) result in overfitting. Task augmentation with mn-) or m m(P()(G())) prevent overfitting, but do not provide performance benefits, possibly because their complexity is insufficient or excessive for 20 adaptation steps. Further more, task augmentation with no adaptation (No = 0) falls back to data augmentation, where samples from transformed distribu- tions are directly used to learn (x, z). Data augmentation with ((-) outperforms no augmentation, but is unable to outperform bascline and suffers from overfitting. It shows that task augmentation provides improvements orthogonal to data augmentation.
APPLICATION: FMRI INTER-SUBJECT CORRELATION (ISC) ANALYSIS
Humans use language to effectively transmit brain representations among conspecifics. For exam- ple, after witnessing an event in the world, a speaker may use verbal communication to evoke neural representations reflecting that event in a listener's 's brain (Hasson et al.) 2012). The efficacy of this transmission, terms of listener comprehension. is predicted by speaker-listener neural synchrony and synchrony among listeners (Stephens et al. (010)). To date, most work has measured brain- to-brain synchrony by locating statistically significant inter-subject correlation (ISC); quantified as the Pearson product-moment correlation coefficient between response time series for corresponding voxels or regions of interest (ROIs) across individuals (Hasson et al., 2004; Schippers et al.) 2010 for statistical de-
Silbert et al. 2014, Nastase al. 2019). Using DEMINE and pendency testing. we can extend ISC analysis to capture nonlinear and higher-order interactions in continuous fMRI responses. Specifically, given synchronized fMRI response frames in two brain
8

---

Under review as a conference paper at ICLR 2020
201 Counzonn N-201 prdd, 611, 20,03,04.037
Busoiia 2-. h- (50.200 (020,00001
1.0
2.3
Migh
Wel Wo---DWWCC-i
23*
bustua
3.0
63
pn04
hw-00
4--020
(a) 20D Gaussian dataset, N 300 samples
(b) 1D Gaussian dataset, p 0.8
230 davoiaa p-0., t00.
Sire wave scD Notoo 100.590.10000777100 s0co)
1.75
Vidu kntinetton (ex00
MetuCheette
COHNENT Net-otriWor Neauotraacg
1.25
2.00
9.0-
2.5
2.9-
2.0
9:29
M-120
A-00C
h=2000
(c) 20D Gaussian dataset, p 0.3
(d) Sine wave datasct, a 8TT
Figure 1: Comparing MI Estimation performance of DEMINE and Meta-DEMINE with the KSG estimator Kraskov Krassot et al. (2004) and MINE- (Belgha et al. 2018 on different datasets using varying number of samples. The bars show estimator mean and standard deviation averaged over 5 runs with different seeds. The error bars show 95% confidence interval (not available for MI-KSG), The statistical significance focused variants DEMINE-Sip sig and Meta- SEtu-DMMINE-Sip achieves the highest 95% confident MI estimation. Meta-DEMINE improves over DEMINE most of the time.
Effect of Data Augmenation (hiner loop Enrations My-
Effect of Task ongmentat Orner loop thenat hens Mo]
Effect of Task Augmentation
(tnner loep itorations (c-
0.6
0.6
0.6
Mutuual Information (ratt)
withett resto loarring
Munuel Infominnommiiionn/o (nots)
0.5
0.5
0.5
Information
0.4
Twik-lay
0.2
0.2
0.2
0.1
0.1
0.1
500
1000
1500
2000
2500
3000
500
1000
1500
2000
7500
3000
0
500
1003
1500
2008 2500 3000
loog Iterabons
nerations
oop iterations Nr:
(a) Meta-DEMINNE-r No = 0. (b) Meta-DEMINNE No = 10.
(c) Meta-DEMINN--r No 20.
Figure 2: To study the effect of task augmentation and number of adaptation steps, DEMINE-VI vr with different task augmentation modes and vary number of adaptation iterations No (0, 10, 20) on Gaussian 20D, p 0.3 dataset. Combinations of permutation and mirror- ing operations are effective in reducing overfitting and improving performance.
Meta-
Table Number of HCP-MMP1 regions with significant correla- tion (r) and MI (DEMINE, Meta-
Table 2: Segment classification accuracy for NeuralMI versus
Pearson's 's correlation in 1-vs-rest*.
DEMINE) during listening.
Classification
Mask
dDMN Mask
Accuracy (%)
F
Br
Bk
P
Br
Bk
MI
No. shared
r DEMINE
Meta DEMINE
3.7
1.8
2.6
1,9
3.7
NIA
Pearson's I. 1vR
35.0
20.4
25.8
31.5
NIA
14,8
6,4
11.8
9.9
N/A
-sig
DEMINE-VE 1vR
42.8
28.0
32.8
35.9
0.637
16.5
7.9
11.6
12.0
0.035
37
24
23
Meta: VE IvR
47.2
32,5
39,9
41.0
0.752
13.7
7.9
8.2
8.9
0.031
DEMINE-Sig
24
Abbreviations: P: Pieman; F: Forgot; Br: Brenx: Bk: Black, ML: Mutual Information.
Meta: DEMINE-SS sig
23
26
29
*Noe that all the results averaging the subjects.

---

Under review as a conference paper at ICLR 2020
regions X and Z across K subjects X,, Z,, 10,i=1a =::=1,... l,...., K as random variables. We model the condi- tional mutual information I [X;;; Z + j) as the MI form of pair-wise ISC analysis. By definition, [X;;; Zylů lē * j) first computes MI between activations X, and Zj from subjects and j respectively, and then average across pairs of subjects i + j. It can be lower bounded using Eq. 7 by learning a To(x, 2) shared across all subject pairs.
Dataset. We study MI-based and correlation-based ISC a fMRI story comprehension dataset by Nastase et al.] (2019) with 40 participants listening to four spoken stories. Average story duration II minutes. An IMRI frame with full brain coverage captured at repetition time TR 1.5 sec- onds with 2.5mm isotropic spatial resolution. We restricted our analysis to subsets of voxels defined using independent data from previous studies: functionalll-ddffin
masks of high ISC voxels (ISC; 3,800 voxels) and dorsal Default-Mode Network voxels (dDMN; 3,940 voxels) from Simony et al. (2016) as well as 180 HCP-MMPI multimodal cortex parcels from Glasser et al. (2016). All masks were defined in MNI space.
Implementation. We compare MI-based ISC using DEMINE and Meta- DEMINE with correlation- based ISC using Pearson's s correlation. DEMINE and DEMMINE setup follows Section 5 The fMRI data were partitioned by subject into a train set of 20 subjects and validation set of 20 different subjects. Residual ID CNN is used instead of MI.P as the encoder for studying temporal dependency. For Pearson's 's correlation, high-dimensional signals are reshaped to ID for correlation analysis. Effective sample size for confidence interval calculation is the number of unique non- overlapping fMRI samples.
Results. We first examine, for the fine grained HCM-MMP1 brain regions, which have p < 0.05 statistically significant MI and Pearson's 's correlation. Table D shows the result. Overall, more re- gions have statistically significant correlation than dependency. This is expected because correlation requires less data to detect. But Meta- DEMINE is able to find 6 brain regions that have statistically significant dependency but lacks significant correlation. This shows that MI analysis can be used to complement correlation-based ISC analysis.
By considering temporal ISC over time, fMRI signals can be modeled with improved accuracy. In we apply DEMINE and Meta-DEMINE with I, 10TRs (15s) sliding windows random variables to study amount of information that can be extracted from ISC and dDMN masks. We use between-subject time-segment classification (BSC) for evaluation (Haxby et al. 2011 I Guntupalli et al. 2016). Each IMRI scan is divided into K non-overlapping L 10 TRs time segments. The BSC task is one versus rest retrieval: retrieve the corresponding time segment of an individual given a group of time segments x excluding that individual, measured by top-1 accuracy. For re- trieval score, (X, 2) is used for DEMINE and Meta-DEMINE and p( X, Z) is used for Pearson's correlation as a simple baseline. With CNN as encoder, DEMINE and Meta-DEMINE model the signal better and achieve higher accuracy. Also. Meta-DEMINE is able to extract 0.75 nats of MI from the ISC mask over 10 TRs or 15s, which could potentially be improved by more samples.
Table 20
7 CONCLUSION
We illustrated that a predictive view of the MI lower bounds coupled with meta-learning results in data-efficient variational MI estimators, DEMINE and Meta-DEMINE, that are capable of perform- ing statistical test of dependency. We also showed that our proposed task augmentation reduces over- fitting and improves generalization in meta-learning We successfully applied MI estimation to real world, data-scarce, fMRI datasets. Our results suggest a greater avenue of using neural networks and meta-learning to improve MI analysis and applying neural sowwwokk-bsedd information theory tools to enhance the analysis of information processing in the brain. Model-agnostie lower bound estimation approaches- - including MINE, DEMINE and Meta-DEMINE- are limited to estimating small MI lower bounds up Olog N) as pointed out in (McAllester & Statos 2018), where N is the number of samples. In real fMRI datasets, however, strong dependency is rare and existing MI estimation tools are limited more by their ability to accurately characterize the depen- dency. Nevertheless, when quantitatively measuring strong dependency, cross-entropy (McAllester & Statos 2018) or model-based quantities, alternatives to MI, such as correlation or CCA, may be measured with high confidence.
nigh-confidence, MI
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Barber Felix Agakov. The IM algorithm: a variational approach to information maximization. Advances in Neural Information Processing Systems, 16:201, 2004.
Ibrahim Ahmad and Pi-Erh Lin. A nonparametric estimation of the entropy for absolutely continuous distributions (corresp.). IEEE Transactions on Information Theory, 22(3):372-375, 1976.
Brian Avants, Charles I. Epstein, Murray Grossman, and James C Gee. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurode- generative brain. Medical Image Analysis, 12(1):26-41, 2008.
Francis R Bach and Michael Jordan. Kernel independent component analysis. Journal of machine learning research, 3(Jul):1-48, 2002.
Yashar Behzadi, Khaled Restom, Joy Liau, and Thomas T Liu. A component based noise correction method (CompCor) for BOLD and perfusion based fMRL. Neurolmage, 37(1):90-101, 2007.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Devon Hjelm, and Aaron Courville. Mutual information neural estimation. In International Conference Machine Learning, pp. 530-539, 2018.
James Bergstra, Daniel Yamins, and David Daniel Cox. Making a science of model search: Hyper- parameter optimization in hundreds of dimensions for vision architectures. 2013.
Thomas B Berrett and Richard Samworth. Nonparametric independence testing via mutual infor- mation. Biometrika, 106(3):547-566, 2019.
Robert W Cox. AFNI: software for analysis and visualization of functional magnetic resonance
neuroimages. Computers and Biomedical research, 29(3):162-173, 1996.
Carol Daniel. knew you
black.
http:///thench.oorr/sttriie
knew seoo-were-blach 2018. Accessed: 2018-10-12.
Oscar Esteban, Christopher Markiewicz, Ross W Blair, Craig Moodie, Ayse Ilkay Isik, Asier Er- ramuzpe Aliaga, James Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, Hiroyuki Oya, Satrajit Ghosh, Jessey Wright, Joke Durnez, Russell Poldrack, and Krzysztof Jacek Gor- golewski. FMRIPrep: robust preprocessing pipeline for functional MRI. bioRxiv, 2018. doi: 10.1101/306951.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1126-1133 135, 2017a
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning In Conference on Robot Learning, pp. 357-368, 2017b.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Ad- vances Neural Information Processing Systems, pp. 9537-9548, 2018
Vladimir S Fonov, Alan C Evans, Robert C McKinstry, CR Almli, and DL Collins. Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. Neurolmage, (47); $102, 2009.
Neil Gaiman. The man who forgot ray b htwp:bbww...tppss/ssoooolooooooo
http:///ouuunndddod..ccoooeeellr hww////igennengna....... Chee-mn--ee man- who- Seann-whoo-rroeee forgor ray brabburyy 2018. Accessed: 2018-10-12.
Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual informa- tion for pp. 5986-5997, 2017.
mixtures. In Advances in Neural Information Processing Systems,
Weihao Gao, Sewoong Oh, and Pramod Viswanath. Demystifying fixed k-nearest neighbor infor- mation estimators. IEEE Transactions Information Theory, 64(8):5629-5661, 2018.

---

Under review as a conference paper at ICLR 2020
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 11617615):171,0111
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neu- ral networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS10). Society for Artificial Intelligence and Statistics, 2010.
Krzysztof Gorgolewski, Christopher Burns, Cindee Madison, Dav Clark, Yaroslav Halchenko, Michael Waskom, and Satrajit Ghosh. Nipype: a flexible, lightweight and extensible neuroimag-
ing data processing framework in python. Frontiers Neuroinformatiics 5:13, 2011. ISSN
1662-5196. doi: 10.3389/fminf.22/ 201 1.00013.
URL https: www www.wfrrone wer.iiiiiiiinn.mmmm..m org/
article/10. 3389/ fninf...f. 2011 00013
Krzysztof Gorgolewski, Tibor Auer, Vince Calhoun, Cameron Craddock, Samir Das, Eugene Duff, Guillaume Flandin, Satrajit S Ghosh, Tristan Glatard, Yaroslav O Halchenko, et al. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data, 3:160044, 2016.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 13-77.30 Springer, 2005a.
Arthur Gretton, Alexander J Smola, Olivier Bousquet, Ralf Herbrich, Andrei Belitski, Mark Augath, Yusuke Murayama, Jon Pauls, Bernhard Schölkopf, and Nikos K Logothetis. Kernel constrained covariance for dependence measurement. In AISTATS, volume 10, pp. 112-119, 2005b.
Douglas N Greve and Bruce Fischl. Accurate and robust brain image alignment using boundary- based registration. Neurolmage, 48(1):63-72, 2009.
J Swaroop Guntupalli, Michael Hanke, Yaroslav O Halchenko, Andrew C Connolly, Peter J Ra- madge, and James Haxby. model of representational spaces human cortex. Cerebral Corfex, 26(6):2919 (000:22999-29300 2934,, 2016.
Uri Hasson, Yuval Nir, Ifat Levy, Galit Fuhrmann, and Rafael Malach. Intersubject synchronization of cortical activity during natural vision. Science, 403456644:1634416644 2004.
Uri Hasson, Asif A Ghazanfar, Bruno Galantucci, Simon Garrod, and Christian Keysers. Brain-to- brain coupling: mechanism for creating and sharing social world. Trends in cognitive sciences, 16(2): 1000:114-121,1 2012.
James V Haxby, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Con- roy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2):4044416, 2011.
Caroline M Holmes and Ilya Nemenman. Estimation of mutual information for real-valued data with error bars and controlled bias. arXiv preprint arXiv: 1rXiv:1903.09280, 2019.
Mark Jenkinson, Peter Bannister, Michael Brady, and Stephen Smith. Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neurolmage, 17(2):8258844 841, 2002.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. 2018.
Bayesian model-agnostic meta-learning. arXiv preprint
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1rXiv:1412.6980. 2014.
A. Kraskov, H. Stogbauer, and P. Grassberger. Estimating mutual information. Physical review E, 2004.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-basse hyperparameter optimiza- tion through reversible learning. In International Conference on Machine Learning, pp. 2113- 2122, 2015.
12

---

Under review as a conference paper at ICLR 2020
David McAllester and Karl Statos. Formal limitations the measurement of mutual information. arXiv preprint 1rX.v:1811.044 2018.
Samuel A Nastase, Valeria Gazzola, Uri Hasson, and Christian Keysers. Measuring shared responses across subjects using intersubject correlation. Social Cognitive and Affective Neuroscience. 14(6):
667-685, 05 2019. ISSN 1749-5016. doi: 10. 00.10933/cam//s2203.. URL https://aoi........ org/
10. 10.1093/500 scan 01933/caa/ns2237 n8x037
Jim O'Grady.
Running from the Bronx. https: //
/ete:///sunnee //s/unduccll ludd com/
the seeesstorr--ooloeee http::/olllle////oooorrro------- 2018-10-12.
corn-rom-thhe 2018a.
Accessed:
Jim O'Grady, Pie Man. httpps///hhemmhh.rrr////tt//////
2018b. Accessed:
2018-10-12.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:/807.033744 2018.
Hieu Pham, Melody Guan, Saaret Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning, pp. 4092-4101, 2018.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On vari- ational lower bounds of mutual information. In Bayesian Deep Learning Workshop, NeurIPSW, 2018.
Jonathan D Power, Anish Mitra, Timothy O Laumann, Abraham Snyder, Bradley L Schlaggar, and Steven E Petersen. Methods to detect, characterize, and remove motion artifact in resting state IMRI. Neurolmage, 84:320-341, 2014.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv: 2017.
Marleen B Schippers, Alard Roebroeck, Remco Renken, Luca Nanetti, and Christian Keysers, Map- ping the information flow from one brain to another during gestural communication, Proceedings of the National Academy of Sciences, pp. 201001791, 2010.
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4):551-555 2010.
Lauren Silbert, Christopher Honey, Erez Simony, David Poeppel, and Uri Hasson. Coupled neural systems underlie the production and comprehension of naturalistic narrative speech. Proceedings of the National Academy of Sciences, 11644))64667-44666 2014.
Erez Simony, Christopher Honey, Janice Chen, Olga Lositsky, Yaara Yeshurun, Ami Wiesel, and Uri Hasson. Dynamic reconfiguration of the default mode network during narrative comprehen- sion. Nature Communications, 7:12141, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087,
Greg Stephens, Lauren J Silbert, and Uri Hasson. Speaker-listener neural coupling underlies successful communication. Proceedings of the National Academy of Sciences, 107(32):14425- 14430, 2010.
Jeffrey Mark Treiber, Nathan S White, Tyler Christian Steed, Hauke Bartsch, Dominic Holland, Nikdokht Farid, Carrie McDonald, Bob S Carter, Anders Martin Dale, and Clark Chen. Characterization and correction of geometric distortions in 814 diffusion weighted images. PLOS ONE, 1113):c015247,, 2016.
J. Tustison, B. B. Avants, P. A. Cook, Y. Zheng, A. Egan, A. Yushkevich, and J. C. Gee. N4itk: improved n3 bias correction. IEEE Transactions on Medical Imaging, 131001131011322 June 2010. ISSN 0278-0062. doi: 2010000002200000066666
13

---

Under review as a conference paper at ICLR 2020
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. (630-3638, 2016.
Sijia Wang, Daniel J Peterson, Christopher Gatenby, Wenbin Li, Thomas J Grabowski, and Tara M Madhyastha. Evaluation of field map and nonlinear registration methods for correction of suscep- tibility artifacts in diffusion mri. Frontiers in Neuroinformatics, 11:17, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint orXiv:/611.083 2016.
Qinyi Zhang, Sarah Filippi, Arthur Gretton, and Dino Sejdinovic. Large-scale kernel methods for independence testing. Statistics and Computing, 28(1): 2018.
Yongyue Zhang, Michael Brady, and Stephen Smith. Segmentation of brain MR images through hidden markov random field model and the expectation-mantt masiiiooommaiizziiionn algorithm. IEEE Transac- tions on Medical Imaging, 20(1):45-57, 2001.
14

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 THE DEMINE ALGORITHM
Algorithm DEMINE
Input Data: {(x, 2) 2)aain (I, 2)aa
Parameters: Batch B, Iterations No. Learning rate n
Output: MI, TxX 2)
1: g(0) < Xavier Initialization (Glorot & Bengio 2010)
2: fori= 1: : No do
3: Sample a batch of (#r; 21) (x, z)main
4: Compute L (mi, 2),,, 0(t-1)
5: Compute ( C -rradinett for
6: Update g(() using Adam (Kingma & Ba 2014 with 17
7: end for
8: MI = T(X, 7),, 7),, (.
9: return MI, ((No)
A.2 ADDITIONAL DETAILS OF THE FMRI DATASET
The dataset we used contains 40 participants (mean age 23.3 years, standard deviation 8.9, range: 1853; 27 female) recruited listen to four spoken stories The stories were renditions of "Pie Man" and "Running from the Bronx" by Jim OGrady (O'Grady 2018b a). "The Man Who Forgot Ray Bradbury" by Neil Gaiman (Gaiman 2018), and "I Knew You Were Black" by Carol Daniel (Daniel 2018); story durations were 7, 9. 14, and 13 minutes, respectively. After scanning, participants completed a questionnaire comprising 25-30 questions per story intended to measure narrative comprehension. The questionnaires included multiple choice, True/False, and fill-in-the- blank questions, as well as four additional subjective ratings per story. Functional and structural images were acquired using 3T Siemens Prisma with 64-channel head coil. Briefly, functional images were acquired in an interleaved fashion using gradient-echo echo-planar imaging with a multiband acceleration factor of (TR/TE= 1500/31 ms where TE stands for "echo time", resolution = 2.5 mm isotropic voxels, full brain coverage).
All fMRT data were formatted according to the Brain Imaging Data Structure (BIDS) standard (Gor- golewski et al. 2016) and preprocessed using the iMRIPrep library (Esteban et al. 2018). Functional data were corrected for slice timing, head motion, and susceptibility distortion, and normalized to MNI space using nonlinear registration. Nuisance variables comprising head motion parameters, framewise displacement, linear and quadratic trends, sine/cosine bases for high-pass filtering (0.007 Hz), and six principal component time series from cerebrospinal fluid (CSF) and white matter (WM) were regressed out of the signal using the Analysis of Functional Neurolmages (AFNI) software suite (Cox) 1996).
The fMRI data comprise X E V for each subject, where Vú represents the flattened and masked voxel space and Trepresents the number of samples (in TRs) during auditory stimulus presentation.
Additional Details on Dataset Collection Functional and structural images were acquired using a 3T Siemens Magnetom Prisma with a 64-channel head coil. Functional, blooddoxyyeeaaiion-leee dependent (BOLD) images were acquired in interleaved fashion using gradient-echo echo-planar imaging with pre-scan normalization, fat suppression, a multiband acceleration factor of 3, and no in-plane acceleration: TR/TE = 1500/31 ms, flip angle 67%, bandwidth 2480 hz per pixel, resolution 2.5 mm3 isotropic voxels, matrix size = 96 x 96, Field of view (FoV) = 240 x 240 mm, 48 axial slices with roughly full brain coverage and no gap, anteriorposterior phase encoding. At the beginning of each scanning session, Tl- Tl-weighted structural scan (where T1 stands for
'Two of the stories were told by a professional storyteller undergoing an fMRI scan; however, fMRI data for the speaker were not analyzed for the present work due to the head motion induced by speech production. 'The study was conducted in compliance with the Institutional Review Board of the University
15

---

Under review as a conference paper at ICLR 2020
"longitudinal relaxation time"), was acquired using high-resolution single-shot Magnetization- Prepared 180 degrees radio-frequena pulses and RApid Gradient- (MPRAGE) sequence with an in-plane acceleration factor of 2 using GeneRalized Autocalibrating Partial Parallel Acquisition (GRAPPA): TR/TE/TI = 2530/3.3/1 100 ms where TI stands for inversion time, flip angle 7 resolution 1.0 1.0 x 1.0 mm voxels, matrix size 256 x 256, FoV 256 256 176 mm, 176 sagittal slices, ascending acquisition, anteriorposterior phase encoding, no fat suppression. 5 min 53 total acquisition time. At the end of each scanning session T2-weighted (where T2 stands for "transverse relaxation time") structural scan was acquired using the same acquisition parameters and geometry as the TI- Tw-eeighted structural image: TR/TE 3200/428 ms, 4 minutes 40 seconds total acquisition time. A field map was acquired at the beginning of each scanning session, but was not used in subsequent analyses.
Additional Details on Dataset Preprocessing Preprocessing was performed using the fMRIPrep library Estehan et al. (2018), a Nipype library 8 Gorgolewski et al., 2011) based tool. Te-whihteed images were corrected for intensity non-uniformity using the N4 bias field correction algorithm Tustison et al. 2010) and skull-stripped using Advanced Normalization Tools (ANTs) (Avants
et al. 2008). Nonlinear spatial normalization to the International Consortium for Brain Mapping (ICBM) 152 Nonlinear Asymmetrical template version 2009c (Fonov al. 2009) was performed using ANTs. Brain tissue segmentation cerebrospinal fluid, white matter, and gray matter was was performed using FSL library' 2 FAST tool Zhang et al. (2001). Functional images were slice timing corrected using AFNI software's 's 3dT'shift (Cox (196) and corrected for head motion us- ing FSL library's MCFLIRT tool (Jenkinson et al. 2002). "Fieldmap-less" distortion correction was performed by co-registering each subject's 's functional image to that subject's 's intensity-inverted T1-weighted image (Wang et al., (017) constrained with an average field map template (Treiber et al. 2016). This was followed by co-registration to the corresponding Tl T1-weighted image using FreeSurfer software's boundary-baaaae registration (Greve & Fischl 2009) with degrees of free- dom. Motion correcting transformations. field distortion correcting warp. BOLD-to-TI transforma- tion and TI-to-template (MND) warp were concatenated and applied in a single step with Lanczos interpolation using ANTs. Physiological noise regressors were extracted applying "a Component Based Noise Correction Method" aCompCor (Behzadi et al. 2007). Six principal component time series were calculated within the intersection of the subcortical mask and the union of CSF and WM masks calculated in T1w (TI weighted) space, after their projection to the native space of each functional run. Framewise displacement (Power et al.) (2014) was calculated for each functional run. Functional images were downsampled to 3 mm resolution. Nuisance variables comprising six head motion parameters (and their derivatives), framewise displacement, linear and quadratic trends, sine/cosine bases for high-pus pass filtering (0.007 Hz cutoff), and six principal component time series from an anatomically-definee mask of cerebrospinal fluid and white matter were regressed out of the signal using AFNI's 3dTprojecct (Cox 1996). Functional response time series were z-scored for each voxel.
A.3 COMPARISON BETWEEN HSIC AND DEMINE
We first review the Hilbert-Schhidt independence criterion (HSIC), widely-studied correlation- based independence criterion and discuss its connections with the MINE family of mutual informa- tion lower bound methods, and then study DEMINE and spectral HSIC implementation on the synthetic datasets.
The HSIC approach (Gretton et al. 2005b a) is based on a necessary and sufficient condition of independence: two random variables X and Z are independent if and only if for all bounded or positive functions f and g Id + R. Exz f (Z) Ezg (2) O. A proof can be constructed by showing equivalence
Ex (xx (Z) = O, or equivalently
Exz(f(X) Ex (x)) to the definition of independence, P(X, 2) P(Z).
To construct an independence test, existing approaches (Gretton et al., 2005b a) use Reproducing Kernel Hilbert Spaces (RKHS) for f and g. function space that not only covers all functions
between [0, 1). but also allows computationally efficient estimation or bounding of
imrieppp
httpp:
16

---

Under review as a conference paper at ICLR 2020
COCO o (xx
given samples, and test COCO(X, 2) = 0. Confidence intervals are derived through McDiarmid's inequality, or using dosed-form distributions to approximate the test statistics to certain order of moments, and compute the confidence interval from the closed-form distribution.
The COCO(X, 2) used by HSIC estimators bears great resemblance to the MINE family of mutual information estimators. In fact, it can be shown that
COCO Z)
s. Pxx fx))((2)
Eg (1))
s. P(a,a) f (z)g(zz Px log (x)g(a)
M
sup ,9 (xx (z) Px log B ((0))(z)) Ri FEAI
(10)
[(x,s)=-Px I (z)g(z) log En- am (s))(()) z IAINE
s.
s.p (ax f()))(() PR (c)o(a)
R TAINBA,
means that within a family of decomposable functions where (x( COCO(X.Z) is upperbound to the MINE estimates. In addition, the equivalence of COCO(X.2)
(2))
0 and [(X, Z) = 0 seems to suggest form of mutual information bound. On the other hand, MINE allows the use of non-decomposable To (X, Z). Existing results on MINE (Poole et al. 2018)) seem to suggest that a non-dccomposable TX(,, 2) gives superior empirical mutual information estima- tion performance over decomposable Tx(X, Z). The necessity of non-decomposable To(X, Z) designs and mutual information lower bounds under decomposable designs of To (X, Z) may be
subjects of further research.
Similar to the MINE estimators, HSIC-based estimators tend to have loose confidence intervals due to the need to bound generalization error of kernels f and g on unseen data points. We expect a pros-validatioon-baeed approach like DEMINE to also improve the performance of the HSIC-based estimators.
Comparison between DEMINE and HSIC on synthetic benchmarks. We compare Canonical Correlation Analysis (CCA), DEMINE, DEMINE-meta and HSIC for independent testing on our synthetic Gaussian and sine wave benchmarks presented in Section 5 Results for single random seed is reported for a compact presentation, but we have ran experiments using multiple random seeds and find the result of single random seed representative enough.
For CCA, we compute p-value using the x² test. For HSIC, we report p-value using a publicly available implementation for a spectral HSIC test (Zhang et al., 2018 I| The default kernel is used. Hyperparameters set recommended setting when available. For DEMINE and DEMINE-meta, the setup is identical to Section 5 A 2-sided 95% confidence interval is reported, but showing only the lower side.
Experiment results are compiled in Table Statistically significant dependence detections with p 0.05 are bolded. Results show that spectral HSIC requires less data to test dependency for the simple Gaussians dataset. But the more challenging sine wave dataset, DEMINE-sig and DEMINE- meta-sig perform better. Overall, we find DEMINE more complementary to linear correlations for dependency testing on complex signals. Note that Gaussian kernels are used for spectral HSIC. More complex kernels have potential to improve results.
B SANITY CHECK ON STATISTICAL DEPENDENCY TESTING
We performed sanity check of our approach, as well as several statistical dependency testing imple- mentations that we compare against. We run different statistical dependency testing implementations on our 1D Gaussian = 0.0. N 30 samples dataset where X and Z are independent. A large number of runs with different random seeds are performed. False positive rate of p < 0.05 statistical
com oxmm cs kerpy We also experimented with classic HSIC with gamma ap- proximation Gretton et al. (2005a) https: :ttp:::///gttttt com amber0309 /HSII ISIC and a block HSIC imple- find that they both re-
pecttp::/githhtt
mentation (Zhang et al. (2188) from https: thub. com oxm cs /eeryyrpy port significantly more than 5% false positives for independent ID and 20D gaussians = a = (30, 100} across 100,000-1,000 100011000,0000000000000000000000 random seeds, indicating errors in confidence interval calculations.
This false positive case for CCA, because for this sine wave data ground truth correlation is O.
17

---

Under review as a conference paper at ICLR 2020
Table 3: Statistical dependency testing comparison between CCA, spectral HSIC and DEMINE algorithms on synthetic datasets. Statistically significant dependency with < 0.05 detections are bolded. See text for detailed experimental setups.
Problem
Samples
CCA
Spectral HSIC
DEMINE-sig
DEMINE-meta-iig
ID Gaussian = 0.8
30
p=0.000
p-0.001
1 2 0.006 0.006.0.011
0.017
I 2 0.006 0.017
0.006-0.017
Gaussian P
p-0.001
ID Gaussian 0.8
300
p=0.000
p=0.001
I 2
0.143-0.132
2 0.146-0.132
Gaussian P 0.8
1000
p=0.000
p=0,001
1 0.278-0.168
0.292-0.168
Gaussian 0.8
3000
p=0.000
p=0.01
I 0.365-0.146
0.344-0.146
p
30
p=0.115
P=0.19
-0.003-0.017
0.001-0.017
2013
Gaussian p 0.3
100
p=0.000
p=0001
I 2 0.000-0.009
2 0.001 0.0099
20D
Gaussian p 0.3
300
p=0.000
p=0.001
I 0.005-0.005
0,007-0.005
20D Gaussian p 0.3
1000
p=0.000
1=0001
I 0,322-0.170
0.376-0.170
20D
Gaussian & 0.3
3000
p=0.000
p-0.01
2 0.632-0.253
0.689-0.253
2013
Gaussian 0.0
300
p=0.624
1 2 0.00ID-0.00S
2
0.000-0.005
Gaussian &
p=0.000
P-0.14
20D Gaussian
0.2
300
p=0.000
P00001
T 0.002-0.005
2
0.003-0.005
20D Gaussian م 0.3
300
p=0.000
p=0.001
I 0.005-0,005
0,007-0.005
201 Caussian 0.4
300
p=0.000
0=000
I 0,191-0.140
0.260-0.140
20D Gaussian P 0.5
300
p=0.000
0=0.01
Z 0.621-0,388
0,815-0.388
Sine wave 8
30
p-0.826
P-0.556
7
0.000-0.005
0.002-0.017
Sine wave 8
100
p=0.962
p-0.113
7
0.006-0.009
0.006-0.009
Sine wave 8.
300
p=0.093
p=0,988
1 0.448-0.306
0.511-0.306
Sine wave 8
1000
p=0.047
p=0111
1 1.351-0.450
2
1.316-0.450
Sine
p=0.166
p=0.994
Table 4: Sanity check of several statistical dependency testing implementations. See text for details.
Appronch
False Positive Rate
Number of Runs
Note
(95% confidence)
sig Spectral HSIC
0.000
0,040
612500
https://githin httpchuhb..oomccao oa rpy
Block HSIC
0.418
612500
ALL /q1 Lhub. com/ oxn los coskeepp kerpy
HSIC with Gamma Approximation
0.055
1000000
bEtps://gAt.00
significance was recorded to validate different implementations actually follow such false positive rates. Correct implementations should have false positive rate lower or equal 0.05. Results are summarized in Table 4 Statistically significant deviations (under Hoeffding inequality) are marked in bold font. The number of runs for DEMINE relatively low, but no false positives were found. Low false positive rate of DEMINE might be due to partly the conservative estimation provided by Hoeffding inequality, and partly the generalization gap between train and test splits.
18


Reviews:
[{"review_id": "SJxb8_g0FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a neural-network-based estimation of mutual information, following the earlier line of work in [A]. The main focus has been to develop an estimator that can reliably work with small dataset sizes. They first reduce the sample complexity of estimating mutual information by decoupling the network learning problem and the estimation problem by creating a training and validation set and then using the validation set for estimating mutual information. Of course, there is still the problem of learning the network with smaller sized data. For this, they propose the strategy of creating multiple tasks from the same dataset, where the dataset is run through transformations that do not affect mutual information.\n\nI am inclined to accept (weak) the paper for the following reasons:\n1. The paper uses some nice ideas to reduce the variance of the MI estimates and to be able to work with smaller dataset sizes. Both splitting data into training and validation and then using task augmentation to make learning robust are pretty nice ideas.\n2. The results on the synthetic datasets show that the resulting estimator does have low variance and the estimates are less than or equal to the true MI value, which is consistent with the fact that it is a lower bound estimation.\n3. The results on fMRI dataset were interesting and showed that the method gives improvements over baselines were the estimates were made on a smaller sized dataset.\n\nSome improvement suggestions:\n1. I don't see why MINE cannot be applied to the fMRI dataset and results be reported. I know that the variance in estimation is large, but it would still be useful to look at the performance of MINE in comparison to DEMINE.\n2. There are several errors in the writing - hyperparamters in Abstract, repetition of the word \"Section\" in fMRI experiment section etc. - which needs to be fixed.\n\n[A] Mutual Information Neural Estimation, ICML 2018", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byl0ceB1cH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript studies mutual-information estimation, in particular variational lower bounds, and focuses on reducing their sample complexity. The first contribution is based on adapting the MINE energy-based MI estimator family to out-of-sample testing. MINE involves fitting a very flexible parametric form of the distribution, such as a neural network, to the data to derive a mutual information lower bound. The present work separates the data fitting from the mutual information evaluation to decrease sample complexity, the argument being that the function class is no longer a limiting factor to sample complexity of the mutual information estimation. The second contribution uses meta learning to decrease the sample complexity required to fit the neural network, creating a family of tasks derived from the data with data transformation that do not modify the mutual information. The approaches are demonstrated on synthetic data as well as fMRI data, to detect significant inter-subject dependencies in time-series of neural responses.\n\nThere are some very interesting and strong contributions of this manuscript. However, I worry that one of the central theoretical arguments does not seem correct to me. Indeed, the manuscript introduces sample complexity results to justify the benefits of the out-of-sample procedure (th 1), but it seems to me that these give an incomplete picture. Th 1 gives the number of validation samples required to bound error between the mutual information estimate at finite samples and asymptotically for a function T parametrized by \\tilda{theta}. This control is of a very different nature from the control established by MIME which controls the error to the actual best possible variational bound. In other terms, the control of th 1 does not control the estimation error of T. This is the reason why it is independent from the function class. The total error in estimating the mutual information must take this error in account, and not only the validation error. Hence the theoretical sample complexities contributed are not comparable to those of MIME.\n\nThe meta-learning estimator seems to involve a significant implementation complexity, for instance heuristic switchs between estimation approaches. The danger is that could hard to make reliable in a wide set of applications. It would be more convincing to see more experiments. Along this direction, it is problematic that, in the synthetic examples, the relative  performance of methods changes significantly from experiment to experiment and there does not seem to be a simple way to control that. On the other hand, the MIME-f-ES tends to have a reasonably good failure mode.\n\nI would have been interested in \"false detection\" experiments: comparing estimator in a variety of problems where the mutual information is zero, but for different marginal distribution. This is particularly important when the application is to test for independence, as in the fMRI experiments.\n\nFor hyper-parameter search (using hyperopt), the manuscript should make it explicit what metric is optimized. Is it the data fit of the neural networks? With what specific measure?\n\nWith regards to the fMRI experiments, good baselines are missing: DEMINE is compared to Pearson correlation. Additionally, CNNs are not a particularly good architecture for fMRI, as fMRI is not locally translation invariant (see Aydöre ICML 2019 for instance). Finally, it seems that the goal here is to test for independence. In such a situation, there are better strategies for higher-order tests of independence than estimating mutual information, in particular estimator that give a control p-value.\n\nMinor comments: \n\nAlg 1 seems a fairly generic neural-network fitting algorithm. In its current state, I am not sure that it adds a lot to the manuscript.\n\nThere are many acronyms that are never defined: MINE, TCPC", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklLqYip9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Second update:\nWe thank the authors for updating their paper.\n\nThe work is now improving, and is on the right track for publication at a future conference.  There are a few comments on the new results, and suggestions for further improvement:\n\n* The issue of possible false positives due to sample dependence in fMRI data has again been ignored in the rebuttal. Without careful treatment of this effect, these results are vacuous.\n\n* There is still no comparison with competing nonparametric tests on the fMRI data.\n\n* the results linking the COCO and MINE estimators are interesting. Some statements don't make sense, however, eg. \"HSIC-based estimators tend to have loose confidence intervals due to the need to bound generalization error of kernels f and g on unseen data points.\" First, f and g are functions, not kernels. Second, testing with HSIC or COCO does not require generalisation to unseen data points: this is why testing is an easier problem than regression.\n\nFor HSIC testing, I am surprised to read in the footnote that the Gamma approximation report significantly more than 5% errors, especially given the Table 4 results that show the correct level. In any case, I very strongly suggest using a permutation approach to obtain the test threshold for HSIC, which is by far the most robust and reliable method. The Gamma approximation has no statistical guarantees, as stated explicitly in the HSIC testing paper of NeurIPS 2007. Permutation gives a guarantee of the correct level. See the NeurIPS paper for details. Once you have verified the correct false positive rate for the permutation threshold, then you can compute the p-value on the alternative.\n\nWhile the comparison with HSIC is a helpful one, it is also required to compare with the competing method closest to yours, i.e. the Berrett  and Samworth test. In addition, HSIC is a non-adaptive test, but your test is adaptive, so a fairer comparison would be to a modern adaptive test such as \"An Adaptive Test of Independence with Analytic Kernel Embeddings.\"\n\n* The false positive rate in the sanity check is far below the design level of 0.05. This is as I expected, given the use of the Hoeffding bound. This should be stated clearly in the main text, and not disclosed in the final sentence of the final page of the appendix.\n\n\n====================\n\nUpdate: thank you for your rebuttal. \n\n\"the necessary and sufficient condition of dependency, is more general and is complementary to other techniques that make stronger assumptions about the data. \"\n\nThe alternative tests listed are nonparametric. That is to say, unlike Pearson correlation, they do not make specific parametric assumptions on the nature of the dependence. Rather they make generic smoothness assumptions (your test also makes such assumptions by the choice of neural network architecture). Thus, comparison with the prior work in Statistics and Machine Learning is relevant, since these tests have the same aims and scope as your tests. \n\nThe cited Berrett and Samworth MI test uses a permutation approach to obtaining the test threshold, not an asymptotic approach  (see the results of Section 4 of that paper).   Several of the other cited tests also use a permutation approach for the test threhsold. These tests are therefore relevant prior work.  \n\n\" If things change very little from one second to the next, the signals could be very similar and may not really be, intuitively, independent samples and may bias result of the study. However, which independence assumptions to use is not in scope for our paper,  because our fMRI study is trying to show that dependency testing works \"\n\nIn the work cited by Chwialkowski et al, failure to account for the dependence between samples results in excessive false positives. This is because, for dependent data, the effective sample size is reduced, and the tests must be made more conservative to correct for this effect. It is therefore the case that the fMRI results may be false positives.\n\nRe level:\"This proof could be experimentally verified ...\"  This should be verified. In particular, Hoeffding can be very loose in practice, which is likely to be observed in experiments. \n\n\n======\n\nThe authors propose a procedure for improving neural mutual information estimates, via a combination of data augmentation and cross validation. They then use these estimates in hypothesis testing, on low dimensional toy datasets and on high dimensional real-world fMRI data.\n\nThe improved training procedures for MI estimation are of interest, however the hypothesis testing parts of the paper could still be improved.\n\nIn hypothesis testing, it is important to verify that the test has the correct level (false positive rate). This is all the more essential when the estimate has required optimisation over parameters. It is not clear from the presentation that this has been confirmed.\n\nThere are a number of prior approaches to testing for multivariate statistical dependence in the machine learning and statistics literature (including a 2017 paper which uses mutual information). A small selection is given below, although a literature search will reveal many more papers. In the absence of citation or comparison with any of the prior work on multivariate statistical dependence testing, the current submission is not suitable for publication. \n\n\nIn statistics:\n---------------\n\nhttps://arxiv.org/abs/1711.06642\nNonparametric independence testing via mutual information\nThomas B. Berrett, Richard J. Samworth\n2017\n\n\nMeasuring and testing dependence by correlation of distances\nGábor J. Székely, Maria L. Rizzo, and Nail K. Bakirov\nAnn. Statist.\nVolume 35, Number 6 (2007), 2769-2794.\n\n\nLarge-scale kernel methods for independence testing\nQinyi ZhangEmail Sarah Filippi, Arthur Gretton, Dino Sejdinovic\nStatistics and Computing\nJanuary 2018, Volume 28, Issue 1, pp 113–130| Cite as\n\n\n\nIn machine learning:\n---------------------\n\nMultivariate tests of association based on univariate tests\nHeller, Ruth and Heller, Yair\nAdvances in Neural Information Processing Systems 29\n2016\n\nA Kernel Statistical Test of Independence\nGretton, Arthur and Fukumizu, Kenji and Choon H. Teo and Song, Le and Sch\\\"{o}lkopf, Bernhard and Alex J. Smola\nAdvances in Neural Information Processing Systems 20\n2008\nhttp://papers.nips.cc/paper/3201-a-kernel-statistical-test-of-independence.pdf\n\nhttp://proceedings.mlr.press/v70/jitkrittum17a/jitkrittum17a.pdf\nAn Adaptive Test of Independence with Analytic Kernel Embeddings\nWittawat Jitkrittum, Zoltán Szabó, Arthur Gretton ; ICML 2017, PMLR 70:1742-1751\n\nTime dependence\n----------------\n\nIt is also the case that if the variables have time dependence, then appropriate corrections must be made for the test threshold, to avoid excessive false positives. Does the fMRI data exhibit time dependence?  For the case of multivariate statistical dependence testing, such corrections are described e.g. in:\n\nhttps://papers.nips.cc/paper/5452-a-wild-bootstrap-for-degenerate-kernel-tests.pdf\nA Wild Bootstrap for Degenerate Kernel Tests\nKacper Chwialkowski, Dino Sejdinovic, Arthur Gretton\nNeurIPS 2014\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper deal with a mutual information based dependency test. 

The reviewers have provided extensive and constructive feedback on the paper. The authors have in turn given detailed response withsome new experiments and plans for improvement. 

Overall the reviewers are not convinced the paper is ready for publication.  
Decision: Reject