Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a combined technique based on matrix factorization and adversarial models to reduce the batch effect problem in fMRI data. The new model, called rshSCP, is designed to be robust to the site-effects in estimation of sparse connectivity pattern components. The idea is to decompose correlation matrices into a set of shared hierarchical pattern of subject components as well as site related components. Then an existing adversarial model is applied in order to increase the reproducibility of them. They also applied a simple initialization method for optimization procedure and evaluated their approach on both simulated and real datasets.
main_review: I found the paper is well written, the proposed idea interesting, and the motivation is clear. I enjoyed that the proposed approach uses matrix factorization to tackle the site-related batch effect problem in fMRI data. However, the base of the formulations for either the matrix factorization or adversarial learning of hSCP is already introduced in other works. The original novelty of the paper includes decomposing the correlation matrix into both hSCP and site components. 

One primary concern regarding this method could be the reproducibility problem and the lack of theoretical guarantees on sub-optimal solutions. Given that this sparse hierarchical pattern is supposed to estimate a proper human brain functionality using fMRI data, it is valuable to design convex objective functions that lead to a unique optimal point rather than multiple sub-optimal solutions. In the current paradigm, the optimal solution for $W$, $D$, $U$, and $V$ is dependent on the initialization procedure as is also mentioned in the paper. Despite the careful initialization of matrices, comparing the performance of the results in either rshSCP and Adv. rshSCP, with and without random initialization, does not show significant improvement (Table 1 and Table 2). 
Moreover, based on the results demonstrated in the experiment section, more justification is required for higher computational cost in the proposed method, given the slight performance improvement comparing to baselines. 

A couple of further comments: 
1) Try to define the dimensionality of matrices explicitly as soon as you introduce them. I am not sure what are the dimensionalities of $U^s$ and $V$? How are they defined? What do their dimensionalities represent, given that we want $U^s V$ to fit the dimensionality of $\theta^n$? 
2) Designing a [group] regularizer that can capture the biological/functional feasibility of subject-specific components or site-related properties might help with non-convex optimization problems and reproducibility issues. 

Overall, I feel like the contribution in this paper is not of great enough significance for this venue right now. Yet, I would be open to increasing my overall score based on the authors' responses since the idea behind the paper is interesting.
limitations_and_societal_impact: The authors adequately addressed the limitations of the work under either future directions or the weaknesses of the model in the conclusion section. They also provided discussions on different challenges along with the paper and tried to alleviate the corresponding limitations by suggesting and implementing new propositions.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces the method site-robust hierarchical sparse connectivity patterns (rshscp) and its adversarial extension. It builds on sparse connectivity patterns and its adversarial extension introduced in references 22 and 23. These propose a factorization of correlation matrices extracted for functional connectivity analysis. The factorization consists of a symmetric product of  sparse factors around a central diagonal weighting matrix. The present contribution introduces a site-dependent component into the optimization problem and adds a classifier to predict site, which is then adversarially optimized to make the learned structure lose information about site. In synthetic and real-data experiments it is shown that the method is indeed capable of discarding site information while retaining other relevant information.



main_review: While this contribution shows that site information can be discounted, I see several issues with it as it stands, which transcend through all evaluation criteria of interest.

Technical questions and comments:
- What is the trace(Lambda) = 1 constraint for? Does this scale correctly with n? It seems like the W need to grow bigger and bigger to accommodate trace(correlation matrix) = n. Have you considered other constraints?
- Why is Gamma_n created by adding 1 to Theta_n? Why is the adversarial perturbation a matrix of 1s and not e.g. a noise vector? Also, 1_P is not defined in the notation, but J_P is. Is 1_P == J_P or does it mean something else? Furthermore, adding anything to Theta without care or renormalization takes it away from the set of feasible correlation matrices.
- Eq (5) necessarily creates non-symmetric matrices if U^s is not a multiple of the identity matrix. The diagonal should be applied on both sides if anything. How is this not a problem for the algorithm?
- How are all the lambda_r set? (also: using lower-case and upper-case lambda for different things is a bit confusing. At the beginning I thought that lower-case lambda were the diagonal entries of upper-case lambda)
- Is the W_n in formula (1) a typo and should be W_j?

Experiment questions and comments
The fact that the proposed method works better under the evaluated metrics than the same method without site information is the absolute minimum bar: If it doesn’t do that, you need to look for another method. The more interesting comparisons are ones that compare to existing literature. While there is a paragraph stating that this is somehow impossible, I do not believe that to be the case.
Two things should be done:  1) known preprocessing methods (the ones cited) should be used to remove confounds, followed by the extraction of the hierarchical decomposition 2) Other methods for correlation-matrix extraction should be used in combination with deconfounding methods, e.g. group sparse inverse covariance methods.
If evaluation ever becomes a problem, here is a suggested common evaluation metric: Generate BOLD time series. For example, given the BOLD time series in some ROIs or maybe a full hemisphere, use the extracted correlation matrix (+ variances if necessary) to generate BOLD time series for the missing part of the brain, for every voxel. This evaluation method also takes into account the parcellation chosen.

Can this method, being hierarchical, support going down to correlations at the voxel-level? This would be an interesting step forward.

It would be very interesting to see this method elevated to a general confound elimination method. Sometimes subject age is a confound. It seems like this framework can accommodate this confound in the same way as site is a confound. It would be interesting to see this method predicting a pathological variable with e.g. age, sex and site as confounds. One could evaluate its robustness against choices of test data at very different propensity scores (such as evaluated in the recent https://www.biorxiv.org/content/10.1101/2021.06.16.448764v1).


Conceptual questions and comments
There seems to be some semantic fudging and absence of logical connection around the use of the term “hierarchical”. First of all, if at all applicable at the level of neurons, this does not mean that this term is applicable at the level of voxels and vice versa. More importantly, there is no requirement or indication that incorporating a hierarchical matrix factorization idea into a connectivity estimate captures any of the aforementioned structure. One way of proving that there might be a link would be to show that the particular factorization method introduced here is better than flat methods such as TV+L1-extracted regions -> correlation matrix, followed by conditional BOLD generation.

The title is phrased way too general. This paper is not about fMRI data analysis, but very specifically about connectivity estimation. At minimum this should be rephrased. 


limitations_and_societal_impact: There is unlikely to be any societal impact. The possible societal impacts would be positive for medical diagnosis, but fMRI has not established itself as a tool of choice in any field of medical diagnosis.

The obvious limitations of any approach related to fMRI "connectivity" (first and foremost the neglect of first-order driving effects, second the acknowledgement of all possible other confounding effects) should be required to be mentioned in every paper that addresses it. Sadly, this is at odds with the incentives for publishing, so it would need to become a requirement.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper deals with the important problem of how to pool fMRI data from multiple sites for connectivity estimation. The authors propose a simple extension to adversarial hierarchical sparse connectivity pattern (hSCP) estimation to remove site effects. The argument against adopting methods, such as the widely-used COMBAT, is to preserve the manifold structure of connectivity matrices. The method is tested on simulated data with higher accuracy shown. The method is also tested on real data with higher reproducibility demonstrated and age effects retained.
main_review: Originality
The proposed methods, rshSCP and Adv. rshSCP, build upon previous methods, hSCP and its adversarial extension (Adv. hSCP), partly by using the usual trick of adding a loss to discourage prediction of confounds, in this case site effects. The idea at a high level is not particularly original, but the implementation details are useful. A bit surprised domain adaptation and optimal transport are not cited and discussed.

Quality
The proposed methods are technically sound, but there are a few concerns regarding assumptions and results.
1. Is retaining the positive semidefinite property essential for downstream analysis? Comparisons with DL methods that do not retain positive semidefinite property are needed.
2. Following the theme of modeling the manifold property of correlation matrices, why use l2 loss in (1) as opposed to Gaussian likelihood as in standard Gaussian graphical models or intrinsic distance between covariance?
3. Will there be an identifiability problem since W1*W2 = W1*A* A’*W2 for A orthogonal?
4. For estimating rshSCP with only one site s in the reproducibility experiments, would using the same V estimated from all sites except s for both reproducibility assessment splits result in inflated reproducibility values?
5. How much did site predictability drop using adv. rshSCP compared to adv. hSCP?
6. Age-related biological variability seems to be retained but just on par with hSCP. Does that imply there is no need to account for site effects? What happens if the two studies with quite different age range were removed? Also, how is the performance of adv. hSCP? Overall, to claim biological variability is retained, need to show results for more than just age.
7. Not sure what to make of the qualitative analysis in Figures 1 and 2. Are there no exceptions where adv rshSCP pools “wrong” regions into a network at any of the hierarchical levels? 

Clarity
The paper is clearly written but there are some debatable statements in the Introduction.
1. Is hierarchical representation always better than single level parcellation or is it application dependent?
2. Is “binarized” components always better than their continuous counterparts, e.g. derived from PCA and ICA? The argument was higher interpretability with binarization, yet in an earlier statement, it was mentioned that ICA, NMF, and dictionary learning provide “interpretable” components. Along this line, allowing for components to overlap matches biology but this is provided by PCA and ICA.
3. Which biological predictors are likely to be correlated with site effects? If we remove site effects, wouldn’t biological effects be removed as well.

Significance
The problem being tackled is important and the approach in the proposed methods is applicable to other methods though implementation details would likely need to be altered to a certain extent. The results, however, are not particularly impressive. Whether the proposed methods are suitable for EEG and MEG is slightly questionable since temporal dynamics is more important for those modalities.

Summary after Author Responses
While some comments were well addressed, a number of issues remained, such as reproducibility using the same V, how much site predictability dropped using adv. rshSCP compared to adv. hSCP not reported for real data, need for prediction beyond age (which is only correlated with site at 0.2), and using limited subjective prior neuroscience findings to evaluate the extracted components (in a subjective manner). Hence, I will keep my original score.
limitations_and_societal_impact: Limitations are briefly mentioned in the conclusions, and for the current prediction task, there is no negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method called rshSCP (robust-to-site hSCP) to estimate hierarchical sparse connectivity patterns (hSCPs) from multi-site fMRI data while reducing the impact of site-specific variation due to scanner and experiment details rather than biological effects. Their approach is applied on top of the previous hSCP method as well as an extension that uses adversarial learning to improve reproducibility of estimated patterns (Adv. hSCP). The proposed method includes an additive component to the decomposition of the correlation matrix in hSCP, which models site effects as a matrix factorization with diagonal site-specific and full shared (across-site) matrix components. The method then additionally applies an adversarial classifier (a two hidden-layer feed-forward neural network) to predict the site from the subject-specific components of the hSCP decomposition. The entire optimization is formulated as a minimax problem, where the classifier maximizes its ability to predict site information while the remaining parameters are optimized to reconstruct the correlation matrices while minimizing the classifier's accuracy. The authors also propose an SVD-based initialization which improves reproducibility of the estimated components. Experiments on simulated data as well as fMRI data from multi-center imaging studies show that rshSCP improves accuracy (on simulated data) as well as reproducibility (on simulated and real data) of the estimated components, both when applied with hSCP and Adv. hSCP. The proposed method also leads to a reduction in accuracy when predicting site from the estimated subject-specific components using an SVM, as desired. Predicting age from the estimated subject-specific components with a random forest and visualizing the shared components shows that rshSCP preserves relevant biologically-related variation.
main_review: Overall, the paper is clear and well-written. The primary contribution is the inclusion of an additional additive term to the hSCP decomposition, as well as an adversarial loss and classifier to remove site-specific variation from the learned components. While an adversarial approach to removing confounding effects is not novel, this seems to be its first application to hSCP and the authors carefully demonstrate the ability of this technique to learn more accurate and reproducible components than versions of hSCP that do not use this technique.

One interesting comparison that could have been made is applying a harmonization method (e.g., ComBat) to ${\bf \Lambda}$ directly (as the authors discuss), then addressing the issue of altered relative contributions of the global components by fixing the harmonized ${\bf \Lambda}$ values and re-fitting the ${\bf W}$ matrices. Determining whether this approach achieves comparable reproducibility and/or interpretability of the learned components would provide a simple baseline to compare to the adversarial approach. However, the authors demonstrate the utility of their proposed method rigorously enough even when compared to other versions of hSCP that I don't feel the lack of this comparison to be a major issue.

Another choice that could be altered is using an SVM to demonstrate the reduced ability to predict site information from the learned subject-specific components. The authors do not justify this other than to say "we already used feed forward network". It is unclear whether a feed forward neural network (such as the architecture used to train the adversarial classifier) would be better at extracting site information even from the learned rshSCP components, and showing the results with the same or similar choice of classifier would strengthen that result.

Other minor comments:
- The subscript for the second ${\bf W}$ in the top row of Equation 1 should be $j$, not $n$.
- If my understanding is correct, the cross-entropy loss term $\mathcal{L}$ in Equation 7 should have a negative sign (so that the cross-entropy is maximized with respect to the hSCP parameters and minimized with respect to $\zeta$ in the joint optimization)
limitations_and_societal_impact: Yes, the authors discuss the potential negative impact and limitations of their work when considering the validity/interpretability of findings in neuroscience.
review_title: null

Meta-review: Dear authors,

despite some initial concerns reviewers have now, after discussion and reading
your rebuttal, endorsed the paper (although not strongly). I will therefore suggest
a rather positive outcome for this work.

Best regards,
The AC
Decision: Accept (Poster)