Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: The authors present two metrics to make inference about how two different brain zones are affected by the same experimental stimulus. The first metric, zone generalization, is the correlation between the predicted task-evoked signal for one zone and the actual measured task-evoked signal for another zone. The second metric, zone residuals, is the average across pairs of participants of the correlation between the residuals of regressing the actual task-evoked signal of one zone from another zone, for each participant in a pair. Using these metrics the authors propose four types of inferences that combine conclusions about the similarity of the effect of one stimulus on two different brain zones and about stimulus-related effects not captured by the stimulus model used. The authors illustrate the application of the metrics on simulations that manipulate the degree of similarity of response and the degree of stimulus properties not captured by the stimulus model. (In the supplementary material an analysis of signal to noise ratio is also presented.) The authors then apply the metrics to two empirical fMRI movie watching datasets and 34 language regions of interest, illustrating the four different inferences allowed by their approach. In the supplementary material the authors show in simulations that functional connectivity and some implementations of representational similarity analysis do not allow for the inferences proposed in this framework.
Main review: This is a well written paper proposing a quantitative framework to expand the inferences that can be done in the study of the causal effects of task stimuli on brain signals. The originality of the proposal is based on the explicit modeling of a unique and shared task causal effect (which the zone generalization metric aims to capture), and in the explicit modeling of unique and shared real task features causal effects not captured by the chosen task stimuli model (ie. difference between stimulus properties and stimulus representation) (which the zone residuals metric aims to capture). The manuscript contains a supplementary material file with useful code to track simulations and analysis. The supplementary material also contains a useful guideline explaining how the metrics can be normalized according to the research problem at hand and a very useful figure (Fig. 6) illustrating the relationship between brain zones task-evoked effects, stimulus properties and stimulus representations. Some points require clarification:

If you apply a standard task linear regression for zones Y1 and Y2: Y1 = bT + e1, where b, that measures the causal effect of T on Y1, is statistically significant, and Y2 = aT + e2, where a is also statistically significant and  a != b, it seems that I could claim that the same task representation T produced different causal effects on zones Y1 and Y2, given that a!=b. Similarly if a == b, I could claim that task representation T produced the same causal effect on zone Y1 and Y2. How is this inference different from what you measure with the zone generalization metric?

The zone residuals metric takes into account the number of possible pairs of participants. Does the metric approximate to its real value when the number of participants in the dataset tends to infinity? Or, can the metric have a bias problem if we do not have enough participants?

A fundamental point of the paper is the ability to make the four inferences proposed by the framework based on assessments of large and small values of the metrics. In the empirical application the authors used a cutoff based on simulations, and in the discussion this point is pushed to future research. I consider that given the importance of this cutoff to make the correct inference, more should be discussed in the manuscript about how the selection of the cutoff could be done, for example data-driven, previous-knowledge, comparison to other tasks. This of course will be only ideas, but they could give the readers a sense of how this problem can be solved. In my view, without a way to solve this problem the framework loses utility.

The two metrics, zone generalization and zone residuals, are pairwise measures. Can they be thought from a multivariate perspective? For example, by controlling for the effect of a third zone? In other words, can a third zone, for example, confound the metrics, as it may occur in functional connectivity analysis?
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper considers the problem of whether the stimulus properties affect different brain zones in the same way. To address this problem, the authors proposed a method (Figure 1 in the paper) which involves two new metrics: zone generalisation and zone residuals. Synthetic simulations and real data applications are implemented to illustrate the performance of the proposed method.
Main review: Pros:
1. the considered problem is interesting and important.
2. the real data application in the paper is interesting.
3. Although without any theoretical/mathematical justification, the proposed metrics are intuitively reasonable.

Cons:
1. Section 2 of this paper is a bit of confusing to me, because three different models are presented in this section, and the main metrics, zone generalisation and zone residuals, are defined based on different models. So it is unclear to me about which is the model that you main method (Figure 1) based on?
2. Although the considered problem has some "causal flavour", I feel that the proposed method is rather irrelevant to causality. So I am not sure whether this paper would be a good fit for this conference.
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: An intriguing question about causal effects in neuroscience, and tools from structural equation modeling to answer them.
Main review: The paper proposes an intriguing question, to study if two causal effects indeed share a common stimulus and if there are unobserved stimulus properties that can act as a common cause. The proposed metrics to answer the question are simple and easy to follow; the review below focuses only on the weaknesses of the paper.

Figure 1 is very helpful. Consider adding a Figure 0 that illustrates encoding model performance (i.e. Stimulus properties -> Stimulus representation via e.g., ELMo; recording activity in target brain zone; measuring predictability via "variance overlap" as in the first row of Fig 1).

Eqn 3: For the zone generalization metric, it seems important to normalize by the encoding model performance corr(hat{Y}_1, Y_1) and corr(hat{Y}_2, Y_2)? If not, how should we calibrate large vs. small zone generalization values corr(hat{Y}_1, Y_2)? The text also suggests that small values for zone generalization, *in addition to* good encoding model performance, tells us that two zones respond differently. Can we define the metric using relative correlation measures? Another naive question: Should zone generalization be a symmetric metric? I.e. combine corr(hat{Y}_1, Y_2) and corr(hat{Y}_2, Y_1) somehow?

For the regression in Eqn2, should the model still be fitted using the cross-validation procedure for fitting Eqn1 (for each zone in isolation)? Eqn2 suggests that we should fit the two encoding models jointly... Please clarify how hat{g}_1 and hat{g}_2 are fitted to training data.

Zone residuals: It will be helpful to give intuition for why zone residuals are not suitably computed for a single participant. Is there a fundamental non-identifiability of g and h in Eqn 4 if we only get to see a single participant? (The linear assumption on g likely avoids this non-identifiability). Is there non-identifiability between h and noise epsilon_i? Would structural assumptions on h help? How do multiple participants help to avoid this difficulty?

Again, Eqn4 suggests that we should fit the encoding models across all participants jointly. Please clarify how hat{g} and the residual regressions Y_1,P <- Y_2,P etc. are fitted to data.


Section 3 Simulations: All the simulations are well-specified/realizable. I.e., brain activity in each zone is indeed a function of X, Z and epsilon. It would be helpful to simulate mildly mis-specified settings: E.g. Zone 1 activity is a mediator or effect modifier for Zone 2 activity. (Does not fit the additive shared noise epsilon_12 model but could plausibly be true in actual fMRI data).
In general, Z being assumed as a common cause of Y_1 and Y_2 is a specific causal modeling assumption. It will be helpful to spell this out, and discuss alternate models for when zone generalization is large (e.g., that Y_1 is a cause of Y_2 or vice versa). Note that the zone residuals will have to be modified in the case of these alternate causal models. <-- This clarification around the modeling assumptions is the biggest opportunity to strengthen the paper.

Section 4: Naive question (connecting to the asymmetric nature of zone generalization and zone residual metrics) -- Do the asymmetric relationships inferred between different ROIs suggest a different underlying mechanism than a common cause Z (or is it fully explained by different base SNR for the different ROIs, contributing to an asymmetric zone generalization value)?

-----
(After author response)
I appreciate the authors' clarification to Q3 and Q8 (asymmetry in the metrics) and the response to Q7 (which was the weakest point) taken in conjunction with response Q4 to Nc86 have satisfied my concerns about the utility of these metrics in mis-specified settings.
review_title: null

Meta-review: The authors develop two metrics for evaluating how different brain zones may be affected by the same stimuli. The framework created shows promising results when applied to actual and simulated data. This is a well-written paper with a particular application in mind. It designs a framework for discussing causal effects for applications in neuroscience. This paper's developed metrics and framework are potentially impactful and present a promising start for future research in this direction. The reviewers and I found the author's replies and rebuttal especially helpful in reaching our decision, so I hope some of these comments and arguments are included in the final version of the paper.

Minor comment: In addition to the comments made by the reviewers below, I would encourage the authors to make the references more consistent. At the moment some authors' first names are included and others are not.
Decision: Accept (Poster)