An unexpected confounder: how brain shape can be used to classify MRI scans ?

Authors: valentine  wargnier-dauchelle
thomas  grenier
michaël  sdika
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 18:00:02

Abstract: Although deep learning has proved its effectiveness in the analysis of medical images, its great ability to extract complex features makes it susceptible to base its decision on spurious confounders present in the images. However, especially for medical applications, network decisions must be based on relevant elements. Numerous confounding factors have been identified in the case of brain scans such as gender, age, MRI sites or scanners, etc. Nevertheless, although skull stripping is a classic preprocessing step for brain scans, brain shape has never been considered as a possible confounder. In this work, we show that brain shape is used in the classification of brain MRI scans from different databases, even when it should not be considered as a clinically relevant factor. To this purpose, we introduce a rigorous two steps method to assess whether a factor is a confounder or not, and we apply it to identify the brain shape as a confounding variable in brain images classification. Lastly, we propose to use a deformable registration in the data preprocessing pipeline to align the brain contours of the images in the datasets, whereas standard pipelines often do nothing more than affine registration. Including this deformable registration step makes the classification free from the brain shape confounding effect.

Main paper: 
 introduction: 
 Deep learning has emerged as a powerful tool in the field of medical imaging. Its ability to automatically learn and extract complex patterns from vast amounts of data has revolutionized the way we analyze images. However, the great performances of deep learning come with the price of the black-box nature of these methods: deep neural networks, with their non-linearity and their large number of parameters, are difficult to explain. Training explainable and interpretable networks is therefore a key issue for medical image analysis as the lack of transparency can hide the fact that the network decision may be based on wrong reasons: a bias in the training set can make a confounding factor plays an important role in the decision. Classifiers are especially subject to this problem whether they are used for pure classification problems or as guidance for adversarial networks or diffusion models.Several tricks can be used to reduce or remove the influence of a known confounding factor from a model. The simplest is to carefully collect the training dataset such that the confounding variables are matched in the different classes as in (Leming et al., 2022) . However, this approach makes it tedious to create large datasets. If possible, normalization preprocessing can also be used to discard variations of this variable across the dataset. For example, in (Wargnier-Dauchelle et al., 2021) , the MRI signature is removed using brain tissue probability maps instead of MRI scans as input of a deep classifier. Data augmentation can also be used to make the model invariant to a set of variables or transforms. In the literature, works have also been proposed to train models free from the influence of a known confounder. In (Zhao et al., 2020) , the features of a network are trained for the prediction of the objective task but also trained adversarially for the prediction of the gender, considered here as the confounding variable, making the features invariant to this confounder. In (Wang et al., 2018) , the model is first trained for the objective task and the top layer is then fine-tuned to predict the confounding variable: the gender, the subject or the contrast material. During this fine-tuning step, weights sensitive to the confounding factor are identified and discarded. For these methods to be used, the potential confounders need to be identified first. Moreover, for the two latter methods, the potential confounding factor should be available as a scalar of categorical data and the value of the confounding variable should be known during training. To identify confounding factors in images, attribution maps can be used to localize the information used by the network to make its decision. For example, in (Sun et al., 2023) , an attribution maps comparison protocol was proposed by visualizing confounding factors artificially added to the images. In (Wargnier-Dauchelle et al., 2021 , 2023) , these maps are used to validate and/or improve the interpretability of the network by verifying that the decision is based on brain lesions.In this work, we assess the importance of the brain shape as a confounding factor in the classification of brain images. To do so, we propose a rigorous protocol to verify that the brain shape is indeed a confounding factor used as a part of the network decision despite the standard affine spatial normalization. The first step of our protocol is to verify if it is possible to classify various datasets using the confounding factor only: for the concrete case we investigate, we propose to use the brain mask as a brain shape representation and our first non-intuitive result is that it is possible to classify the datasets using only this mask. Then, we check that the brain shape is indeed part of the decision when original grayscale images are used as input of the network. To this end, we modify the image such that the identified factor could be from the opposite class and evaluate the impact on the classification task. We come up with two solutions to change the brain shape from one class to another: by trimming the borders of the brain or with deformable registration. Finally, we show that by complementing the standard affine registration of the preprocessing with a deformable registration to normalize the brain shape, we can classify brain images while canceling out the confounding effect of brain shape in classification. 
a generic two steps confounding factor identification procedure: 
 Our procedure to assess that a variable is indeed a confounding factor is in two steps. In the first step, we verify that the suspicious variable is a potential confounding factor by checking whether or not it is possible to classify the data using this variable only. To do so, a classifier, having this variable only as input, is trained to classify the subjects using the same class label as the original problem. If the classifier is random, with an accuracy close to 0.5, the variable can be discarded from the potential confounders of the problem.If the classification is possible in the first step, we check in the second step whether this factor is indeed used by a model trained with the original images as input. To do so, the model is trained conventionally on the original images. Then, test data are transformed such that the value of the suspicious factor lies within the distribution of this factor for the opposite class, while modifying the image as little as possible. The difference in classification performances between the original and the transformed test data is then measured. A lower classification performance for transformed data would indicate that the identified factor is used by the model to correctly classify and is indeed a confounder.For the concrete case of the brain shape, the first step is achieved by trying to classify several brain datasets using the brain masks only as input of the network. This binary mask, indicating whether a voxel is inside the brain (value of 1) or outside (value of 0), is used as a representation of the brain shape. For the second step, we need a transform to make the brain shape of a subject match the shape of subjects in the opposite class, while changing the image content as little as possible. Two such transforms are investigated: brain mask crop or brain mask registration.Brain mask crop A brain mask is randomly drawn from the opposite class and used to crop the grayscale image of the current subject: pixels outside the mask are set to the background value. This cropped image is given as input to the classification model at test time. The image remains the same inside the brain mask but it is changed at its border. Note that this technique modifies only a part of the shape: the part outside the mask drawn from the opposite class.Brain mask registration A brain mask is also randomly drawn from the opposite class but this time, it is used as a reference to realign the brain shape of the current subject. To realign a moving brain mask B m on a reference brain mask B r , we solve the following optimization problem:EQUATION)where T is the transformation we look for, ∂B m and ∂B r are the border of the two brain masks and d(., ∂B m ) is the Euclidean distance to the border of the moving brain mask. To penalize strong deformations, a bending energy term with a coefficient λ is added and the Jacobian of the transformation J(x) is constrained to be higher than a given threshold t for all voxels. Any registration algorithm could be used with a null image as the fixed image, the distance transform of ∂B m as the moving image and a cost function masked with ∂B r . The transformation T is then applied to the original grayscale image to obtain an image with a brain shape of the opposite class.Affine registration to a reference template is usually included in the data preprocessing to spatially normalize the datasets. We advocate that this affine registration step is not sufficient to avoid the brain shape confounding effect in deep learning. We propose to add a deformable registration step to normalize the brain shape. To do so, we extract the brain mask of all subjects as well as of the reference template, and realign each subject's brain mask to the reference template brain mask by solving the registration problem of Equation 1. The computed deformable transforms are then applied to the corresponding images to create shape-normalized brain datasets that can be used to train any network. 
data: 
 Seven T1w MRI datasets are used in our experiments: the five public healthy databases IXI1 , HCP2 (Babayan et al., 2019) , kirby (Landman et al., 2011) , MPI (Babayan et al., 2019) and IBC (Pinho et al., 2018) , the OFSEP/EDMUS multiple sclerosis (MS) dataset3 from the "Observatoire français de la sclérose en plaques", the MS french registry (Vukusic et al., 2020; Confavreux et al., 1992) , the MICCAI BraTS 2020 glial tumors public dataset (Bakas et al., 2017 (Bakas et al., , 2018;; Menze et al., 2014) that also includes the manual tumors segmentation, and the Alzheimer's disease (AD) ADNI-1 dataset4 (Weiner et al., 2010) which also includes healthy subjects (CN). Division in training, validation and test sets is given in Table 1 .MR images are preprocessed using FSL FLIRT affine registration on the T1 MNI atlas (Jenkinson and Smith, 2001; Jenkinson et al., 2002) , HD-BET brain extraction (Isensee et al., 2019) and N4 bias field correction (Tustison et al., 2010) except for the BraTS dataset. As this dataset is provided preprocessed using the CaPTk pipeline5 that includes the brain extraction (different from HD-BET), we only applied the affine registration and the bias field correction. The final image size is 91 × 109 × 91 with a 2mm voxel size. Binary classifiers are trained to classify the brain MRI datasets (either healthy, with multiple sclerosis or with tumor subjects). In the following, "shape normalized" datasets refer to the datasets normalized using the procedure of Section 2.3. We also evaluated the impact of elastic deformations data augmentation during training (denoted as "Elastic DA"). The deformations were chosen to be strong enough to hide the differences between brain shapes of the different datasets. Classification performances are evaluated using the true positive/negative rate (TPR/TNR) and the balanced accuracy (BA). In Section 4.2, we analyze the feasibility of distinguishing the datasets using only the confounding factor that-is-to-say using only the brain binary masks. In Section 4.3, we evaluate if the brain shape is indeed a confounding factor for classification models trained with MRI input6 .The classifier, implemented in Pytorch, is a 3D PatchGan (Isola et al., 2017) , trained with the Adadelta optimizer (Zeiler, 2012), class balanced minibatches, an initial learning rate set to 1 and the cross entropy loss. This CNN is defined as C64-C128-C256-C512 where Ck denotes a Convolution-BatchNorm-LeakyReLU (slope 0.2) layer with k filters, except for the first layer on which no BatchNorm is applied. At the end, a convolution is applied to obtain a 1-dimensional output. In Section 4.1, attributions were computed using gradient maps (Simonyan et al., 2013) . The brain shape registration algorithm of Section 2.2 is based on the algorithm described in (Sdika, 2008) . We set t = 0.85, λ = 10 -3 and the transformation is represented by a B-Spline vector field with a node spacing of 4 voxels. 
attributions highlight brain borders: 
 To visualize the confounding factors, we use attribution maps which indicate the relevance of each voxel in the network decision. Figure 1 shows some attribution examples for the IXI vs BraTS classification. In Figures 1(a ) and 1(b), attributions are focused at the top and bottom of the brain, near the borders. Especially, high attributions (in absolute value) are in the brain stem. Yet, we might expect the areas inside the brain to be the most useful for decision-making. Although these attributions indicate that the brain shape can be involved in the network decision, they are not sufficient alone to draw a definitive conclusion.We apply the first step of our method as described in Section 2.2: binary classifiers are trained to classify several datasets using the brain masks only as input. As shown in Table 2 , the first startling result is that, except for the intra-dataset ADNI task, it is possible to distinguish all pairs of datasets based on the brain shape only, without using any tissue or texture information. Brain masks from the tumors dataset BraTS can be classified from the healthy dataset IXI with a perfect classification score. Note that this classification performance could be partly explained by the difference between the BraTS preprocessing pipeline and the pipeline used for the other datasets. Classifying MS brain masks from the healthy ones is more difficult but it is still possible with a balanced accuracy of 62%. Thus, it is possible to classify healthy vs pathological subjects using only the brain shape. One might wonder to what extent the difference in brain shape is due to the pathology itself or to some differences due to the way datasets are built. To investigate the dataset construction effect, we consider the brain mask classification task between several healthy datasets and within the ADNI dataset. For example, we obtain a perfect accuracy for the IXI vs HCP problem. Even when several databases are aggregated in one class, the distinction is always possible with an accuracy higher than 85% for IXI vs IBC/kirby/MPI. Population age in the databases is another element that can influence brain shapes (with normal aging brain atrophy). However, despite the subjects of HCP and IBC/kirby/MPI match in age, it is possible to classify these datasets almost perfectly. In the intra-ADNI experiment, classes are from the same dataset and match in age but one is healthy and the other pathological. One can see that, despite the possible atrophy due to AD, brain masks are more difficult to distinguish with a balanced accuracy as low as 55%. This reinforces the idea that in general, the way datasets are built is a stronger factor than the disease itself for the brain mask classification. Thus, it is possible to distinguish various databases based on the brain shape only and the difference seems to be linked to a dataset construction effect that is not eliminated when disease or age factors are not present.In this part, we applied the second step of the process described in Section 2.2: for each subject, a brain mask is randomly drawn from the opposite class and used to crop the current image ("Cropped images") or realigned its brain shape with a deformable registration ("Registered images"). An image for which the crop or the registration changes the predicted class, as well as the corresponding modified images, are shown in Figure 2 . We can see that the brain shapes of the two classes are different. With the crop, the shape is partially modified: for example, the area around the stem or the frontal lobe do not change, whereas the registration allows to better fit the shape. Most of the tumor area is left untouched with the crop and it is probably not this loss of information that changes the classification. Indeed, on average in the test set (without shape normalization), only 3.6% ± 5.0% of the tumor is cropped (3.3% ± 5.5% for the images still classified as patho- logical and 5.6% ± 3.3% for the others). The classification results are presented in Figure 3 . For tumors, without elastic data augmentation, the classification performances on the original images are perfect. Conversely, when the shape is modified the accuracy of both classes falls. The TNR is lower than 50% for the cropped images and it falls to 16% for the registered images. Thus, the brain shape seems to be a key factor learned by the network to classify the images. For MS, the impact is not as strong but still present with a mean loss of accuracy of 3 points for cropped images and 16 points for registered images. This is consistent with the fact that brain masks are harder to distinguish for the MS dataset as shown in Section 4.2. When elastic data augmentation is used, the classification is more difficult on the original images for the tumors dataset but the decision seems less based on the brain shape as the accuracy decreases less when the modified images are tested: the accuracy is in average 27 points lower for cropped images and 7 points lower for registered images. This data augmentation improves the robustness, especially for MS as the classification is slightly better on the original images. In this case, the brain deformation seems to have hardly any impact with only 3-point accuracy difference on the registered images.The previous experiments validate that the brain shape is a part of the decision for MRI classification. We advocate that if the images are realigned not only with an affine registration but also such that the brain shapes are realigned to the reference template, the network decision can be free from the brain shape confounder. Figures 2 and 3 present the results of the same experiments as in Section 4.3 but using images with the shape normalization of Section 2.3. Visually, with the shape normalization, there is virtually no longer difference between the shapes. In terms of classification, we obtain similar or better performances than without normalization for both tumors and MS. When the modified images are used at the inference, the impact is minor, with an accuracy loss of around 5 points for tumors cropped images and around 1 point for tumors registered images. This is inferior to the model using images without shape normalization, even with elastic data augmentation. For MS, the results are equivalent or slightly better than without shape normalization and with elastic data augmentation. With the shape normalization, elastic data augmentation seems less useful. Moreover, in Figures 1(c ) and 1(d), attributions are no longer localized on the borders but all over the brain. Therefore, the shape normalization seems to be enough to eliminate the confounding effect of the brain shape. 
conclusion: 
 In this work, we propose a generic method to assess whether a variable is a confounding factor or not. We apply the proposed protocol to several public MRI datasets to identify the brain shape as a non-intuitive confounder for brain scans classification. In addition, we proposed to add a non-rigid brain shape realignment in the preprocessing pipeline to eliminate the confounding effect of the brain shape. As this step does not degrade the classification performances, our recommendation is to systematically use it (even when the brain shape is not a confounder) in addition to the affine registration conventionally used in standard pipelines. The elements highlighted in this paper could also be used in stateof-the-art methods like (Zhao et al., 2020; Wang et al., 2018) , which so far have only been applied to solve the problem for scalar confounding variables. For this, the confounding variable predicted in these methods would be the brain mask through a segmentation loss.with the attributions without shape normalization. The shape normalization is as efficient than on the tumors dataset, as the shapes of the two databases match better. 
 
 Appendix: brain-development.org/ixi-datasethumanconnectome.org/study/hcp-young-adultofsep.orgadni.loni.usc.educbica.github.io/CaPTk/preprocessing brats.htmlHealthy vs healthy datasets results are in the supplementary material for space consideration 


Decision: The paper explores brain shape as a confounder in brain MR classification; this is in line with some recent attention to confounders/shortcuts in the medical imaging community. The reviewers agree on the relevance of the topic and clarity of the paper. During the discussion phase, the main concerns revolved around the validity of the reached conclusions based on the presented experiments and explanations. The authors have addressed this in the revision, with several reviewers raising their scores. I agree with several of the reviewers that this work would make an interesting contribution to the conference and am happy to recommend acceptance.

Reviews: review: paper_summary: paper_summary: The study investigates whether the brain shape is a potential confounder for brain MRI classification tasks. The authors use skull-stripped data from the IXI, HCP, MPI, kirby21, IBC, OFSEP, and BraTS2020 collections. In the first experiment, the brain masks are used to classify the origin dataset of scans. Later, cropping and registration to the target dataset brain mask are applied to demonstrate that the classification performance indeed drops at test time, indicating that the shape is relevant. Deformable registration and data augmentation with elastic deformation is shown to be able to partially mitigate the issue.
main_review: main_review: Thank you for your detailed replies to my comments and for your effort to improve the manuscript. The submission is now in my opinion significantly improved, important points clarified, and I therefore have adjusted my rating.
strength_weakness: strength_weakness: - The method was tested on many different datasets, but the results are only presented/discussed for a small subset of the parts in 4.3/4.4. Please either add the full results in the supplementary or briefly discuss them to avoid the impression of "cherry-picking".
- The motivation for the choice of classifier is a bit unclear and unusual (PatchGAN). Since it is a patch-based approach, details regarding patch sizes, etc., are missing.
questions: questions: - Did you also test the grayscale-based confounding for the rest of the experiments listed in Table 2? With you only mentioning the IXI vs. OFSEP and IXI vs. BraTS this would be interesting
- Please provide more details regarding the selection of the PatchGAN classifier as mentioned above
paper_summary: paper_summary: This paper discuss the potential for brain shape to be a confounder despite the standard affine spatial normalization, and try to prove that it is by attempting the possibilities to classify brain MRI scans into the database sources. In doing so, the authors propose two step methods: 

(1) database source classification firstly using only the assessed confounder (brain shape) represented by 3D binary brain masks from different databases

(2) checking whether the brain shape is indeed a confounder by conventionally train the classifiers using the original brain MRI scans, then test the classifiers with two different transformation methods applied to the test data of the opposite class: (a) with and (b) without additional deformable registration. If the performance of (a) is lower than (b), it indicates that brain shape is indeed a confounder.
main_review: main_review: The paper has addressed the majority of the review comments and revisions have significantly improved the clarity and validity of the research. Although questions about what cause the brain shape confounders remain, this could spark more research in the future. Overall, the paper adds valuable insights to the field and deserves to be included in the conference.
strength_weakness: strength_weakness: Although I found the overall methodology to be valid, I have concerns regarding the methodology’s first step to assess brain shape as a confounder. 3D binary brain masks are used to classify brain MRI database source in this step. The results show that these masks alone can distinguish database sources, even among datasets with similar demographics in terms of age and brain health condition. This raises important questions, particularly about the causes of such differentiation. My concern centers on the potential oversight of other indirect confounders (i.e. sex and/or race distribution within database) that could inadvertently influence the classification’s success. 

At the same time, based on the findings of the methodology’s first step, the authors claim that age and brain health condition are irrelevant to the classification performance. This seems counterintuitive to the conventional understandings. Other than race and sex, age and brain condition are intricately linked to brain morphology and may act as indirect confounders, impacting the outcomes and interpretations of the study. It worries me if the authors come to the wrong conclusions for the first step’s results.

These concerns, however, do not diminish the validity of the results and conclusions derived from the methodology’s second step, which recognizes brain shape as a confounder. The suggestions following these concerns (see detailed comments) may help clarify the indirect effects of population demographics on brain shape, improving the study's conclusions.
questions: questions: In the detailed comments, I highlighted the need for several considerations, which include:

- transparency in disclosing information of other potential confounding factors that might influence brain shape
- additional comparison and analyses to support the study’s conclusions and statement, especially related to the methodology’s first step
- reevaluation of the study’s conclusions due to potential oversight of other confounding factors that might have had influence the brain shape
- a rationale behind the exclusion of comparison between normal datasets in the methodology’s second step experiments
paper_summary: paper_summary: The paper investigates into the issue of confounding factors in models used for the classification of brain MRI scans, with a focus on the previously unconsidered factor of brain shape. It introduces a two-step method to first identify and then mitigate the confounding influence of brain shape on classification accuracy through the use of deformable registration. The approach aims to enhance the reliability of MRI classification outcomes by ensuring that decisions are based on clinically relevant features rather than spurious confounders. The authors test this and compare across different datasets and populations.
main_review: main_review: According to the update from the authors, the paper now clearly describes the issues related to brain shape associations. Although another significant improvement would be testing on known confounders, I still believe it has its merits without those experiments and can lead to future research.
strength_weakness: strength_weakness: Some conclusions in the paper are reached without sufficient evidence. For example, the authors pose the question "whether the difference in brain shape we observe is due to the pathology, and therefore clinically relevant, or due to some differences in the population used to build the datasets." They then find that there exists a population difference and state "the difference seems to be linked to a population effect and not to relevant factors like disease or age." The fact that an observation is different between two populations does not shed any evidence at all on whether there is also a clinically relevant factor. The authors know this since the result of another experiment was that "it is possible to classify healthy vs pathological subjects using only the brain shape." Suggesting an association between the two.

Furthermore, it is well known that brain shape changes with age, and the authors mention this, contradicting the conclusion. I suggest the authors 1) rephrase this conclusion to the posed question and 2) clear up the seemingly contradictory phrasing, perhaps they are meant to be general vs specific but that is not very clear. 

In terms of the proposed method to assess whether variables are confounders, it would be much stronger to show this method demonstrated on known confounders/non-confounders to show the efficacy of the method, before testing an unknown variable with a new method. 

These weaknesses are fixable and I believe that there could be an interesting finding here but the current experimental set-up does not back-up the claims.
questions: questions: Do the authors expect a different outcome had the images been aligned with the nonlinear MNI152 mapping vs the affine-only MNI used here?

There has been significant prior work on brain shape in the medical field. Pathology such as inflammation can have a direct impact on the shape of the brain [1], the effects of hypertension [2] or dementia [3]. How do the findings from this work align with the current literature on brain shape and pathology?

[1] O. Guillemot-Legris and G. G. Muccioli, “Obesity-induced neuroinflammation: beyond the hypothalamus,” Trends Neurosci, vol. 40, no. 4, pp. 237–253, 2017.

[2] Strassburger, Terri L., et al. "Interactive effects of age and hypertension on volumes of brain structures." Stroke 28.7 (1997): 1410-1417.

[3] Wang, Lei, et al. "Changes in hippocampal volume and shape across time distinguish dementia of the Alzheimer type from healthy aging." Neuroimage 20.2 (2003): 667-682.
paper_summary: paper_summary: The paper proposes several methods to assess the importance of the brain shape (bias) in classification tasks based on T1-weighted MRI datasets and a solution to get rid of this bias. Large databases from healthy, MS and tumor’s datasets are used to train binary. After several tests (classification on original images, cropped images and images transformed with deformable transformation; generation of saliency maps), the authors conclude that brain shape is indeed used by the classifier to take the decision while it does not contain any medically relevant information about the pathology. Finally, the authors show that performing non-linear registration as a preprocessing step allows to obtain less biased classifiers.
main_review: main_review: Thank you for the clarifications and updates. I think that the paper brings valuable insights to the community about common biases present in our datasets and will lead to interesting discussions around that topic at MIDL.
strength_weakness: strength_weakness: Although the methods for identifying confounders and correcting for it (for brain shape specifically) are proposed, no discussion is made about the importance and impact of the architecture of the classification model chosen (see Questions To Address In The Rebuttal).
questions: questions: The authors decided to use a 3D PatchGan model as classifier. It would be good to add a justification for the choice of that model, which does not seem to be very commonly used for brain MRI classification tasks.

Is it possible that the choice of model also influence the strength of the bias? Have you tried different model architectures and made the same observations?

Although affine registration to an atlas was performed on all databases, substantial differences in brain volume remain between databases that were processed with the authors pipeline (Tables 3, IXI and OFSEP). Did the authors do any evaluation of the registration quality?

Comments: title: Response to Rebuttal
Invitation to reply to authors
comment: The results seem indeed to indicate that the deformable registration is not necessary in this case. Nevertheless, the results in Figure 3 and 6 show that applying this spatial normalization to the images does not reduce the classification performances for all experiments. Therefore, as mentioned in Appendix C, our take-home recommendation is to apply it as a systematic preprocessing step. We added this recommendation in the conclusion.
Thank you for addressing the review comments and making revisions. I am updating my score to reflect the paper's merits for publication.

One last suggestion/question: Based on the intra-database classification results from the ADNI dataset, it appears that brain masks alone cannot distinguish brain abnormalities within a single database. Does this indicate that additional deformable registration isn't needed for intra-database classification? Clarifying this could help readers understand when deformable registration is or isn't necessary, potentially simplifying future research methodologies
I thank the authors for their detailed responses and clarification. I am updating my score to reflect that.
Dear reviewers,

The authors have prepared responses to your comments, which you should now be able to see in OpenReview. We encourage you to reply to their comments, and where necessary, adjust your rating. Please do so before the 27th of March.
**General comment:**

We would like to thank all the reviewers for their comments, which helped improve the document.
The manuscript modifications appear in color in the new submission. 

We corrected some misprints: the number of test samples is 500 and not 600 for HCP. The TNR and TPR in Table 2 for the HCP vs IBC/kirby/MPI classification were incorrect: the correct ones are TNR=0.99 and TPR=0.95 (instead of TNR=1.00 and TPR=0.82).
The conclusion does not change and it is even strengthened since we can almost perfectly distinguish healthy datasets with matching age.

As suggested by the reviewers, step 1 experiment for the ADNI dataset was added in the manuscript and step 2 experiments for the healthy datasets were added as supplemental material.

**Response to the reviewer:**

* On the choice of PatchGAN

We tried to use other classical architectures (ResNet, DenseNet, MobileNet, etc) but we had overfitting problems on some of our small datasets. PatchGAN (a simple 4-layer CNN) was robust on all dataset classifications. The choice of the architecture can indeed have an impact on the confounder strength in the network decision but like the other hyperparameters (regularization, preprocessing, etc). The influence of the architecture would be interesting to investigate in an extended version, with the difficulty of finding a trainable architecture on our small datasets. However, our paper highlights the brain shape as a possible confounder for brain scans classification. The proposed methodology to detect and remove the brain shape confounder would be applicable to any architecture. 

* On the registration quality

The affine registration has only been visually inspected on our databases. We indeed rely on the robustness of the chosen software (FLIRT) which is an established registration procedure in the community. As the affine registration only performs scaling and rotation, the brain shapes cannot perfectly match and it seems normal to have these volume variations.

* Typos

We corrected the identified typos.
**General comment:**

We would like to thank all the reviewers for their comments, which helped improve the document.
The manuscript modifications appear in color in the new submission. 

We corrected some misprints: the number of test samples is 500 and not 600 for HCP. The TNR and TPR in Table 2 for the HCP vs IBC/kirby/MPI classification were incorrect: the correct ones are TNR=0.99 and TPR=0.95 (instead of TNR=1.00 and TPR=0.82).
The conclusion does not change and it is even strengthened since we can almost perfectly distinguish healthy datasets with matching age.

As suggested by the reviewers, step 1 experiment for the ADNI dataset was added in the manuscript and step 2 experiments for the healthy datasets were added as supplemental material.

**Response to the reviewer:**

* On the origin of the brain shape difference

The conclusion was indeed not clearly stated in the initial submission and has hopefully been clarified. We did not (mean to) claim that age or disease are irrelevant for the brain mask classification, but that these two factors *alone* cannot explain the brain shape difference, and that there are multiple factors that we regroup under the term "dataset construction effect" (previously "population effect"). As we mentioned in the original manuscript, the brain shape difference can indeed be linked to multiple factors: disease (as in the papers the reviewer cites), age, sex, preprocessing pipeline, etc. To investigate the brain shape effect in the case of the classification of the healthy vs pathological images (MS or tumors), we considered the same classifier trained to distinguish healthy databases with similar age distribution. In this case, age or the considered pathology are not involved in the brain shape difference but the datasets can still be distinguished one from another. This implies that there are other factors, hard to identify and a priori unwanted for the healthy vs pathological classification, that impact the brain shape. Our intuition is that when matching datasets for factors such as age, sex or ethnicity, other unknown or unintuitive factors influencing the brain shape might remain, and the brain shape may still be a confounder.
We reformulated the text for more clarity.

* On applying our method on known confounders

Thank you for the comments. It could be interesting to evaluate our method on a known confounder as the factors considered in [1]. It could be done in an extended journal version of this work.

[1] *Sun, S., Koch, L.M., Baumgartner, C.F. (2023). Right for the Wrong Reason: Can Interpretable ML Techniques Detect Spurious Correlations?. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14221. Springer, Cham. https://doi.org/10.1007/978-3-031-43895-0_40*

* On the MNI atlas

Both the affine registration and the nonlinear brain shape normalization were done using the non-linear MNI atlas (MNI152_T1_2mm and not MNI152lin_T1_2mm). It seems that MNI152lin_T1 was removed from recent versions of FSL. We did not do the experiments to quantify the influence of the chosen reference template but our intuition is that this influence is negligible. There is a priori no reason for the spatial normalization (either the affine registration or the nonrigid registration on the brain shape) to be better or worse by choosing between two carefully built reference templates and that would influence the classification output.
**General comment:**

We would like to thank all the reviewers for their comments, which helped improve the document.
The manuscript modifications appear in color in the new submission. 

We corrected some misprints: the number of test samples is 500 and not 600 for HCP. The TNR and TPR in Table 2 for the HCP vs IBC/kirby/MPI classification were incorrect: the correct ones are TNR=0.99 and TPR=0.95 (instead of TNR=1.00 and TPR=0.82).
The conclusion does not change and it is even strengthened since we can almost perfectly distinguish healthy datasets with matching age.

As suggested by the reviewers, step 1 experiment for the ADNI dataset was added in the manuscript and step 2 experiments for the healthy datasets were added as supplemental material.

**Response to the reviewer:**

* On the intra-database classification

We have added an intra-database (ADNI CN vs ADNI AD) experiment for the first step. The results show that the classifier cannot distinguish these two sets based on the brain shape. Here, the disease does not seems to influence the brain shape. The Brats LGG vs HGG was not added or discussed for space reasons. In this case, the balanced accuracy is 57%.

* About the claim that disease/age would be irrelevant for the classification

The conclusion was not clearly stated in the initial submission and has hopefully been clarified. We did not (mean to) claim that age or disease are irrelevant for the brain mask classification. We indeed agree with the reviewer that disease (like MS) or age can modify the brain shape as mentioned in the original submission. Nevertheless, as it is possible to distinguish brain shape between healthy datasets with similar age distribution, it seems that these two parameters *alone* are unable to explain the difference of the brain shape between two databases (but are not irrelevant). The difference should come from a combination of factors that we regroup under the term of "dataset construction effect" (previously "population effect").  Our intuition is that when matching datasets for factors such as age, sex or ethnicity, other unknown or unintuitive factors influencing the brain shape might remain, and the brain shape may still be a confounder. 
We reformulated the text for more clarity.
 
* On the sex/race influence

Race was not disclosed in the datasets' information. 
The percentage of women, for each dataset (except for BraTS2020 as the information is not provided), is given below: 

IXI     : 65; 
HCP     : 59; 
MPI     : 19; 
kirby   : 45; 
IBC     : 17; 
OFSEP   : 68; 
ADNI-CN : 47; 
ADNI-AD : 48

Age and disease were taken as examples for discussion on the factors that can influence the brain shape. Other factors can indeed be considered. Although a deeper discussion on other factors would be interesting, there is no space to add this discussion that could appear in an extended journal version of the work.

* On the exclusion of comparison between normal datasets in second step experiments

We did not do the step 2 experiments for the healthy vs healthy datasets for the initial submission. We chose to only do the second step for the healthy vs pathological (BraTS or OFSEP) classification because it seemed to us it has a more practical and clinical interest than classifying among healthy images. Nevertheless, to avoid the impression of "cherry-picking", we did the experiments since the reviewers feedback and added the results on the healthy databases classification in the Appendices of the new version.

* On the random classifier 

We mean an accuracy close to 0.5 as this would mean the classifier is not able to distinguish from one class to the other.  A classifier with an accuracy much lower than 0.5 would mean that it can distinguish brain masks among different classes but would inverse the datasets: in this case, it is worth considering whether the training was conducted correctly.

* On the Figure 2 caption

Yes, the first line is without shape normalization and the second line is with shape normalization.
**General comment**:

We would like to thank all the reviewers for their comments, which helped improve the document.
The manuscript modifications appear in color in the new submission. 

We corrected some misprints: the number of test samples is 500 and not 600 for HCP. The TNR and TPR in Table 2 for the HCP vs IBC/kirby/MPI classification were incorrect: the correct ones are TNR=0.99 and TPR=0.95 (instead of TNR=1.00 and TPR=0.82).
The conclusion does not change and it is even strengthened since we can almost perfectly distinguish healthy datasets with matching age.

As suggested by the reviewers, step 1 experiment for the ADNI dataset was added in the manuscript and step 2 experiments for the healthy datasets were added as supplemental material.

**Response to the reviewer:**

* On the choice of showing only the step 2 results for the healthy vs pathological classification

We did not do the step 2 experiments for the healthy vs healthy datasets for the initial submission (it was not "cherry-picking").
We chose to only do the second step for the healthy vs pathological (BraTS or OFSEP) classification because it seemed to us it has a more practical and clinical interest than classifying among healthy images. Nevertheless, to avoid the impression of "cherry-picking", we did the experiments since the reviewers' feedback and added the results on the healthy databases classification to the Appendices of the new version.

* On the choice of PatchGAN

We indeed tried to use other classical architectures (ResNet, DenseNet, MobileNet, etc) but we had overfitting problems on some of our small datasets. PatchGAN (a simple 4-layer CNN) was robust on all dataset classifications. The model is trained on the complete image. The "patch" name comes from the fact that the output is a reduced 3D maps of logits (patches are the fields of view of these logits). Each one of these logits is trained for the classification. At test time, we used the average of these logits to classify.  

* On the databases split

We perform a random split based on 80/10/10% for the small databases and we adjust the number of samples for the bigger databases to avoid excessive imbalance at training time.

* On the impact of the deformable registration

In this work, we show that the brain shape can be used in a classification task. It can indeed be a clinically relevant factor or not. And even when it is a clinically relevant factor, we might also want to discard its influence to force the classifier to focus on other factors (lesions, internal structure, etc). Therefore, using the normalization or not is application depending, based on whether it is known that the brain shape is clinically relevant (or not). Note that, when the relevance of the brain shape is not known, our methodology can also be used to measure its influence in the classification. Note also that the proposed deformable registration is only performed to fit the borders and not the internal structures as we can expect from usual deformable registrations. These structures, their size and their shape are then mostly preserved by our registration.   

* On the random classifier

During the training, the batches are balanced to cancel the effect of our imbalanced datasets (this point was added in the manuscript). At test time, when the classifier is random (return class 1 with a probability 0.5), both accuracy and balanced accuracy will be 0.5 whatever the balancing of the classes in the test dataset.

* On the Figure 3 legend

Indeed, the image type (original, cropped, registered) is at test time. We have added it to the legend.

* Typos

We corrected the identified typos.